The Mythical DevOps Engineer

Reading Time: 8 minutes

I’m always a little suspicious of job specs looking for the so-called DevOps Engineer role. They often mention a vast variety of duties and responsibilities.

Are they hiring for a single role or a whole team?

Roles having DevOps in their title hardly share the same meaning. They often have something in common, though. They try to cover for what traditionally would have been the specialization of different professionals.

Don’t get me wrong: cross-functional expertise is definitely important. But I don’t think DevOps means replacing a multitude of specialization with a single role. Different specializations like Operations, Security, Testing, Development, Product Management and so on, are vast and require specific knowledge.

I think the key differentiator of successful DevOps organizations is that they enable effective collaboration. They have as clear North Star the goal to deliver value to the end user.

Overall, I don’t think we should be talking about a DevOps Engineer, but rather about DevOps culture in organizations.

But let’s take a step back first.

What does DevOps mean, really?

I tweeted my own definition of DevOps some time ago.

DevOps organizations incentivise different specialities to collaborate. The intrinsic existing tension between Dev, making changes to the system, and Ops, wanting to keep the system stable, dissolves. The greater good is now the value stream.

A stable system that delivers nothing is as useless as an unstable system that keeps offering new functionality.

Dev and Ops understand the importance of working together to maximise this flow to figure out which bets worked out and which ones didn’t.

Organizations that embrace the DevOps mindset can be more effective than the competition at experimenting with new functionality. They quickly validate their assumptions, activating and deactivating functionality by flipping a switch on a dashboard.

Incidents become an opportunity for learning rather than a chance of blaming someone.

In general, DevOps organization learn to adapt and evolve to any situation.

Overall, I think there shouldn’t be a single DevOps role but, rather, a set of specific specialities collaborating effectively.

This ideal view of the terminology, though, might sometimes clash with the reality of the job market. Companies willing to attract the best talent with the most current skills may end up advertising for roles that are counterproductive in the context of DevOps principles.

But let’s have a look at a few interesting job specs.

work harder neon sign photo
Photo by Jordan Whitfield on Unsplash

What are companies looking for?

Let’s read through a few excerpts from job specs I found out there in the wild.

The flexible problem solver

[…] Devops Engineers are IT professionals who collaborate with software developers, system operators and other IT staff members to manage code releases. They cross and merge the barriers that exist between software development, testing and operations teams and keep existing networks in mind as they design, plan and test. Responsible for multitasking and dealing with multiple urgent situations at a time, Devops Engineers must be extremely flexible. […]

A job spec on the internet

This is one of those classic examples where the organization believes that the DevOps principles should be delegated to a single team.

The spec mentions the myriad of duties that are responsibility of the Devops Engineers in the company. A Devops Engineer is expected to “multi-task and deal with multiple urgent situations at a time”. Therefore, they “must be extremely flexible”.

Multitasking and dealing with multiple urgent situations at a time is, for sure, likely to happen anywhere: I don’t think this should be a peculiarity of a role in an organization. On the contrary, a healthy environment empowers every engineer to handle urgent situations and learn from them.


Coming across this role, I’d think that the organization is not really trying to adopt DevOps practices. Instead of encouraging people to collaborate and improve, they’re building a dedicated team to throw issues and urgent situations at.

This job spec would be a big red flag for me.

The productivity booster

A DevOps Engineer combines an understanding of both engineering and coding. A DevOps Engineer works with various departments to create and develop systems within a company. From creating and implementing software systems to analysing data to improve existing ones, a DevOps Engineer increases productivity in the workplace.

Another job spec on the internet

In a DevOps organization engineers do work with various departments. But what’s the point then of having a dedicated DevOps Engineer role? Do the other type of engineers not work with the various departments of the organization? Do non-DevOps Engineers not analyse data and improve existing systems? Additionally, the job spec claims that a DevOps Engineer increases productivity in the workplace. How? Does it radiate productivity?

The Release Manager… but DevOps!

A DevOps Engineer works with developers and the IT staff to oversee the code releases. […] Ultimately, you will execute and automate operational processes fast, accurately and securely.

My favourite so far

This is quite a condensed one but what strikes me as interesting is the release aspect mentioned in it.

It is quite a complex aspect of the DevOps culture in my opinion. I tend to separate the concept of deployment from the one of release. Product updates as experienced by the user are governed by a release policy that may or may not be the same as the deployment policy. This really depends on the strategy of the organization.

Regardless of this distinction, though, I believe that constraining the capability of delivering value to the end user to a specific role undermines the agility of an organization.

The teams should be able to continuously release code into production. The release of functionality should be controlled through mechanisms such as feature flags so that the code that reaches production does not necessarily activate. This makes it possible for the organization to control when the functionality actually reaches the user.

In general, a deployment should be a non-event: nothing special, just another merge into the main branch that causes code to end up in production. Moreover, releasing the actual functionality to the user should not require a dedicated engineer to be performed: the relevant stakeholders in the company, usually departments other than engineering, should be able to enable the functionality to the user, perform experiments and autonomously decide when it is appropriate to release new functionality.

Job specs like this one feel like they’re trying to repurpose the role of the Release Manager to keep up with the latest trends by just changing a few words.

I don’t think release management goes away in a DevOps organization. What changes is that the concept of deployment gets decoupled from the one of release. This is done to enhance the agility of the engineering organization and reduce the risk that’s intrinsic of the changes reaching production environments.

At the same time, the relevant technological changes are implemented so that releasing new functionality to the users can become a non-event, in the hands of the strategic initiatives of the organizations.

A Platform Engineer. But cooler!

The DevOps Engineer will be a key leader in shaping processes and tools that enable cross-functional collaboration and drive CI/CD transformation. The DevOps Engineer will work closely with product owners, developers, and external development teams to build and configure a high performing, scalable, cloud-based platform that can be leveraged by other product teams.

This job spec is one of those that I consider the least bad. It describes a set of responsibilities that usually pertain to a Platform or Infrastructure Team. Most of these teams often get renamed to DevOps Team and their members become DevOps Engineers for fashion reasons.

The Platform Engineering team becomes the key enabler for organizations that want to embrace the DevOps principles. But thinking that such principles will only be embraced by that team will hardly result in a successful journey.

The Platform Engineering team will surely be responsible to build the relevant infrastructure that enables the other teams to build on top but they can’t be left alone in the understanding and application of those principles.

Developer teams will need to become autonomous in adopting and making changes to those systems; they will need to understand the implications of their code running in production; understand how to recognize if the system is not behaving as expected and be able to react to it.

At the same time, even the Product team should spend time understanding what new important capabilities derive from successfully adopting DevOps practices. Code continuously flowing into production behind feature flags, containerization technologies, improved monitoring and alerting, and so on, open endless opportunities of improved user experience and experimentation that should be leveraged to maximise the company competitiveness.

people riding boat on body of water photo
Photo by Matteo Vistocco on Unsplash

What should companies be looking for?

We’ve just gone through a few job specs that look for variations of a DevOps Engineer role and I’ve outlined what aspects I think are flawed in those roles. But what should companies look for, then?

Before blindly starting to hire for roles driven by industry fashion trends, organizations should rather invest in understanding what’s holding them back from being DevOps.

In the Unicorn Project, Gene Kim mentions the Five Ideals of successful DevOps organizations. I think they’re an effective set of principles to take the temperature of your organization in terms of DevOps practices. Those ideals are as follows:

  • Locality and Simplicity
  • Focus, Flow and Joy
  • Improvement of Daily Work
  • Psychological Safety
  • Customer Focus

Locality and Simplicity

Making changes to the system, in order to deliver greater value to the end user, should be easy: easy in terms of team’s autonomy to make changes to an area of the product as well as being easy in terms of friction that the technology in use imposes on the changes.

Focus, Flow and Joy

Developers should be able to focus on their work and be able to make software with minimum impediments. This is facilitated by making sure that the software development lifecycle infrastructure is working for the benefit of the engineering organization.

Improvement of Daily Work

Continuously learning and improving the conditions in which the work gets done is the key to maximise the flow of value and the happiness of the people doing the work. Successful organizations facilitate a continuously improving environment by enabling engineers to build tools and practices that improve their daily operations.

Psychological Safety

An organization will hardly be able to improve if the people that are part of it are not incentivised to raise issues and discuss them. This is not something you solve for by hiring a specific role. It the organization’s responsibility to facilitate an environment where constructive feedback is the norm.

Customer Focus

Last but not least, the engineering organization, just like any other department in the company, should be sharply focused on the customer. All the efforts should be balanced against what’s best for the customer and, ultimately, for the company.


What should companies be looking for then? I think the priority should be on understanding what’s blocking the organization from fully embracing a DevOps mindset. Once that’s established, the expertise needed to get there is probably going to be easier to identify.

The most important aspect for me, though, is understanding the importance of specialities. Every role will have incredible value to add to the journey towards DevOps. What’s fundamental is understanding the importance of collaboration between the different roles. The organization will have to put the relevant practice changes in place to facilitate collaboration. Specific DevOps knowledge in terms of technology, tools and best practices, will be required, for sure, but it won’t be something a single role should be responsible of.

nude man statue photo
Photo by Roi Dimor on Unsplash

A mythical role

It feels like the DevOps Engineer is a mythical figure that certain organizations pursue in the hope of finding the holy grail of a Software Engineer capable of doing anything.

This, of course, will hardly be the case. Recognizing the importance of the single specializations is what makes organization successful and capable of maximising the expertise of the people that they are made of.

What happens in a DevOps organization is that responsibilities are redistributed: developers are empowered to make changes to production environments because organizations recognize the importance of moving fast. This means opportunities for success increase together with the opportunities of failure.

Eliminating barriers and creating a safe space for collaboration helps Devs and Ops work together to resolve issues when they occur. This is what ultimately leads to high performing teams that are incentivised to follow the North Star of the continuous value stream to the end user.

Instead of pursuing a mythical role then, let’s go after the much more plausible alternative of creating a well oiled machine where all the people are incentivised to work together in harmony with the clear goal of maximising the value to the end user.


Thanks for getting to the end of this article. I sincerely hope you’ve enjoyed it. Follow me on Twitter if you want to stay up-to-date with all my articles and the software I work on.

Cover photo by Rhii Photography on Unsplash

Business outcome language: an introduction for software engineers

Reading Time: 4 minutes

Using a business outcome language helps keeping the problem definition focused on the value that you should deliver to your customers. Let’s explore together how to make sure our language is not polluted by technical details.

As a technological leader at your company, you are the citizen of two worlds: the business world and the implementation world. Each world speaks its own language, so it’s vital that they communicate clearly and unambiguously.

Your first days in the implementation world

Software engineers typically start their career in the implementation world, with few contact opportunities with the business world.

As a software engineer what excites you is the challenge of solving a problem. You crave more problems and you feel more satisfied as you solve them. Your brain is triggered by the problem definition and it starts spitting out all of the possible ways you can achieve the expected result.

The problem definition at this stage is probably in an implementation-oriented language with little indication as to what business outcome the company is looking for. It’s likely that you don’t care about that though because you only care about the coding challenge.

Your language evolves around the how

In the implementation world you develop a language that is specific to the solution to the problems and the tools you have to solve them.

You communicate with your peers by focusing on how to achieve the solution. The discussions often develop around the technical details of the existing software.
The foreigners living outside of the implementation world look at you with surprise and curiosity: your words don’t really make much sense from the outside.

The first contact with the foreigners

There comes a point in your career when you have to interact with the people from the business world and work together.

Depending on the approach your company has for building product, this interaction can often be an opportunity for the two worlds to collaborate and find a sensible outcome to pursue.
The most effective way of doing this is by a common language: the business outcome language.

Certainly, this is often challenging for the implementation people because their native language is not well understood outside their world. Using the implementation language comes naturally for them. Explaining how something can or cannot be achieved through implementation details feels incredibly easy.

The trouble with the implementation language

Let’s see a few examples of why using the implementation language for every interaction can be counter-productive.

We can’t do that!

dilbert, technical debt, comic strip, business outcome language

“What you’re asking is impossible to do by that date. The amount of technical debt we have accumulated means that we need 3 months to refactor the service classes to be capable of supporting this new functionality. We also have to update to the latest version of the database in order to support that change…”

– An implementation world native

While those might all be valid reasons as to why the specific change in question is hard to achieve, they are contributing to setting the wrong tone for the conversation.

Raising very specific technical concerns is going to shift the conversation to an explicit technical negotiation.
The people from the business world will inevitably, and often subconsciously, try to petition the implementation people for a middle-ground solution. This achieves a similar outcome but overcomes some of the technical limitations that have been raised.

When the negotiation is successful it might feel like a WIN-WIN situation but in reality, it isn’t.

First of all, the business people might have just agreed to build functionality that is polluted by workarounds and has evolved around technical constraints instead of around the customer.
Similarly, the implementation people might have just accepted to build more technical debt into their world.

I speak your language too, I’ll tell you exactly what you have to do!

dilbert, feature creep, comic strip, business outcome language

“We need to add a new button to the page so that the users can export the data as PDF.”

– A business world native speaking the implementation language

The above example is outlining exactly which technical solution should be used without even specifying what the problem to be solved is.
Imagine the following problem definition, instead: the users want to share the insights data with their colleagues with minimum effort.

While an “Export as PDF” button might try and address the issue, it only partially solves for it. Exporting as PDF, saving it to your disk, finding it again to attach it into an e-mail doesn’t necessarily mean minimum effort.

Having clarified that the users want to share insights with minimum effort helps the engineers come up with a much more effective solution. One possibility is providing the users with a “Share” call to action that sends the insights to the desired recipients, without having to export anything.

Clarifying what the problem is instead of mandating a specific solution will let the engineers and the designers explore the best approach within the current constraints.

A technical language anchors down the thinking

The implementation language is a powerful way of describing solutions. It is very detailed and it helps engineers be unambiguous as to what has to be done.
Its power, though, makes it dangerous if used in the wrong context.

Bringing too many implementation details into the discussions anchors the thinking to the status quo, potentially preventing from identifying the right value for the users of your product. It is important to defer the implementation evaluation to the very end in order to let the thinking evolve and articulate in the space of the business outcome that the company wants to achieve.

Help free the language from the technical constraints

As a technological leader, you should be fluent in both languages. Take advantage of this and make sure the conversation stays away from implementation details by ensuring that all the people involved speak a business outcome language.

Understand the technical limitations and concerns raised by your compatriots and translate them into the appropriate language, weighing what is really worth including in the conversation.

Make sure not to preclude innovation and avoid constraining the thinking to the status-quo of the technical limitations. This exercise will help you evaluate many more implementation opportunities than before.

Cover photo by Headway on Unsplash

Technical debt kills your company

Reading Time: 9 minutesThe early days of a startup are often a chaotic mix of excitement, rush and fear. Software-enabled startups are learning about their product as they iterate with their first customers. This leads to frequent and unplanned changes that will help rapidly evolve their product to fit the market they’re trying to disrupt. Nevertheless, the decisions taken, consciously or unconsciously, on how to evolve their product will inevitably affect their future.

The compromise culture

If you have the chance to live the exciting experience of the early days of a startup as a software engineer, you’ll most likely lead on the technical decisions and will be working hands-on on the code of the product around which the company will be built. And you’ll be the first cause for technical debt.

Technical debt is the compromise you make to get things done faster and grow rapidly. Compromising on quality is justified by the short term goals you want to achieve to establish yourself in the market as soon as possible. You have identified a problem, you’ve found a solution that helps your potential customers do the job and you work hard to sell your solution. But since most startups at such an early stage are just losing money, their focus is mainly around shipping features to sell. This means the tension is all towards rushing to the delivery. Anything stopping you from delivering earlier is an obstacle for the company.

One such obstacle is the “Engineering” bit in Software Engineering.

Engineering is the application of scientific knowledge and mathematical methods to practical purposes of the design, analysis, or operation of structures, machines, or systems. (Wikipedia)

The application of a methodological engineering approach to the crafting of your early stage software product is most likely overkill for any early stage startup because it requires time and thorough thinking. Often, you can’t afford to spend too much time on it because you can’t even be sure what you’re building will be useful to your potential customers.

Conscious vs Unconscious compromise

My code works. I have no idea why.What’s critical in this early stage of the journey is to remain conscious about the compromises you’re going to make to sacrifice quality over speed. Consciousness implies that you’re trying hard to plan to address the technical debt you’re already accumulating. Hoping that the technical debt you’ve accumulated will go away by itself is not a strategy and will end up biting you in the ass as you grow. Worse, not realizing you’re accumulating technical debt will most surely lead to the most painful growth for your engineering organization.

As a good technical leader you should plan for technical debt as if one day your business will be extremely successful.


Planning for technical debt

Not all startups are technological, of course, but if technology is the enabler for your business then you have to make sure technical debt is understood across the company. Understanding the impact technical debt has on your future is hard enough, but it’s especially hard to understand for the less technical people in the company.

As a technical leader, one responsibility you have is ensuring that the state of technical debt is very clear across the board.

If the compromises you’ve made on the quality of the software you’re producing have helped you getting your second client then you’ve only been half-successful as an early-stage startup engineer. The other half of your success resides in making sure the company realizes how much slower you are now on acquiring your third client or on-boarding your next engineer, or fixing that new bug that has just been logged. Every compromise you make has to be clearly understood in order for your future engineering organization to be effective in the next iterations of your product.

Planning for technical debt is definitely not easy and there’s no single success recipe for it. But keeping technical debt in mind should be part of the job of a technical leader. I’ve tried to list a few questions here that it might help to ask yourself from time to time when growing together with the rest of the organization.

  • Are the functionalities of your product documented?
  • Will it work tomorrow when success brings trouble?
  • How much do you know about your software in production?
  • How long will it take for your next hire to get going on the code-base?

Are the functionalities of your product documented?

Goat head issue

Implementing new features and iterating on them happens at a very fast pace in the early stages of a startup. Agreement on what to ship to production often happens verbally between founders and there’s no room for process overhead to formalize those decisions. This means there’s probably no documentation about how your product is behaving for your customers.

Lack of documentation is a form of technical debt. This particular form of technical debt will become hard to address as your company grows and you start hiring more people.

There’s an update your company wants to ship as part of a feature that’s considered very important across the board. This new bit of functionality relies on the existing feature you have implemented in the very early days. You join the conversation and help the new developers understand how the existing functionality behaves. There’s no documentation or automated tests that define how the software in production is behaving so your help is very important at this stage. Your team successfully delivers the change and ships it in production. Everything seems great until one of your customers receives the update and realizes they can’t use the feature they were relying on anymore in the way they intended the feature to be used. Turns out other stakeholders in the company had sold the feature as something else to some of the more recent clients, something not really close to how you expected that feature to be used. Your team now has to rush and “fix” what you have just shipped to restore the old functionality otherwise your most recent client will be very upset.

I’ve seen this happening multiple times. This causes immense frustration across the company and specifically among engineers. It affects the confidence they have in you as a technical lead because you’ve just led them to ship something that has caused more harm than anything. It dramatically impacts the confidence they have in what themselves are shipping to production subsequently. And, undeniably, it affects the perception the other stakeholders in the company have of the engineering organization. Therefore, it is your responsibility to make sure this doesn’t happen.

Documenting is vital for your engineering organization to stay healthy. BDD (Behavior-Driven Development) techniques can really help in this regard as they help you merge the documentation phase with the test phase. This helps you in creating a contract between the functionality and the rest of the organization. It also helps the engineering organization be more confident about changes to the product. Automated tests and documentation help the engineers know what to expect from their changes and it also represents the artifact that defines the product functionality, helping reduce the ambiguity across the company.

Will it work tomorrow when success brings trouble?

Scalability is one of those aspects you might have compromised on in the early days in order to be quicker at delivering. If you’ve done this consciously your company is aware that the current software is not fit for a larger scale yet and will invest resources in scaling it before things go bad. Your role as a technical leader is consult on the planning for resources.

Nowadays this problem seems almost non-existent since the majority of the software runs on managed infrastructures as a service that allow for instant horizontal scaling. And yet, I’ve seen poor technical decisions and insane quantities of technical debt strangling platforms and make them unscalable even if running on cloud platforms.

This can be because of:

  • lack of documentation on specific details to be aware of when scaling
  • insane number of dependencies on third-party providers
  • code base so coupled with the underlying infrastructure that it had hard-coded assumptions on which server it was running on (seriously)
  • asynchronous jobs to be run manually

and so on.

Scalability issues will definitely hit your company if your product is successful. You have to do something to prepare your business to handle the success that it deserves.

Communication is vital for addressing this aspect of the technical debt. Make sure it’s clear across the board every time compromises are being made on the scalability. Be honest to yourself and the rest of the organization about it and act promptly before it’s too late.

How much do you know about your software in production?

Collecting metrics about how your software is doing in production might not seem obvious at the beginning of your startup journey but is vital for your company to be successful. Relying only on your clients to know how it is going is not scalable (see above) and the later you address this the worse it is for your organization.

Measuring is the only way you have to build your organization’s confidence on how the product is being used and how reliable it is in production while at the same time removing any ambiguity around it. Making sure usage patterns and service reliability data points are clearly collected and radiated will help define an unambiguous picture of the software that runs in production, allowing your company to make informed decisions.

Lack of monitoring and instrumentation is not only technical debt: it’s organizational debt. A company that grows without measuring what it does will deliver a product that is the result of the decisions of who shouts the loudest in the organization. And this kind of organizational debt is one of the hardest to address because it shapes the culture of your company.

It’s your responsibility, as a technical leader, to campaign for ensuring that monitoring and instrumentation is adopted as early as possible.

How long will it take for your next hire to get going on the code-base?

As a technical leader you’ll most likely take part in hiring at least the first engineers working on the code that you have crafted for your product to hit the market. You’ll most likely look for someone that fits your company culture and maybe has experience with the technologies your product is currently taking advantage of. But what you should ask yourself, regardless of the type of engineer you’re going to hire, is whether your next hire is going to be able to become autonomous on making changes to the code-base and, ideally, be able to lead future hires as your company grows.

The compromises on the quality, documentation, architecture and sensibility in general of your technological asset will affect the ability of your future hires to impact positively your company. There will be no 10X developer able to overcome insane assumptions on which your software relies on. And this is especially true when your product has been built with technologies that require a high degree of discipline in order to keep everything manageable.

wtfs per minute

Ruby is one of those technologies that I consider requires a high degree of discipline. It gives you the freedom to do pretty much any thing you want. You can easily override any method you like and inject different behavior for already existing functionality pretty much everywhere. From great power comes great responsibility said someone. This is particularly true with this kind of technology.

I’ve seen so many surprising changes to core parts of used libraries that the original intended behavior had completely been altered, leaving newcomers completely knocked out learning how things were behaving completely unpredictably. Someone called this The principle of most surprise and is measurable by the number of WTFs per minute shouted by the developers at work.

This is one of those things I consider to dramatically affect the morale and health of the engineering organization. On top of that, such insensible things are so technical that they are even harder to explain to the rest of the organization when it’s time to address them.

Conclusion

Crafting a product from scratch and building a company around it is not an easy job. The fact that the company turns out to have a market for the product you’ve built is a big success on its own. You should feel proud of it as you’ve been one of the main factors for this success. But different phases in the growth of the company require an adaptive mindset as things evolve. The skillset required to build a company from scratch are different from the ones needed to scale it. And the most important thing is acknowledging this bit.

Technical debt is everywhere and is inevitable. Technical companies will always struggle trying to manage it but the successful ones are the ones that plan for it rather than hoping it goes away by itself.

As you go through the wonderful experience that an early stage startup is, try to think about the questions we’ve just gone through and imagine if things can be done at least slightly better before it’s too late. Try to always question the way you are achieving your results and understand how many shortcuts you’re taking to get there. Pick technologies carefully and be ready to dismiss them as you realize they don’t fit your purpose. Don’t force nasty workarounds to make them work the way they’re not intended to work. Do not blindly fall in love with your code but, rather, be always ready to get rid of it for the better.