Organizational Anti-Patterns that Impact DevOps

Part Four in a Series on DevOps Anti-Patterns

Sometimes it’s hard to be sure that the culture is fully DevOps. After all, this is a path of continuous improvement, and always striving to become better is part of the DNA. There’s also a lot of nuance and opinion on what DevOps means, and a lot of ways to get there.

So, I was thinking: what about an alternative symptom-based checklist to measure how far you are on the journey? I came up with some ideas to use as a gauge. Essentially… How DevOpsey are you?

By DevOpsey, of course, I mean all the things in the DevOps Loop (plan, create, verify, package, release, configure, monitor). But it also applies to things like the DevOps Super-pattern, and probably a bunch of other things too.

I kept adding stuff to the article until I realized that it was getting too big and needed to split it out. There are some things I’m not going to repeat, like why silos, heroes, and yak shaving are bad, but this final part addresses some of the anti-pattern symptoms you might come across in the organization and organizational structure. If you’re interested in the previous article in the series, you can find it here.

Let me know what you think.

You don’t have a diverse workforce

Diversity, of course, has many attributes like gender, race, orientation, outlook, experience, age, skills, school, background, and so on. Amongst many advantages, it will give us new perspectives, helps eliminate fear and improve empathy, and can help mitigate impacts of cognitive biases like group-think. In turn, this helps make us better at solving problems.

This is becoming increasingly important: with the demand for software developers doubling every six years or so, we will fail if we persist in our biases.

Some things should not be diverse: competence, collaboration, a sense of innovation, and a selfless desire to drive results and make things better come to mind.

I read somewhere recently that “inclusion is free”… the thought being that eliminating personal biases really shouldn’t cost you anything, just make sure that you’re hiring the best for the job. I spent a lot of my career thinking this way.

But it’s not that simple: it implies a level playing field, and that there is a diverse community out there banging on your door looking for work. I feel that this isn’t good enough, and inclusion has to be worked at. By this, I mean two things:

  • How do we encourage those who have self-excluded themselves from consideration, be it through lack of empowerment, bad experiences, or something else;

  • How do we ensure that we have the right culture, facilities, and environment in place for so that everyone has a chance to thrive?

Do you have the right culture in place to empower and develop every individual in the organization? Companies that truly believe in inclusion and embedded it in everyday routine are better equipped to succeed than those who give just a little deference as a side project.

One of the things that Enova does incredibly well is that it’s not passive in it’s inclusion: it strives to proactively grow and encourage the community, both through events such as ChiWitCon and more broadly. I think that these are exemplary behaviors and it would be great to see them embraced more.

You have a DevOps team

If you’ve built your DevOps team from an operations team, and they are still heavily involved in building and deploying applications and infrastructure on the part of developers, then it’s not DevOps, it’s a sheepish looking Ops with a ‘Dev’ glued in front.

This silo was likely designed with the intent of making development more efficient. In practice, though:

  • A disjoint development life-cycle may mean that there are integration surprises when an application finally makes its way out of the developer’s hands.

  • It’s hard to keep developers thinking about how the code will actually run, and the isolation leaves opportunity for bad or poorly performant design.

In contrast, a DevOps Tooling Team (or Delivery Team, or Platform Team, or…) is a bit different since it’s about building and maintaining homogeneous infrastructure and automation that makes it trivial for developers to deliver early and often. But that team is a DevOps team just like any other, responsible for delivering incremental value for services in their domain.

Eliminating operational silos to empower developers through the use of simplification and automation so that they don’t have to wait is crucial to success.

Your company prioritizes short-term profits over long term value

https://www.atlassian.com/devopshttps://www.atlassian.com/devops

Okay, so perhaps you’re thinking it’s a stretch to include company profits in a DevOps blog post?

Remember though the various flavors of the DevOps cycle: planning begins with the product teams, and ideally a company has enough telemetry that the business is getting fast and accurate feedback on the value that feature changes are delivering.

Certainly if your organization behaves like a Feature Factory where there is pressure from the business to deliver unverifiable short-term incremental profit over ensuring long-term well-being of the company, then there is a problem.

The Seven Diseases of Western Management Lack of consistency of purpose Emphasis on short-term profits Personal review systems including evaluation by performance, merit-rating or annual review Mobility of management Running a company on visible figures alone rather than ‘invisible’ or non-numerical data Excessive medical costs Excessive liability costs — W. Edwards Deming, Out of the Crisis

From a development perspective, the effort expended in advancing functionality whilst maintaining quality have to be balanced carefully.

Although it’s impractical for any organization to be run by perfectionists (since nothing of value will ever be delivered), I’ve seen the consequences a purely feature-led agenda without balance, and its not pretty.

One anti-pattern that is not uncommon is related to feature prioritization, and I’ve seen several places where product teams had several hundred P1 requirements. When everything is high priority, then nothing is. I’d talked earlier about the responsibilities of the development organization, but I think it’s also incumbent upon the product side of the company to be thoughtful and disciplined in how it makes decisions around feature prioritization.

Developers are not first-line-of-defense

If performance and availability aren’t first-order concerns for software teams, then how realistic is it to expect that the software they produce will either be performant, or highly available?

In spite of the best intentions, it’s very easy to develop a sense of “somebody else’s problem”.

At one company I worked with — and I’m pretty sure they aren’t unique — there was a team named Site Reliability Engineering. This particular SRE team was responsible for monitoring health of all production systems and acting as first responder when something went wrong. Because of the size and substantial technology variations in the portfolio of applications in the company, it took six months for a new member of the SRE team to get up to speed.

It also meant that development teams — although quick to help when called — were focused on features rather than reliability, and prioritization decisions were made that inevitably had a negative impact on the maintainability of the code.

I think it comes back to the company’s culture, leadership, and core values that help prioritize what is important. If performance and availability are prioritized, measured, and continuously improved then the behaviors and organization will change for the better.

Your company favors manual testing over automated testing

The role of a QA engineer is, of course, to assure quality.

It turns out, though, that it’s really hard to prove that a piece of software is fit for purpose.

So how does a QA engineer prove that they are adding value. Oftentimes (and as a generalization), I’ve seen a couple of ways:

  • They build extensive and complex test suites that either take a long time to run or a lot of effort to maintain

  • In looking for bugs and attempting to show leadership, they stretch the definition of their remit and submit feature enhancements because they don’t like the way something works.

Whereas there is merit in exploratory testing, adding a silo for building automated tests will encourage proliferation of unnecessary tests, and gating release on a manually operated silos will impact delivery far more than a more minimal (but still adequate) approach.

You’d like to be able to use advanced practices like automated canaries and chaos engineering, but your tooling is slowing you down

It’s easy to believe that you have a unique set-up and won’t be able to leverage prior art. Implementing your own CI/CD solution sounds easy and fun doesn’t it? Like how hard could it really be?

A common disease that afflicts management and government administration the world over is the impression that “Our problems are different.” They are different, to be sure, but the principles that will help to improve quality of product and of service are universal in nature. — W. Edwards Deming

I think, though, you have to ask yourself: can you truly commit the long-term resources required to learn (and stay up to date with) current best practices in tooling, as well as implementing, maintaining, supporting, and extending tooling as demand dictates?

This is not trivial. Amongst many other companies, Target has some great stories on the relief that came from abandoning their own multi-generations of custom tooling and switching to Spinnaker. Not that Spinnaker is your only option, but anywhere that a community has your back and can share the pain of investment has to be a win.

You have shadow IT, or several teams performing similar functions

Maybe IT had some operational responsibilities for engineering, but they were either stretched thinly or haven’t had time to stay up to date with current thinking and best practices. Maybe it’s an engineering services team, or cloud infrastructure… or perhaps it’s several teams.

I’ve seen this at a handful of places, and usually what happens is that frustration levels will build until someone decides to take matters into their own hands and develop their own technical solution. This might work for a while, but:

  • since they are instituted to solve a short-term problem, and — since the solution may not be part of the instigator’s “day job” — they aren’t always maintainable;

  • they may operate distinct from corporate systems such as single-sign-on, and can open the possibility of intrusion or exposure of confidential information

An alternative I’ve seen is where an organizational unit — unhappy with the service they are provided and recognizing how productivity is affected — formally sponsors a set of shadow systems. This may have the advantage of being longer lasting, but is ultimately divisive and means that:

  • there is potential confusion and analysis paralysis about preferred direction for newcomers to the organization;

  • there are diverging roadmaps between implementations, risking duplication of infrastructure and third-party sourcing;

  • similarly, now multiple teams are performing the same or similar function, leading to inefficiencies and reduced economies of scale

These shadow organizations won’t address the underlying organizational or focus issues. They will likely make a true transformation harder, since as well as changing the mindsets of individuals, there will be organizational momentum to overcome (which requires it’s own article).

There’s a trick to having conversations at the right level in the organization to effect efficient, long-lasting, and positive change.

Things aren’t right, but if we just do [insert trendy framework] everything will be fine

Although it’s better than can’t even syndrome or analysis paralysis, if the culture of an organization is broken then adding Scrum, SAFe, Continuous Delivery, Lean Six Sigma, or any one of a dozen other paradigms won’t make a bit of difference.

Developing a highly performant culture with people who are competent, engaged, and effective isn’t something that happens by sending them to a class or have them follow a set of rituals.

The process of organizational change management is hard to do well. It’s not to say that these frameworks do not have value, it’s just that to follow ritual without a true understanding and competence in the actual work will not yield good results. Johanna Rothman does a great job of explaining this.

It’s also important to not let ritual fool you into thinking you’re being productive, and it’s useful to be introspective about the value that each of your scaffold meetings and tasks (things like retrospectives, sprint planning, and so on) have on productivity. Meetings are expensive, and in this way you can evaluate how much better (or worse) that time affects productivity for each one of the individuals involved.

You heard about DevOps, and you’re going to figure it out…

Maybe you read about DevOps in a blog post or at a conference. Perhaps someone in your management chain thought it was a good idea.

By its very nature, DevOps is incredibly interdisciplinary, and there are many pitfalls. You can’t be just organizational change management, or just change management, or just development, or just operations, or just process, or just quality, or just IT, or just…

It is not enough to do your best; you must know what to do, and then do your best. — W. Edwards Deming

Please don’t embark on this journey alone: experienced companies and individuals are within reach to help keep you on track.

Summary

I sometimes hear about improving the way that software is delivered by deploying tools like Jenkins to help with continuous integration and deployment.

Whereas these inevitably have their place, they will allow you to win a few battles but not the war. Take another look at the Manifesto for Agile Software Development:

https://agilemanifesto.orghttps://agilemanifesto.org

The solutions to the symptoms presented here start with conversations and a change in mindset. They are intended to be consistent with that mantra of individuals and interactions over processes and tools, although fixing process is a lot of what comes next and automation is ultimately crucial.

If you’re new to this journey, then I urge you to consider outside help. But either way, as you and your organization look to continually improve, don’t let the promise of technology or a magical framework fool you into thinking that’s all that’s needed.

p.s. Thanks to Kevin Rischow for multiple rounds of review and feedback across this entire series, and Ana Lebron for her help on the diversity and inclusion section.

p.p.s. For another angle, take a look at The DevOps Checklist.

p.p.p.s. I ended up quoting W. Edwards Deming a lot through this. His work is worth checking out if you’re not already familiar with it.

p.p.p.p.s. Can’t even syndrome needs an article to itself, but it’s basically where an organization learns to fear change.

p.p.p.p.p.s. If you’re still in any doubt about why diversity and inclusion matters, then I hope you’ll follow the link to this article from McKinsey & Company.

This article originally appeared on Medium.