Illustrated by Matt Peet

DevOps has certainly evolved over the past several years. Sure, it’s still a super-set of principles and practices that help development teams move more quickly and efficiently. But veteran engineering leader Mike Rasmussen understands DevOps as a broader cultural shift.

“In fact, you can implement CI/CD and automation and not have a DevOps culture,” Rasmussen says. “It’s really about a shared understanding between development and operations, QA and product and every piece of the value chain. It’s about all those folks working together to deliver value to customers.”

Rasmussen was most recently VP of Sling TV’s engineering team, where he oversaw major digital transformation. In previous roles, he led as VP of Product at PayClip and headed Product and Engineering at Electronic Arts and other companies in the gaming industry. He ran live games and free-to-play games with millions of customers, where he learned how to create compelling products.

“At each of those places, if the team had a very strong DevOps culture—even before it was really called DevOps—they could move quickly,” he says. “They had cultures of shared responsibility. They worked across silos. They had fully empowered product teams. DevOps has emerged as a way to articulate some of those principles.”

In this article, Rasmussen digs into how to shape a DevOps culture through trust, including how to establish trust as a leader and continue earning trust through development teams’ results. He illustrates how these cultures benefit from prioritizing outcomes over output and describes the core elements of achieving successful outcomes. These transitions can seem radical for leaders and teams alike, and Rasmussen shares how leaders can work with their developers through this cultural shift into an atmosphere of trust and ownership.

Early in a past job, Rasmussen asked his manager why QA was not embedded in the engineering organization. The answer surprised him.

“He said that was like having the fox in the hen house,” Rasmussen recalls. “He asked, why would we do that? Who would keep the engineers honest?”

He has since translated that boss’s answer in different terms. Keeping QA and Engineering as separate as foxes and chickens sends the message that they can’t be trusted to collaborate. That each team has certain self-interests that, once the outside check is removed, would conflict with organizational goals.

Such sentiments run counter to Rasmussen’s ideas about DevOps: it’s about people owning their performance and their results.

“No developer is trying to pull the wool over anybody’s eyes,” he says. “They’re trying to deliver value. They want to know what’s wrong with their software and what’s not working in production. They want to know whether or not customers are experiencing issues.”

A culture of trust grants engineers access to that information by shifting disparate groups (such as Engineering and QA) into one team that owns the entire value chain. It’s about empowerment, Rasmussen says, and allowing these teams to do what they naturally want to do.

A lack of trust in an organization leads to excessive processes and bureaucracy that calcify the value chain. The different groups within the company have little to no communication.

“It’s a kind of command-and-control environment, or a feature factory,” Rasmussen explains. “The feature ideas come from on high, and the Engineering or Product team cranks through them. That’s not DevOps. DevOps is about empowering teams, and having a shared ownership of the customer experience. A subset of DevOps is CI/CD: test automation, deployment automation, tools like feature flags and canary releases. But you can implement automation and still not have a DevOps culture, because at its core, it’s about trust.”

The best way to make a development team trustworthy? Simple, he says. Trust them.

“DevOps is about giving teams the responsibility of the outcomes,” he says. “You have to cede some control, and that requires trust.”

However, leaders can’t just talk about a culture of trust. That trust has to manifest somehow, and Rasmussen sees it rise up in cultures of learning and failing fast. When a blameless culture builds itself around learning from mistakes and preserving psychological safety, those teams can move much more quickly than ones trying to identify fault and assign blame.

Trust alone does not buttress a DevOps team. “You still have to have canary releases, automated deployment, automated integration, gated check-ins—all the actual mechanics of DevOps,” Rasmussen says. “Those have to drive outcomes. But you can’t just do all those things and ignore trust issues. You have to have both.”

Along with the culture of trust, a development team still needs to show results. That’s the ultimate way for leaders and teams alike to continue earning trust from each other and the organization.

How to create meaningful results—or, which results matter—is the question.

“I don’t think you gain that trust by just moving quickly and releasing in production, willy nilly, without having any ability to manage or mitigate that risk,” Rasmussen says. “We’re engineers. We’re constantly pushing things into production. Without a plan for how to manage that, Operations might view us as a marauding horde. Suddenly battle lines are drawn. So we have to be cognizant of the results and how they build that trust.”

On a journey to DevOps, teams accomplish trustworthy results in stages. Rasmussen highlights these stages for a development team:

• Teams should have a full self-service platform that enables them to manage their releases all the way through to production. “Independently, a team—which includes an Operations individual—can release to production without any sort of outside approval,” Rasmussen says.
  
  However, teams need some automation in place, and reliable testing, to earn this platform. Which   leads to:
• Teams require the ability to do canary deployments. One of the first things Rasmussen’s team put in place was a feature flagging system, which supported the ability to roll back releases when things broke. “We were able to demonstrate that we can release to a small cohort, we can track the results, and we can control the blast radius” he says. “That meant we could actually test in production, which is necessary in many cases. This showed Ops that we understood the risks, and where committed to mitigating them.
• Teams need to verify their automated deployment process. “You need to start demonstrating that you can actually cover the same use cases and the same risks that your current manual testing processes are covering with automated testing,” he says. “Ultimately, you can have the entire CI/CD pipeline automated because you’ve built that trust.”

A significant part of DevOps is measuring everything to understand if teams are moving in the right direction with efficiency and productivity. 

“So often companies implement OKRs and don’t have empowered product teams in DevOps,” Rasmussen says. “Thankfully, most teams aren’t just measuring lines of code anymore, but they are continuing to measure output. That completely misaligns the incentives. It reduces alignment and shared responsibility.”

Rasmussen’s solution? Value outcomes over output.

Let’s say that a team’s mission is to get a payload across a river.

• The company expects output-driven results if they provide the team with specifications for the bridge. Design, aesthetics, tolerances. The leaders likely want reports when certain milestones are reached so they can measure a sense of progress.

• The company allows outcome-driven results if it simply presents the team with the mission: We need to get this payload across that river. How they accomplish that is up to the team.

“Engineers might build a rocket that launches across the river that’s way more effective than a bridge, or they might build a floating bridge that uses less resources,” Rasmussen explains. “They might tunnel under the river, or they might build a zip line. It ultimately doesn’t matter to the business. They just want to get across the river. And if you save money and resources in doing it, even better.”

Transitioning from output to outcomes is a mind-shift for many leaders, because stakeholders are so often used to pulling output-based levers. That’s how they feel they add value as leaders.

But the opposite is much more true. Even (or especially) when the going gets tough, leaders need to lean back on DevOps principles and trust and empower their teams to deliver the outcomes they want.

“Most dysfunctions are the result of the leadership, not the teams,” Rasmussen says. “It’s on leadership to model vulnerability and trust. Help yourself out by assuming positive intent. Your teams are not trying to fail. They’re trying to succeed. Hold them to a high standard, and show you expect results by putting the ownership on them.”

Success in DevOps is observed in outcomes and achieved by trust. And the results are still measurable and quantifiable. Rasmussen highlights some big elements that DevOps teams can incorporate to attain more concrete success.

Rasmussen started out pretty broad with the team at Sling TV by measuring these four high-level metrics: deployments, lead time, change failure rate and mean time to recovery. That gave the team a baseline and underlined some issues right away.

For instance, one development team had an average 45-60 day lead time, meaning an engineer’s code took nearly two months to hit production. By that point, any engineer has switched context dozens of times, and problems in production are laborious to address.

“So the first thing we focused on was how we could get more deployments out and reduce our lead time,” Rasmussen says. “We all needed to get a little bit uncomfortable with how fast we were moving.”

But they also needed some mitigation for the risk. That led to the second element.

A different one of Rasmussen’s teams had already ventured down the pathway of automation. So, the other teams leaned on that one as the flagship team to demonstrate how to successfully automate.

The automation itself was pretty simple. The right programs exist. The troubled team actually had much of it in place already. But its culture didn’t trust Operations or QA, the testing or the automation.

“We put some basic functionality in place, like feature flags, so we could do canary releases, push the boundary, all while mitigating risk,” Rasmussen says. “We did a few examples to try and gain the team’s trust, to actually start releasing into production.”

Over that quarter, that team worked through the automation issues together with Operations and QA—and its lead time shrank from 45 days to 45 minutes. That’s how much automation was already in place. The team simply lacked the leadership to instill trust in it.

“Once that happened, the flood gates opened. Every team saw the benefits and wanted to move in that direction,” Rasmussen says. “Every team saw they didn’t have to go through a crazy approval process. That they had an embedded ops person who largely approved what they needed to get into production. So we started automating every team.”

The next level up was to start measuring the actual automation. A 45-minute lead time was impressive enough, but what was the percentage of unit test coverage? What was the percentage of integration test coverage? How much of the CI/CD pipeline was automated? Which teams had already integrated security?

“We created an assessment that would measure all that at a finer level of granularity,” Rasmussen says. “We started out with those four high-level metrics. Once we accomplished those, we knew there was headroom to look at the additional details. We could chunk things down to the actual nuts-and-bolts of the automation.”

These transitions into a trusting DevOps culture can be as radical and unfamiliar to engineers as they are to their leaders. So it’s up to those leaders to help coach their team members through these changes.

Rasmussen sees three layers for leaders aiming to accomplish that: champion those who embrace the changes; hold struggling teams accountable; and own the process for their companies.

While implementing these changes, Rasmussen plucked out two Sling TV teams who were closely coupled and thriving in this environment. He allowed them to demonstrate the benefits of the new environment.

“We basically held them up and said, look, this team was able to go all the way to production,” he says. “They had automated testing here, and they used this tool for deployment. And these tools are available for everybody. If you want to have the same results as this team, here’s who you go talk to and how you do it.”

Note that Rasmussen did not impose these changes on all the teams—even though the strategies were proving successful. That returns to his philosophy of pushing ownership onto the teams and allowing them to take responsibility for their own success.

“It’s kind of cart and horse,” he explains. “The cart is that teams want to release into production without an arduous approval process. The horse that pulls that cart is that they have to automate their testing and deployment. They have to be able to manage the potential risks of deploying all the way to production. That’s ownership.”

Now, allowing teams to assume their own ownership doesn’t mean that automation is a Wild West free-for-all. Rasmussen’s leadership establishes certain requirements for releasing into production: automated testing, roll-back ability, canary releases. They may implement some standardization across tools, for example. But the creativity itself is completely in the team’s hands.

“We try to let them have as much autonomy as possible about how they actually solve problems,” he says.

That autonomy comes with definite responsibilities: teams must be accountable for the outcomes they signed up for. Results matter and trust matters, Rasmussen says.

He started the DevOps process with teams by establishing high-level achievable goals, such as increasing deployments by 50%. Not everyone hit those marks.

“The teams that didn’t reach the goals didn’t take ownership,” he says. “It became very clear when we would review our results at the end of the quarter. Why did this not work? They always had competing priorities.”

Rasmussen could work to smooth out those conflicting priorities and clear any other roadblocks. He sees that his role is to mitigate those problems. But the results? Only the team can be responsible for those.

Companies oftentimes go through this DevOps transformation at the same time that they’re trying to create empowered product teams, and at the same time they’re trying to move to outcomes-based goals. That was the case for Rasmussen and Sling TV. And it’s not easy to do any one of those things, let alone all three at once.

Teams are responsible for their results. But leaders are responsible for the entire process teams use to achieve outcomes. Teams own the work, but leaders own the culture—and the journey of altering it.

“It ultimately is on leadership to create very clear outcome-based guardrails for how those teams need to operate, where they need to go and then holding the teams accountable for it,” Rasmussen says.

Rasmussen explains how a culture of trust is a necessary element in a DevOps atmosphere.

• Trusting cultures place ownership and responsibility on the teams, and integrate teams to shorten the value chain. Leaders facilitate more trust simply by trusting their teams—developers want to succeed and most often just need the opportunity. Showing concrete results from automation, testing and risk mitigation continues to earn deeper trust from the rest of the organization.

• Leaders create stronger ownership by shifting from output-driven goals to outcome-focused ones. Asking for outcomes rather than output shifts the team’s incentives and expectations and increases their freedom to solve problems creatively and effectively. The shift is often more difficult for leaders who are used to pulling output-driven levers, and their teams need them to lean instead on DevOps principles.

• Identifying elements of measuring and attaining successful outcomes offers clearer strategies for implementing trust-based DevOps practices. Rasmussen identifies four high-level metrics (deployments, lead time, change failure rate and mean time to recovery) that his teams improved through automation—which itself needs to be assessed for efficacy.

• Leaders must work with teams through these big-time cultural changes. Some development teams will embrace the changes—and those champions become the exemplars. Struggling teams need to be held accountable for what they signed up for, and leaders must own the transition process into DevOps for their organizations.

“Culture is almost always at the root of the problems on your team,” Rasmussen says. “It’s on leadership to find a way through those problems. You have to lean on the principles and values of your culture or create them if they’re not there.”