Portrait illustrations by Matt Peet

The term “data-driven improvement” is as close to ubiquitous in the realm of software engineering as any other mantra. We’re a numbers-savvy bunch, and we love experimenting, testing and measuring our work. We generally think of data as a really good idea. 

But data-driven improvement is also like a modified version of a certain card in Cards Against Humanity. (No, not that card.) Fill in the blank:

• Step 1: Data.

• Step 2: _______.

• Step 3: IMPROVEMENT.

What exactly do we do with data to bring about that vision of data-driven improvement? How do we take these raw metrics and convert them into a better world for engineers, better results for organizations, better products for customers?

Pluralsight sat down (remotely) with several tech leaders to glean their insights and strategies for that elusive Step 2, where they convert the numbers into continuous improvement for their engineering organizations. Here, we’ve collected the wisdom of Brenda Jin, founder and CEO of Guanyin Labs; Mark Hopkins, CIO at Skullcandy Inc.; Scott Lewis, VP of Engineering at APiO; and Amol Kher, VP of Engineering at Vida Health. From distinct perspectives, they ultimately share experiences around a central core idea: that the use of data must align with an organization’s vision to drive real improvement.

Prioritization is key to leading a high performing team. Identifying and fine-tuning an organization’s highest points of impact delivers clarity on what’s worthy of a team’s time, and what needs to be dropped—and the right metrics can cut through the scale and the noise to offer that insight.

“This question of prioritization has been really front-and-center in my mind because the priority of strategic decisions that leaders have to make can be at such a large scale,” Brenda says. “With the insights that we're able to do with computing, we can just zoom in on exactly what matters.” 

What matters, of course, depends on the organization and its goals, and it can change with time. Brenda understands that questions of prioritization need to be weighted on multiple axes as leaders decide where to dedicate their teams’ time and resources. 

“Do you tackle the tech debt today or lay the foundation for tomorrow? Is it better to squash bugs, build tools or ship features?” she asks. “There is a limited total amount of time and energy you can devote at work. Being able to align your efforts with the company will ensure that your contributions have maximum impact.”

Figuring out where a team’s efforts will reap the greatest dividends is just the first half of the equation. Delivering consistent performance in those efforts is the other. 

Brenda relies on her team’s priorities to set realistic expectations for their work. Everyone on the team knows what their goals are, and how to measure the incremental progress toward those targets. As she puts it, “In order to be excellent at work, you’ll need to understand your team, and ultimately, your company.” 

Most every team will deal with a lull in performance. In these times, leaders often turn to the past to glean information about the difference between then and now. But Brenda reminds us that the interpretation of performance data is not a static thing—it can (and mostly should) be revisited continuously.

“What many of us do today is look at employees at a snapshot in time and try to work backwards to figure out exactly what happened,” she says. “Leaders often rely on the metrics that are already available. But nowadays, there are opportunities to enhance or aggregate data in new ways to gain insights that can really isolate the key factors that support strategic decision-making. With better data, we can proactively identify which initiatives or leadership behaviors are highly correlated with success.”

Mark understands continuous improvement as a cyclical movement rather than a linear one. Rather than teams progressing one step in front of the other, they always return to the same ideas and the same points to assess them over and over again. He and his team are always reevaluating how they’ve been doing and what small tweaks they can initiate to make an impact, regardless of how slight. 

That’s right: small tweaks, slight impacts. He believes process improvement is less about making big, far-reaching decisions. It’s about metrics, speed and tracking. 

“The concept is to drive decisions with data, make quick incremental changes and then evaluate what the impact is by looking at the data,” Mark explains. “This creates a flywheel of process improvement driven by information.”

He understands the temptation for leaders to go for ambitious projects and bring in more drastic IT or development changes. But from his experience, the small steps often make the big differences. He and his team opt for making very subtle changes, measuring results as they come in, and pivoting as needed.

“We use this concept of driving decisions quickly with data, making quick incremental changes, then seeing what the impact is by looking at the data,” Mark says. Without such data, decisions are discreet events that aren’t replicable or measurable over time. So true change starts with seeing the results. Their analysis provides new ideas that can be tested, implemented and measured.

This data-driven cycle creates the constant feedback to drive better results, and according to Mark, it brings the most value to the business. 

“It's a very quick and iterative process to develop systems and put processes in place that deliver business benefit,” he says. “It's much more fulfilling when you do something and you can see the business value that it's delivering.”

A danger inherent in being driven by data is that the data replaces the human experience. That has never been the point of incorporating metrics into decision-making for any of the experts we’ve spoken to. In fact, many of them take direct measures to ensure that the numbers enrich the development experience for engineers as a community, and for customers and other end-users.

Scott’s engineering organization at APiO has benefitted from evaluating the frequency of commits and story points over the goal line. The initial reactions from the engineers when he introduced these metrics were not atypical—they worried about their flaws being exposed to their peers and the effects of Big Brother supervision.

“But what it ended up doing was raising all boats,” Scott says. “They started looking at outcomes.”

Now the culture at APiO is that everyone can learn from everyone else. Engineers check in with one another frequently to offer support and code reviews. Interns come in with new ideas that senior engineers pick up, and the senior engineers help out the newest team members to help them get up to speed. Everyone learns from what the best engineers are doing, and the data helps them understand how to replicate those good behaviors.

“The main thing that we as leaders need to always be cognizant of is how do we help them get to where they are trying to go, while at the same time benefiting our company,” Scott says. “But building people is more important than building companies. Build people, and companies build themselves.”

One of the pitfalls of data-driven organizations is that they simply have so. much. data. When Amol served as the VP of Engineering at Life360, his organization processed user feedback from more than 25 million Life360 users. It would have been impossible for his teams to take action on every piece of that feedback. He knew full well that no one can listen to all of it.

“You can create or put in systems pretty quickly to know if there are any emerging trends going on,” he says. “Systems that can help you get data about your users, customer service metrics, any sort of internal testing. You’re not addressing feedback individually, but you’re seeing what the trend lines are. When fifteen people are taking the time to tell you that there must be something going on, don’t dismiss it.”

Amol adds that teams should listen to all their channels for customer feedback. Talking to customers doesn’t mean focusing on any one single avenue. Twitter might be as insightful as the app’s help feature, and other departments—like customer service—can relay the input they receive.

“Form alliances with everyone else in the company, so everyone’s focused on solving those problems,” he says.

With the inherent business need for data in setting and measuring progress toward organizational goals, the numbers need to be as consumable as possible. Metrics have a story to tell. But humans, unlike computers, are visual learners. Recalling her time at Slack, Brenda highlights how her team used visual dashboards to deliver key information in a timely manner. 

“When we changed rate limits to API tokens, one of the first things we did was set up a dashboard so that our business partners across partnerships, product and customer experience could have real-time visibility into which developers and teams were affected by the new rate limits,” she says. “Because we had this visibility, we were able to stay in sync as we deployed the changes while minimizing negative end-user impact. It would have been much harder to do if any of the business stakeholders had to pull ad hoc reports.” 

Developing a strategy behind the metrics an organization is tracking and making them readily available, accessible and understandable enables tech leaders to inform their decisions and implement them quickly. According to Brenda, the right data allows leaders to see the issues at hand and act much faster. 

“Data can help you discover the number one thing that we can influence and change today,” she says. “Companies can course correct much faster if they can isolate issues at that level.”

Data is powerful. It can help organizations make more informed decisions. And the more data available to you, as a leader, the more insights you can extract from it. For leaders who want to find their weak points, data can spot-check their instincts and open investigations into deeper issues—if they are truly willing.

“Data is only as good as our own biases,” Brenda reminds us. “In order to truly find gaps and blind spots, leaders need to be as open to seeing those in the data as they are about finding them in their organizations.”

Tech leaders must truly engage with the metrics they have in order to observe their own biases at play. They will already be well on the way to constantly improving their own use of data in their organizations if they observe these key ideas from the leaders we spoke with: 

• Organizations perform more effectively when they use data to prioritize their highest-impact pushes, as well as to perform consistently in the short and long term. 

• Continuous improvement comes much more readily from small, incremental, well-tested changes than from the headline-grabbing, big-ticket implementations.

• Data needs to bolster the human experience, rather than subjecting it. Developers can learn (individually and collectively) how to improve their performance and what new skills will have greatest impact. Teams can view customer data in the aggregate to better understand the end-user experience. And in any case, data made visual is more easily utilized by all stakeholders than raw numbers.

The one thread tying all their ideas together?

• Data alone cannot lead to improvement. Leaders need to align their analysis with the organization’s strategy for success—whether that goal is improving an engineering team, a product, a profit margin or a customer experience. 

Data-driven improvement is about more than collecting as many metrics as possible. It’s about mapping a team’s efforts to its organization’s goals and objectives, then creating metrics to seek out understanding regarding their progress.

“You could just look at lines of code committed or number of pull requests per engineer,” Brenda says, “but that doesn’t speak to the quality of an engineer’s output, or the strategic initiatives they’ve contributed to. That’s why companies really need to think about their data strategy and the infrastructure they’re going to need to support the data insights that they want.”