Pitfalls in Measuring SLOs

By Gremlin
In this talk, we will discuss how we brought the theory of SLOs to practice, and what we learned that we hadn’t expected in the process.
Course info
Level
Intermediate
Updated
Nov 6, 2020
Duration
28m
Table of contents
Pitfalls in Measuring SLOs
Description
Course info
Level
Intermediate
Updated
Nov 6, 2020
Duration
28m
Description

We built support for SLOs (Service Level Objectives) against our event store so we could monitor our own complex distributed system. In the process of doing so, we learned that there were a number of important aspects that we didn’t expect from carefully reading the SRE workbook. This talk is the story of the missing pieces, unexpected pitfalls, and how we solved those problems. We’d like to share what we learned and how we iterated on our SLO adventure. As an SLO advocate and a design researcher, we collected user feedback through iterative deployments to learn what challenges users were running into. This conversation will discuss how we iterated our design, based on user feedback; how we deployed, what we learned, and re-deployed; and how we collected information from our users and from the alerts our system fired. In this talk, we will discuss how we brought the theory of SLOs to practice, and what we learned that we hadn’t expected in the process. We’ll discuss implementing the SLO feature and burn alerts; and our experiences from working with the SRE team who started using the alerts. Our hope is that when you buy or build your SLO tools, you’ll know what to look for, and how to get started. implementors will be able to start with a more solid ground, and that we will be able to advance the state of SLO support for all teams that wish to implement them. The major design points will be broken into a discussion of what we actually built; a number of unexpected technical features; and ways that we had to educate users beyond the standard SLO guidelines. The talk is largely conceptual: no live code will be shown, although some innocent servers may well die in the process of being visualized.

About the author
About the author

Gremlin's enterprise Chaos Engineering platform makes it easy to build more reliable applications in order to prevent outages, innovate faster, and earn customer trust.

More from the author
Failing over without Falling over
Intermediate
21m
Nov 30, 2020
More courses by Gremlin