Skip to content

Contact sales

By filling out this form and clicking submit, you acknowledge our privacy policy.

Control failure with chaos engineering

Chaos Engineering is introducing controlled disruptions into distributed systems, studying behavior and improving resiliency with automation. Learn More!

Jun 08, 2023 • 3 Minute Read

Please set an alt value for this image...

Chaos Engineering is about introducing controlled disruptions into a distributed system, carefully studying the behavior, identifying the weak areas, and improving resiliency with automation.

Evolve your systems to be fireproof with automated fire drills.

Distributed systems are becoming more complex as microservice architectures evolve, and more unpredictable as the velocity of development increases.

A distributed system may have distinct services that work just fine in isolation, but interactions with other system components can create unpredictable behavior — often only exposed and observed in production environments.

“Everything fails all the time. We lose whole data centers! Those things happen.”

Werner Vogels

So how do we identify these weaknesses before they spread into our production systems and impact the customer experience?

Controlled disruptions

By carefully designing experiments that introduce controlled disruptions, we can proactively identify and address weaknesses — and break away from the dysfunctional and reactive incident response model.

The disruptions can be as simple as killing a process on a Linux server, or inducing errors for segment of live traffic serving customers in production.

Chaos Engineering example experiments that introduce controlled disruptions into complex distributed system include:

  • stopping, rebooting, and terminating virtual machines
  • removing network services, routers, and load balancers
  • simulating the failure of an entire region
  • introducing latency between services, missing messaging topics, random errors, and crashing docker containers
  • mimic the unavailability of 3rd party APIs or create additional latency
  • simulate operating system issues with I/O, CPU, and memory spikes

While Chaos Engineering techniques are still being learned and adopted by smaller organizations, the methods are already institutionalized across large organizations like Amazon and Netflix.

Failure as a Service

To support controlled disruption, the Failure as a Service (FaaS) architecture was proposed in a research study from the University of Berkeley.

The goal is analogous to that of fire drills. That is, before experiencing unexpected failure scenarios, a cloud service could perform failure drills from time to time to find out the real-deployment scenarios in which its recovery does not work.

The architecture is based on three important characteristics:

  1. Failure drills are based on the large-scale injection of disruption
  2. Failure drills are conducted against online production environments
  3. Failure drills are available to the organization as an easy-to-use service

The architecture includes the following components:

  • The FaaS Controller is a fault-tolerant service and sends fire drill commands to the agents running in VMs.
  • The FaaS Agent runs on the same VM as the target service, and receives the drill commands that invokes controlled disruption.
  • The Monitoring Service collects data about the target services that is used for the failure-drill scenarios and specifications.

The architecture provides a basic model for designing a system for introducing common failure modes in large-scale distributed system.

By leveraging the principles of Chaos Engineering to exercise large-scale on-line failures, your organization and systems can evolve into a more resilient — and fireproof — culture.

What’s your experience with Chaos Engineering, or the challenges in your organization with adopting the techniques? Drop a comment below or connect with me on twitter!

Get the skills you need for a better career.

Master modern tech skills, get certified, and level up your career. Whether you’re starting out or a seasoned pro, you can learn by doing and advance your career in cloud with ACG.