Tips for Adopting GitHub Kubernetes in AWS

Last week I was lucky enough to attend the AWS EMEA Summit Online. Containers were a big theme of the summit, and one of the most popular topics on the agenda was Kubernetes GitOps on AWS with Jason Umiker.

So what is GitOps, and why is it becoming such a big deal for businesses trying to run Kubernetes on AWS?

Modern software development practices focus around three main activities: 

1. Source code management using tools like Git
2.  Automated building and testing of applications using tools like Jenkins or CodeBuild
3.  Automated code deployment using tools like CodeDeploy

These activities are often combined together as a Continuous Integration pipeline where the build service monitors the Git repo and automatically triggers a build in response to changes in the source code. 

Taking the idea of automation to the next level, the practice of Continuous Deployment extends the pipeline a stage further by automatically deploying the new code as soon as it has successfully passed through the build and test stages. 

A combined CI/CD approach handles every step in the process, from the moment a code change is committed, right through to deployment, with the huge benefit that your applications work in every environment, be it development, integration, staging or production!

GitOps leverages these tried and tested principles, allowing us to apply the same philosophy to handling changes to Infrastructure as Code (IaC). If infrastructure is managed as code, then why not store that code in a source control system like Git and automatically trigger updates to our infrastructure every time a change is committed?

The migration of workloads into the cloud has significantly accelerated the adoption of Infrastructure as Code as an approach to provisioning resources. Tools like CloudFormation and the AWS Cloud Development Kit (CDK) allow you to codify, model and provision cloud resources using familiar programming languages like JSON and YAML, as well as Python, Java, and .Net. 

Git is the tool that a lot of developers are already using, so it makes sense to bring the operations and infrastructure teams onboard as well, so that everybody can benefit from consistent, streamlined and efficient working practices for managing code.

Kubernetes supports a declarative approach to creating, updating and deleting objects. This means that the desired state of objects in Kubernetes can be  defined by configuration files. Kubernetes deployments, pods, and services like load balancers can all be defined as code, which ideally should be version controlled and stored in a suitable source code repository like Git. 

Tools like CodePipeline and Flux CD can be configured to notice changes in your Git repository and take action to trigger updates to your AWS infrastructure.

Flux CD is a simple utility which runs on your Kubernetes cluster.  Instead of manually running commands like kubectl apply -f ./mydeployment.yaml every time you want to update your application, Flux CD will periodically pull the code changes from your Git repository and apply the changes for you!

By default, the  Git pull frequency is set to 5 minutes, but you can tell Flux to sync the changes immediately using the following command: fluxctl sync --k8s-fwd-ns flux

Now that we have some background, let's run through a few tips for getting the most out of GitOps in your Kubernetes deployment workflow.

With Git you can track the full history of changes to your code. This means you’ll understand who made a change, what they changed, when they made the change, why it was necessary, and who approved it.  Of course that means it’s very important to make sure people include appropriate comments when committing a change and it’s great practice to reference a ticket relating to a feature request or bug.  

Changes merged to the branch can then trigger a deployment in the corresponding environment. You might also consider each developer having their own branch, so that they can iterate their own changes independently and when ready merge changes to a development or integration environment and test against everyone else’s changes, eventually merging to the production branch when ready to release to production.

Enforce peer review in your pull request mechanism for every single change. If you think of your production environment as a castle, then let Git be your drawbridge. Force all changes through the peer review process in Git and configure your pipeline to ensure that all changes go through integration testing before being released to production. This way only successfully tested changes can make their way into prod, ensuring that your application works in every environment.  

With Git, the only thing you need to do to reverse a change is revert back to a previous commit! For example, you could use a command like git revert HEAD to roll back the latest changes and return to the previous commit.

The ultimate goal of course is to have everything automated, so that any change in your Git repo is detected and triggers an update to the corresponding environment. Flux CD is a tool which monitors all of the container image repositories that you specify, detects new images, triggers deployments, and automatically updates the desired running configuration of your Kubernetes cluster. So it basically ensures that the state of your Kubernetes cluster matches the configuration you’ve supplied in Git.

For any resources which are outside of Kubernetes, like RDS databases or EC2 instances, you can use CodePipeline to automatically trigger changes to be applied using CloudFormation.

Using a GitOps approach to managing your Kubernetes infrastructure in AWS is a natural choice.

Your app works in every environment. 

You have your development and operations teams using the same tools.

You’re in control of your environment and you understand the "who, what, why, when and who approved" for every change. Plus you have the ability to roll back to a known and trusted, earlier state whenever you need to!

• Git Definitions and Terminology Cheat Sheet
• The serverless approach to testing is different and may actually be easier
• You need SRE skills to thrive in a serverless world
• How Many, How Well, How Much? Measuring Cloud Success
• Kubernetes Deep Dive