GitOps is a fairly new (2017) style of implementing DevOps practices that has quickly grown in popularity. This 3-part blog series will:
- Explain the fundamentals of GitOps and the tools you will need in your repertoire to make the principles and practical approach a success in your enterprise.
- Provide a walkthrough of the steps needed to setup and install all the necessary components for a successful GitOps automation process.
- Outline the inner workings of the Caylent GitOps Accelerator and all its benefits.
Origin of GitOps
The term GitOps was coined in a blog post titled “GitOps—Operations by Pull Request” published on August 7, 2017 by Alexis Richardson, the CEO of Weaveworks. However, the fundamental concepts that underpin the GitOps methodology were largely devised by Google and codified in the now-famous Site Reliability Engineering book that was published in March 2016. However, SRE is a very Google-specific methodology that at the time was difficult to implement anywhere else. This has changed in the intervening years, but in the meantime GitOps has evolved to fill in some of the tools and practices that make it possible for everyone to manage systems the way Google does internally.
What Is GitOps?
At a high level, GitOps is a loose methodology, or collection of tools, processes and principles that collectively make it easier and more cost-effective to develop and operate cloud-native software. I call it a “loose methodology” because, as methodologies go, it is not particularly prescriptive and leaves a lot of room in the tent for people and organizations to make their own decisions.
The fundamental problem that any DevOps methodology (GitOps included) is attempting to address is that developers and operators are inherently at odds. Developers are trying to release new features as quickly as possible, increasing the rate of change to the system. Operators are trying to keep the system operating reliably, and thus want to minimize changes to the system. Various DevOps methodologies address this problem in different ways:
- Traditional DevOps takes a cultural approach, bringing the incentives of developers and operators more in line with each other and improving communication between them.
- SRE takes a role-based approach, combining the responsibilities of developers and operators into a single role called a Site Reliability Engineer, and establishing a set of rules for how those people should split their time between development and operations tasks.
- GitOps takes an automation approach, replacing human operators with software automation and allowing developers the freedom to pursue feature development.
So you may not be surprised to find that the glue which holds Git and Kubernetes together in the GitOps world is a piece of software called an “Operator”. This software lives inside of a Kubernetes cluster and is responsible for constantly watching the Git source repositories. Any time a change is made within a Git repository, the Operator leaps into action, making whatever changes it needs to inside the cluster to match the current description of the system in Git.
GitOps Tool Stack
There are several fundamental tool types that are required to properly implement GitOps:
- Source Code Repository
- Container Registry
- Continuous Integration (CI) Tool
- Continuous Deployment (CD) Tool (a.k.a. Kubernetes Operator)
- Secrets Management Tool
- Infrastructure as Code (IaC) Tool
As you can see from the diagram above, for the Caylent GitOps Accelerator used in the examples we have chosen a specific stack which includes GitHub, Cloud Build, Container Registry, Secret Manager, GKE and ArgoCD. These are by no means the only choices or even the best choices for you, but they were chosen to provide the most easily accessible and cloud-native setup for a broad audience.
In the next few sections we will take a look at the available options for each of these tool types and discuss their relative strengths, so that you can make your own choices when looking to implement GitOps in your organization.
Source Code Repository
As the name GitOps would imply, Git is a fundamental part of the required tool stack. While git is an open-source project available on its own, there are many popular managed Git implementations that provide additional tooling above and beyond just managing the source code. While it is possible to implement GitOps with a vanilla Git installation, it is not the recommended approach.
By far the most popular SaaS-style Git implementation is GitHub. GitHub is free to use but also provides Enterprise subscriptions for more powerful features. Many of the world’s free and open source projects are hosted on GitHub, and it has many very useful features that make it easy to work with. This includes issue tracking, release management, built-in container repositories, the ability to bundle multiple repositories into projects, and much more.
The accelerator used to demonstrate GitOps is hosted on Caylent’s Enterprise GitHub, and we will be using it later in our coding examples.
GitLab is a popular and opinionated alternative to GitHub that was specifically built with DevOps in mind. It has not seen the same broad adoption as GitHub, largely because it requires a paid subscription for use. However, if your organization is willing to pay for the functionality, GitLab is a powerful choice for the basis of a GitOps implementation. It has many of the same features as GitHub, but also provides turnkey automation for the overall flow of software development.
Bitbucket is the git-based source code repository managed by Atlassian, and is frequently used as part of the broader Atlassian cloud suite that includes JIRA, Confluence and other tools. Although it is not as popular as GitHub, because the Atlassian suite is so broadly adopted within Enterprises many people have access to Bitbucket repositories at work.
Cloud Source Repositories
Google Cloud provides Cloud Source Repositories for you to store, manage and track code. Cloud Source Repositories are more than just a Git repository, you can use them to quickly design, develop and manage code securely. This can be a good choice if you are developing code to be deployed on the Google Cloud Platform, because it is tightly integrated with Cloud Build and other tooling.
Continuous Integration (CI) Tools
Cloud Build is a GCP service that executes the build step using “cloud builders”, which are pre-packed docker containers with appropriate tooling installed. Cloud build is a completely serverless platform that gives you the flexibility to create custom build steps, so each step in the process can use a completely independent set of tools.
The accelerator used to demonstrate GitOps makes use of Cloud Build primarily for the fact that it is modifying GCP resources and the built-in tooling and service account access make this a breeze.
CircleCI is a SaaS-style build engine that lives in the cloud or can be hosted on-premise. By including a simple circleci.yaml file to configure in your repository to define the build steps, CircleCI can act as your CI/CD engine.
Similar to CircleCI, TravisCI is another SaaS-style build engine that provides both cloud and on-premise versions. TravisCI has been around longer, and has closer ties with the open source community. The most prominent feature of TravisCI is that it is designed to support parallel builds simultaneously in multiple environments, although the pricing plan to do so can get expensive quickly.
Jenkins X is under the Jenkins umbrella of products, but takes quite a different approach from the original Jenkins and is specialized to work with containers and Kubernetes. We refer specifically to Jenkins X here because although you can use either one for the Continuous Integration side of things, Jenkins X is the only one that plays well with Continuous Deployment in a Kubernetes environment. Jenkins follows more of a “kitchen sink” approach, trying to solve all the problems in your stack, rather than just keeping a tight focus on CI or CD, it also provides testing tools and a Kubernetes Operator. If you want one tool to rule them all, Jenkins X is the place to look.
Personally, I prefer a small collection of very focused tools over a single uber-tool like Jenkins X, but make your own assessment here.
Continuous Deployment (CD) Tools
The particular way that GitOps implements a Continuous Deployment pipeline is what really sets it apart from other DevOps workflows. GitOps defines a “Kubernetes Operator”, which is a custom resource within Kubernetes responsible for deploying your packaged application container images into the cluster. There are several choices available, some very bare bones and others with a whole lot of additional tooling included. Here are the most popular options.
ArgoCD is a continuous delivery, declarative GitOps tool for Kubernetes. Like Flux, ArgoCD is also a GitOps operator with a web UI. Using ArgoCD’s web UI, we can simulate a GitOps pipeline. ArgoCD gives a visual representation of your application and environment configurations. ArgoCD has gained immense popularity recently and is one of the most widely adopted operators.
Watch this space for the follow up to this article:
One of the benefits of ArgoCD is that it supports a whole lot of different tools for managing Kubernetes manifests within your code repository, including popular tools like Helm, Kustomize, Ksonnet, Jsonnet and others. It also provides a robust plug-in architecture so that third parties can write additional tooling of the same variety.
Additionally, ArgoCD is very lightweight and so is easy to install in a local minikube environment for testing, which sets it apart from Spinnaker which is a more robust tool, but significantly heavier-weight and harder to get installed.
The accelerator used to demonstrate GitOps makes use of ArgoCD primarily because it has a “live view” of deployment operations that make it great for understanding what is happening in the cluster.
Spinnaker Operator is a free to use Kubernetes Operator. Spinnaker was initially developed by Netflix to manage their huge container environment on AWS, but it was later open-sourced and adapted to act as a Kubernetes Operator. Spinnaker is actually a fantastic tool, well-polished and heavily adopted amongst the community. It supports a robust deployment workflow, mult-cloud and multi-cluster management, and an impressive set of observability features.
There are only two real drawbacks to using Spinnaker. The first is that because it predates GitOps by several years, it has a lot of legacy features and tools left over from the days of Netflix pre-Kubernetes. The second is that it is a large and complex system that is not the easiest to get up and running compared to the other options, and so it is hard to get a local development environment running with Spinnaker.
Personally, my choice would always be between ArgoCD and Spinnaker based on the size of the organization adopting it, with ArgoCD winning for smaller organizations and Spinnaker taking the title for larger more ops-heavy organizations.
Weaveworks introduced Flux in 2016. It is a Kubernetes cluster GitOps operator. Flux is responsible for ensuring that the cluster state must match with the configurations on Git. We can use Flux for deployments on a Kubernetes cluster. Flux periodically pulls the Git repository to check if there is any change in the repo. If Flux discovers any change, Flux applies the changes to the cluster.
Flux was actually the “original” Kubernetes Operator that kicked off the GitOps movement, but the project is in a state of -ahem- flux at the moment. Flux v1 has gotten significantly behind the curve compared to the other available options, and Flux v2 is still under development. The Flux development team at one point decided to use the core libraries from ArgoCD (called the GitOps Engine), but have since pivoted and chosen to build their own version of the same tools they call the GitOps Toolkit. When released, Flux v2 may be a significant improvement over other options, but at the present time I find Flux v1 to be clunky and hard to manage, and Flux v2 is still not in a particularly usable state.
Jenkins X is a powerful CI/CD solution, but it’s different from the classic Jenkins tool. I have already described this in the detail in the Continuous Integration Tools section, so I won’t repeat myself here. But along with everything else, Jenkins X provides a Kubernetes Operator and can act as your Continuous Delivery solution as well.
Secrets Management Tools
While Git can do most of the things you need in terms of storing GitOps configuration and code, the one thing you absolutely never want to put in Git is secrets. And by secrets I mean anything that is used to verify identity or access, including TLS certificates, GPG Signing Keys, SSH Keys, Tokens or Credentials. Many security breaches in the world of tech have been caused by developers accidentally checking secrets into an accessible Git repository. What you need for these things is a tool that is built to encrypt and manage secret values.
Vault is undoubtedly the best in class secrets manager out there. It is free, open source and brought to you by the team at HashiCorp that is also responsible for Terraform. It has a convenient Kubernetes manifest available so you can install it directly in your cluster along with your running applications for easy access. Vault can manage multiple environments with ease.
The only reason Vault is not included by default in the accelerator is because we only use a few secrets for the setup, and the overhead of installing additional tools didn’t seem worth it when we already had the Google Cloud SDK installed for Secret Manager access. For any real GitOps implementation, I would immediately switch all the secrets management to Vault.
Secret Manager is a cloud-native tool provided as part of the Google Cloud Platform Security Tools. It keeps secrets encrypted and access is managed through cloud-native Google IAM.
Kubernetes itself can also manage secrets. It stores both secret values and configuration maps in memory which are accessible directly from within the cluster. This is a good option for secret values that are only used within the cluster (and we use them extensively), but if you need the secret to be shared with any service or user outside the cluster it is best to keep them in an external secret management tool.
Infrastructure as Code (IaC) Tool
The GitOps workflow itself does not tend to need any kind of Infrastructure as Code (IaC) tool to function, because Kubernetes replaces most of the infrastructure that used to be created in the cloud by this style of tool. For example, things like load balancers and managed instance groups for auto-scaling are replaced by internal Kubernetes primitives such as ingress controllers and replica sets.
However, in order to create the environment in which Kubernetes operates, such a tool is still required for bootstrapping. For example, before we can deploy anything into Kubernetes, we have to first create the Kubernetes cluster and the Virtual Machines that populate the Node Pools within the cluster.
Terraform is by far and away the most popular IaC tool in use right now. It is developed by HashiCorp and maintained as free and open source software. Its popularity is largely derived from its open plug-in architecture that allows a standard configuration language (called HCL—HashiCorp Configuration Language) to be used across multiple clouds and any other tool that can be managed in a declarative fashion.
The accelerator makes extensive use of Terraform to bootstrap the initial GitOps environment, and I would strongly recommend it for your environment.
Google Cloud Deployment Manager
Google also has a built-in IaC tool called Cloud Deployment Manager which takes JSON or YAML formatted templates and uses them to generate infrastructure elements within GCP. It is a very well-built tool but is strictly limited to working with GCP infrastructure elements.
GitOps provides a powerful solution for managing Kubernetes-based deployments in a declarative and automated fashion. If you’d like assistance from Caylent getting to the next step please reach out to us here. We offer a complimentary half-day workshop to work with your team on adapting this approach to work for your organization.
If you’re keen to learn more about GitOps, join us for our upcoming Caylent webinar: GitOps: What It Is and Why You Need It on April 29th 2 PM ET / 11 AM PT. Register here today!
Stay tuned for the next blog in this series, Part 2: Getting Started with GitOps on Google Cloud Platform. Coming soon.
Caylent provides a critical DevOps-as-a-Service function to high growth companies looking for expert support with Kubernetes, cloud security, cloud infrastructure, and CI/CD pipelines. Our managed and consulting services are a more cost-effective option than hiring in-house, and we scale as your team and company grow. Check out some of the use cases, learn how we work with clients, and read more about our DevOps-as-a-Service offering.