05.04.21

Part 3: Optimizing the Caylent GitOps Accelerator on Google Cloud Platform

By James Adams
Part 3 - Optimizing the Caylent GitOps Accelerator on Google Cloud Platform

In case you missed it, Part 1 and Part 2 of this 3-part series covered the necessary tool stack and initial GitOps implementation steps necessary for making the methodology a success on Google Cloud Platform. Dive into the previous blogs to catch up here:

Bootstrapping the Caylent GitOps Accelerator

Now that we’ve done all the heavy lifting steps, we get to enjoy the fun and magic of the Caylent GitOps Accelerator! You are just a few commands away from having a working GitOps environment. I’m going to take some time to explain what the accelerator is doing behind the scenes, but let’s go ahead and run our bootstrap command.

In your terminal, make sure your working directory is your local cloned copy of the caylent-gitops-infra repository. Mine happens to be sitting in ~/dev/projects/caylent-gitops/caylent-gitops-infra, but your location will be wherever you ran the gh clone command earlier.

In this directory is a shell script called bootstrap.sh that is going to build out the GitOps environment for you. Before we run it, you should be aware that it takes some command-line flags.

FlagValueDescription
-c (or –cloud)gcp(required) Indicates the cloud provider to use. Currently gcp is the only option in this example, but the accelerator can be expanded to work with any cloud provider.
-o (or –owner)<string>(required) Your GitHub username. The bootstrap script will automatically replace all variables that refer to this with the value you provide. This is required the first time you run the bootstrap script.
-p (or –project)<string>(optional) The name of your GCP project. The bootstrap script will automatically replace all variables that refer to this with the value you provide. This is needed if you named your project something other than ‘caylent-gitops’
-a (or –app-repo)<string>(optional) Name of your forked copy of the caylent-gitops-app repository. This is only needed if you renamed the forked repo.
-e (or –config-repo)<string>(optional) Name of your forked copy of the caylent-gitops-config repository. This is only needed if you renamed the forked repo.
-i (or –infra-repo)<string>(optional) Name of your forked copy of the caylent-gitops-infra repository. This is only needed if you renamed the forked repo.

So, if you left the repository names alone earlier when you forked them to your own GitHub account and used a GCP project called ‘caylent-gitops’, then you can use a simple version of the bootstrap script like this:

$ ./bootstrap.sh -c gcp -o <GITHUB_USER>

But if you renamed all the repository forks and called your GCP project something other than ‘caylent-gitops’, then you’ll need to include all the flags like this:

$ ./bootstrap.sh -c gcp -o <GITHUB_USER> -p <GCP_PROJECT> -a <APP_REPO> -e <CONFIG_REPO> -i <INFRA_REPO>

Pick the appropriate version of the command and execute it, making sure to replace the items in <> with the appropriate value.

Once you run this command, you will see a whole slew of output on the command line. Let’s take a minute and break down what is happening here.

First, the bootstrap script is validating that all of the command line tools are installed correctly. It checks for git, terraform, gcloud and gh on the command line to verify they are available on the path. If any of the tools are not installed or configured, the bootstrapper will error out with some instructions on where to download the tools.

Once the tools are validated, the bootstrapper will check to see if you provided any of the -o, -p, -a, -e or -i flags. It takes these values and does a search and replace in all of the *.tfvars files which provide variable values for terraform. If any changes are made, it also commits those changes to your local clone of the caylent-gitops-infra repository and pushes those changes up to the forked copy that lives on GitHub (so that they are visible to Cloud Build, which is connected to the forks).

NOTE: Even if you don’t provide any of these flags, the bootstrapper will commit and push changes because it generates a random value to append to the name of the terraform state bucket (which has to be globally unique) so that more than one person in the world can run this accelerator without problems.

The bootstrapper then checks the cloud flag and based on the value provided changes the working directory (in this case into the ‘gcp’ directory). It generates a temporary directory where ssh keys and passwords will be stored during the bootstrapping process.

It then iterates through the four environments that are defined (dev, qa and prod). For each of these environments, three secret values are created:

  • An administrator password for ArgoCD
  • A private SSH Key that ArgoCD will use to communicate with GitHub
  • A public SSH Key that GitHub will use to recognize requests from ArgoCD

The public SSH Key is added to your GitHub account, while the ArgoCD administrator password and private SSH Key are stored in the Google Cloud Secret Manager for later use.

A Cloud Storage bucket is then created to store the terraform state files. It is worth pointing out that when you use the open-source version of terraform, by default it will write state information out to a set of files on your local machine. This isn’t very useful if you are working in an environment where there is more than one person working with the cloud environment, so we create a Cloud Storage bucket to hold these files and tell terraform to use that instead of the local file system.

And now the bootstrapper executes terraform and tells it to build the base environment. This does several things.

First, it enables all of the necessary Google Cloud Services (just like what we did manually earlier for Cloud Build).

Then, it creates all the Cloud Build Triggers that are needed to build the infrastructure and the application whenever changes are made to the GitHub repositories. These triggers are set up to start a new build whenever a new semver tag is pushed to the forked repository being tracked by Cloud Build.

However, what this setup does not do is actually trigger the builds that will construct the dev, qa and prod environments. For that we have to push a new tag to the infrastructure repository.

Tagging the Infrastructure Repository

Now that the automated builds are in place for all our environments, we need to create a new tag and push it to our fork of the caylent-gitops-infra repository. This can be done manually with a few git commands, but to make life easier I have included a handy little script called rev.sh that will automatically calculate the next semantic version, create a tag with that version and push it from your local clone up to the GitHub-tracked fork that will trigger Cloud Build.

The public caylent-gitops-infra repository comes with a pre-built tag of 0.0.1, so to get to 0.0.2, just run the following command:

$ ./rev.sh

And that’s it! You have just pushed a fresh tag to the infrastructure repository, which will trigger the Cloud Build triggers on GCP and start creating all of the infrastructure for your dev, qa and prod environments. I should warn you, it does generally take 5-10 minutes for the environments to spin up, so while you are waiting let’s take a look at what this is doing for you.

To begin with, each environment needs a VPC (or Virtual Private Cloud) in a particular region. Inside that VPC, the accelerator is going to create a subnet with 256 private IPs in the address space. And then inside that subnet, it is going to create a GKE cluster that is configured with two nodes (VMs) in a node pool that is attached to a control plane.

Once those environments are set up, the accelerator will then create a Kubernetes namespace inside the GKE cluster where ArgoCD will run called argocd. It then installs the latest stable release of ArgoCD (v2.0.0 at the time of this writing), and waits for it to spin up before configuring the system.

I should note that ArgoCD out of the box comes with an insecure default administrator password that is stored within a Kubernetes “secret” as plain text. We don’t want this value just hanging around for anyone to pick up, so the first thing the accelerator does once ArgoCD is up and running is retrieve this default password, use it to log in and replace the password with the secure version the bootstrapper created in the Google Secret Manager. This version of the password is strongly encrypted and secured by Google IAM. Once it has done that, it deletes the plaintext password from Kubernetes.

The accelerator then configures ArgoCD with your fork of the caylent-gitops-config repository, where the Kubernetes manifests for the caylent-gitops-app are stored. There are two manifests per environment, one for the Kubernetes Service which exposes the app to the outside world, and another for the Kubernetes Deployment which indicates which version of the app container image should be deployed into the cluster.

Once the repository has been configured in ArgoCD, the accelerator then tells ArgoCD to deploy the application itself. At the moment, this process will fail, because we haven’t actually built the application image. But let’s pause for a moment now that your infrastructure is up and running to take a look at what we have here.

Go back to the Google Cloud Console and take a look under the Kubernetes Engine panel. If you look under the Clusters tab, you’ll see that there are three clusters running, one for each of the environments: dev, qa and prod.

NOTE: In the following several screenshots I have the view filtered to just show dev so it’s less of an eye chart. You can do the same if you like with the dropdown.

GCP Kubernetes clusters screen

If you then click on the Workloads tab, you will see a whole bunch of deployments related to ArgoCD, which tells you that it is properly installed.

GCP Workloads tab

Then if you click on the Services & Ingress tab, you will see matching services for ArgoCD.

GCP services and ingress tab

You may notice that one of those services has an “External load balancer” (rather than a Cluster IP) with a little link icon next to it. This means that the service is configured as externally accessible, and if you click on it that should open a new tab in your browser that loads up the ArgoCD UI.

Argo success screen

It is asking you to login. Remember when I mentioned that ArgoCD comes with a built-in administrator account? Well the username you need to enter is ‘admin’. But the password is stored within Google Secret Manager. So head over there in the console and you’ll find a set of secrets.

GCP secret manager

If you choose a particular environment (say dev) and click on the argocd-admin-password-dev secret, you’ll see that there is a single version of this secret defined. If you click on the little icon with the three dots to the right of that version, there is a menu option for “View secret value”. Clicking on this will show you the secure password for that ArgoCD installation.

Argo installation

Copy that value and flip back over to the browser tab where you have the ArgoCD UI sitting at the login screen, and paste in that password. Now you should see the main ArgoCD dashboard with a failed deployment for the caylent-gitops-app. The status here says “Missing” and “OutOfSync”.

Argo outofsync screen

As I mentioned before, the reason this failed is because we haven’t built the application yet. Building the application is a breeze (all we have to do is give it a new tag), but I wanted you to have this window open so you can see what ArgoCD does when a new version of your application becomes available. If you click on the deployment within the ArgoCD UI, it will show a live graph of what is deployed inside your Kubernetes cluster. Keep this open when we move onto the next step of building the application so you can see what happens.

Building the Application

Fortunately, building the application is no big deal, since the accelerator already set everything up for us. All you have to do is go to your terminal and change directories into your local clone of the caylent-gitops-app repository (whatever you chose to call it), and run a few simple git commands.

When we did this last time I made it super simple by providing a rev.sh script to do this for you. This time let’s do it by hand with git so you can see what is going on under the hood (don’t worry, it’s just a few extra commands). Assuming you are starting with a fresh clone of the public repository I provided, there should be an existing tag of 0.0.1. That means our next release version should be 0.0.2. Run the following commands in your local repository to create a new tag and push it up to the fork on GitHub.

$ git tag 0.0.2
$ git push origin 0.0.2

That’s it! You just created a new tag in your local repository and then pushed it up to GitHub, where the Cloud Build trigger will detect it and build a fresh copy of the application in a docker container, and then push that image into the Google Container Registry where ArgoCD can find it.

GCP success screen

The rev.sh script actually does a little more than just tag the repository, it does the work of figuring out what the next version number should be by looking at the git log output and finding the last tag that was applied, and then incrementing it.

NOTE: There is a minor bug in the rev.sh script where it will fail to figure out the next version number if you have multiple version tags on the same git commit. I will get around to fixing this soon.

But now that you have pushed a new version tag, let’s go back to the ArgoCD console and watch what happens.

Argo gitops app screen

Well that’s weird, the deployment failed again. Why is that? This is because we built a container image with a tag of 0.0.2 while the default manifest for the deployment specifies a version number of 0.0.1, which never got built.

I’ll admit, I did this on purpose. I wanted to demonstrate how to update your deployment manifest, which we’ll do next. We could have just manually triggered the application build for version 0.0.1 in the Cloud Build console, but that wouldn’t have demonstrated the magic of GitOps nearly as well.

Updating the Deployment Manifest

So now we have our infrastructure up and running with ArgoCD installed, and we have built version 0.0.2 of our application. There is a docker image in the Google Cloud Repository that is tagged as caylent-gitops-app:0.0.2 (or whatever you renamed it to), and now we need to tell ArgoCD to deploy that version instead of the 0.0.1 version it is currently attempting to deploy. So go to your local clone of the caylent-gitops-config repository and take a look under the dev folder at caylent-gitops-deployment.yaml file.

caylent gitops deployment yaml

If you look under the spec/template/spec/containers/image key of this yaml file it should say something like ‘gcr.io/caylent-gitops/caylent-gitops-app:0.0.1’. It may have a different name if you named your GCP project or your caylent-gitops-app repository clone something else. But the important part here is the 0.0.1 after the colon, which is the version number of the container image ArgoCD will be trying to deploy. We never built the 0.0.1 version, but we just built the 0.0.2 version and it’s ready to go. So let’s update that version number to 0.0.2 to deploy the version we just built.

caylent gitops deployment screen

And of course you will need to stage that change with git, commit it locally and then push it up to the repository in GitHub.

$ git add dev/cayent-gitops-deployment.yaml
$ git commit -m "Releasing version 0.0.2 to dev"
$ git push

And now that the new version of the deployment manifest has been pushed up to GitHub, ArgoCD will pick it up (it is configured to look for changes every 5 minutes) and deploy the new version of our app. If you get impatient waiting for ArgoCD to refresh, you can just manually click the “Sync” button in the ArgoCD UI. It may not seem like it, but this is some of the crucial magic of GitOps. Notice that what we did to deploy the application was done entirely through git and our source code repository. We didn’t have to put in a release ticket to an operations team or go manually run a CI/CD pipeline. All we had to do was make a change to a repository.

And on top of that, we added a comment to our git log that explained what we were doing. This means every release of our software is easy to audit, and if we want to know what version of our application is deployed in each environment all we have to do is go look in the caylent-gitops-config repository. Even better, we now have a version of the application deployed in dev that is not deployed in qa or prod, so we have fine-grained control over our releases through git operations. Of course, at some point we probably want to automate the progression of a release through all the environments. ArgoCD has lots of hooks to manage those kinds of workflows (and a really cool side project called Argo Workflows that can help you configure the entire flow), but I’ll leave that for another time.

Before we wrap up the example, let’s go back to our ArgoCD UI and take a look at what happened when we pushed that update. You should be able to see ArgoCD pick up the new release and deploy a new replica set with a new pod in Kubernetes, and perform some health checks against it to verify that it is running and reachable before destroying the pod with the previous release.

Argo screenshot

Now that you’ve seen ArgoCD in action deploying your application, let’s connect to the application itself and verify that it is working as expected. Go back to your Kubernetes Engine console and click on the Services tab. You should now see a caylent-gitops service with an External load balancer assigned.

services and ingress screen

If you click on that External IP, your browser will open up another tab connected to your deployed application.

hello dev image

I know, not very exciting, is it? But it isn’t supposed to be, because this is just an example to help you get to a place where you can use GitOps to manage and deploy your very own application. I can only imagine that your application will be much more exciting than this one.

What’s Next?

You are now at the very starting point of discovering the joys of GitOps. It was a lot of work to get here (or at least a lot of explanation, hopefully the accelerator did most of the work). But now you can start making changes to this to be more in line with your own environment.

For example, it’s probably not very realistic to have all of your environments defined in a single configuration repository, because then everybody has access to deploy to all of them, right? In the real world, you probably want to split up your dev, qa and prod into separate repositories that have the appropriate permissions on them limited to the teams that manage those environments.

And there are lots of other interesting things you can do with this. What if you want to require approval for someone to release a new version of the software? This is commonly implemented using Pull Requests in GitHub that only allow changes to be merged after the PR has been approved. That means you don’t need that external ticketing system like ServiceNow to manage release approvals, it can all be done with Git tools!

You probably want to set up proper certificates and TLS as well, since our example used self-signed certificates that probably produced a nasty browser warning when you tried to open ArgoCD or the deployed application. This is not particularly difficult to set up, but requires a domain registration I didn’t want to include in the example.

The point is, there is plenty more to explore, but everything beyond this point is going to be specific to your organization and your application, and there is no way I could do it justice in a blog post with a broad audience. I highly recommend you read more about GitOps and experiment with it on your own.

Cleaning Up

When you are done playing around with the GitOps environment that we just built, you’ll probably want to clean up the resources in GCP so you aren’t getting charged for them while they aren’t in use. To help out with this, I have included a cleanup.sh script in the caylent-gitops-infra repository that will do the reverse of what the bootstrap.sh script does, tearing down the environment completely. The script works in much the same way, but doesn’t have nearly as many flags and options. Just run the following command from the terminal inside the caylent-gitops-infra repository:

$ ./cleanup.sh -c gcp

IMPORTANT NOTE: Unfortunately the GitHub CLI does not currently support deleting SSH Keys, so the cleanup.sh script is unable to do this for you. You will need to use the GitHub web console to manually remove SSH Keys from your GitHub account.

Conclusion

GitOps provides a powerful solution for managing Kubernetes-based deployments in a declarative and automated fashion. If you have additional questions or want some help, feel free to reach out. Caylent is a Professional Services organization that helps companies with exactly this sort of customization. We regularly run webinars that explore this and other topics and would be happy to set up a half-day workshop with you and your team.

If you’re keen to learn more about GitOps, watch the recording of our recent Caylent webinar: GitOps: What It Is and Why You Need It on April 29th 2 PM ET / 11 AM PT. Watch here today!


Caylent provides a critical DevOps-as-a-Service function to high growth companies looking for expert support with Kubernetes, cloud security, cloud infrastructure, and CI/CD pipelines. Our managed and consulting services are a more cost-effective option than hiring in-house, and we scale as your team and company grow. Check out some of the use cases, learn how we work with clients, and read more about our DevOps-as-a-Service offering.