04.27.21

Part 2: Getting Started with GitOps on Google Cloud Platform

By James Adams
Part 2: Getting Started with #GitOps on Google Cloud Platform

In Part 1 of this series, we laid the groundwork with a background of Gitops and a deep dive into the tools needed to optimise the principles of the methodology. In Part 2 of this 3-part series, we’ll get started with the initial implementation steps that will set you up for success in optimizing GitOps on Google Cloud Platform.  So, now that we have a general understanding of GitOps principles and the available tools, let’s dive into how you can use these tools to get up and running.

For the purposes of this example, I have made the following tool selections:

  • Cloud Service Provider -> GCP
  • Source Code Repository -> GitHub
  • Continuous Integration -> Cloud Build
  • Continuous Delivery -> ArgoCD
  • Container Registry -> GCP Container Registry
  • Kubernetes Implementation -> GKE
  • IaC Tool -> Terraform

Caylent has developed an accelerator (the Caylent GitOps Accelerator) using this toolstack that will allow you to bootstrap a GitOps environment in GCP by running a simple shell script.  This accelerator is provided in three publicly-available GitHub repositories that we will reference for this example.

Please Note: The Caylent GitOps Accelerator relies on a Unix/Linux style environment with Bash installed.  It has been tested on MacOS and a number of common Linux Distributions such as Debian/Ubuntu/RedHat.  The accelerator *should* work on a Windows machine with the Windows Subsystem for Linux 2 (WSL2) installed, but this has not been tested.  If you happen to test this out using a WSL2 environment and run into issues, please let us know.

Create a GCP Account and Project

Before doing anything else, you will need to create a GCP account if you don’t already have one.  GCP will require a credit card, but for new accounts you will get a 90-day trial and a $300 cloud credit to use the platform.

IMPORTANT: The examples given throughout the rest of this article will create cloud infrastructure that costs money while running.  You do this at your own risk with the understanding that Caylent and the author of this article are not responsible for any charges you incur while following along with these instructions.

To create your account, follow these steps:

  1. Open Google Cloud Console in a web browser
  2. When prompted to sign in, click the “Create Account” link in the bottom left
  3. Follow the instructions to register your email address as a Google account
  4. Continue to the Google Cloud Console and accept the terms

Now create a Google Cloud Project to hold the cloud resources:

  1. Click the Project Selector in the top left
  2. Click New Project in the top right corner of the dialog box
  3. Enter a name for you project in the Project name field (I called mine caylent-gitops)
  4. In the Billing account field, select the billing account that will pay for your project.  If you do not see this field but instead see an Organization field, then your company already has a Google Cloud Organization set up which will manage billing for your project.
  5. Click Create to create your project

Create a GitHub Account

While the accelerator code is published to a public repository in GitHub which you can view without a GitHub account, you will not be able to make any changes to that code.  In order to fork the repositories and make changes, you will need to set up a GitHub account.

To create a GitHub account:

  1. Open a web browser and navigate to GitHub
  2. Click the Sign Up button in the upper right corner
  3. Enter a value into the Username field
  4. Enter an Email address
  5. Choose a Password
  6. Click the Verify button, and solve the visual puzzles presented to prove you are a human
  7. Click Create account

Now that we have created an account, in order to do anything interesting with it from the command line we will need to generate an SSH Key, and add it to our GitHub account.  GitHub has already written a very comprehensive set of instructions for how to do this, so click on the link and follow the instructions for your operating system before proceeding.  If you do not create a keypair and add the public key to your GitHub account, the accelerator’s bootstrap script will fail.

Install the Command Line Tools

To successfully execute the bootstrap script provided by the accelerator, you will need to install some command-line tools and have them available on the path.

Install Git

You will need to have the Git command-line utility installed on your local machine.  Many development tools come bundled with git, so you may already have it installed.  You can check if git is installed on your machine by opening a terminal window and running the command git –version.  If your system cannot find git, then you will need to install it.

git version 2.25.1

If you are running on a Mac, git comes with the Xcode developer tools which you can install through the App Store.  Alternatively, you can install git with homebrew.

Install Terraform

You will need to have terraform installed on your local machine to bootstrap the initial environment.  Make sure that you are using a version that is at least 0.13.0 or you may have issues with running the example.

Terraform can be downloaded from the website or installed through a package manager on linux if your platform supports it.  Verify that terraform is properly installed and available on the path by running the ‘terraform -version’ command in a terminal.

Terraform v0.14.10

Install the GitHub CLI

The GitHub command-line tools are used by the accelerator scripts to manage SSH Keys and handle some of the more common GitOps workflow operations.  You can download the tools from the releases page.  Verify that gh is properly installed and available on the path by running the ‘gh –version’ command in a terminal.

gh version 1.8.1 (2021-04-02)
https://github.com/cli/cli/releases/tag/v1.8.1

Install Python

Python is required by the Google Cloud SDK, so make sure you install it first.  Python comes pre-installed on many systems, so first verify if python is installed by running the python –versioncommand in a terminal.

Python 3.8.5

If python is not installed, you should be able to install this through your system App Store or package manager.  If you prefer you can download and install the binaries directly from the Python downloads page.

Install the Google Cloud SDK

The Google Cloud SDK comes bundled with a number of useful tools for managing GCP and things within it (such as Kubernetes clusters).  The SDK requires python to run, so you should have installed that in the previous step.

Once you have verified that python is installed and available on the path, install the Google Cloud SDK.  Google provides hosted packages for nearly every platform, and you can find detailed installation instructions in the quickstart guide.

Once you have installed the Google Cloud SDK, verify that it is available on the path by running the ‘gcloud –version’ command in a terminal.

Google Cloud SDK 336.0.0
alpha 2021.04.09
beta 2021.04.09
bq 2.0.66
core 2021.04.09
gsutil 4.61

(Optionally) Install Kubectl and the ArgoCD CLI

You may additionally want to install ‘kubectl’ as it comes in handy when troubleshooting Kubernetes deployments, but is not required by the accelerator bootstrapping process.  While experimenting with ArgoCD it may also be useful to install the ArgoCD CLI tools, but again this is not required.

Fork and Clone the GitHub Repositories

While the caylent-gitops-* repositories of our accelerator are publicly available, you will not be able to write to them directly to configure them for your environment.  ArgoCD needs to be able to write to the caylent-gitops-config repository to update the Kubernetes manifests, so you’ll need to have your own writable copy (called a “fork”).

Additionally, while you will want an internet-accessible copy of your forked repository on GitHub, you will also want a local copy of that fork on your development machine (called a “clone”) so you can run the bootstrapping commands locally.

Since you have already installed the GitHub command line tools, you can do this conveniently from the command line with a single command to fork and then clone each repository.  But unfortunately, the GitHub command line tools do not (as of this writing) support renaming repositories.  So if you want to change the names of your forked repositories you will need to do that through the GitHub web interface.

Originals, forks and  clones

I highly recommend creating a parent folder on your local machine to keep the three repositories together in one place.  I call mine “caylent-gitops”, but feel free to name this directory whatever you like.  This parent directory isn’t referenced anywhere else in the examples, so it won’t have any impact on future instructions.

Authenticate the CLI with GitHub

Before you can run any GitHub CLI commands that deal with repositories, you will need to authenticate yourself with GitHub so they can verify it is really you making the requests.  Run the below GitHub command from the terminal and follow the prompts in your web browser to authenticate.

gh auth login

This will prompt you with a series of questions before opening a web browser to authenticate you through OAuth.  When asked about GitHub.com or GitHub Enterprise, this will most likely be GitHub.com (unless your company has an Enterprise subscription).  Select SSH as your preferred protocol, and skip uploading a public ssh key (we already did this when creating your GitHub account earlier).  When it asks you how you would like to authenticate, select Login with a web browser.

At this point the GitHub CLI will generate a one-time authentication token, which you will need to copy before hitting enter.  A web browser will open and ask you to paste in that authentication token, and then you will be asked to grant GitHub CLI permissions to your GitHub account.  Go ahead and accept and then login to your GitHub account.  At this point you can close the browser and return to the terminal, at which point you have been authenticated and can run additional GitHub CLI commands.

But what did that just do exactly?  Well, a file called “hosts.yaml” was created under your account home within a hidden “.config/gh” directory with some information that can be used to authenticate your future GitHub CLI commands.  Try running “cat ~/.config/gh/hosts.yaml” and see what is in there.

github.com:
user: james-caylent
oauth_token: gho_[REDACTED]
git_protocol: ssh

The value to the right of “oauth_token:” is what the GitHub CLI uses instead of a password to authenticate API requests it makes to GitHub.com on your behalf.  Just keep in mind that if you run into GitHub CLI issues later on, you may need to remove this file and re-create it by running “gh auth login” again.

Fork and Clone the Application Repository

From a terminal window, navigate to the directory where you want your clone of the caylent-gitops-app repository to reside.  Run the following GitHub CLI command to make a fork of the repository in GitHub and then clone that fork locally in your current directory.

gh repo fork caylent/caylent-gitops-app --remote=true --clone=true

NOTE: The GitHub CLI does not currently support renaming of repositories.  If you want to rename the repository, set the remote and clone flags to false when forking, go to the GitHub web page and rename the forked repository there, and then issue a separate clone command as below.  (Though to be honest, you will save yourself a lot of typing if you just leave the repository names alone for the remainder of the examples)

gh repo fork caylent/caylent-gitops-app --remote=false --clone=false
----RENAME YOUR FORK USING THE WEB UI----
gh repo clone <YOUR_GITHUB_USERNAME>/<YOUR_FORKED_APP_REPO_NAME>
caylent-gitops-app

Fork the Configuration Repository

From a terminal window, navigate to the directory where you want your clone of the caylent-gitops-config repository to reside.  Run the following GitHub CLI command to make a fork of the repository in GitHub and then clone that fork locally in your current directory.

gh repo fork caylent/caylent-gitops-config --remote=true --clone=true

NOTE: The GitHub CLI does not currently support renaming of repositories.  If you want to rename the repository, set the remote and clone flags to false when forking, go to the GitHub web page and rename the forked repository there, and then issue a separate clone command as below:

gh repo fork caylent/caylent-gitops-config --remote=false --clone=false
----RENAME YOUR FORK USING THE WEB UI----
gh repo clone <YOUR_GITHUB_USERNAME>/<YOUR_FORKED_APP_REPO_NAME>

Fork the Infrastructure Repository

From a terminal window, navigate to the directory where you want your clone of the caylent-gitops-infra repository to reside.  Run the following GitHub CLI command to make a fork of the repository in GitHub and then clone that fork locally in your current directory.

gh repo fork caylent/caylent-gitops-infra --remote=true --clone=true

NOTE: The GitHub CLI does not currently support renaming of repositories.  If you want to rename the repository, set the –remote and –clone flags to false when forking, go to the GitHub web page and rename the forked repository there, and then issue a separate clone command as below:

gh repo fork caylent/caylent-gitops-infra --remote=false --clone=false
----RENAME YOUR FORK USING THE WEB UI----
gh repo clone <YOUR_GITHUB_USERNAME>/<YOUR_FORKED_APP_REPO_NAME>

Configuring Command Line Tools

Before we can bootstrap the Caylent GitOps Accelerator, there are a few housekeeping items to run through.  While we have installed some very powerful command line tools, we need to tell some of them who we are and what resources within GitHub and GCP we will be using.  This process is very similar to how we previously authenticated with GitHub using the GitHub command line.

To begin with, let’s authenticate with GCP using the gcloud command line tool.  Run the following command in your terminal.

$ gcloud auth application-default login
Your browser has been opened to visit:
https://accounts.google.com/o/oauth2/auth?[REDACTED]

At this point your default browser will be opened to an OAuth authentication page, where you will need to select the email account you used to create your GCP account earlier.

Google sign in screen

Select the account to use and then you will be shown another screen asking to grant the necessary permissions to the command line tool.

Google SDK authorization screen

After clicking Allow, you will be redirected to a success screen and you can close the browser.  At this point if you return to the terminal it will have printed out something similar to the below:

You are now logged in as [james.adams@caylent.com].

Since I already described the OAuth token process for the GitHub CLI, I won’t repeat that here.  But if you are curious, all of the local configuration data for the gcloud tool is stored in your home directory under .config/glcoud/.

While we are setting up the gcloud tool, let’s go ahead and set the current project.  Mine is called ‘caylent-gitops’, but use whatever name you gave to your project in GCP.

$ gcloud config set project caylent-gitops
Updated property [core/project].

There is one more thing we should do before moving on to setting up the Caylent GitOps Accelerator, which is to set a default name and email address for our Git installation.  If you are a software developer, you have probably already done this, but if you just installed Git recently you should set these values so that they are attached to any code modifications you make.

$ git config --global user.name "James Adams"
$ git config --global user.email "james.adams@caylent.com"

The –global flag tells git to use these values for all operations on your local machine.  If you do not include that flag, the values would be set just for the current repository (assuming your working directory is inside a cloned git repository).

In case you are curious about where these values end up, all git configuration is stored within your home directory in a hidden file called .gitconfig ( or .config/git/config on some systems).

Setting Up Cloud Build

Now that we have forked copies of the Caylent GitOps Accelerator repositories into our private GitHub account and cloned those forks on our local machine, we need to do a little bit of manual setup for Cloud Build.

When you create a new project inside of a Google Cloud account, by default all of the key Services are disabled.  Before you can make use of a particular service, you need to enable it either with a gcloud command or through the web console.  For most of the services needed by the Caylent GitOps Accelerator this is done automatically using a terraform template that is invoked by the bootstrap script.

Unfortunately, while GCP has made it possible to connect Cloud Build to external GitHub repositories, they have not yet (as of this writing) made this functionality available through the command line or APIs.  So for this particular task we must resort to the web console.

If you are curious, the gcloud command to enable Cloud Build is shown below:

$ gcloud services enable cloudbuild.googleapis.com

Alternatively, we can enable services through the web console by visiting the API Library screen on Google Cloud, selecting our project, searching for “cloud build”, and clicking the button to enable it.

GCP Caylent gitops search

Now that Cloud Build is enabled, visit the Cloud Build Triggers screen in the web console.  Here you will see a button along the top labeled “Connect Repository”, which you should click.

Cloud build triggers

This will cause a panel to open up on the right side of the screen that will let you enter some details about the repository.

connect repository

A quick note on the method by which Cloud Build will connect to your repository.  Google has created a GitHub App which will live in GitHub and monitor your repositories, and then push out and changes to your Google Cloud environment when they are detected.  This is a good thing, because the alternative approach would be for GCP to constantly poll your GitHub account to look for changes, which means there would likely be a delay between the time a change was made in GitHub and the point at which that change would be noticed by Google Cloud.

To install the GitHub App, select “GitHub (Cloud Build GitHub App)” and click “Continue” to see the Authenticate section.

connect repositor 2

In the screenshot above, I have already selected my GitHub account and my fork of the caylent-gitops-app repository.  You will need to open the GitHub Account dropdown and click the “Add” button, which will take you through an OAuth process that will allow Cloud Build to connect to your GitHub account.  Since we have already gone through this process twice previously with the gh command line tool and the gcloud command line tool, I will assume you are familiar with the process and won’t describe it in detail here.

Once you have authenticated your GitHub account and selected the forked repositories (be sure to select all three) within the GitHub Cloud Build App, check the box indicating you understand the terms and click the “Connect” button.

Cloud Build will now give you an opportunity to create a trigger.  It is not necessary to create any triggers through the web console because the Caylent GitOps Accelerator bootstrap script will do this automatically using Terraform later.  Simply click the “Done” button to close out the dialog box.

connect repositiory 3

You should now see that all three of your forked repositories are connected within Cloud Build.

forked repository management screen

That is all the manual setup required before bootstrapping your GitOps environment.

  • You now have a GitHub account with forked repositories and an SSH Key
  • A Google Cloud account with an appropriate project to hold your cloud resources
  • Cloud Build is enabled and attached to your forked repositories
  • You have installed and configured all the necessary command line tools and cloned the forked repositories on your local machine.

From this point forward we can kick back and let the Caylent GitOps Accelerator do the heavy lifting. 

Remember, GitOps provides a powerful solution for managing Kubernetes-based deployments in a declarative and automated fashion. If you’d like assistance though from Caylent getting to the next step please reach out to us here. We offer a complimentary half-day workshop to work with your team on adapting this approach to work for your organization.

If you’re keen to learn more about GitOps, join us for our upcoming Caylent webinar: GitOps: What It Is and Why You Need It on April 29th 2 PM ET / 11 AM PT. Register here today!

Jump into Part 3 of this series next: Optimizing the Caylent GitOps Accelerator on Google Cloud Platform


Caylent provides a critical DevOps-as-a-Service function to high growth companies looking for expert support with Kubernetes, cloud security, cloud infrastructure, and CI/CD pipelines. Our managed and consulting services are a more cost-effective option than hiring in-house, and we scale as your team and company grow. Check out some of the use cases, learn how we work with clients, and read more about our DevOps-as-a-Service offering.