Currently, 56 million developers are using GitHub to develop their applications today, and that statistic is predicted to grow to 100 million by 2025....
Explore the difference between AWS certifications and competencies, why competencies are a more rigorous, experience-based validation of a partner’s ability to deliver real-world cloud solutions, and why they matter when choosing the right AWS Partner.
Explore how we helped our customer in the financial sector migrate from High-Performance Computing (HPC) workloads on an on-premise Slurm cluster to AWS ParallelCluster, detailing the process, challenges, and benefits.
Migrating to the cloud is deeply desirable due to ease of the management, scalability and many other factors, however poor choices in the migration process can lead to increased costs, poor performance and tech debt. Learn about the top 7 cloud migration mistakes and how to avoid them.
Currently, 56 million developers are using GitHub to develop their applications today, and that statistic is predicted to grow to 100 million by 2025. Developers optimize GitHub for multiple tasks, from writing a piece of code for their application to deploying it in an environment. This requires a solution that can automate all the mundane tasks such as code reviews, issue tracking, branch management, etc., so that instead developers can focus on developing applications. GitHub Actions and runners provide the solutions for automating all the internal workflows from idea through to production.
As GitHub Action supports automated developer workflows, the feature means you can implement a CI/CD pipeline. By leveraging GitHub actions, every time something happens in your repository, you can configure automatic actions to execute in response. An engineer creating a pull request, creating an issue, joining as a contributor, and merging pull requests into the master branch are examples of GitHub action events.
GitHub Actions allow you to define pipeline style templates for workloads such as jobs, builds, and other various tasks in your GitHub repository. Start with a YAML file where you define a trigger which can be a pull request. Once your trigger kicks off, you can configure requests to compile code, package code, run docker containers, run arbitrary code, run a testing cycle, build or deploy the application.
To do all these tasks, you require GitHub runners.
GitHub Runners are the applications responsible for running the jobs inside a GitHub action workflow. Host this runner in a virtual environment on any cloud provider (AWS, Azure, GCP), or it can be a self-hosted runner in your own environment. Self-Hosted Runners allow you to save a lot of time and effort by executing your pipelines with pre-installed and pre-configured tools needed for your pipeline jobs. These self-hosted runners can also run inside containers on a Kubernetes cluster.
In this article, we will learn how to optimize GitHub Actions with self-hosted runners deploying them on a Kubernetes cluster step by step.
To follow the steps in this article, you need a GitHub account and a repository that is storing files. Go to the “Actions” tab inside your GitHub repository and click on “Set up this workflow”.
This will create a YAML file automatically in the workflow which will build any new commit in the GitHub repository. Since a new file has been added, we can go ahead and commit the changes in the repository by clicking on start commit.
Once you commit the changes, a workflow will start, which will run the build for this commit. Click on “Create main.yaml” workflow to get the details of the build.
Click on the build.
Now, you can see all the details of the build steps defined in main.yaml.
The GitHub Action is ready. Let us set up a self-hosted GitHub runner now.
To set up a self-hosted GitHub runner, go to Settings > Actions in your GitHub repository, and scroll to the bottom. Here you will find “Self-hosted runners”, click on add runner to create a new self-hosted runner.
This will provide the steps to install a GitHub action runner on your machine, depending on the operating system you have. These are the steps to set up an action runner on an Ubuntu machine.
This command will start the GitHub Actions self-hosted runner registration. It will ask the name, label, and workspace of the runner you want to have and finally save all the settings.
Finally, start the self-hosted runner to listen to the jobs running.
Now go back to Settings > Actions in your GitHub repository and refresh the page, you will see the runner is up and running and in the idle state.
You need to use the self-hosted label for all the self-hosted runners. Now to run the GitHub actions in your self-hosted runner, edit the “runs-on” parameter in the main.yaml file as self-hosted and commit this change.
The commit will create another workflow run with a build, running inside your self-hosted Github runner.
Now that the self-hosted runner is running on a linux machine, let’s look at how to deploy it on Kubernetes. To follow the next steps, you’ll need a Kubernetes cluster running.
Firstly, I will need a dockerfile to build a docker image. In this dockerfile, I will install GitHub runner binary and start the container using GitHub Personal Access Token.
The entrypoint will register the GitHub token and install the runner’s software.
Now, build the docker image. It will run the dockerfile and execute all the steps one by one to create the runner docker image.
When you check now, you find a new docker image “runner”.
Push this runner docker image on DockerHub. This docker image will be used for the Kubernetes deployment in the coming steps.
Next, you need to create a namespace on the Kubernetes cluster where the runner deployment will happen.
Go to GitHub Account Settings > Developer settings > Personal access tokens and create a new token to authenticate runner to the deployment.
Create a secret file to pass the GitHub personal access token to the deployment.
Apply the secret using kubectl.
Now the most crucial part, create the deployment to run the containers of runner docker image inside the Kubernetes pods. This deployment will use the secret stored in secret.yaml and spin one runner on the Kubernetes cluster inside the gitaction namespace.
Apply the deployment using kubectl.
Check the status of the deployment.
This command will show the single runner pod running inside the gitaction namespace.
On Kubernetes cluster, you can easily scale the runner container by increasing the replicas. The command below starts 2 more runner pods on the cluster.
Now if you check the number of runner pods, it will be 3. Two new pods are starting.
Congrats, you have successfully deployed self-hosted GitHub action runners on a Kubernetes cluster.
Optimize GitHub Action as a CI/CD solution for your pipelines to build, test, deploy using YAML configurations. Self-hosted runners can help you save a lot of costs that you would incur using a cloud provider. Go ahead and leverage your own GitHub Actions and self-hosted runners before deploying them on a Kubernetes cluster.
Caylent provides a critical DevOps-as-a-Service function to high growth companies looking for expert support with Kubernetes, cloud security, cloud infrastructure, and CI/CD pipelines. Our managed and consulting services are a more cost-effective option than hiring in-house, and we scale as your team and company grow. Check out some of the use cases, learn how we work with clients, and read more about our Infrastructure and DevOps Modernization services.
Caylent Team
# Create a folder
osboxes@osboxes:~$ mkdir actions-runner && cd actions-runner
# Download the latest runner package
osboxes@osboxes:~$ curl -o actions-runner-linux-x64-2.278.0.tar.gz -L https://github.com/actions/runner/releases/download/v2.278.0/actions-runner-linux-x64-2.278.0.tar.gz
# Extract the installer
osboxes@osboxes:~$ tar xzf ./actions-runner-linux-x64-2.278.0.tar.gz
# Create the runner and start the configuration experience
osboxes@osboxes:~/actions-runner$ ./config.sh --url https://github.com/<GITHUB_USER_ALIAS>/<REPOSITORY_NAME> --token ANDM2G24R4H4WTD3GCPW58LARVT2Qosboxes@osboxes:~$ ./run.sh
osboxes@osboxes:~$ gedit dockerfile
FROM debian:jessie
ARG RUNNER_VERSION="2.278.0"
ENV GITHUB_PERSONAL_TOKEN ""
ENV GITHUB_OWNER ""
ENV GITHUB_REPOSITORY ""
RUN apt-get update \
&& apt-get install -y \
curl \
sudo \
git \
jq \
tar \
gnupg2 \
apt-transport-https \
ca-certificates \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
RUN useradd -m github && \
usermod -aG sudo github && \
echo "%sudo ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
USER github
WORKDIR /home/github
RUN curl -O -L https://github.com/actions/runner/releases/download/v$RUNNER_VERSION/actions-runner-linux-x64-$RUNNER_VERSION.tar.gz
RUN tar xzf ./actions-runner-linux-x64-$RUNNER_VERSION.tar.gz
RUN sudo ./bin/installdependencies.sh
COPY --chown=github:github entrypoint.sh ./entrypoint.sh
RUN sudo chmod u+x ./entrypoint.sh
ENTRYPOINT ["/home/github/entrypoint.sh"]osboxes@osboxes:~$ gedit entrypoint.sh
#!/bin/sh
registration_url="https://api.github.com/repos/${GITHUB_OWNER}/${GITHUB_REPOSITORY}/actions/runners/registration-token"
echo "Requesting registration URL at '${registration_url}'"
payload=$(curl -sX POST -H "Authorization: token ${GITHUB_PAT}" ${registration_url})
export RUNNER_TOKEN=$(echo $payload | jq .token --raw-output)
./config.sh \
--name $(hostname) \
--token ${RUNNER_TOKEN} \
--url https://github.com/${GITHUB_OWNER}/${GITHUB_REPOSITORY} \
--work ${RUNNER_WORKDIR} \
--unattended \
--replace
remove() {
./config.sh remove --unattended --token "${RUNNER_TOKEN}"
}
trap 'remove; exit 130' INT
trap 'remove; exit 143' TERM
./run.sh "$*" &
wait $!osboxes@osboxes:~$ docker build . -t runner
Sending build context to Docker daemon 2.107GB
Step 1/14 : FROM debian: jessie
jessie: Pulling from library/debian
---> 48e774d3c4f5
Step 2/14 : ARG GITHUB_RUNNER_VERSION="2.278.0"
Step 3/14 : ENV RUNNER_NAME "runner"
Step 4/14 : ENV GITHUB_PAT ""
Step 5/14 : ENV GITHUB_OWNER ""
Step 6/14 : ENV GITHUB_REPOSITORY ""
Step 7/14 : ENV RUNNER_WORKDIR "_work"
Step 8/14 : RUN apt-get update && apt-get install -y curl sudo git jq && apt-get clean && rm -rf /var/lib/apt/lists/* && useradd -m github && usermod -aG sudo github && echo "%sudo ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
Step 9/14 : USER github
Step 10/14 : WORKDIR /home/github
Step 11/14 : RUN curl -Ls https://github.com/actions/runner/releases/download/v${GITHUB_RUNNER_VERSION}/actions-runner-linux-x64-${GITHUB_RUNNER_VERSION}.tar.gz | tar xz && sudo ./bin/installdependencies.sh
Step 12/14 : COPY --chown=github:github entrypoint.sh ./entrypoint.sh
Step 13/14 : RUN sudo chmod u+x ./entrypoint.sh
Step 14/14 : ENTRYPOINT ["/home/github/entrypoint.sh"]
Successfully built d10b1931c41e
Successfully tagged runner:latestosboxes@osboxes:~$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
runner latest d10b1931c41e 4 minutes ago 422MB
osboxes@osboxes:~$ kubectl create ns gitaction
osboxes@osboxes:~$ gedit secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: github-secret
namespace: gitaction
type: Opaque
stringData:
GITHUB_PERSONAL_TOKEN: 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osboxes@osboxes:~$ kubectl apply -f ./secret.yaml
secret/github-secret createdosboxes@osboxes:~$ gedit Kubernetes.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: runner
namespace: gitaction
labels:
app: runner
spec:
replicas: 1
selector:
matchLabels:
app: runner
template:
metadata:
labels:
app: runner
spec:
containers:
- name: runner
image: geekman/runner:latest
env:
- name: GITHUB_OWNER
value: geekman
- name: GITHUB_REPOSITORY
value: jenkins-pipeline
- name: GITHUB_PERSONAL_TOKEN
valueFrom:
secretKeyRef:
name: github-secret
key: GITHUB_PERSONAL_TOKEN
osboxes@osboxes:~$ kubectl apply -f kubernetes.yaml
deployment.apps/linux-runner configured
osboxes@osboxes:~$ kubectl -n gitaction get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
runner 1/1 1 0 2m
osboxes@osboxes:~$ kubectl -n gitaction get pods
NAME READY STATUS RESTARTS AGE
runner-74c97495cb-8s7m7 1/1 Running 1 15s
osboxes@osboxes:~$ kubectl -n gitaction scale deployment runner --replicas=3
deployment.extensions/runner scaled
NAME READY STATUS RESTARTS AGE
runner-74c97495cb-6r5f7 1/1 ContainerCreating 0 15s
runner-74c97495cb-7y5r6 1/1 ContainerCreating 0 15s
runner-74c97495cb-8s7m7 1/1 Running 0 2m