04.09.20

Building Cloud-Native Applications with Kubebuilder and Kind

By Gabriel Garrido
Building Cloud-Native Applications with #Kubebuilder and #Kind

Introduction

In this article, we will explore how to use Kubebuilder and Kind to create a local test cluster and an operator. Following that operation, we will then deploy that operator in the cluster and test it. All of the code is included below to port-forward to private endpoints the Kubernetes way. Also, if you want to learn more about the idea and the project check out the Forward operator page here.

Essentially, what the code does is to create an alpine/socat pod. You can specify the host, port, and protocol and it will make a tunnel for you, so then you can use port-forward or a service or ingress or whatever to expose things that are in another private subnet.

While this might not sound like a good idea at first, it does have some specific and essential use cases. Check your security constraints though before doing any of this⁠—though in a normal scenario it should be safe. In terms of use cases, this project is useful for testing or for reaching a database while doing some debugging or testing. The tools used in this project are what makes it really interesting as this is for building cloud native applications, since it native to Kubernetes, and that’s what we will explore here.

While Kind is not actually a requirement, I used that for testing and really liked it, it’s much faster and simpler than Minikube.

Also, if you are interested in how I came up with the idea to make this operator, check out this GitHub issue here.

Prerequisites

Create the Project

In this step, we need to create the Kubebuilder project, so in an empty folder we run:

$ go mod init techsquad.rocks
go: creating new go.mod: module techsquad.rocks

$ kubebuilder init --domain techsquad.rocks
go get sigs.k8s.io/controller-runtime@v0.4.0
go mod tidy
Running make...
make
/home/kainlite/Webs/go/bin/controller-gen object:headerFile=./hack/boilerplate.go.txt paths="./..."
go fmt ./...
go vet ./...
go build -o bin/manager main.go
Next: Define a resource with:
$ kubebuilder create api

Create the API

Next let’s create an API, something for us to use to run our controller.

$ kubebuilder create api --group forward --version v1beta1 --kind Map
Create Resource [y/n]
y
Create Controller [y/n]
y
Writing scaffold for you to edit...
api/v1beta1/map_types.go
controllers/map_controller.go
Running make...
/home/kainlite/Webs/go/bin/controller-gen object:headerFile=./hack/boilerplate.go.txt paths="./..."
go fmt ./...
go vet ./...
go build -o bin/manager main.go

Up until this point here, we have only created a boilerplate/basic or empty project with defaults. If you were to test it now, it will work, but it won’t do anything interesting. However, it covers a lot of ground and we should be grateful that such a tool exists.

Add Our Code to the Mix

First, we will add our code to api/v1beta1/map_types.go, which will add our fields to our type.

/*

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package controllers

import (
	"context"
	"fmt"
	"strconv"
	"strings"

	"github.com/go-logr/logr"
	"k8s.io/apimachinery/pkg/api/errors"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/types"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
	"sigs.k8s.io/controller-runtime/pkg/reconcile"

	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

	forwardv1beta1 "github.com/kainlite/forward/api/v1beta1"
)

// +kubebuilder:rbac:groups=maps.forward.techsquad.rocks,resources=pods,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=map.forward.techsquad.rocks,resources=pods,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=forward.techsquad.rocks,resources=maps,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=forward.techsquad.rocks,resources=pods/status,verbs=get;update;patch
// +kubebuilder:rbac:groups="",resources=pods,verbs=get;list;watch;create;update;patch;delete

// MapReconciler reconciles a Map object
type MapReconciler struct {
	client.Client
	Log    logr.Logger
	Scheme *runtime.Scheme
}

func newPodForCR(cr *forwardv1beta1.Map) *corev1.Pod {
	labels := map[string]string{
		"app": cr.Name,
	}
	var command string
	if strings.EqualFold(cr.Spec.Protocol, "tcp") {
		command = fmt.Sprintf("socat -d -d tcp-listen:%s,fork,reuseaddr tcp-connect:%s:%s", strconv.Itoa(cr.Spec.Port), cr.Spec.Host, strconv.Itoa(cr.Spec.Port))
	} else if strings.EqualFold(cr.Spec.Protocol, "udp") {
		command = fmt.Sprintf("socat -d -d UDP4-RECVFROM:%s,fork,reuseaddr UDP4-SENDTO:%s:%s", strconv.Itoa(cr.Spec.Port), cr.Spec.Host, strconv.Itoa(cr.Spec.Port))
	} else {
		// TODO: Create a proper error here if the protocol doesn't match or is unsupported
		command = fmt.Sprintf("socat -V")
	}

	return &corev1.Pod{
		ObjectMeta: metav1.ObjectMeta{
			Name:      "forward-" + cr.Name + "-pod",
			Namespace: cr.Namespace,
			Labels:    labels,
		},
		Spec: corev1.PodSpec{
			Containers: []corev1.Container{
				{
					Name:    "map",
					Image:   "alpine/socat",
					Command: strings.Split(command, " "),
				},
			},
			RestartPolicy: corev1.RestartPolicyOnFailure,
		},
	}
}

func (r *MapReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
	reqLogger := r.Log.WithValues("namespace", req.Namespace, "MapForward", req.Name)
	reqLogger.Info("=== Reconciling Forward Map")
	// Fetch the Map instance
	instance := &forwardv1beta1.Map{}
	err := r.Get(context.TODO(), req.NamespacedName, instance)
	if err != nil {
		if errors.IsNotFound(err) {
			// Request object not found, could have been deleted after
			// reconcile request—return and don't requeue:
			return reconcile.Result{}, nil
		}
		// Error reading the object—requeue the request:
		return reconcile.Result{}, err
	}

	// If no phase set, default to pending (the initial phase):
	if instance.Status.Phase == "" || instance.Status.Phase == "PENDING" {
		instance.Status.Phase = forwardv1beta1.PhaseRunning
	}

	// Now let's make the main case distinction: implementing
	// the state diagram PENDING -> RUNNING or PENDING -> FAILED
	switch instance.Status.Phase {
	case forwardv1beta1.PhasePending:
		reqLogger.Info("Phase: PENDING")
		reqLogger.Info("Waiting to forward", "Host", instance.Spec.Host, "Port", instance.Spec.Port)
		instance.Status.Phase = forwardv1beta1.PhaseRunning
	case forwardv1beta1.PhaseRunning:
		reqLogger.Info("Phase: RUNNING")
		pod := newPodForCR(instance)
		// Set Map instance as the owner and controller
		err := controllerutil.SetControllerReference(instance, pod, r.Scheme)
		if err != nil {
			// requeue with error
			return reconcile.Result{}, err
		}
		found := &corev1.Pod{}
		nsName := types.NamespacedName{Name: pod.Name, Namespace: pod.Namespace}
		err = r.Get(context.TODO(), nsName, found)
		// Try to see if the pod already exists and if not
		// (which we expect) then create a one-shot pod as per spec:
		if err != nil && errors.IsNotFound(err) {
			err = r.Create(context.TODO(), pod)
			if err != nil {
				// requeue with error
				return reconcile.Result{}, err
			}
			reqLogger.Info("Pod launched", "name", pod.Name)
		} else if err != nil {
			// requeue with error
			return reconcile.Result{}, err
		} else if found.Status.Phase == corev1.PodFailed ||
			found.Status.Phase == corev1.PodSucceeded {
			reqLogger.Info("Container terminated", "reason",
				found.Status.Reason, "message", found.Status.Message)
			instance.Status.Phase = forwardv1beta1.PhaseFailed
		} else {
			// Don't requeue because it will happen automatically when the
			// pod status changes.
			return reconcile.Result{}, nil
		}
	case forwardv1beta1.PhaseFailed:
		reqLogger.Info("Phase: Failed, check that the host and port are reachable from the cluster and that there are no networks policies preventing this access or firewall rules...")
		return reconcile.Result{}, nil
	default:
		reqLogger.Info("NOP")
		return reconcile.Result{}, nil
	}

	// Update the At instance, setting the status to the respective phase:
	err = r.Status().Update(context.TODO(), instance)
	if err != nil {
		return reconcile.Result{}, err
	}

	// Don't requeue. We should be reconcile because either the pod
	// or the CR changes.
	return reconcile.Result{}, nil
}

func (r *MapReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&forwardv1beta1.Map{}).
		Complete(r)
}

Essentially, we just edited the MapSpec and the MapStatus structure.

Now, we need to add the code to our controller in controllers/map_controller.go

In this controller, we have now added two functions: one to create a pod and the other to modify the entire Reconcile function (this one takes care of checking the status and make the transitions⁠—in other words, it makes a controller work like a controller. Also, have you noticed how the Kubebuilder annotations generates the RBAC config for us? Pretty handy, right?

Starting the Cluster with Kind

Next, we will use Kind to create a local cluster to test:

$ kind create cluster --name test-cluster-1
Creating cluster "test-cluster-1" ...
 ✓ Ensuring node image (kindest/node:v1.16.3) 🖼 
 ✓ Preparing nodes 📦 
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-test-cluster-1"
You can now use your cluster with:

kubectl cluster-info --context kind-test-cluster-1

Could it really be that easy!? Well, yes, it is!

Running our Operator Locally

For testing, you can run your operator locally like this:

$ make run
/home/kainlite/Webs/go/bin/controller-gen object:headerFile=./hack/boilerplate.go.txt paths="./..."
go fmt ./...
go vet ./...
/home/kainlite/Webs/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go run ./main.go
2020-01-17T21:00:14.465-0300    INFO    controller-runtime.metrics      metrics server is starting to listen    {"addr": ":8080"}
2020-01-17T21:00:14.466-0300    INFO    setup   starting manager
2020-01-17T21:00:14.466-0300    INFO    controller-runtime.manager      starting metrics server {"path": "/metrics"}
2020-01-17T21:00:14.566-0300    INFO    controller-runtime.controller   Starting EventSource    {"controller": "map", "source": "kind source: /, Kind="}
2020-01-17T21:00:14.667-0300    INFO    controller-runtime.controller   Starting Controller     {"controller": "map"}
2020-01-17T21:00:14.767-0300    INFO    controller-runtime.controller   Starting workers        {"controller": "map", "worker count": 1}

Testing it

First, we spin up a pod, and launch nc -l -p 8000.

$ kubectl run -it --rm --restart=Never alpine --image=alpine sh
If you don't see a command prompt, try pressing enter.

# ifconfig
eth0      Link encap:Ethernet  HWaddr E6:49:53:CA:3D:89  
          inet addr:10.244.0.8  Bcast:10.244.0.255  Mask:255.255.255.0
          inet6 addr: fe80::e449:53ff:feca:3d89/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:698 (698.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
/ # nc -l -p 8000
test

Now, we edit our manifest and apply it⁠—checking that everything is in place. Then we do the port-forward and launch another nc localhost 8000 to test if everything went well. First, the manifest:

$ cat config/samples/forward_v1beta1_map.yaml 
apiVersion: forward.techsquad.rocks/v1beta1
kind: Map
metadata:
  name: mapsample
  namespace: default
spec:
  host: 10.244.0.8
  port: 8000
  protocol: tcp

Then port-forward and test the code.

$ kubectl apply -f config/samples/forward_v1beta1_map.yaml
map.forward.techsquad.rocks/mapsample configured

# Logs in the controller
2020-01-17T23:38:27.650Z        INFO    controllers.Map === Reconciling Forward Map     {"namespace": "default", "MapForward": "mapsample"}
2020-01-17T23:38:27.691Z        INFO    controllers.Map Phase: RUNNING  {"namespace": "default", "MapForward": "mapsample"}
2020-01-17T23:38:27.698Z        DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "map", "request": "default/mapsample"}

$ kubectl port-forward forward-mapsample-pod 8000:8000                                                                                                                                                                       
Forwarding from 127.0.0.1:8000 -> 8000                                                                                                                                                                                                                                           
Handling connection for 8000                                               

# In another terminal or tab or split
$ nc localhost 8000
test

Making it publicly ready

Here, we just build and push the Docker image to Docker Hub or to your favorite public registry.

$ make docker-build docker-push IMG=kainlite/forward:0.0.1
/home/kainlite/Webs/go/bin/controller-gen object:headerFile=./hack/boilerplate.go.txt paths="./..."
go fmt ./...
go vet ./...
/home/kainlite/Webs/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go test ./... -coverprofile cover.out
?       github.com/kainlite/forward     [no test files]
?       github.com/kainlite/forward/api/v1beta1 [no test files]
ok      github.com/kainlite/forward/controllers 6.720s  coverage: 0.0% of statements
docker build . -t kainlite/forward:0.0.1
Sending build context to Docker daemon  45.02MB
Step 1/14 : FROM golang:1.13 as builder
1.13: Pulling from library/golang
8f0fdd3eaac0: Pull complete
...
...
...
 ---> 4dab137d22a1
Successfully built 4dab137d22a1
Successfully tagged kainlite/forward:0.0.1
docker push kainlite/forward:0.0.1
The push refers to repository [docker.io/kainlite/forward]
50a214d52a70: Pushed 
84ff92691f90: Pushed 
0d1435bd79e4: Pushed 
0.0.1: digest: sha256:b4479e4721aa9ec9e92d35ac7ad5c4c0898986d9d2c9559c4085d4c98d2e4ae3 size: 945

Then you can install it with make deploy IMG=kainlite/forward:0.0.1 and uninstall it with make uninstall.

Closing notes

Be sure to check out the Kubebuilder book if you want to learn more and also read the Kind docs. If you enjoyed this blog, please follow me on Twitter or GitHub!

Errata

If you spot any error or have any suggestions, please send me a message so that I can update the code.

Also, you can check the source code and changes in the generated code and the sources here.

This blog was originally published here.

License:MIT License


Caylent provides a critical DevOps-as-a-Service function to high growth companies looking for expert support with Kubernetes, cloud security, cloud infrastructure, and CI/CD pipelines. Our managed and consulting services are a more cost-effective option than hiring in-house, and we scale as your team and company grow. Check out some of the use cases, learn how we work with clients, and read more about our DevOps-as-a-Service offering.