Introduction
In this article, we will explore how to use Kubebuilder and Kind to create a local test cluster and an operator. Following that operation, we will then deploy that operator in the cluster and test it. All of the code is included below to port-forward to private endpoints the Kubernetes way. Also, if you want to learn more about the idea and the project check out the Forward operator page here.
Essentially, what the code does is to create an alpine/socat pod. You can specify the host, port, and protocol and it will make a tunnel for you, so then you can use port-forward or a service or ingress or whatever to expose things that are in another private subnet.
While this might not sound like a good idea at first, it does have some specific and essential use cases. Check your security constraints though before doing any of this—though in a normal scenario it should be safe. In terms of use cases, this project is useful for testing or for reaching a database while doing some debugging or testing. The tools used in this project are what makes it really interesting as this is for building cloud native applications, since it native to Kubernetes, and that’s what we will explore here.
While Kind is not actually a requirement, I used that for testing and really liked it, it’s much faster and simpler than Minikube.
Also, if you are interested in how I came up with the idea to make this operator, check out this GitHub issue here.
Prerequisites
Create the Project
In this step, we need to create the Kubebuilder project, so in an empty folder we run:
$ go mod init techsquad.rocks
go: creating new go.mod: module techsquad.rocks
$ kubebuilder init --domain techsquad.rocks
go get sigs.k8s.io/controller-runtime@v0.4.0
go mod tidy
Running make...
make
/home/kainlite/Webs/go/bin/controller-gen object:headerFile=./hack/boilerplate.go.txt paths="./..."
go fmt ./...
go vet ./...
go build -o bin/manager main.go
Next: Define a resource with:
$ kubebuilder create apiCreate the API
Next let’s create an API, something for us to use to run our controller.
$ kubebuilder create api --group forward --version v1beta1 --kind Map
Create Resource [y/n]
y
Create Controller [y/n]
y
Writing scaffold for you to edit...
api/v1beta1/map_types.go
controllers/map_controller.go
Running make...
/home/kainlite/Webs/go/bin/controller-gen object:headerFile=./hack/boilerplate.go.txt paths="./..."
go fmt ./...
go vet ./...
go build -o bin/manager main.goUp until this point here, we have only created a boilerplate/basic or empty project with defaults. If you were to test it now, it will work, but it won’t do anything interesting. However, it covers a lot of ground and we should be grateful that such a tool exists.
Add Our Code to the Mix
First, we will add our code to api/v1beta1/map_types.go, which will add our fields to our type.
/*
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package controllers
import (
"context"
"fmt"
"strconv"
"strings"
"github.com/go-logr/logr"
"k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/types"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
forwardv1beta1 "github.com/kainlite/forward/api/v1beta1"
)
// +kubebuilder:rbac:groups=maps.forward.techsquad.rocks,resources=pods,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=map.forward.techsquad.rocks,resources=pods,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=forward.techsquad.rocks,resources=maps,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=forward.techsquad.rocks,resources=pods/status,verbs=get;update;patch
// +kubebuilder:rbac:groups="",resources=pods,verbs=get;list;watch;create;update;patch;delete
// MapReconciler reconciles a Map object
type MapReconciler struct {
client.Client
Log logr.Logger
Scheme *runtime.Scheme
}
func newPodForCR(cr *forwardv1beta1.Map) *corev1.Pod {
labels := map[string]string{
"app": cr.Name,
}
var command string
if strings.EqualFold(cr.Spec.Protocol, "tcp") {
command = fmt.Sprintf("socat -d -d tcp-listen:%s,fork,reuseaddr tcp-connect:%s:%s", strconv.Itoa(cr.Spec.Port), cr.Spec.Host, strconv.Itoa(cr.Spec.Port))
} else if strings.EqualFold(cr.Spec.Protocol, "udp") {
command = fmt.Sprintf("socat -d -d UDP4-RECVFROM:%s,fork,reuseaddr UDP4-SENDTO:%s:%s", strconv.Itoa(cr.Spec.Port), cr.Spec.Host, strconv.Itoa(cr.Spec.Port))
} else {
// TODO: Create a proper error here if the protocol doesn't match or is unsupported
command = fmt.Sprintf("socat -V")
}
return &corev1.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: "forward-" + cr.Name + "-pod",
Namespace: cr.Namespace,
Labels: labels,
},
Spec: corev1.PodSpec{
Containers: []corev1.Container{
{
Name: "map",
Image: "alpine/socat",
Command: strings.Split(command, " "),
},
},
RestartPolicy: corev1.RestartPolicyOnFailure,
},
}
}
func (r *MapReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
reqLogger := r.Log.WithValues("namespace", req.Namespace, "MapForward", req.Name)
reqLogger.Info("=== Reconciling Forward Map")
// Fetch the Map instance
instance := &forwardv1beta1.Map{}
err := r.Get(context.TODO(), req.NamespacedName, instance)
if err != nil {
if errors.IsNotFound(err) {
// Request object not found, could have been deleted after
// reconcile request—return and don't requeue:
return reconcile.Result{}, nil
}
// Error reading the object—requeue the request:
return reconcile.Result{}, err
}
// If no phase set, default to pending (the initial phase):
if instance.Status.Phase == "" || instance.Status.Phase == "PENDING" {
instance.Status.Phase = forwardv1beta1.PhaseRunning
}
// Now let's make the main case distinction: implementing
// the state diagram PENDING -> RUNNING or PENDING -> FAILED
switch instance.Status.Phase {
case forwardv1beta1.PhasePending:
reqLogger.Info("Phase: PENDING")
reqLogger.Info("Waiting to forward", "Host", instance.Spec.Host, "Port", instance.Spec.Port)
instance.Status.Phase = forwardv1beta1.PhaseRunning
case forwardv1beta1.PhaseRunning:
reqLogger.Info("Phase: RUNNING")
pod := newPodForCR(instance)
// Set Map instance as the owner and controller
err := controllerutil.SetControllerReference(instance, pod, r.Scheme)
if err != nil {
// requeue with error
return reconcile.Result{}, err
}
found := &corev1.Pod{}
nsName := types.NamespacedName{Name: pod.Name, Namespace: pod.Namespace}
err = r.Get(context.TODO(), nsName, found)
// Try to see if the pod already exists and if not
// (which we expect) then create a one-shot pod as per spec:
if err != nil && errors.IsNotFound(err) {
err = r.Create(context.TODO(), pod)
if err != nil {
// requeue with error
return reconcile.Result{}, err
}
reqLogger.Info("Pod launched", "name", pod.Name)
} else if err != nil {
// requeue with error
return reconcile.Result{}, err
} else if found.Status.Phase == corev1.PodFailed ||
found.Status.Phase == corev1.PodSucceeded {
reqLogger.Info("Container terminated", "reason",
found.Status.Reason, "message", found.Status.Message)
instance.Status.Phase = forwardv1beta1.PhaseFailed
} else {
// Don't requeue because it will happen automatically when the
// pod status changes.
return reconcile.Result{}, nil
}
case forwardv1beta1.PhaseFailed:
reqLogger.Info("Phase: Failed, check that the host and port are reachable from the cluster and that there are no networks policies preventing this access or firewall rules...")
return reconcile.Result{}, nil
default:
reqLogger.Info("NOP")
return reconcile.Result{}, nil
}
// Update the At instance, setting the status to the respective phase:
err = r.Status().Update(context.TODO(), instance)
if err != nil {
return reconcile.Result{}, err
}
// Don't requeue. We should be reconcile because either the pod
// or the CR changes.
return reconcile.Result{}, nil
}
func (r *MapReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&forwardv1beta1.Map{}).
Complete(r)
}Essentially, we just edited the MapSpec and the MapStatus structure.
Now, we need to add the code to our controller in controllers/map_controller.go
In this controller, we have now added two functions: one to create a pod and the other to modify the entire Reconcile function (this one takes care of checking the status and make the transitions—in other words, it makes a controller work like a controller. Also, have you noticed how the Kubebuilder annotations generates the RBAC config for us? Pretty handy, right?
Starting the Cluster with Kind
Next, we will use Kind to create a local cluster to test:
$ kind create cluster --name test-cluster-1
Creating cluster "test-cluster-1" ...
✓ Ensuring node image (kindest/node:v1.16.3) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to "kind-test-cluster-1"
You can now use your cluster with:
kubectl cluster-info --context kind-test-cluster-1Could it really be that easy!? Well, yes, it is!
Running our Operator Locally
For testing, you can run your operator locally like this:
$ make run
/home/kainlite/Webs/go/bin/controller-gen object:headerFile=./hack/boilerplate.go.txt paths="./..."
go fmt ./...
go vet ./...
/home/kainlite/Webs/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go run ./main.go
2020-01-17T21:00:14.465-0300 INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":8080"}
2020-01-17T21:00:14.466-0300 INFO setup starting manager
2020-01-17T21:00:14.466-0300 INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
2020-01-17T21:00:14.566-0300 INFO controller-runtime.controller Starting EventSource {"controller": "map", "source": "kind source: /, Kind="}
2020-01-17T21:00:14.667-0300 INFO controller-runtime.controller Starting Controller {"controller": "map"}
2020-01-17T21:00:14.767-0300 INFO controller-runtime.controller Starting workers {"controller": "map", "worker count": 1}Testing it
First, we spin up a pod, and launch nc -l -p 8000.
$ kubectl run -it --rm --restart=Never alpine --image=alpine sh
If you don't see a command prompt, try pressing enter.
# ifconfig
eth0 Link encap:Ethernet HWaddr E6:49:53:CA:3D:89
inet addr:10.244.0.8 Bcast:10.244.0.255 Mask:255.255.255.0
inet6 addr: fe80::e449:53ff:feca:3d89/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:698 (698.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
/ # nc -l -p 8000
testNow, we edit our manifest and apply it—checking that everything is in place. Then we do the port-forward and launch another nc localhost 8000 to test if everything went well. First, the manifest:
$ cat config/samples/forward_v1beta1_map.yaml
apiVersion: forward.techsquad.rocks/v1beta1
kind: Map
metadata:
name: mapsample
namespace: default
spec:
host: 10.244.0.8
port: 8000
protocol: tcpThen port-forward and test the code.
$ kubectl apply -f config/samples/forward_v1beta1_map.yaml
map.forward.techsquad.rocks/mapsample configured
# Logs in the controller
2020-01-17T23:38:27.650Z INFO controllers.Map === Reconciling Forward Map {"namespace": "default", "MapForward": "mapsample"}
2020-01-17T23:38:27.691Z INFO controllers.Map Phase: RUNNING {"namespace": "default", "MapForward": "mapsample"}
2020-01-17T23:38:27.698Z DEBUG controller-runtime.controller Successfully Reconciled {"controller": "map", "request": "default/mapsample"}
$ kubectl port-forward forward-mapsample-pod 8000:8000
Forwarding from 127.0.0.1:8000 -> 8000
Handling connection for 8000
# In another terminal or tab or split
$ nc localhost 8000
testMaking it publicly ready
Here, we just build and push the Docker image to Docker Hub or to your favorite public registry.
$ make docker-build docker-push IMG=kainlite/forward:0.0.1
/home/kainlite/Webs/go/bin/controller-gen object:headerFile=./hack/boilerplate.go.txt paths="./..."
go fmt ./...
go vet ./...
/home/kainlite/Webs/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go test ./... -coverprofile cover.out
? github.com/kainlite/forward [no test files]
? github.com/kainlite/forward/api/v1beta1 [no test files]
ok github.com/kainlite/forward/controllers 6.720s coverage: 0.0% of statements
docker build . -t kainlite/forward:0.0.1
Sending build context to Docker daemon 45.02MB
Step 1/14 : FROM golang:1.13 as builder
1.13: Pulling from library/golang
8f0fdd3eaac0: Pull complete
...
...
...
---> 4dab137d22a1
Successfully built 4dab137d22a1
Successfully tagged kainlite/forward:0.0.1
docker push kainlite/forward:0.0.1
The push refers to repository [docker.io/kainlite/forward]
50a214d52a70: Pushed
84ff92691f90: Pushed
0d1435bd79e4: Pushed
0.0.1: digest: sha256:b4479e4721aa9ec9e92d35ac7ad5c4c0898986d9d2c9559c4085d4c98d2e4ae3 size: 945Then you can install it with make deploy IMG=kainlite/forward:0.0.1 and uninstall it with make uninstall.
Closing notes
Be sure to check out the Kubebuilder book if you want to learn more and also read the Kind docs. If you enjoyed this blog, please follow me on Twitter or GitHub!
Errata
If you spot any error or have any suggestions, please send me a message so that I can update the code.
Also, you can check the source code and changes in the generated code and the sources here.
This blog was originally published here.
License：MIT License
Caylent provides a critical DevOps-as-a-Service function to high growth companies looking for expert support with Kubernetes, cloud security, cloud infrastructure, and CI/CD pipelines. Our managed and consulting services are a more cost-effective option than hiring in-house, and we scale as your team and company grow. Check out some of the use cases, learn how we work with clients, and read more about our DevOps-as-a-Service offering.
