02.25.21

Kubernetes Cluster Federation with Admiralty

By Mathias Phoyu
#clusterfederation

Kubernetes today is a hugely prevalent tool in 2021, and more organizations are increasingly running their applications on multiple clusters of Kubernetes. But these multiple cluster architectures often have a combination of multiple cloud providers, multiple data centres, multiple regions, and multiple zones where the applications are running. So, deploying your application or service on clusters with such diverse resources is a complicated endeavor. This challenge is what the process of federation is intended to help overcome. The fundamental use case of federation is to scale applications on multiple clusters with ease. The process negates the need to perform the deployment step more than once. Instead, you perform one deployment, and the application is deployed on multiple clusters as listed in the federation list.

What Is Kubernetes Cluster Federation?

Essentially, Kubernetes cluster federation is a mechanism to provide one way or one practice to distribute applications and services to multiple clusters. One of the most important things to note is that federation is not about cluster management, federation is about application management. 

Cluster federation is a way of federating your existing clusters as one single curated cluster. So, if you are leveraging Kubernetes clusters in different zones in different countries, you can treat all of them as a single cluster.

In cluster federation, we optimize a host cluster and multiple member clusters. The host cluster comprises all the configurations which pass on all the member clusters. Member clusters are the clusters that share the workloads. It is possible to have a host cluster also share the workload and act as a member cluster, but organizations tend to keep the host clusters separate for simplicity. On the host cluster, it’s important to install the cluster registry and the federated API. Now with the cluster registry, the host will have all the information to connect to the member Clusters. And with the federated API, you require all the controllers running on our host clusters to make sure they reconcile the federated resources. In a nutshell, the host cluster will act as a control plane and propagate and push configuration to the member clusters.

If we talk about federation manifests, we need to discuss two particular points of information: cluster replacement and the configuration override. It is crucial to configure where we want the application deployment and which member clusters the deployment should leverage with cluster replacement. In the configuration overrides, we are able to change any configuration and tailor our application to the cluster if needed. Another feature for federation is the namespace scoped control plane. With the namespace scope control plane, we can use just one namespace to federate the namespace across different clusters. This is quite a powerful mechanism if the intention is to use federation in production but don’t wish to dedicate an entire cluster. Instead, you can just dedicate one namespace.

How to set up Federated Kubernetes Clusters?

KubeFeb v2 is the official Kubernetes cluster federation implementation. It uses a single set of APIs for the coordination of configurations between the host clusters and the member clusters. KubeFeb runs two types of information, one is the type configuration, and the other is the cluster configuration. FederatedTypeConfig stores the type configuration, which tells KubeFed which API types it has to handle. KubeFedCluster and KubeFedConfig comprise the cluster configurations which tell KubeFeb which clusters to target. The host cluster has the KuberFeb control plane. The KuberFeb control plane has KubeFedCluster and KubeFedConfig resources for each member cluster.

Now that we have covered the fundamentals, let’s set up Federated Kubernetes Clusters. To set the federation, a prerequisite is to have KubeFed up and running on the cluster. Once you have the KubeFed control plane up and running, download the kubefedctl command-line tool to communicate with it. You can get the latest release on GitHub

Before starting working with multiple clusters, Federated CDRs must be installed in the host cluster to interact with it. 

helm --namespace kube-federation-system upgrade -i kubefed kubefed-charts/kubefed --create-namespace --kube-context cluster1

The first step in this process then is to join the clusters using the kubefedctl tool. Here, we are using two clusters for a federation example.

kubefedctl join cluster1 --cluster-context cluster1 --host-cluster-context cluster1 --v=2

kubefedctl join cluster2 --cluster-context cluster2 --host-cluster-context cluster1 --v=2

Check if the clusters are available in the federation system.

kubectl -n kube-federation-system get kubefedclusters
NAME       READY   AGE
cluster1   True	1m
cluster2   True	1m

Now it’s time to play with the federation. Firstly, create a federated namespace on the host cluster and add the federated clusters in the placement specification.

apiVersion: v1
kind: Namespace
metadata:
  name: dev-namespace
---
apiVersion: types.kubefed.io/v1beta1
kind: FederatedNamespace
metadata:
  name: dev-namespace
  namespace: dev-namespace
spec:
  placement:
	clusters:
	- name: cluster1
	- name: cluster2

If you deploy an application or a service on the federated namespace, it will be deployed on both cluster1 and cluster2. Now let us do a federation deployment on the federation clusters. Here is an example that installs Nginx on all federation clusters in Kubernetes.

apiVersion: types.kubefed.io/v1beta1
kind: FederatedDeployment
metadata:
  name: dev-deployment
  namespace: dev-namespace
spec:
  template:
	metadata:
  	labels:
    	app: nginx
	spec:
  	replicas: 3
  	selector:
    	matchLabels:
      	app: nginx
  	template:
    	metadata:
      	labels:
        	app: nginx
    	spec:
      	containers:
      	- image: nginx
        	name: nginx
  placement:
	clusters:
	- name: cluster1
	- name: cluster2

And there you have it, cluster federation in Kubernetes.

How to Use Admiralty for Kubernetes Cluster Federation?

In this section, we will look at how to leverage Admiralty for the Kubernetes cluster federation. Admiralty is a new player in the multi-cluster Kubernetes domain. Admiralty schedules the workloads across different clusters intelligently using a system of Kubernetes controllers.

Firstly, you need to install Admiralty on all the clusters in the federation list. Once the installation is in place, add multicluster.admiralty.io/elect=”” annotation at the pod level. Now let us consider the same Nginx deployment example shown above. To use Admiralty for multi cluster deployment using cluster federation, the template will change to as below:

template:
 metadata:
  	 labels:
    	 app: nginx
  	 annotations:
    	 multicluster.admiralty.io/elect: ""

To provide an overview of how cluster federation works with Admiralty: When you add multicluster.admiralty.io/elect annotations to the workloads, a multi-cluster-scheduler creates a dummy pod. The actual pod mutates to a proxy pod scheduled on virtual-kubelet nodes. The pod goes into another round of scheduling to decide where the pod will run. The scheduler then creates delegate pods on the member clusters. After this, a feedback loop updates the proxy pods’ annotations with the annotations present in the delegate pods. And there you have it. 

Advantages of Admiralty

Although Admiralty is a new tool with just five contributors working on this project currently, it has multiple advantages. Below are the advantages of Admiralty:

  • High performance with high availability
  • Centralized management for access control and audit logs
  • Can migrate applications during cluster upgrade
  • Register clusters at scale with ease
  • Provides cloud product for edge, computing Internet of Things (IoT)
  • Active disaster recovery
  • Resource federation
  • Distributed workflows 

Conclusion

Cluster federation is all about managing multiple applications into multiple clusters and not about how you manage multiple clusters. If you are working on multiple Kubernetes clusters, cluster federation is your ticket to a unified deployment process. To further optimize your work, go ahead and give Admiralty a try for cluster federation on the Kubernetes clusters.


Caylent provides a critical DevOps-as-a-Service function to high growth companies looking for expert support with Kubernetes, cloud security, cloud infrastructure, and CI/CD pipelines. Our managed and consulting services are a more cost-effective option than hiring in-house, and we scale as your team and company grow. Check out some of the use cases, learn how we work with clients, and read more about our DevOps-as-a-Service offering.