KubeMQ: A Kubernetes-Native Message Queue Solution

These days, the competition between enterprise organizations is cutthroat. Every organization is looking to exchange information between their systems instantly, in real-time or near-realtime, to make better and faster decisions. For such information to flow continuously, the integration between application components needs to be seamless. To take all the benefits of cloud computing, an application built these days needs to be cloud-native, flexible, scalable, and be deployed quickly. Such an application is split into several small components called microservices. Each microservice can run independently of the others. Each service needs to have a means in order to talk to each other so that they can work in tandem. This is where message brokers come into the picture. Message brokers are the communication connector between the cloud components in the hybrid environments and the on-premises systems.

Full Messaging Solution for Enterprises

You can implement message brokers across a variety of business problems in different domains with different enterprise cloud environments. They are optimal for inter-application communication and ensuring that the message (information packets) are delivered as required. You can commonly find message brokers in financial transactions and payment processing, e-commerce order processing and fulfilment, protecting highly sensitive data, etc. The rise in the Internet of Things (IoT) has introduced many electronic devices that generate a massive amount of data. Here the communication is happening from machine to machine where the electronic devices are talking to each other. At this level also, message brokers are used for real-time communication between devices, which is crucial for data analytics and business intelligence to enable you to provide your services to your customers in real-time.

You must store the message reliably and guarantee the delivery of the message in order to make sure the communication between the application is happening correctly. To do so, message brokers use a component called a message queue. We can optimise message queues to order and store the messages until the application consumes them. A message queue stores the messages in the exact order of the transmission and they remain in the queue till the application receives them.

So fundamentally, the message queues are used to facilitate the communications between the applications by sending and receiving message data. Once the application receives the message data, the application uses them to interact with databases, business logic and web browsers. The message queue is not aware of the content in the message data packets. Message queues are a reliable and secure transport layer to move the data unchanged between the applications. They use a bunch of application programming interfaces to send and receive message data. These application programming interfaces support Visual Basic, Java, C, COBOL, etc., across all platforms.

Introduction to KubeMQ

KubeMQ is an enterprise-grade, real-time, highly available, scalable and secure message broker and message queue which is also a Kubernetes native solution. The tool is also lightweight, so we can deploy KubeMQ in a container in just one minute. The KubeMQ container size is just 30 MB. KubeMQ is a Go programming language application. KubeMQ can easily integrate with third party tools like Prometheus, Datadog, Zipkin and many other cloud-native applications. KubeMQ is the message queue that supports efficient memory usage and high volume messaging with low latency. The messaging tool also supports several messaging patterns such as Publish-Subscribe (a.k.a Pub/Sub), persistent queue, CQRS based RPC flows.

Compared to Kafka, RabbitMQ or ActiveMQ, the tool is a relatively new solution. But when it comes to Kubernetes, KubeMQ has a big advantage over others. KubeMQ is a Kubernetes native message broker, so the tool can be deployed on a Kubernetes cluster by just a single command without any additional manifest or templates. This is one of the main reasons why enterprise organizations choose KubeMQ if their applications are running on a Kubernetes cluster. 

If you are an enterprise organization, you can use KubeMQ to save a lot of money and time by integrating your development and operations workflow into a unified system. You don’t need to be an expert to use the tool either, as it is very simple to use and DevOps friendly. KubeMQ helps you accelerate development and production cycles.

Top KubeMQ Features:

• One distributed infrastructure messaging platform for Kubernetes backend
• Super-fast, small and lightweight Docker container.
• Messaging delivery support for at least once, at most once and exactly once delivery models.
• Supports durable FIFO based Queue, RPC Command, Publish-Subscribe with Persistence (Events Store), Publish-Subscribe Events, and Query messaging patterns.
• Supports WebSocket Transport protocols, REST and gRPC with TLS support.
• Built-in tracing, metrics and caching.
• The tool provides a slick monitoring dashboard.
• Provides SDKs for Go, Java, Python, C#, PHP, Ruby, Node, jQuery and cURL.
• Message broker configuration not required.

Advantages/Disadvantages of KubeMQ

Advantages:

• The KubeMQ founder team is a group of IT veterans with 20 years plus of experience in developing robust and scalable technologies.
• KubeMQ now supports billions of messages in real-time in the production environment.
• Integrates easily with financial, healthcare, e-commerce, telecom and cyber systems.
• Speeds up your development and production cycles.
• Saves a lot of operational cost for the organization.
• Supports a rich set of connectors to connect with microservices.

Disadvantages:

• Has a few limitations for integration with SQL systems.
• SDKs for NodeJs is hard to follow.
• No official KubeMQ community yet.
• Tracing support for the C# client library is not yet there.

Solution Use Cases and Considerations

As KubeMQ is a Kubernetes-native solution, the tool has a couple of places where it truly shines and a couple of areas where challenges can, and will, arise. In most complex architectural designs, there are no magic bullets and no one-size-fits-all answers to any critical solution component—KubeMQ is no exception. 

The Upside

• This tool is easy to deploy alongside your Kuberntes workloads that depend on it, making it easy to use KubeMQ all the way from local development to production.  
• Being a Kubernetes workload itself, the tool can be a great fit where you want a consistent messaging/queuing solution across different Cloud providers, or between Cloud and on-premises.  
• KubeMQ solves many messaging needs, making it capable of being a single solution which can work in place of several others. For scenario one, if you need pub/sub, worker message queues, and a stream-style queue with individual tracking of producers and consumers. Scenario two, in AWS you might otherwise need to leverage SQS, SNS, and Managed Streaming for Kafka as discrete services. KubeMQ can do all of it.

The Downside

• Kubernetes doesn’t necessarily give you a great infrastructure environment by default for running all of the use cases KubeMQ can support. Some KubeMQ use cases need high-speed local storage to have good throughput, and if you dropped this into a typical EKS cluster in AWS, you might find the combination of gp2 (or gp3) EBS storage isn’t sufficient for your streaming workload to handle your producer volume. You might also find that you don’t have the network throughput needed in small general purpose worker node instances to not throttle traffic. You might need to have specialty worker nodes with local NVMe and high speed networking and then do advanced setup to ensure that KubeMQ only runs there and uses those high-speed volumes exclusively.
• While KubeMQ addresses many use cases, it may not be the best fit, fastest, or most scalable option for your specific one. For pure worker queue use-cases you will be able to support massive scale with little thought, work, or planning by just plugging in something like AWS SQS. Similar can be said for pub/sub at scale with options like AWS SNS.
• Added Operational overhead for a Kubernetes-based solution like KubeMQ isn’t nearly the same burden as managing dedicated VMs (assuming you are already managing Kubernetes infrastructure today.) But, by and large, it will still require more management and operational overhead than the equivalent managed services offered by the major Cloud providers. Especially if you spend any significant time optimizing it to meet your scaling objectives.

It is up to you to determine if the advantages of flexibility outweigh the advantages of operational overhead, and sometimes things like multi-cloud requirements (and thus the requirement to provision/support different managed services from different providers) or needing to support running a workload both in the Cloud and on-premises will help bring clarity to which is the right architectural fit for you.

However, if you’re still not sure if KubeMQ might work for you? Talk to a Caylent Solutions Architect to help you sort out the best way to solve for your messaging, queueing, and Pub/Sub needs for your unique circumstances

Conclusion

In this digital work, software developers often face problems in sending and receiving message data between applications/services. Before the development of such tools, developers used endpoints for communications and message exchange, but now you have message brokers and message queues. These are the modern, reliable and secure ways that the enterprise organization uses to communicate between applications/services. Since KubeMQ is Kubernetes native, the tool is a perfect fit as a messaging system on your Kubernetes clusters.

Caylent provides a critical DevOps-as-a-Service function to high growth companies looking for expert support with Kubernetes, cloud security, cloud infrastructure, and CI/CD pipelines. Our managed and consulting services are a more cost-effective option than hiring in-house, and we scale as your team and company grow. Check out some of the use cases, learn how we work with clients, and read more about our DevOps-as-a-Service offering.