A few days ago we published a blog post about how to use Apache Kafka’s proprietary authentication and authorization (ACL) with Kubernetes Service Accounts via Supertubes. If you skimmed that post or have actually used it with Supertubes, you might think that this is (as Shakespeare put it) much ado about nothing, as things just work out-of-the-box.
As the CTO of Banzai Cloud, and someone involved in the details of our product development, I can confirm that this has been our single biggest engineering push of 2020 so far (spoiler alert – that record won’t last long: there are three or four parallel efforts already challenging it). Nevertheless, the view from 10,000 feet is unchanged:
The purpose of this post is not to go into details about how Kafka ACL on Kubernetes works, but to highlight some of its more under-discussed aspects – a quick glance under the hood if you like – and a reflection on building non-intrusive software.
See, us engineers, we like building complex things – the more complex the better – and, as a result, we often don’t focus enough on the end user and on legacy software. Similarly, here in Banzai-land we love building complex things, but with the twist that whatever we build on Kubernetes must converge toward two points:
Be it Pipeline, our container management platform for building hybrid clouds, PKE, our CNCF certified Kubernetes distribution, Backyards (now Cisco Service Mesh Manager), the Istio distribution that just works or the subject of the post, Supertubes – the primary objective is to make using our software simple.
Making a feature simple to use is often difficult, and potentially unrewarding. Many competing container management platforms (and there are plenty out there) check off a feature by doing a simple
helm install behind the scenes. However, the rub lies in the flexibility, integration, edge cases and our obsession with going the extra mile – solutions that are usable out-of-the-box, production-ready defaults, but with replaceable batteries (not just included).
But enough rambling, let’s get back to business.
We wanted to accomplish two things without having to modify Kafka client libraries or code:
<namespace>-<serviceaccount>Kafka user to automatically apply for client applications
If you’re familiar with Supertubes, you know that we run Kafka inside an Istio servicemesh (with all the benefits that implies). This has been part of an incremental process on the road toward “perfection“. Initially, we started using Envoy alongside each broker to control access, and benefit from Envoy’s protocol filters for Kafka. As we’ve been deploying larger and larger Kafka clusters, we needed a way to control the Envoy data plane, and this is what Istio does as a control plane. We are big Istio users and fans, and have an Istio distribution called Backyards (now Cisco Service Mesh Manager), thus we added a few additional Istio benefits to Supertubes, for example, automatic mTLS, certificate rotation and management, WASM filters, and so on.
Simplifying things a bit, the authentication and authorization flow looks like this under the hood:
<namespace>-<serviceaccount>. Supertubes reads the
<namespace>-<serviceaccount>from the TCP stream and uses it as the Principal that represents the client application.
From the client application’s point of view this is the least intrusive solution, as it doesn’t require any changes.
Note: There are other solutions out there which provide a custom library that reads the service account of the client application and passes that info to Kafka through the SASL mechanism. In my opinion, this solution is relatively intrusive, since you have to modify the client application to interact with Kafka through this library (which may not be available in all programming languages).
So what did it take to achieve this level of simplicity for the end user?
This project turned out to be more complex than we originally envisioned, but it was worth the effort, as we managed to radically simplify the user experience and align legacy Kafka mechanisms with their Cloud Native equivalents. In the process, we managed to reuse a lot of know-how and components between our teams to uniquely benefit our customers. We believe that simplicity is a competitive advantage, but of course only if it is as simple as it can be, and no simpler…
And yes, we know we’re not the first to have made this point. 😉