Kafka external access

Monday, December 7th, 2020
In this post, we'll describe how to provision your Apache Kafka cluster using either Supertubes or the koperator so that it is accessible by external client applications that run outside the Kubernetes cluster.
There are two methods that allow you to externally publish your Kafka cluster:
- using LoadBalancer type services
- using NodePort type services
The LoadBalancer
method is a very convenient way to
publish your Kafka cluster, as it allows you to forgo
setting up a Load Balancer, provisioning public IPs,
configuring routing rules, etc. since all these are taken
care for you. Also, this method has the advantage of
reducing your attack surface, since you don't have to make
the Kubernetes cluster's nodes directly reachable from
outside, because incoming external traffic is routed to the
nodes of the Kubernetes cluster through the Load Balancer.
The NodePort
method provides access to Kafka for external
clients through the external public IP of the nodes of the
Kubernetes cluster. This access method is a good fit when:
- there is no Load Balancer support in the used Kubernetes distribution for the hosting environment
- the extra hops introduced by the Load Balancer and Ingress controller are rendered unacceptable by business requirements
- the environment where the Kubernetes cluster is hosted is locked down, and thus the Kubernetes nodes are not reachable through their public IPs from outside.
External listeners
You can expose the Kafka cluster outside the Kubernetes
cluster by declaring one or more externalListeners in the
KafkaCluster
custom resource.
listenersConfig:
externalListeners:
- type: "plaintext"
name: "external1"
externalStartingPort: 19090
containerPort: 9094
- type: "plaintext"
name: "external2"
externalStartingPort: 19090
containerPort: 9095
Above, externalListeners creates two external access
points through which the Kafka cluster's brokers can be
reached. These external listeners are registered in the
advertized.listeners
Kafka broker configuration as
EXTERNAL1://...,EXTERNAL2://...
External listeners with LoadBalancer access method
The default access method for external listeners is LoadBalancer, thus, in the example below,
accessMethod: LoadBalancer
can be omitted from the external listener config:
listenersConfig:
externalListeners:
- type: "plaintext"
name: "external1"
externalStartingPort: 19090
containerPort: 9094
accessMethod: LoadBalancer
This will create a Load Balancer for the external listener,
external1
. Each broker in the cluster will receive a
dedicated port number on the Load Balancer which is computed
as broker port number = externalStartingPort + broker id.
This will be registered in each broker's config as
advertized.listeners=EXTERNAL1://<loadbalancer-public-ip>:<broker port number>
.
External listeners that use the LoadBalancer access method
also require an ingress controller to be configured, which
can be configured through the ingressController
field of
the KafkaCluster custom resource:
spec:
ingressController: "envoy"
The ingress controllers that are currently supported are:
- envoy - uses Envoy Proxy as an ingress controller.
- istioingress - uses Istio Gateway as an ingress controller. This is the default controller for Kafka clusters provisioned with Supertubes, since those clusters run inside an Istio mesh.
Additional configurations for these ingress controllers, such as the number of replicas, resource requirements and resource limits, can be configured through the envoyConfig and istioIngressConfig fields respectively.
External access through static URL
In order to expose the Kafka cluster through a URL, rather
than through a load balancer's public IP, specify the URL in
the hostnameOverride
field of the external listener that
resolves to the public IP of the load balancer:
listenersConfig:
externalListeners:
- type: "plaintext"
name: "external1"
externalStartingPort: 19090
containerPort: 9094
accessMethod: LoadBalancer
hostnameOverride: kafka-1.dev.my.domain
This will result in broker access address being advertized
as,
advertized.listeners=EXTERNAL1://kafka-1.dev.my.domain:<broker port number>
.
External listeners with NodePort access method
Using the NodePort access method, external listeners make Kafka brokers accessible through either the external IP of a Kubernetes cluster's node, or on an external IP that routes into the cluster.
listenersConfig:
externalListeners:
- type: "plaintext"
name: "external1"
externalStartingPort: 32000
containerPort: 9094
accessMethod: NodePort
A
NodePort
type service will be created separately for each broker.
Brokers can be reached from outside the Kubernetes cluster
at <any node public ip>:<broker port number>
where the
<broker port number>
is computed as
externalStartingPort + broker id.
The externalStartingPort must fall into the range allocated for nodeports on the Kubernetes cluster, which is specified via --service-node-port-range (see the Kubernetes documentation).
External access through dynamic URL
In order to make the broker accessible through a URL,
specify a suffix in the hostnameOverride
field of the
external listener:
listenersConfig:
externalListeners:
- type: "plaintext"
name: "external1"
externalStartingPort: 32000
containerPort: 9094
accessMethod: NodePort
hostnameOverride: .dev.my.domain
The hostnameOverride
behaves differently here than with
LoadBalancer access method. In this case, each broker will
be advertized as
advertized.listeners=EXTERNAL1://<kafka-cluster-name>-<broker-id>.<namespace><value-specified-in-hostnameOverride-field>:<broker port number>
.
If a three-broker Kafka cluster named kafka is running in
the kafka namespace, the advertized.listeners
for the
brokers will look like this:
- broker 0:
- advertized.listeners=EXTERNAL1://kafka-0.external1.kafka.dev.my.domain:32000
- broker 1:
- advertized.listeners=EXTERNAL1://kafka-1.external1.kafka.dev.my.domain:32001
- broker 2:
- advertized.listeners=EXTERNAL1://kafka-2.external1.kafka.dev.my.domain:32002
NodePort external IP
Kafka brokers can be made accessible on external IPs that are not node IP, but can route into the Kubernetes cluster. These external IPs can be set for each broker in the KafkaCluster custom resource as in the following example:
brokers:
- id: 0
brokerConfig:
nodePortExternalIP:
external1: 13.53.214.23 # if "hostnameOverride" is not set for "external1" external listener, then broker is advertised on this IP
- id: 1
brokerConfig:
nodePortExternalIP:
external1: 13.48.71.170 # if "hostnameOverride" is not set for "external1" external listener, then broker is advertised on this IP
- id: 2
brokerConfig:
nodePortExternalIP:
external1: 13.49.70.146 # if "hostnameOverride" is not set for "external1" external listener, then broker is advertised on this IP
If hostnameOverride field is not set, then broker address is advertized as follows:
- broker 0:
- advertized.listeners=EXTERNAL1://13.53.214.23:9094
- broker 1:
- advertized.listeners=EXTERNAL1://13.48.71.170:9094
- broker 2:
- advertized.listeners=EXTERNAL1://13.49.70.146:9094
If both hostnameOverride and nodePortExternalIP fields are set:
- broker 0:
- advertized.listeners=EXTERNAL1://kafka-0.external1.kafka.dev.my.domain:9094
- broker 1:
- advertized.listeners=EXTERNAL1://kafka-1.external1.kafka.dev.my.domain:9094
- broker 2:
- advertized.listeners=EXTERNAL1://kafka-2.external1.kafka.dev.my.domain:9094
Note that if nodePortExternalIP is set, then the containerPort from the external listener config is used as a broker port, and is the same for each broker.
Future NodePort enhancements
Currently, either hostnameOverride or
nodePortExternalIP, or both, have to be specified for
external listeners in the NodePort access method. In future
versions, if these are not set, the node IP of the node
where the broker pod is scheduled will be used in the
advertized.listeners
broker configuration.