INSIGHTS
11 min read
Published on 04/06/2022
Last updated on 03/21/2024
What is Kurun?
Kurun is a multi-tool to help Kubernetes developers. We can summarize the features in 3 short sentences:The second statement is especially handy during Kubernetes admission webhook developement. It's quite hard to configure an operator running on your local computer and receive admission webhook requests from within a remote Kubernetes cluster.
- Just like go run main.go but executed inside Kubernetes with one command.
- Just like kubectl port-forward ... but the other way around!
- Just like kubectl apply -f pod.yaml but images are built from local source code.
Recap of admission webhooks
Ideally, the operator runs inside the cluster, so the admission controllers send the admission requests to the operator directly. However, during development, it is more practical to keep your operator on your local machine (for debugging and other reasons), but unfortunately, admission controllers cannot access your operator this way. Applications running inside Kubernetes usually cannot open connections to your local workstation, partly for security reasons, but mostly because of IPv4 NAT.Solution
The basic idea is to initiate a full-duplex, keepalive connection from the local machine (client side) into the Kubernetes cluster (server side) and use this connection to reverse-proxy the admission requests from the server to the client and deliver the admission responses.First version
The first version of kurun was created in a strong need and followed the idea of "make it available as soon as possible". Initially Kurun was a bash script as well. Therefore we used third parties to solve problems, like:- Tunneling: A practical third party named
inlets
was responsible for the tunneling. It was able to maintain the communication between the server and client side over a WebSocket (WS) connection. - TLS termination: Since the
inlets
server did not provided an option for serving TLS connections, we embedded another third party named Ghostunnel. It was responsible for terminating TLS before theinlets
server.
kubectl
exec calls to manage Kubernetes resources which was the quickest way of implementing Kubernetes communications.
Why fix it when it works?
The above concept worked well for a long time. Unfortunately the inlets stack became unavailable."I pivoted inlets this year from open source, with a separate commercial version, to a commercial version with open-source automation and tooling. Why? The Open Source model wasn't serving the needs of my business..." - Alex EllisYou can read more details in this blog post. But as developers who are enthusiastic for open-source we took Alex advice:
"So I flipped this around. If you only used inlets because it was free, find an alternative (there are a bunch of them, with different tradeoffs) or pay for it." - Alex EllisWe wrote our version of tuneling between Kubernetes and a local computer.
If you do something do it right!
The new version ofkurun
(0.6.0) upgrades several component:
- Kubernetes resource management: Uses the standard k8s.io Go modules to access the cluster and manage resources. Creates a deployment for the
kurun
server pod and also sets up a service for the pod. - Tunneling:
kurun
client uses the official k8s.io Go modules to create an HTTP client that can access the cluster's API server. It relies on the API server proxy to open and maintain a secure WebSocket connection to thekurun
server. This connection will be used as a tunnel between the local machine and the Kubernetes cluster. - TLS termination: Now kurun has built-in TLS termination, there's no need to use any external tools.
Task | First version | New version |
---|---|---|
Secure proxying | - | Secure WebSocket connection proxied by Kubernetes API server |
Tunneling | Inlets (3rd party) | Native request/response proxy through WSS |
TLS termination | Ghostunnel (3rd party) | Native go/http package |
Accessing cluster resources | kubectl | Standard go k8s.io libraries |
What is under the hood?
Warning, the following paragraph is a deep technical description of kurun's low-level implementation. If you are not interested in this kind of information skip to the Webhook example. Proxying requests through a WebSocket tunnel might seem easy, but it's trickier than you think. Originally, WebSockets are handling bidirectional traffic without state. Hence, when you want to proxy HTTP calls over WebSockets you have two options. Create a new WebSocket connection for every single request, which is trivial but not efficient. The second is to track the parallel connection's packets. And this is what you see in the figure. Let's see this step-by-step:- The server receives the HTTP request from the Kubernetes API server.
- It creates a goroutine to handle the request.
- This routine encapsulates the packet with an
id
and sends it through the WebSocket. - The goroutine then waits for the response with the matching
id
.
- The client receives the encapsulated HTTP request along with its
id
. - It sends the request to the (webhook) handler and waits for the response.
- The response is encapsulated with the same
id
and sent back on the WebSocket.
id
and forwards the response to the Kubernetes API. With this solution, it is possible to do a non-blocking serialized tunnel between the Kubernetes API and your machine.
Webhook example
You can use the following example configuration with a sample application to try outkurun
port-forward feature in a sandbox environment.
Component | Description |
---|---|
cert-manager | A great Kubernetes tool for managing TLS certs and secrets. |
kube-service-annotate | A standalone tool which is responsible for the webhook logic. It annotates newly created services in your Kubernetes cluster. This will run on your local machine in this case. |
kurun | Will set up a tunnel for reverse proxying the admission requests securely between your local machine and your Kubernetes cluster. |
- Configuration available on your local machine for accessing a running Kubernetes cluster. It can also be kind/k3d/minikube
git
to clone repositoriesgolang
to run kurun and kube-service-annotatejq
to read client TLS data from Kubernetes secrets
1. Install cert-manager
Probably the most convenient way to create/manage TLS certs in Kubernetes is usingcert-manager
. Our little example will also use it for obvious reasons. Install cert-manager
into your Kubernetes cluster with the following command:
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.7.1/cert-manager.yaml
2. Generate certificates
Since Kubernetes' admissionregistration controllers use secure connections exclusively for webhooks, you will need some certificates to make things to operate. Please note we also addedlocalhost
to dnsNames
of the certificate. This is required to be able to use the TLS cert on your local machine too.
Further information about Kubernetes webhooks: Dynamic Admission ControlRun the following command to create the following items:
- CA Issuer
- CA Cert
- CA Secret
- Server Cert Issuer
- Server Cert
- Server TLS Secret
kubectl apply -f -<<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: kurun-ca-issuer
namespace: default
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: kurun-ca-cert
namespace: default
spec:
isCA: true
commonName: kurun-ca
issuerRef:
kind: Issuer
name: kurun-ca-issuer
secretName: kurun-ca-secret
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: kurun-issuer
namespace: default
spec:
ca:
secretName: kurun-ca-secret
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: kurun-cert
namespace: default
spec:
dnsNames:
- localhost
- kurun.default
- kurun.default.svc
issuerRef:
kind: Issuer
name: kurun-issuer
secretName: kurun-secret
EOF
Your output should be something like below:
issuer.cert-manager.io/kurun-ca-issuer created
certificate.cert-manager.io/kurun-ca-cert created
issuer.cert-manager.io/kurun-issuer created
certificate.cert-manager.io/kurun-cert created
3. Setup kube-service-annotate
kube-service-annotate
is a simple sample application which handles admission requests and returns admission responses over HTTP and HTTPS. It basicly annotates Kubernetes services based on a ruleset. This application will run on your local machine this time, playing the role of an operator under development. You can clone the repository with the following command:
git clone https://github.com/banzaicloud/kube-service-annotate.git
kube-service-annotate
will look for a config file named rules.yaml
by default which contains the annotation rule set. Create a new file rules.yaml
with the following example content:
- selector:
app: my-service
annotations:
my-label-dependant-annotation: true
- annotations:
always-annotate-this: true
4. Save TLS secrets to local machine
Now we have our certs in Kubernetes, but we will also need them on our local machine to runkube-service-annotate
in secure mode. With the following bash commands you can extract and save the certificate and private key from the server secret to your local machine.
kubectl get secret kurun-secret -o json | jq -r '.data["tls.crt"]' | base64 -d > tls.crt
kubectl get secret kurun-secret -o json | jq -r '.data["tls.key"]' | base64 -d > tls.key
5. Run kube-service-annotate
The following line will start thekube-service-annotate
on your local machine:
go run main.go --tls-key-file tls.key --tls-cert-file tls.crt --rules-file rules.yaml
The output should look something like this:
2022/03/01 11:27:56 [INFO] Reading rules from: "rules.yaml"
2022/03/01 11:27:56 [INFO] There are 2 active rules
2022/03/01 11:27:56 [WARN] no metrics recorder active
2022/03/01 11:27:56 [WARN] no tracer active
2022/03/01 11:27:56 [INFO] Listening TLS on :8080
6. Install kurun
https://github.com/banzaicloud/kurunInstall
kurun
using one of the following methods:
- with go
go install github.com/banzaicloud/kurun
- with brew
brew install kurun
- or clone the git repository
git clone https://github.com/banzaicloud/kurun.git
7. Run kurun port-forward
kurun
runs locally, but it has access to your Kubernetes cluster with the configuration stored in your KUBECONFIG
. kurun
performs the following actions on the cluster:
- Creates a deployment for kurun-server. The kurun-server listens (over HTTPS) inside the cluster and proxies requests to your local machine.
- A service will be generated which points to the kurun-server pod.
kurun
with the example parameter set:
kurun port-forward https://localhost:8080 --tlssecret kurun-secret -v
Explanation of parameters:
- https://localhost:8080 : the target where kurun client will proxy the admission requests to (this is where
kube-service-annotate
is listening) - --tlssecret kurun-secret : reads the client TLS secret from the Kubernetes secret and uses it on the client side
You can exit from kurun client any time by hitting Ctrl+C. This automatically starts a cleanup process which removes all client-generated resources from your Kubernetes cluster, but see also the next step when using mutating webhooks.
7. Setup mutatingwebhookconfiguration
This step is essential when creating mutating webhooks. You need to have a mutating webhook configuration in your Kubernetes cluster, so admission controllers know where/how to send admission requests. Note that the service configuration now points to the kurun service.kubectl apply -f -<<EOF
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
name: kurun-webhook
annotations:
cert-manager.io/inject-ca-from: "default/kurun-ca-cert"
webhooks:
- name: kurun-webhook.banzaicloud.com
clientConfig:
service:
name: kurun
namespace: default
path: "/mutate-secrets"
port: 80
rules:
- operations: [ "CREATE" ]
apiGroups: [""]
apiVersions: ["v1"]
resources: ["services"]
scope: "*"
admissionReviewVersions: ["v1beta1", "v1"]
sideEffects: None
EOF
Don't forget to remove/edit this MutatingWebhookConfiguration after exitingkurun
! Otherwise, it would point to a non-existing service becausekurun
removes its own resources on exit, and requests to the service would fail, resulting in your inability to create services (or resources set in the MutatingWebhookConfiguration).
8. Create service
Now everything is set up! By running the following script you can create a new dummy service that actually does nothing. On the other hand, the following things do happen (simplified):- The Kubernetes API server receives the request, passes it to the admission controller.
- The admission controller checks mutationwebhookconfigurations and sends the admission request to the (kurun) service we set up.
kurun
service receives the admission request formy-service
and proxies it to the pod runningkurun-server
.kurun
server proxies the admission request through its WSS tunnel tokurun
client on the local machine.kurun
client forwards the admission request tokube-service-annotate
.kube-service-annotate
checks the admission request ofmy-service
, and applies annotations to the service descriptor from matching rules inrules.yaml
.kurun
client receives the admission response fromkube-service-annotate
, proxies it through the WSS tunnel to thekurun
server.kurun
server receives the admission response and returns it to the admission controller.- The admission controller passes on the extended
my-service
descriptor.kubectl apply -f -<<EOF apiVersion: v1 kind: Service metadata: name: my-service namespace: default labels: app: my-service spec: clusterIP: None clusterIPs: - None type: ClusterIP EOF
kurun
client logs the request:
2022/03/01 16:17:29 "level"=1 "msg"="handling request" "wsConn"={} "client"={} "id"=824634754304 "request"={"Method":"POST","URL":"/mutate-secrets?timeout=10s","Proto":"HTTP/1.1","ProtoMajor":1,"ProtoMinor":1,"Header":{"User-Agent":["kube-apiserver-admission"],"Content-Length":["1631"],"Accept":["application/json, */*"],"Accept-Encoding":["gzip"],"Content-Type":["application/json"]},"Body":{},"GetBody":"<unhandled-func>","ContentLength":1631,"TransferEncoding":[],"Close":false,"Host":"kurun.default.svc:80","Form":{},"PostForm":{},"MultipartForm":null,"Trailer":{},"RemoteAddr":"","RequestURI":"/mutate-secrets?timeout=10s","TLS":null,"Cancel":"<unhandled-chan>","Response":null}
kurun
server also logs the events:
2022/03/01 16:17:29 "level"=1 "msg"="request received" "server"={} "request"={"Method":"POST","URL":"/mutate-secrets?timeout=10s","Proto":"HTTP/1.1","ProtoMajor":1,"ProtoMinor":1,"Header":{"Content-Type":["application/json"],"Accept-Encoding":["gzip"],"User-Agent":["kube-apiserver-admission"],"Content-Length":["1631"],"Accept":["application/json, */*"]},"Body":{},"GetBody":"<unhandled-func>","ContentLength":1631,"TransferEncoding":[],"Close":false,"Host":"kurun.default.svc:80","Form":{},"PostForm":{},"MultipartForm":null,"Trailer":{},"RemoteAddr":"10.244.0.1:52069","RequestURI":"/mutate-secrets?timeout=10s","TLS":{"Version":772,"HandshakeComplete":true,"DidResume":false,"CipherSuite":4865,"NegotiatedProtocol":"http/1.1","NegotiatedProtocolIsMutual":true,"ServerName":"kurun.default.svc","PeerCertificates":[],"VerifiedChains":[],"SignedCertificateTimestamps":[],"OCSPResponse":[],"TLSUnique":[]},"Cancel":"<unhandled-chan>","Response":null}
2022/03/01 16:17:29 "level"=1 "msg"="request queued" "server"={} "request"={"Method":"POST","URL":"/mutate-secrets?timeout=10s","Proto":"HTTP/1.1","ProtoMajor":1,"ProtoMinor":1,"Header":{"User-Agent":["kube-apiserver-admission"],"Content-Length":["1631"],"Accept":["application/json, */*"],"Content-Type":["application/json"],"Accept-Encoding":["gzip"]},"Body":{},"GetBody":"<unhandled-func>","ContentLength":1631,"TransferEncoding":[],"Close":false,"Host":"kurun.default.svc:80","Form":{},"PostForm":{},"MultipartForm":null,"Trailer":{},"RemoteAddr":"10.244.0.1:52069","RequestURI":"/mutate-secrets?timeout=10s","TLS":{"Version":772,"HandshakeComplete":true,"DidResume":false,"CipherSuite":4865,"NegotiatedProtocol":"http/1.1","NegotiatedProtocolIsMutual":true,"ServerName":"kurun.default.svc","PeerCertificates":[],"VerifiedChains":[],"SignedCertificateTimestamps":[],"OCSPResponse":[],"TLSUnique":[]},"Cancel":"<unhandled-chan>","Response":null} "id"=824634754304
2022/03/01 16:17:29 writeLoop: "level"=1 "msg"="processing request" "server"={} "conn"={} "request"={"Method":"POST","URL":"/mutate-secrets?timeout=10s","Proto":"HTTP/1.1","ProtoMajor":1,"ProtoMinor":1,"Header":{"User-Agent":["kube-apiserver-admission"],"Content-Length":["1631"],"Accept":["application/json, */*"],"Content-Type":["application/json"],"Accept-Encoding":["gzip"]},"Body":{},"GetBody":"<unhandled-func>","ContentLength":1631,"TransferEncoding":[],"Close":false,"Host":"kurun.default.svc:80","Form":{},"PostForm":{},"MultipartForm":null,"Trailer":{},"RemoteAddr":"10.244.0.1:52069","RequestURI":"/mutate-secrets?timeout=10s","TLS":{"Version":772,"HandshakeComplete":true,"DidResume":false,"CipherSuite":4865,"NegotiatedProtocol":"http/1.1","NegotiatedProtocolIsMutual":true,"ServerName":"kurun.default.svc","PeerCertificates":[],"VerifiedChains":[],"SignedCertificateTimestamps":[],"OCSPResponse":[],"TLSUnique":[]},"Cancel":"<unhandled-chan>","Response":null}
9. Validating the results
Checking the newly created service to see the annotations added bykube-service-annotate
.
kubectl describe svc my-service
The output should be similar to:
Name: my-service
Namespace: default
Labels: app=my-service
Annotations: always-annotate-this: true
my-label-dependant-annotation: true
Selector: <none>
Type: ClusterIP
IP Family Policy: RequireDualStack
IP Families: IPv4,IPv6
IP: None
IPs: None
Session Affinity: None
Events: <none>
The complexity of example might be frightening for beginners, but starting to develop webhook based applications will make familiar all the things above. We think kurun
is a useful developer tool which we actively use it in everyday work. We hope it will also make your work easier as a Kubernetes developer.Subscribe to
the Shift!
Get emerging insights on emerging technology straight to your inbox.
Unlocking Multi-Cloud Security: Panoptica's Graph-Based Approach
Discover why security teams rely on Panoptica's graph-based technology to navigate and prioritize risks across multi-cloud landscapes, enhancing accuracy and resilience in safeguarding diverse ecosystems.
Subscribe
to
the Shift
!Get on emerging technology straight to your inbox.
emerging insights
The Shift keeps you at the forefront of cloud native modern applications, application security, generative AI, quantum computing, and other groundbreaking innovations that are shaping the future of technology.