SPIFFE/SPIRE Federation on Kind clusters
Friday, September 24th, 2021
Spiffe.io is a universal identity control plane for distributed systems. SPIFFE is a set of standards to help us achieve zero trust identity framework and SPIRE is a reference implementation of SPIFFE. SPIRE can authenticate and authorize workloads in a distributed system.
Essentially SPIRE maps workloads to SPIFFE identities and distributes credentials for secure micro-services communication between Kubernetes workloads. SPIRE servers can be deployed in different architectures. SPIRE Federation (Federating SPIRE Servers with Different Trust Domains) and Nested SPIRE (Chaining SPIRE Servers to Use the Same Trust Domain). SPIFFE/SPIRE Architectures
In this blog, we will demonstrate how to authenticate two SPIFFE-identified workloads that are identified by two different SPIRE Servers.
The first part of this tutorial demonstrates how to set up
SPIRE federation by showing the SPIRE configuration changes
spire-server commands used to set up a stock quote
webapp frontend and service backend. The second part of this
document lists the steps to show the scenario in action
using Helm charts and
In this demonstration you will learn to:
- Configure SPIRE Server to expose its SPIFFE Federation bundle endpoint using SPIFFE authentication.
- Configure SPIRE Servers to exchange trust bundles from each other.
- Create one-time initial Bootstrap federation bundles between two SPIRE Servers using different trust domains.
- Create registration entries for the workloads so that they can federate with other trust domain.
To demonstrate SPIRE Federation, we will set up two Kubernetes clusters using Kind. Then deploy SPIRE server-agent workloads using SPIRE Helm chart provided in the example for distributing SPIFFE ID and trust credentials. Next, we will deploy MetalLB using the MetalLB Helm cart provided in the example to auto-assign external load balancer IP addresses to our clusters. Lastly, we will configure SPIRE servers to federate with its neighboring cluster using the cluster's external IP.
The stock quote webapp frontend and service backend we deploy in our Kubernetes cluster leverages the SPIFFE workload API to get mTLS credentials. In the first cluster server app servers client endpoint with stocks values to the client, server fetches x509 certificate from SPIRE server and sets up a listening port for the client. Client app in the second cluster with SPIRE workloads fetches TLS bundles from all the trusted domains and selects the right certificate to connect to the server.
A port forwarded to your localhost on port 8080 can be used to access the client-server stockbroker webpage and initiate a mTLS request from client to server, which can be verified from the client-server pods logs.
SPIRE Federation using SPIFFE Authentication
To demonstrate SPIRE Federation, we'll use a simple example from the official SPIRE user guide for the Federation model. Source available here spire-tutorials/docker-compose/federation
I have modified the server-client example that can be deployed in the Kubernetes cluster with SPIRE server-agent model. spire-federation-kind This example demonstrates a stock quote web app frontend and service backend by setting up two SPIFFE-identified workloads that are identified by two different SPIRE Servers.
Let's say we have a stockbroker's web app that displays stock quotes periodically from a stock market web service provider. We can see the scenario as below:
- The user enters the broker's web app stock quotes URL in a browser.
- The web app workload receives the request and makes an HTTP request for quotes to the stock market service using mTLS.
- The stock market service receives the request and sends the quotes in the response.
- The web app renders the stock quotes page using the returned quotes and sends it to the browser.
The server holds a set of secure information (stock quotes) and the client exposes HTTP endpoint to display quotes via a web browser. The communication between server-client is via gRPC protocol endpoints, and security for this connection is mTLS using SPIFFE identity framework.
Each workload has been implemented to use the SPIFFE API to authenticate with SPIRE server and obtain trust bundle credentials. The following describes the details of the server-client application authenticated by SPIFFE and secured by mTLS connection.
Setting It All Up
First, let's clone spire-federation-kind repository
git clone https://github.com/nishantapatil3/spire-federation-kind
Compile and build modified stock quotes server and broker webapp client Docker images
# Replace docker push with your public docker repository ./1-build.sh
Create two clusters using Kind, by following the instructions given in below URL
Let's name the clusters creates as
# Create Kind clusters kind create cluster --name kind-1 kind create cluster --name kind-2 # Export kubeconfig to desired location mkdir -p ~/kubeconfigs kind get kubeconfig --name=kind-1 > ~/kubeconfigs/kind-1.kubeconfig kind get kubeconfig --name=kind-2 > ~/kubeconfigs/kind-2.kubeconfig
Once we have two clusters running, deploy MetalLB load-balancer and SPIRE components(server-agent). MetalLB load balancer is required to allow ingress traffic to the Kubernetes cluster and redirect traffic to target pods in our cluster.
globalPrefix refers to
xx.xx.<globalPrefix>.xx IP address
that MetallLB can be configured to allow subnet range for
the external IP address in Kubernetes. Example: If
globalPrefix=255, external IP address configuration range is
172.17.255.1 to 172.17.255.255
# Apply the Helm chart helm template helm/metallb-system --set globalPrefix="255" | kubectl apply --kubeconfig $cluster1 -f - helm template helm/metallb-system --set globalPrefix="254" | kubectl apply --kubeconfig $cluster2 -f -
After we deploy load balancers in our clusters, let's deploy spire-server and spire-agent
To do that we will apply below Helm charts with SPIRE values
as shown below. We can set trustDomain with any alphanumeric
combination and provide this trust name to federatesWith
argument that the cluster federates with.
federatesWith.address should match the external IP of
SPIRE service that it federates with, you can check this IP
kubectl get svc -n spire. Also,
refers to the federation-endpoint of spire-server service
helm template helm/spire --set trustDomain=cluster1.com --set federatesWith.trustDomain=cluster2.com --set federatesWith.address=172.17.254.1 --set federatesWith.port=8443 | kubectl apply --kubeconfig $cluster1 -f - helm template helm/spire --set trustDomain=cluster2.com --set federatesWith.trustDomain=cluster1.com --set federatesWith.address=172.17.255.1 --set federatesWith.port=8443 | kubectl apply --kubeconfig $cluster2 -f -
Next, bootstrap SPIRE certificates with each other such that the workloads (server and client) can receive the certificate from either trust domains to establish the connection
2-bootstrap.sh uses spire-server's tool
bin/spire-server bundle show to get trust bundle from one
trust domain and using that it sets the trust bundle in the
other clusters spire-server using
bin/spire-server bundle set
$ ./2-bootstrap.sh Setting clusters kubeconfig /home/ubuntu/spire-federation-kind/lab_clusters.sh Bootstrap certificate from cluster1 to cluster2 bundle set. Bootstrap certificate from cluster2 to cluster1 bundle set.
Run the following command to create SPIRE entries such that spire-server can fetch the right certificate from a list of registered trust domains to connect to its target.
3-register.sh uses spire-server's container
tool to register workloads stockbroker server and client in
their respective clusters with SPIRE. We set the
FederatesWith field to authenticate itself with other
cluster's spire-server in order to connect to workloads
deployed in that cluster.
$ ./3-register.sh Setting clusters kubeconfig /home/ubuntu/spire-federation-kind/lab_clusters.sh ------------------------- Registering workload: server Entry ID : 12c07f5f-8629-4654-9961-dd6dbeba577e SPIFFE ID : spiffe://cluster1.com/server Parent ID : spiffe://cluster1.com/spire-agent Revision : 0 TTL : default Selector : k8s:sa:server-service-account FederatesWith : spiffe://cluster2.com ------------------------- ------------------------- Registering workload: client Entry ID : 8082ce59-4bca-43a5-83d5-006890f822b5 SPIFFE ID : spiffe://cluster2.com/client Parent ID : spiffe://cluster2.com/spire-agent Revision : 0 TTL : default Selector : k8s:sa:client-service-account FederatesWith : spiffe://cluster1.com -------------------------
Deploy stockbroker server in cluster1 and web app client in cluster2
$ kubectl apply -f helm/server.yaml --kubeconfig $cluster1 serviceaccount/server-service-account created service/stock-quotes-service created deployment.apps/stock-quotes-service created $ kubectl apply -f helm/client.yaml --kubeconfig $cluster2 serviceaccount/client-service-account created service/broker-webapp created deployment.apps/broker-webapp created
Stockbroker server utilizes SPIRE API to create a trust bundle and initiates a listening port with x509src security. SPIRE servers exchange this trust bundle and distribute it to authenticated workloads that ask for bundles to connect to its target. The client gets the trust bundle for a given domain (Trust domains that the client is registered to federate with) using bundleSrc.
Client gets two certificates from two trust domains and iterates over them to find the right certificate to connect to the stockbroker server workload and establishes mTLS secure connection to it. Client calls SPIRE API for all the trust domain that spire-server is configured to federate with and gets the trust bundle in an iterable-list Each bundle is picked up to connect to the target and if the connection succeeds, that bundle is channeled to caller function via a goroutine.
After a secure channel is established between server-client, the server provides stocks information to the client which can be processed to display information to the user.
Now we can Port forward the client pod to
using kubectl. Replace the below command with client-pod's
name and run the command
Example: kubectl port-forward broker-webapp-85574f4585-cxvxg 8080:8080 - kubeconfig $cluster2 $ kubectl port-forward broker-webapp-85574f4585-q92x4 8080:8080 --kubeconfig $cluster2 Forwarding from 127.0.0.1:8080 -> 8080 Forwarding from [::1]:8080 -> 8080 Handling connection for 8080 Handling connection for 8080 Handling connection for 8080 Handling connection for 8080 Handling connection for 8080
Open any browser and navigate to
http://localhost:8080/quotes, you should see a grid of
randomly generated phony stock quotes that are updated every
Server (left k9s) and client (right k9s), you can see that client fetched two certificated from cluster1.com and cluster2.com, iterated over these certificates, and picked the right certificate to connect to stockbroker server. Source
By demonstrating the above steps we have successfully created SPIRE Federated clusters and secured server-client communication.
What did we cover?
- Deploy SPIRE in Kubernetes clusters and configure SPIFFE Federation bundle endpoints.
- Set up SPIRE Federation configuration and exchange trust bundles between two clusters.
- Create Bootstrap Federation bundles between two Trust Domains.
- Demonstrate an example to show SPIRE Federation in action using SPIFFE's identity API via the golang spiffe/v2 library.
Wrap-up: What SPIRE can do?
- Secure microservices communication.
- Validates cryptographic service identities.
- Eliminates the need for secret management.
- Automatically issues, distributes and renews short-lived credentials.
- Reduces operational overhead associated with credential management.