Outshift | OIDC issuer discovery for Kubernetes service accounts

Applications running in Kubernetes Pods are authenticated against the Kubernetes API with their corresponding ServiceAccount tokens. These JWT tokens are usually mounted into containers as files. JWT tokens are signed by the Kubernetes cluster's private key, and can be validated only with the TokenReview API. This API is not widely recognized and, to access it, external systems must first authenticate against Kubernetes to review ServiceAccounts. This configuration and access review process is considerably more complex than necessary, not to mention that it leaves out widely accepted standards like OIDC. Kubernetes already has an OIDC integration, namely inbound authentication of users against the Kubernetes API. The new integration, which is what this blog post is about, wires OIDC in the opposite direction; the Service Account Issuer Discovery feature enables the federation of Kubernetes service account tokens issued by a cluster (the identity provider) with external systems (relying parties) based on the OIDC Discovery Spec. Projected Service Account Tokens are required for this feature to be enabled. Projected service account JWTs differ from "traditional" tokens in that they expire, have a proper issuer, and their audience fields come filled out so that they act like proper JWTs. Today's post is going to be rather technical, since we'll be discussing authenticating Kubernetes applications with external systems through OIDC issuer discovery. We'll use Vault on Kubernetes as the OIDC consumer and a simple client application running in the cluster to access the Vault instance with a projected ServiceAccount token.

Preparation

We need to create a Kubernetes cluster where the ServiceAccountIssuerDiscovery feature gate is enabled. We are going to use kind to prepare our test cluster with some extra kubeadm patches to enable Service Account Token Volume Projection: Some software you will be required to installed on your machine during this tutorial:

kubectl
kind
curl
jq
step
vault

kind create cluster --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  ServiceAccountIssuerDiscovery: true
networking:
  apiServerPort: 6443
kubeadmConfigPatches:
- |
  apiVersion: kubeadm.k8s.io/v1beta2
  kind: ClusterConfiguration
  apiServer:
    extraArgs:
      service-account-issuer: https://localhost:6443
      service-account-jwks-uri: https://localhost:6443/openid/v1/jwks
      service-account-signing-key-file: /etc/kubernetes/pki/sa.key
      service-account-key-file: /etc/kubernetes/pki/sa.pub
EOF

The smallstep CLI is a great tool to analyze JWT tokens (and it does a lot of other things as well). Alternatively, you can use https://jwt.io/ to do the same thing in your browser. Create a sample application that will mount a projected ServiceAccountToken:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  serviceAccountName: default
  containers:
    - image: nginx:alpine
      name: oidc
      volumeMounts:
        - mountPath: /var/run/secrets/tokens
          name: oidc-token
  volumes:
    - name: oidc-token
      projected:
        sources:
          - serviceAccountToken:
              path: oidc-token
              expirationSeconds: 7200
              audience: vault
EOF

The projected SA JWT token has been mounted to the requested location. Let's analyze it with the step CLI:

kubectl exec nginx -- cat /var/run/secrets/tokens/oidc-token | step crypto jwt inspect --insecure

{
	"header": {
		"alg": "RS256",
		"kid": "Rt3TBA31bh3rH67PQbKImg2ldwhPqBTWF2w1Hxqi84c"
	},
	"payload": {
		"aud": ["vault"],
		"exp": 1592924135,
		"iat": 1592916935,
		"iss": "https://localhost:6443",
		"kubernetes.io": {
			"namespace": "default",
			"pod": {
				"name": "nginx",
				"uid": "aa977398-8a06-4106-8563-972f9ecadd55"
			},
			"serviceaccount": {
				"name": "default",
				"uid": "b2680b48-75df-476f-9d95-2a0441c2bb83"
			}
		},
		"nbf": 1592916935,
		"sub": "system:serviceaccount:default:default"
	},
	"signature": "..."
}

Compare this with the original (non-projected) ServiceAccount JWT:

kubectl exec nginx -- cat /var/run/secrets/tokens/oidc-token | step crypto jwt inspect --insecure

{
	"header": {
		"alg": "RS256",
		"kid": "Rt3TBA31bh3rH67PQbKImg2ldwhPqBTWF2w1Hxqi84c"
	},
	"payload": {
		"aud": ["vault"],
		"exp": 1592924135,
		"iat": 1592916935,
		"iss": "https://localhost:6443",
		"kubernetes.io": {
			"namespace": "default",
			"pod": {
				"name": "nginx",
				"uid": "aa977398-8a06-4106-8563-972f9ecadd55"
			},
			"serviceaccount": {
				"name": "default",
				"uid": "b2680b48-75df-476f-9d95-2a0441c2bb83"
			}
		},
		"nbf": 1592916935,
		"sub": "system:serviceaccount:default:default"
	},
	"signature": "..."
}

To be able to fetch the public keys and validate the JWT tokens against the Kubernetes cluster's issuer we have to allow external unauthenticated requests. To do this, we bind this special role (system:service-account-issuer-discovery) with a ClusterRoleBinding to unauthenticated users (make sure that this is safe in your environment, but only public keys are visible on this URL):

kubectl create clusterrolebinding oidc-reviewer --clusterrole=system:service-account-issuer-discovery --group=system:unauthenticated

Get the CA signing certificate of the Kubernetes API Server's certificate to validate it:

kubectl exec nginx -- cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt > kubernetes_ca.crt

Now you can visit well-known OIDC URLs:

curl --cacert kubernetes_ca.crt https://localhost:6443/.well-known/openid-configuration | jq

{
	"issuer": "https://localhost:6443",
	"jwks_uri": "https://localhost:6443/openid/v1/jwks",
	"response_types_supported": ["id_token"],
	"subject_types_supported": ["public"],
	"id_token_signing_alg_values_supported": ["RS256"]

Visit the JWKS address ("jwks_uri") to view public keys:

curl --cacert kubernetes_ca.crt https://localhost:6443/openid/v1/jwks | jq

{
	"keys": [
		{
			"use": "sig",
			"kty": "RSA",
			"kid": "Rt3TBA31bh3rH67PQbKImg2ldwhPqBTWF2w1Hxqi84c",
			"alg": "RS256",
			"n": "vL0tjBqLDFTyqOCPBQC5Mww_3xkhlkWmeklPjSAhFuqL0U-Oie9E1z8FuhcApBaUs7UEPzja02PEZd4i1UF2UDoxKYEG9hG5vPseTXwN_xGnbhOaBdfgQ7KDvqV-WHfmlrnnCizi1VmNAHsoAg6oZMiUdOuk8kCFxpe0N6THmBKNSKnqoRnhSL4uwHSBWJ5pEyWAqyL8KYaaGYhc2MVUs3I8e-gtQE6Vlwe75_QSp9uIZNZeFr5keqiXhz8BWL76ok-vY8UZ8-rH2VIN5LzXkCvhIFI9W_UBzziSnb9l5dgSQCwGf18zVgT0yJjCz0Z9YE9A1Wgeu-LLrJz3gxR8Hw",
			"e": "AQAB"
		}
	]
}

Configuring Vault as an OIDC consumer

We will use the Vault's JWT/OIDC Auth Method to consume the projected Service Account tokens from Kubernetes and validate them with the help of the OIDC Discovery endpoint exposed above.

vault server -dev

In another terminal we need to configure the JWT Auth backend to federate Kubernetes JWT tokens with the OIDC endpoint:

vault server -dev
vault auth enable jwt
vault write auth/jwt/config \
        oidc_discovery_url=https://localhost:6443 \
        oidc_discovery_ca_pem=@kubernetes_ca.crt \
        bound_issuer=https://localhost:6443

vault write auth/jwt/role/demo \
        role_type=jwt \
        bound_audiences=vault \
        bound_subject="system:serviceaccount:default:default" \
        user_claim=sub \
        policies=default

Grab the projected token and save it into a variable, then send the token to Vault's JWT authentication endpoint to exchange it for a Vault token:

JWT=$(kubectl <span class="hljs-built_in">exec</span> nginx -- <span class="hljs-built_in">cat</span> /var/run/secrets/tokens/oidc-token)

curl http://127.0.0.1:8200/v1/auth/jwt/login --data <span class="hljs-string">"{\"jwt\": \"<span class="hljs-variable">$JWT</span>\", \"role\": \"demo\"}"</span> | jq

{
	"request_id": "c635533b-cfad-ba2d-c421-77eb18b45cd6",
	"lease_id": "",
	"renewable": false,
	"lease_duration": 0,
	"data": null,
	"wrap_info": null,
	"warnings": null,
	"auth": {
		"client_token": "s.TLRJddMCIo6d3BM70TjmVhkc",
		"accessor": "koYXrht8K7rTlWZwgaBDGnBe",
		"policies": ["default"],
		"token_policies": ["default"],
		"metadata": {
			"role": "demo"
		},
		"lease_duration": 2764800,
		"renewable": true,
		"entity_id": "87b90fff-c019-5ebb-93e3-51677f538a53",
		"token_type": "service",
		"orphan": true
	}
}

Now we save this token to another variable and check to make sure it's working by having it look itself up on the Vault API:

VAULT_TOKEN=$(curl http://127.0.0.1:8200/v1/auth/jwt/login --data "{\"jwt\": \"$JWT\", \"role\": \"demo\"}" | jq -r .auth.client_token)

curl -H "X-Vault-Token: ${VAULT_TOKEN}" http://127.0.0.1:8200/v1/auth/token/lookup-self | jq

{
	"request_id": "5c8a033d-8f6f-a360-25ca-1ff32f5a69b8",
	"lease_id": "",
	"renewable": false,
	"lease_duration": 0,
	"data": {
		"accessor": "2Q6PSJ1L9FLYcqBxNZA5tuuu",
		"creation_time": 1592919300,
		"creation_ttl": 2764800,
		"display_name": "jwt-system:serviceaccount:default:default",
		"entity_id": "bb1159b8-7b1c-bf88-509a-130e6666818b",
		"expire_time": "2020-07-25T15:35:00.512007+02:00",
		"explicit_max_ttl": 0,
		"id": "s.nJm1aUQ6JsB39Yv3xankAXMe",
		"issue_time": "2020-06-23T15:35:00.512019+02:00",
		"meta": {
			"role": "demo"
		},
		"num_uses": 0,
		"orphan": true,
		"path": "auth/jwt/login",
		"policies": ["default"],
		"renewable": true,
		"ttl": 2764797,
		"type": "service"
	},
	"wrap_info": null,
	"warnings": null,
	"auth": null
}

Automating configuration and client access with Bank-Vaults

The JWT auth configuration of Vault and client access can be automated with the help of Bank-Vaults. This was introduced in a recent PR that added support for projected ServiceAccount tokens. The Bank-Vaults repository contains a fully-fledged Kubernetes OIDC federation example, where the OIDC endpoint is exposed internally, inside the cluster, on a special URL: https://kubernetes. To set up a kind cluster with a JWT authenticated Vault instance, and run a client example, we have to check the repository and apply some manifests. Note: This example requires kurun to be installed (brew install banzaicloud/tap/kurun), because the example container is built directly from Go code found in the repository at kubectl apply time. Other requirements:

go
helm (v3)

git clone git@github.com:banzaicloud/bank-vaults.git
cd bank-vaults

# Create the OIDC issuer enabled cluster for in-cluster use
kind create cluster --config hack/kind.yaml

# Install the Banzai vault-operator
helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
helm upgrade --install vault-operator banzaicloud-stable/vault-operator

# Create the Vault instance configured automatically for OIDC/JWT Auth
kubectl apply -f operator/deploy/rbac.yaml
kubectl apply -f operator/deploy/cr-oidc.yaml

# Run the example cluent which authenticates with a projected ServiceAccount JWT
kurun apply -f hack/oidc-pod.yaml

# Check the logs to make sure it works
kubectl logs -f oidc

This brief in-cluster example concisely demonstrates how OIDC issuer discovery can be enabled for Kubernetes Service Accounts consumed by cluster-external entities, like Vault (as in this case). If you're interested in contributing, check out the Bank-Vaults repository, or give us a GitHub star.

Learn more about Bank-Vaults:

Secret injection webhook improvements

Backing up Vault with Velero

Vault replication across multiple datacenters

Vault secret injection webhook and Istio

HSM support

Injecting dynamic configuration with templates

OIDC issuer discovery for Kubernetes service accounts

Show all posts related to Bank-Vaults