One of the most popular feature of Bank-Vaults, the Vault swiss-army knife for Kubernetes is the secret injection webhook. With the growing popularity of Istio, recently the most requested feature was to support for running Bank-Vaults alongside Istio. We are big fans of Istio (a year ago we open sourced an Istio operator) and we have built an automated and operationalized service mesh, Banzai Cloud Backyards. As both components (Bank-Vaults and Backyards (now Cisco Service Mesh Manager)) are part of our hybrid cloud container management plaform, Pipeline, we went ahead and made them work together smoothly.
We support the following three scenarios:
Register for the free tier version of Cisco Service Mesh Manager (formerly called Banzai Cloud Backyards) and follow the Getting Started Guide for up-to-date instructions on the installation.
backyards install
? Install istio-operator (recommended). Press enter to accept Yes
? Install canary-operator (recommended). Press enter to accept No
? Install and run demo application (optional). Press enter to skip No
STRICT
:
❯ backyards mtls require mesh
INFO[0000] switched global mTLS to STRICT successfully
After this, we can check that mesh is configured with mTLS
turned on which applies to all applications in the cluster in Istio-enabled namespaces. You can change this if you would like to use another policy.
$ backyards mtls get mesh
mTLS rule for /mesh
Policy Targets MtlsMode
/default [] STRICT
Now your cluster is properly running on Istio with mTLS enabled globally.
vault-system
. You can enable Istio sidecar injection here as well, but Kubernetes won’t be able to call back the webhook properly since mTLS is enabled (and Kubernetes is outside of the Istio mesh). To overcome this, apply a PERMISSIVE
Istio authentication policy to the vault-secrets-webhook
Service itself, so Kubernetes can call it back without Istio mutual TLS authentication.
$ kubectl create namespace vault-system
namespace/vault-system created
$ backyards sidecar-proxy auto-inject on vault-system
INFO[0002] auto sidecar injection successfully set to namespace default
$ backyards mtls allow vault-system/vault-secrets-webhook
INFO[0001] policy peers for vault-system/vault-secrets-webhook set successfully
mTLS rule for vault-system/vault-secrets-webhook
Policy Targets MtlsMode
vault-system/vault-secrets-webhook-rw6mc [vault-secrets-webhook] PERMISSIVE
helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
helm upgrade --install vault-secrets-webhook banzaicloud-stable/vault-secrets-webhook --namespace vault-system
helm upgrade --install vault-operator banzaicloud-stable/vault-operator --namespace vault-system
Soon the webhook and the operator become up and running. Check that the istio-proxy
got injected into all Pods in vault-system
.
To recap Scenario 1: Vault runs outside an Istio mesh, whereas the namespace where the application runs and the webhook injects secrets has Istio sidecar injection enabled.
First, install Vault outside the mesh, then install an application within the mesh.
kubectl create namespace vault
kubectl apply -f rbac.yaml -f cr-istio.yaml
$ kubectl get pods -n vault
NAME READY STATUS RESTARTS AGE
vault-0 3/3 Running 0 22h
vault-configurer-6458cc4bf-6tpkz 1/1 Running 0 22h
If you are writing your own Vault CR make sure that istioEnabled: true
is configured, this influences port naming so the Vault service port protocols are detected by Istio correctly.
vault-secrets-webhook
can’t inject Vault secrets into initContainers
in an Istio-enabled namespace when the STRICT
authentication policy is applied to the Vault service, because Istio needs a sidecar container to do mTLS properly, and in the phase when initContainers
are running the Pod doesn’t have a sidecar yet. If you wish to inject into initContainers
as well, you need to apply a PERMISSIVE
authentication policy in the vault
namespace, since it has its own TLS certificate outside of Istio scope (so this is safe to do from networking security point of view).
$ backyards mtls allow vault
INFO[0001] policy peers for vault/ set successfully
mTLS rule for vault/
Policy Targets MtlsMode
vault/default [] PERMISSIVE
In this scenario Vault is running outside the Istio mesh (as we have installed it in the previous steps) and our demo application runs within the Istio mesh. To install the demo application inside the mesh, complete the following steps:
kubectl create namespace app
backyards sidecar-proxy auto-inject on app
kubectl apply -f app.yaml
app
itself and the istio-proxy
:
$ kubectl get pods -n app
NAME READY STATUS RESTARTS AGE
app-5df5686c4-sl6dz 2/2 Running 0 119s
$ kubectl logs -f -n app deployment/app app
time="2020-02-18T14:26:01Z" level=info msg="Received new Vault token"
time="2020-02-18T14:26:01Z" level=info msg="Initial Vault token arrived"
s3cr3t
going to sleep...
To run Vault inside the mesh, complete the following steps. Note that these instructions assume that you have Scenario 1 up and running, and modifying it to run Vault inside the mesh.
app
namespace by removing the istio-injection
label:
backyards sidecar-proxy auto-inject off app
backyards sidecar-proxy auto-inject on vault
vault
namespace, so they will get recreated with the istio-proxy
sidecar:
kubectl delete pods --all -n vault
$ kubectl get pods -n vault
NAME READY STATUS RESTARTS AGE
vault-0 4/4 Running 0 1m
vault-configurer-6d9b98c856-l4flc 2/2 Running 0 1m
kubectl delete pods --all -n app
The app pod got recreated with only the app
container (1/1) and Vault access still works:
$ kubectl get pods -n app
NAME READY STATUS RESTARTS AGE
app-5df5686c4-4n6r7 1/1 Running 0 71s
$ kubectl logs -f -n app deployment/app
time="2020-02-18T14:41:20Z" level=info msg="Received new Vault token"
time="2020-02-18T14:41:20Z" level=info msg="Initial Vault token arrived"
s3cr3t
going to sleep...
In this scenario, both Vault and the app are running inside the mesh. You can configure this scenario right after completing the Prerequisites.
backyards sidecar-proxy auto-inject on app
backyards sidecar-proxy auto-inject on vault
kubectl delete pods --all -n app
kubectl delete pods --all -n vault
$ kubectl logs -f -n app deployment/app
time="2020-02-18T15:04:03Z" level=info msg="Initial Vault token arrived"
time="2020-02-18T15:04:03Z" level=info msg="Renewed Vault Token"
s3cr3t
going to sleep...
The Bank-Vaults alongside Istio feature, Backing up Vault with Velero, Vault replication across multiple datacenters and HSM support with the Bank-Vaults operator are three major features in the upcoming Bank-Vaults release, so stay tuned.
Update: since releasingBank-Vaults 1.0, we also providecommercial support for Bank-Vaults. If you’re interested in commercial support, or anything else from our suite of products, make sure you get in touch with us,here.
If you’re interested in contributing, check out the Bank-Vaults repository, or give us a GitHub star.
Learn more about Bank-Vaults:
- Secret injection webhook improvements
- Backing up Vault with Velero
- Vault replication across multiple datacenters
- Vault secret injection webhook and Istio
- HSM support
- Injecting dynamic configuration with templates
- OIDC issuer discovery for Kubernetes service accounts
- Show all posts related to Bank-Vaults
Banzai Cloud’s Backyards (now Cisco Service Mesh Manager) is a multi and hybrid-cloud enabled service mesh platform for constructing modern applications. Built on Kubernetes and our Istio operator, it gives you flexibility, portability, and consistency across on-premise datacenters and cloud environments. Use our simple, yet extremely powerful UI and CLI, and experience automated canary releases, traffic shifting, routing, secure service communication, in-depth observability and more, for yourself.