CI/CD secret management on Kubernetes
At Banzai Cloud we're building a managed Cloud Native application and devops platform, called Pipeline. Pipeline supercharges the development, deployment and scaling of container-based applications with native support for multi- and hybrid-cloud environments.
Pipeline's built-in CI/CD solution is capable of creating Kubernetes clusters, running and testing builds, packaging and deploying applications as Helm charts, and lots more—all while its secrets are stored and managed by Vault.
Note: The Pipeline CI/CD module mentioned in this post is outdated and not available anymore. You can integrate Pipeline to your CI/CD solution using the Pipeline API. Contact us for details.
Secrets and CI
These days CI/CD systems do way more than just build code: they access SCM repositories, push artifacts, create clusters, and more—all in multi-tenant and (often) open source environments. Simultaneously, developers within a given organization may have different roles and authorizations, however, they all need to be able to start builds and check their outcomes and logs. It's difficult to manage all that: requires lots of maintenance, and storage way in excess of environment variables, files, or being stuck in CI descriptors.
Also, secrets should be short lived, or
should be easily and often revoked and replaced (especially
when they are compromised), all of which should be
We quickly realized this while working on Pipeline and:
- we standardized and open sourced a Vault operator for Kubernetes
- our enterprise customers were unhappy with Kubernetes' secrets (base64 encoding, tsk-tsk), so we allowed them to inject secrets directly into pods from Vault
- we run all CI builds wihtin Kubernetes in unprivileged mode
- secrets that are pulled from Vault are ephemeral and persistent
- in Pipeline
CI/CD descriptor secrets are
referencedonly, and binding is automatically occurs only when necessary and for only as long as it's required
- additionally, build artifacts and images can be security scanned as part of the CI pipeline
Secret management in CI, the Pipeline way
In Pipeline, users are members of organizations. Users may invite other users into organizations, and a user can be a member of more than one organization at a time. These organizations can have different types of associated resources: Kubernetes clusters created inside organizations, object storage buckets, spotguide catalogs, etc. One feature that all these have in common is that they require secrets to work properly. Usually, these secrets allow Pipeline to interact with your cloud provider account in order to provision and manage any underlying resources. Secrets may also be associated with applications running on your Kubernetes clusters, like a TLS certificate chain for MySQL, or a Kubernetes cluster’s own credentials. Secrets are stored in Vault with added layers of abstraction that allow them to be provided to applications and build pipelines.
Using secrets in build steps
At some point, you'll probably be executing a build step
that needs to use a credential to do its job. In the CI/CD
flow, which runs natively on Kubernetes,
Pipeline secrets can
be injected directly into Pods that represent build steps.
If you need one of your secrets for a specific build step,
you can specify it in the step's
This binds the value of a secret's field to an environment
variable supplied to your build step. For example, you can
use a secret that's holding Docker Hub credentials to build
and push an image:
pipeline: # ... build_container: image: plugins/docker repo: foo/bar secretFrom: DOCKER_USERNAME: name: my-docker-secret keyRef: username DOCKER_PASSWORD: name: my-docker-secret keyRef: password
To use a Pipeline secret in the steps of a specific CI/CD project, you must first attach the secret. On the web interface, go to the CI/CD page, click on the project's settings button, and attach the secret you want to use.
A plugin's documentation will specify the name of the
environment variables that the plugin expects the
credentials to be in, but you have the freedom to choose how
you store them in
speaking, it's possible to create a
and add fields to it with the same names as the variables.
Then the keys in the
secretFrom object will be repeated as
keyRefs, and the
name parameter will be the same for all
variables. However, the example of a docker plugin above
Password type secret, which has
password fields. These are mapped to anticipated variable
Installing secrets to clusters
If your application (your Helm charts) needs credentials in the form of Kubernetes Secrets, you can install them to the cluster in order to let the Pods of a given namespace consume them. The following code snippet is an example configuration for installing a secret:
pipeline: install_secret: image: banzaicloud/ci-pipeline-client:0.7 action: InstallSecret clusterSecret: sourceSecretName: pipeline-secret-name name: installed-secret-name namespace: default
With the help of the Pipeline, users can define how these secrets should be injected into the Kubernetes cluster their build is running on.
Learn more about Banzai Cloud Pipeline platform
If you're interested in learning more about using Pipeline with a CI system, contact us.