The usage and management of secrets still carries its own set of technical challenges. Setting up standards and tooling, navigating different provider offerings, defining proper access rules, and establishing mechanisms for secure usage and monitoring of secrets are just some of them. At Bank-Vaults, we try to make secret management in the cloud-native world an accessible, understandable, and straightforward process for everyone. This post shows you how to use different secret service providers using the new Secret Sync tool while also addressing common pitfalls when dealing with secrets.

Why are secrets difficult?

Workloads often need secrets for initialization or during runtime via configuration parameters, environment variables, or SDKs that interact with different services. Ideally, workloads should avoid direct communication with secret service providers. The challenges can then be linked to:

• Additional dependencies
• Complex flows for accessing sensitive data
• Lifecycle management across environments
• Uniform audit trail
• Operational costs

Approach A: Native secret management

In a typical cloud-native scenario, workloads access sensitive data from k8s secrets or in other instances via some additional and intermediary tooling. An external service (such as external-secrets) can be used to synchronize required secrets from different secret services into k8s secrets. However, this might be cumbersome to coordinate. In addition, native k8s secrets could simply be a security concern for your specific requirements (this blog post gives a good summary on this).

Approach B: Bank-Vaults managed secrets handling

Bank-Vaults Secret Sync tackles these problems by exposing a generic way to interact with external secret services and providing a way to synchronize secrets between them. Let's look at an example from the image above. In this scenario, there's a Vault instance acting as a central source of truth. Currently, we can only have one source of truth, but we're working on adding features to support multiple sources and additional secret stores. The secrets required for specific teams and workloads can then be synchronized from the selected source to our local/tenant Vault instance. In summary, this approach aims to:

• Provide a safe and simple way to work with secrets
• Expose common API for secret management regardless of the store backend
• Give total control of the secret synchronization process 

The main difference between the two approaches is that:

1. Approach A: You go all-in for k8s secrets, having tools in place that orchestrate synchronization of secrets from different secret services.
2. Approach B: You want to decide ad-hoc (or mix and match) which secret services to use depending on requirements. In addition, you also want to explicitly control the synchronization process in a generic and reusable way.
   1. Another benefit of this approach is that it integrates well with the rest of Bank-Vault offerings such as mutating-webhook and vault-secrets-reloader which can be used for added benefits such as direct k8s secret injection and live workload reload.

Synchronize secrets to and from Vault with Secret Sync

Following the Approach 2, we will show how you can use this tool to:

Case 1: Synchronize secrets from one secret store to another.
We will use our local file store as a source of truth to synchronize some database secrets into Vault.

Case 2: Consume secrets to bootstrap an application.
We will use Vault instance to fetch database secrets to our local store in the form of a configuration file for an application. Note: The same logic applies to any other combination of secret stores.

(This is an early alpha version and breaking changes are expected. As such, it is not recommended for usage in production. We are actively working on expanding the list of supported stores and consolidating our APIs.)

Step 1: Prepare the environment

You will need the following tools to continue:

• Git
• Makefile
• Golang >= 1.21 To set up the environment, you can build from the source:

git clone https://github.com/bank-vaults/secret-sync.git /tmp/secret-sync
cd /tmp/secret-sync
make build
alias secret-sync="/tmp/secret-sync/build/secret-sync"

Alternatively, you can also use only Docker:

alias secret-sync="docker run --net=host --rm -v /tmp:/tmp ghcr.io/bank-vaults/secret-sync:v0.1.1 secret-sync"

Step 2: Define secret stores

Local store

Create a directory and a config file to use as the local secret store.

# Create a local store directory
mkdir -p /tmp/example/local-store
# Create a local store config file
cat <<EOF > /tmp/example/local-store.yml
secretsStore:
  local:
    storePath: "/tmp/example/local-store"
EOF

Vault store

Deploy Vault and create a config file to use as the Vault secret store.

# Deploy a Vault instance
docker compose -f dev/vault/docker-compose.yml up -d
# Create Vault store config file
cat <<EOF > /tmp/example/vault-store.yml
secretsStore:
  vault:
    address: "http://0.0.0.0:8200"
    storePath: "secret/"
    authPath: "userpass"
    token: "root"
EOF

Documentation and examples on how to use other secret stores can be found in Docs: Secret Stores.

Step 3: Define sync plans

Database secrets

Define a sync plan for db-host, db-user, db-pass secrets. These secrets will be synced from our local to Vault secret store.

cat <<EOF > /tmp/example/db-secrets-sync.yml
sync:
  - secretQuery:
      path: /
      key:
        regexp: db-(host|user|pass)
EOF

Documentation and examples on how to create a more extensive sync plan can be found in Docs: Sync Plan.

Application access secret

Define a sync plan for app-specific secret app-access-config created from various other secrets (e.g. database). This secret will be synced from Vault to our local secret store (as a file). It can also be synced against the same store to refresh the secret.

cat <<EOF > /tmp/example/app-access-config-sync.yml
sync:
  - secretSources:
      - name: selector
        secretQuery:
          path: /
          key:
            regexp: db-(host|user|pass)
    target:
      key: app-access-config
    template:
      data:
        appID: "12345"
        # ...some additional secrets for the given app...
        # Secrets fetched from Vault will be encoded, we need to decode
        hostname: "{{ .Data.selector.dbHost | base64dec }}"
        username: "{{ .Data.selector.dbUser | base64dec }}"
        password: "{{ .Data.selector.dbPass | base64dec }}"
EOF

Step 4: Create database secrets

Create database access secrets in our local secret store.

echo -n "very-secret-hostname" > /tmp/example/local-store/db-host
echo -n "very-secret-username" > /tmp/example/local-store/db-user
echo -n "very-secret-password" > /tmp/example/local-store/db-pass

Step 5: Perform sync

Secret synchronization is performed using the CLI by executing the sync plan between source and target secret stores.

Database secrets

To synchronize database secrets from our local to Vault secret store, run:

secret-sync --source "/tmp/example/local-store.yml" --target "/tmp/example/vault-store.yml" --sync "/tmp/example/db-secrets-sync.yml"

If successful, your output should contain something like:

{"level":"info","msg":"Successfully synced action = 0 for key /db-user"}
{"level":"info","msg":"Successfully synced action = 0 for key /db-pass"}
{"level":"info","msg":"Successfully synced action = 0 for key /db-host"}
{"level":"info","msg":"Synced 3 out of total 3 keys"}

You can also navigate to the local Vault instance and verify these secrets.

Application access secret

To synchronize application access secret from Vault to our local secret store, run:

secret-sync --target "/tmp/example/local-store.yml" --source "/tmp/example/vault-store.yml" --sync "/tmp/example/app-access-config-sync.yml"

If successful, besides logs, you should also be able to find the app access secret via:

$ cat /tmp/example/local-store/app-access-config
{"appID":"12345","hostname":"very-secret-hostname","password":"very-secret-password","username":"very-secret-username"}

Step 6: Cleanup

# Destroy Vault instance
docker compose -f dev/vault/docker-compose.yml down
# Remove example files
rm -rf /tmp/example

Don't miss out

You can support us with your feedback, bug reports, and feature requests.

• Join our Discussion Board 💬
• Support Secret Sync with a star ⭐
• Check other Bank-Vaults projects 📌
• Visit our Roadmap to see or request new features 🚩