Security series: Authentication and authorization of Pipeline users with OAuth2 and Vault Dynamic credentials with Vault using Kubernetes Service Accounts Dynamic SSH with Vault and Pipeline Secure Kubernetes Deployments with Vault and Pipeline Policy enforcement on K8s with Pipeline The Vault swiss-army knife The Banzai Cloud Vault Operator Vault unseal flow with KMS Kubernetes secret management with Pipeline Container vulnerability scans with Pipeline Kubernetes API proxy with Pipeline

All of the following operations are automated by Pipeline, thus, for clusters/control plains launched by Pipeline, these steps may not necessarily need to be taken.

Dynamic SSH credentials

Requirements:

• A working Vault server accessible inside your infrastructure (automated by Pipeline)
• A GitHub organization (can be GitHub Enterprise as well, but this demo uses the public version)
• A VM (the examples below are based on Ubuntu)
• An SSH keypair on your machine

Client Key Signing

Administrative steps

First and foremost you have to log in as an administrator into Vault:

# Point your Vault CLI to the Vault instance
export VAULT_ADDR=https://vault.yourdomain.com

vault login

vault write sys/policy/dev policy='
    path "ssh-client-signer/sign/dev" {
        capabilities = ["update"]
    }'

vault write sys/policy/dev policy='
    path "ssh-client-signer/sign/dev" {
        capabilities = ["update"]
    }'

vault secrets enable -path=ssh-client-signer ssh
vault write ssh-client-signer/config/ca generate_signing_key=true
# This policy makes the CA signings valid for 24h, and is acceptable for daily workflow
vault write ssh-client-signer/roles/dev -<<EOH
{
  "allow_user_certificates": true,
  "allowed_users": "*",
  "default_extensions": [
    {
      "permit-pty": ""
    }
  ],
  "key_type": "ca",
  "default_user": "ubuntu",
  "ttl": "24h"
}
EOH

Download the public key from Vault onto the machine you would like to SSH into:

curl https://${VAULT_ADDR}/v1/ssh-client-signer/public_key > /etc/ssh/trusted-user-ca-keys.pem

Change /etc/ssh/sshd_config and add the following line so it will accept users with signed keys:

TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem

Restart sshd to realize changes:

service ssh restart

User steps

# Point your Vault CLI to your Vault instance
export VAULT_ADDR=https://vault.yourdomain.com

# Enter your GitHub access token here with read:org capabilities, or create one at https://github.com/settings/tokens
export GITHUB_TOKEN=...

vault login -method github $GITHUB_TOKEN

vault write -field=signed_key ssh-client-signer/sign/dev public_key=@$HOME/.ssh/id_rsa.pub > ~/.ssh/id_rsa-cert.pub

Now you can log in to the VM:

ssh ubuntu@xy.xyz.xyz.xyz

Host Key Signing

How many times have you seen this message?

The authenticity of host 'xy.xyz.xyz.xyz (xy.xyz.xyz.xyz)' can't be established.
ECDSA key fingerprint is SHA256:wu5UbyRRuKw+JwCotA2QzQa7dsgs627pzwIYpXhprqA.
Are you sure you want to continue connecting (yes/no)?

Administrative steps

vault secrets enable -path=ssh-host-signer ssh
vault write ssh-host-signer/config/ca generate_signing_key=true
vault secrets tune -max-lease-ttl=87600h ssh-host-signer

Configure the hostsigning policy and hostrole role:

vault write ssh-host-signer/roles/hostrole \
    key_type=ca \
    ttl=87600h \
    allow_host_certificates=true \
    allowed_domains="*" \
    allow_subdomains=true

vault write sys/policy/hostsigning policy='
    path "ssh-host-signer/sign/hostrole" {
        capabilities = ["update"]
    }'

vault write sys/policy/dev policy='
    path "ssh-client-signer/sign/dev" {
        capabilities = ["update"]
    }
    path "auth/approle/role/hostrole/role-id" {
        capabilities = ["read"]
    }
    path "auth/approle/role/hostrole/secret-id" {
        capabilities = ["update"]
    }'

Setup AppRole auth method so that hosts will be able to lease Vault tokens for host key (self) signing:

vault write auth/approle/role/hostrole policies=hostsigning \
    secret_id_ttl=10m \
    token_num_uses=10 \
    token_ttl=20m \
    token_max_ttl=30m \
    secret_id_num_uses=40

Sign the key on the host (log in with AppRole auth method first):

# Get a token with Vault AppRole
VAULT_TOKEN=$(vault write -field token auth/approle/login role_id=${VAULT_ROLE_ID} 
secret_id=${VAULT_SECRET_ID})

# Log in with that token
vault login ${VAULT_TOKEN}

# Sign the key of this host
vault write -field=signed_key ssh-host-signer/sign/hostrole \
    cert_type=host \
    public_key=@/etc/ssh/ssh_host_rsa_key.pub > /etc/ssh/ssh_host_rsa_key-cert.pub

# Set permissions on the certificate to be 0640:
chmod 0640 /etc/ssh/ssh_host_rsa_key-cert.pub

Change /etc/ssh/sshd_config and add the host certificate path to it:

# For host keys
HostKey /etc/ssh/ssh_host_rsa_key
HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub

Restart SSH to realize the changes:

service ssh restart

User steps

When starting a machine:

# Point your Vault CLI to your Vault instance
export VAULT_ADDR=https://vault.yourdomain.com

# Enter your GitHub access token here with read:org capabilities, or create one at https://github.com/settings/tokens
export GITHUB_TOKEN=...

vault login -method github $GITHUB_TOKEN

VAULT_ROLE_ID=$(vault read -field role_id auth/approle/role/hostrole/role-id)
VAULT_SECRET_ID=$(vault write -f -field secret_id auth/approle/role/hostrole/secret-id)
# Pass VAULT_ROLE_ID and VAULT_SECRET_ID to the newly started VM.

When logging into that machine:

# Setup your local ssh configuration to allow hosts signed by this CA (coming from Vault)
CERT_AUTHORITY=$(vault read -field=public_key ssh-host-signer/config/ca)
echo "@cert-authority * $CERT_AUTHORITY" >> ~/.ssh/known_hosts

# If you are using client-side signed keys, sign your key temporarily
vault write -field=signed_key ssh-client-signer/sign/dev public_key=@$HOME/.ssh/id_rsa.pub > 
~/.ssh/id_rsa-cert.pub

# SSH into target machines as usual
ssh ubuntu@xy.xyz.xyz.xyz

Learn by the code

We already start our Pipeline Control Plane instances with this method, which you can check out, here, in our GitHub repository.

Learn more about Bank-Vaults:

• Secret injection webhook improvements
• Backing up Vault with Velero
• Vault replication across multiple datacenters
• Vault secret injection webhook and Istio
• HSM support
• Injecting dynamic configuration with templates
• OIDC issuer discovery for Kubernetes service accounts
• Show all posts related to Bank-Vaults