Update: Logging operator v3 (released March, 2020)

We're constantly improving the logging-operator based on feature requests of our ops team and our customers. The main features of version 3.0 are:

• Log routing based on namespaces
• Excluding logs
• Select (or exclude) logs based on hosts and container names
• Logging operator documentation is now available on the Banzai Cloud site.

Check The Kubernetes logging operator reloaded post for details.

Logging series: Centralized logging under Kubernetes Secure logging on Kubernetes with Fluentd and Fluent Bit Advanced logging on Kubernetes

Client Certificate Authentication

Setting up the Certificates

• Generate the Root CA (for self-signed certificates)
• Generate the Intermediate Server Cert (Singed with CA)
• Generate the Client certificate (Signed with CA)

Prepare the working directory

mkdir certs csr private touch
index.txt echo "1000" > serial

[ ca ]
# `man ca`
default_ca = CA_default

[ CA_default ]
# Directory and file locations.
dir               = .
certs             = $dir/certs
crl_dir           = $dir/crl
new_certs_dir     = $dir/newcerts
database          = $dir/index.txt
serial            = $dir/serial
RANDFILE          = $dir/private/.rand

# The root key and root certificate.
private_key       = $dir/private/ca.key.pem
certificate       = $dir/certs/ca.crt.pem

# SHA-1 is deprecated, so use SHA-2 instead.
default_md        = sha256

name_opt          = ca_default
cert_opt          = ca_default
default_days      = 365
preserve          = no
policy            = policy_strict

[ req ]
# Options for the `req` tool (`man req`).
default_bits        = 4096
distinguished_name  = req_distinguished_name
string_mask         = utf8only

# SHA-1 is deprecated, so use SHA-2 instead.
default_md          = sha256

# Extension to add when the -x509 option is used.
x509_extensions     = v3_ca

[ req_distinguished_name ]
# See <https://en.wikipedia.org/wiki/Certificate_signing_request>.
countryName                     = Country Name (2 letter code)
stateOrProvinceName             = State or Province Name
localityName                    = Locality Name
0.organizationName              = Organization Name
organizationalUnitName          = Organizational Unit Name
commonName                      = Common Name (required)
emailAddress                    = Email Address

# Optionally, specify some defaults.
countryName_default             = US
stateOrProvinceName_default     = CA
#localityName_default           = Mountain View
0.organizationName_default      = Your company name
#organizationalUnitName_default =
emailAddress_default            = foo@example.com

[v3_ca]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always, issuer
basicConstraints = critical,CA:true
keyUsage = critical, cRLSign, digitalSignature, keyCertSign

[ client_cert ]
# Extensions for client certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = client, email
nsComment = "OpenSSL Generated Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, emailProtection

[ server_cert ]
# Extensions for server certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

[ policy_strict ]
# The root CA should only sign intermediate certificates that match.
# See the POLICY FORMAT section of `man ca`.
countryName             = match
stateOrProvinceName     = match
organizationName        = match
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

Generate files for Certification Authority (CA)

openssl genrsa -aes256 -out
private/ca.key.pem 4096

It's highly recommended that you protect keys with passwords, although you can skip that step by removing the -aes256 option.

Generate cert for Certification Authority (CA)

openssl req -config openssl.cnf \
 -key private/ca.key.pem \
 -new -x509 -days 365 -sha256 -extensions v3_ca \
 -out certs/ca.crt.pem

Generate Server Private Key

openssl genrsa -aes256 -out
private/server.key.pem 4096

Create Server Certificate Sign Request (csr)

openssl req -config openssl.cnf \
 -key private/server.key.pem \
 -new -sha256 -out csr/server.csr.pem

Self Sign Server CSR with CA

openssl ca -config openssl.cnf
-outdir certs \
 -cert certs/ca.crt.pem \
 -keyfile private/ca.key.pem \
 -extensions server_cert -days 365 -notext -md sha256 \
 -in csr/server.csr.pem \
 -out certs/server.crt.pem

Fill out Common Name as hostname or fqdn, ie: server

Create the Client certificates

• Create a private key: client.key.pem

  openssl genrsa -aes256 -out
  private/client.key.pem 4096

• Create a certification request: <code>client.csr.pem

  openssl req -config openssl.cnf \
   -key private/client.key.pem \
   -new -sha256 \
   -out csr/client.csr.pem

• Sign the Client CSR with CA: client.crt.pem

  openssl ca -config openssl.cnf
  -outdir certs \ 
  -cert certs/ca.crt.pem \ 
  -keyfile private/ca.key.pem \ 
  -extensions client_cert -days 365 -notext -md sha256 \ 
  -in csr/client.csr.pem \ 
  -out certs/client.crt.pem

Fill out Common Name as hostname or fqdn, ie: client

Check the certificates

Secure logging in action

Enable TLS on Fluent-bit

[OUTPUT]
    Name          forward
    Match         *
    Host          {{ .Values.backend.forward.host }}
    Port          {{ .Values.backend.forward.port }}
    Retry_Limit False
    tls               On
    tls.verify        {{ .Values.backend.forward.tls.verify }}
    tls.ca_file       /fluent-bit/ssl/ca.crt.pem
    tls.crt_file      /fluent-bit/ssl/client.crt.pem
    tls.key_file      /fluent-bit/ssl/client.key.pem
    tls.key_passwd    ${TLS_PRIVATE_KEY_PASSPHRASE}
    Shared_Key        ${SHARED_KEY}

The ${VARABLE_NAME} values indicate the values that come from Environment variables.

kubectl create secret generic
fluentbit-tls \
 --from-file=ca.crt.pem=./certs/ca.crt.pem \
 --from-file=client.crt.pem=./certs/client.crt.pem \
 --from-file=client.key.pem=./private/client.key.pem \
 --from-literal 'server.key.passphrase=1111' \
 --from-literal 'fluent.shared.key=shared1234'

Enable TLS on Fluentd

<source>
  @type   forward
  port    24231
  @log_level debug
  <security>
    self_hostname fluentd
    shared_key fluentd
  </security>
  <transport tls>
    version                TLSv1_2
    ca_path                /fluentd/etc/ssl/ca.crt.pem
    cert_path              /fluentd/etc/ssl/server.crt.pem
    private_key_path       /fluentd/etc/ssl/server.key.pem
    private_key_passphrase "#{ENV["TLS_PRIVATE_KEY_PASSPHRASE"]}"
    client_cert_auth       true
  </transport>
</source>

The #{ENV["VARABLE_NAME"]} values indicate that the values come from Environment variables.

kubectl create secret generic
fluentd-tls \
 --from-file=ca.crt.pem=./certs/ca.crt.pem \
 --from-file=server.crt.pem=./certs/server.crt.pem \
 --from-file=server.key.pem=./private/server.key.pem \
 --from-literal 'server.key.passphrase=1234' \
 --from-literal 'fluent.shared.key=shared1234'

Now put it together