Outshift Logo

PRODUCT

15 min read

Blog thumbnail

by Janos Matyas

Published on 08/08/2020
Last updated on 03/21/2024

Hardened FIPS compliant Istio with Backyards

Share

Companies providing services for government, healthcare, or finance sectors must ensure that their products adhere to government approved security requirements that ensure the protection of sensitive data. The Federal Information Processing Standards (FIPS) outlines the rules for cryptographic modules employed as mechanism that implement security. A cryptographic module is a combination of hardware and/or software that handles cryptographic functions such as encryption, decryption, digital signatures, authentication techniques and random number generation. The FIPS 140-2 standard provides four increasing, qualitative levels of security: Level 1, Level 2, Level 3, and Level 4. These levels are intended to cover a wide range of potential applications and environments into which cryptographic modules may be employed. In this post we'll discuss the approach we took to reaching FIPS 140-2 Security Level 1 compliance for our Istio distribution, Backyards (now Cisco Service Mesh Manager). backyards

Note that there is a newer/updated FIPS version, FIPS 140-3. FIPS 140-3 testing will begin on September 22, 2020.

tl;dr

Istio mesh_Backyards

FIPS at Banzai Cloud

While, for now, we're discussing how we built a FIPS 140-2 compliant Istio with Backyards (now Cisco Service Mesh Manager), several of the steps we'll be covering are shared by other products we are working to make FIPS compliant:
  • PKE, our CNCF certified Kubernetes distribution,
  • Pipeline, the container management platform for hybrid clouds (supporting 5 major cloud providers, bare metal and VMware),
  • Supertubes, production ready Kafka on Kubernetes,
  • Bank-Vaults, the Vault Swiss Army knife and secret injection webhook,
  • and One Eye, the ultimate observability tool for Kubernetes.
You might want to check out how we manage AWS GovCloud Kubernetes clusters with Pipeline

FIPS challenges

FIPS Security Level 1 mandates the usage of approved security algorithms or functions used by cryptographic modules for generating keys that are used to protect sensitive data. This means that any exchange of sensitive data between applications must occur over secure channels, using only FIPS validated cryptographic modules. For the list of FIPS validated cryptographic modules see Cryptographic Module Validation Program (using Google as a vendor example).
For a guide on how to read an FIPS Validation Listing, follow this link
In microservices architecture, as its name implies, multiple services (applications) are employed to implement scalable, flexible, fault-tolerant and secure products that serve complex business requirements. The components that such a system comprise are often written in different programming languages using different technologies. Ensuring that each of these components use an FIPS validated cryptographic module can be easily complicated:
  • there might be no FIPS validated library (cryptographic module) available that is compatible with the component
  • third party components may not allow replacing the used cryptography with a FIPS validated one
  • each component may require its own custom solution which increases the maintenance burden

FIPS 140-2 encryption using Istio

If we take a look at a typical microservice deployment running on our Istio distribution, Backyards (now Cisco Service Mesh Manager), we can see that the above mentioned challenges apply to it as well. Also, in the Istio community there is a longstanding feature request of using FIPS 140-2 compliant versions of Istio (which has resulted in lots of discussion on forums, internal WG or Slack). It is our pleasure, then, to announce that the wait is over: Backyards (now Cisco Service Mesh Manager) and all the components that ship as part of the [Banzai Cloud Istio distribution] are now built with FIPS compliant crypto modules.

In a nutshell

So what steps did we take? What details make Istio FIPS compliant? As discussed in the tl;dr: section:
Note: as FIPS introduces lots of restrictions on the accepted cipher suites and can introduce cryptographic incompatibilities, the default version of Backyards (now Cisco Service Mesh Manager) is still built with the standard SSL libraries. If you'd like to use an FIPS compliant version of Istio/Backyards (now Cisco Service Mesh Manager), get in touch with us.
FIPS 140-2 encryption using Istio

Under the hood

The bottom line for FIPS compliance from the application standpoint is that it must use FIPS verified crypto libraries.

Go-based applications

Unfortunately, the native crypto in Go is not FIPS validated, so we need a way to use a validated crypto library with Go. Luckily, it's Google's BoringSSL to the rescue.
BoringSSL is a fork of OpenSSL that is designed to meet Google’s needs.
BoringSSL as a whole is not FIPS validated. However, there is a core library (called BoringCrypto) that has been FIPS validated. Google also maintains a Go fork that can use BoringCrypto for cryptographic functions.
The distribution name for a Go+BoringCrypto release has the form < GoVersion>b< BoringCryptoVersion>, where < GoVersion> is the Go version the release is based on, and < BoringCryptoVersion> is an integer that increments each time there is a new release with different BoringCrypto bits.
You can use the Go+BoringCrypto distribution as a drop-in replacement, no code changes usually needed for simple compilation. Indeed, it takes more than just a plain re-compile to get our application to make use of the BoringCrypto. The application has to be re-compiled with specific build configuration:
  • The build must be GOOS=linux, GOARCH=amd64.
  • The build must have cgo enabled.
  • The android build tag must not be specified.
  • The cmd_go_bootstrap build tag must not be specified.
To check whether a given binary is using BoringCrypto, run go tool nm on it and check that it has symbols named *_Cfunc__goboringcrypto_*. The program rsc.io/goversion will report the crypto implementation used by a given binary when invoked with the -crypto flag.
Furthermore, we made changes in the Istio codebase to make sure that FIPS is enforced properly in each and every binary. Our build has a more strict TLS configuration than FIPS 140-2 requires, and places similar limitations on TLS versions and used ciphers. This required additional changes in the codebase, but it further enhanced application security. Compare the built binaries The compiled binaries can be verified by the aforementioned goversion tool. If we check the upstream binary of Istiod, for example, we'll find that it uses the standard non-FIPS compliant standard Go crypto: 
❯ goversion ./pilot-discovery
pilot-discovery go1.14.2 (standard crypto)
While doing the same check with the Backyards FIPS 140-2 compliant version, we can see that it uses the validated BoringCrypto library: 
❯ goversion ./pilot-discovery
pilot-discovery go1.14.6b4 (boring crypto) +crypto/tls/fipsonly

Every Istio component the FIPS compliant Backyards distribution deploys uses custom built binaries that are using FIPS 140-2 validated BoringCrypto library.

Envoy

Istio uses Envoy in its data-plane, so the next step is to add an FIPS compliant version of it as well. Envoy uses BoringSSL already and it can be built in an FIPS 140-2 compliant mode by using the --define boringssl=fips build option. The correctness of the resulting FIPS build can be verified by checking the presence of BoringSSL-FIPS in the --version output. Besides the FIPS compliant build of Envoy, we've also made changes in Istio to ensure that the generated Envoy configuration distributed through XDS has strict and compliant TLS settings. Compare the Envoy builds The upstream Istio proxy image contains a non-FIPS 140-2 compliant Envoy build: 
~ ❯ docker run -ti --rm --entrypoint envoy istio/proxyv2:1.6.7 --version

envoy  version: ccae1bd37085ecd78415dc06b50233b3b97e30c0/1.14.4/Clean/RELEASE/BoringSSL
While the FIPS 140-2 compliant Backyards Istio proxy image contains a custom built compliant version of Envoy: 
~ ❯ docker run -ti --rm --entrypoint envoy banzaicloud/istio-proxyv2:1.6.7-bzc-fips --version

envoy  version: 136ca7b03ff5821697d68b77e7411dbc6bfdd5ef/1.14.4/Clean/RELEASE/BoringSSL-FIPS

cert-manager

Backyards installs cert-manager and relies on it for various certificate management related tasks (webhook and ingress certs for example). This was another stumbling block on our path to FIPS 14 0-2 compliance. We used the same tooling, configuration and code base changes as with Istio's components.
If you are interested in cert-manager on Kubernetes, we recently blogged [about > it]({{< relref "/blog/cert-management-on-kubernetes.md" >}}), specifically about [cert management on > Istio]({{< relref "/blog/cert-management-on-istio.md" >}}) as well.
Compare the built cert-manager binaries If we check the upstream version of the cert manager controller we'll find that it is uses standard Go crypto: 
❯ goversion -crypto ./controller
./controller go1.14.2 (standard crypto)
While the FIPS 140-2 compliant Backyards version uses BoringCrpto with strict FIPS settings: 
❯ goversion -crypto ./controller
./controller go1.14.2 (standard crypto)

Showtime

We are going to use two Kubernetes clusters that were created with Banzai Cloud Pipeline. One cluster has an Istio mesh installed and configured using our FIPS 140-2 compliant Backyards version, while the other cluster has a non-compliant mesh installed, which will allow us to demonstrate the differences between the two. Backyards contains a microservice-based demo application which is also installed on both clusters and will be used in the following steps. We are going to use the frontpage service which serves requests on port 8080. The Istio mesh is configured with strict auto mTLS on both clusters. Since the most visible aspects of FIPS 140-2 compliance are the TLS settings, the last item in the demo toolbox today is a handy TLS/SSL testing tool called testssl.sh.

Deploy testssl.sh pod on both clusters

We deployed the testssl pods on each cluster into the default namespace using the following snippet: 
❯ kubectl create -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: testssl
  namespace: default
spec:
  containers:
  - name: app
    image: drwetter/testssl.sh
    command: [ "/bin/sh", "-c", "--" ]
    args: [ "while true; do sleep 3000; done;" ]
EOF

Check the frontpage service endpoint

We can access detailed information about a TLS endpoint by running testssl.sh. The following is the output of such a check against the frontpage service on the cluster that uses the non-FIPS compliant build. The workload runs inside an Istio mesh, thus the TLS here is provided by Istio sidecar Envoy proxy. 
non FIPS compliant cluster ❯ kubectl exec -ti testssl -- testssl.sh https://frontpage.backyards-demo.svc.cluster.local.:8080
 Start 2020-08-06 13:46:47        -->> 10.11.241.177:8080 (frontpage.backyards-demo.svc.cluster.local.) <<--

 rDNS (10.11.241.177):   frontpage.backyards-demo.svc.cluster.local.
 Service detected:       certificate-based authentication => skipping all HTTP checks


 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      offered (deprecated)
 TLS 1.1    offered (deprecated)
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 NPN/SPDY   not offered
 ALPN/HTTP2 h2 (offered)

 Testing cipher categories

 NULL ciphers (no encryption)                      not offered (OK)
 Anonymous NULL Ciphers (no authentication)        not offered (OK)
 Export ciphers (w/o ADH+NULL)                     not offered (OK)
 LOW: 64 Bit + DES, RC[2,4], MD5 (w/o export)      not offered (OK)
 Triple DES Ciphers / IDEA                         not offered
 Obsoleted CBC ciphers (AES, ARIA etc.)            offered
 Strong encryption (AEAD ciphers) with no FS       offered (OK)
 Forward Secrecy strong encryption (AEAD ciphers)  offered (OK)


 Testing server's cipher preferences

 Has server cipher order?     yes (OK) -- only for < TLS 1.3
 Negotiated protocol          TLSv1.3
 Negotiated cipher            TLS_AES_256_GCM_SHA384, 253 bit ECDH (X25519)
 Cipher per protocol

Hexcode  Cipher Suite Name (OpenSSL)       KeyExch.   Encryption  Bits     Cipher Suite Name (IANA/RFC)
-----------------------------------------------------------------------------------------------------------------------------
SSLv2
 -
SSLv3
 -
TLSv1 (server order)
 xc013   ECDHE-RSA-AES128-SHA              ECDH 256   AES         128
 x2f     AES128-SHA                        RSA        AES         128
 xc014   ECDHE-RSA-AES256-SHA              ECDH 256   AES         256
 x35     AES256-SHA                        RSA        AES         256
TLSv1.1 (server order)
 xc013   ECDHE-RSA-AES128-SHA              ECDH 256   AES         128
 x2f     AES128-SHA                        RSA        AES         128
 xc014   ECDHE-RSA-AES256-SHA              ECDH 256   AES         256
 x35     AES256-SHA                        RSA        AES         256
TLSv1.2 (server order)
 xcca8   ECDHE-RSA-CHACHA20-POLY1305       ECDH 253   ChaCha20    256
 xc02f   ECDHE-RSA-AES128-GCM-SHA256       ECDH 253   AESGCM      128
 xc013   ECDHE-RSA-AES128-SHA              ECDH 253   AES         128
 x9c     AES128-GCM-SHA256                 RSA        AESGCM      128
 x2f     AES128-SHA                        RSA        AES         128
 xc030   ECDHE-RSA-AES256-GCM-SHA384       ECDH 253   AESGCM      256
 xc014   ECDHE-RSA-AES256-SHA              ECDH 253   AES         256
 x9d     AES256-GCM-SHA384                 RSA        AESGCM      256
 x35     AES256-SHA                        RSA        AES         256
TLSv1.3 (no server order, thus listed by strength)
 x1302   TLS_AES_256_GCM_SHA384            ECDH 253   AESGCM      256
 x1303   TLS_CHACHA20_POLY1305_SHA256      ECDH 253   ChaCha20    256
 x1301   TLS_AES_128_GCM_SHA256            ECDH 253   AESGCM      128


 Testing robust forward secrecy (FS) -- omitting Null Authentication/Encryption, 3DES, RC4

 FS is offered (OK)           TLS_AES_256_GCM_SHA384
                              TLS_CHACHA20_POLY1305_SHA256
                              ECDHE-RSA-AES256-GCM-SHA384
                              ECDHE-RSA-AES256-SHA
                              ECDHE-RSA-CHACHA20-POLY1305
                              TLS_AES_128_GCM_SHA256
                              ECDHE-RSA-AES128-GCM-SHA256
                              ECDHE-RSA-AES128-SHA
 Elliptic curves offered:     prime256v1 X25519
If you do the same thing on the cluster with the FIPS 140-2 compliant Backyards mesh, the outputs will look totally different. 
FIPS compliant cluster ❯ kubectl exec -ti testssl -- testssl.sh https://frontpage.backyards-demo.svc.cluster.local.:8080
 Start 2020-08-06 14:32:22        -->> 10.63.243.56:8080 (frontpage.backyards-demo.svc.cluster.local.) <<--

 rDNS (10.63.243.56):    frontpage.backyards-demo.svc.cluster.local.
 Service detected:       certificate-based authentication => skipping all HTTP checks


 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      not offered
 TLS 1.1    not offered
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 NPN/SPDY   not offered
 ALPN/HTTP2 h2 (offered)

 Testing cipher categories

 NULL ciphers (no encryption)                      not offered (OK)
 Anonymous NULL Ciphers (no authentication)        not offered (OK)
 Export ciphers (w/o ADH+NULL)                     not offered (OK)
 LOW: 64 Bit + DES, RC[2,4], MD5 (w/o export)      not offered (OK)
 Triple DES Ciphers / IDEA                         not offered
 Obsoleted CBC ciphers (AES, ARIA etc.)            not offered
 Strong encryption (AEAD ciphers) with no FS       offered (OK)
 Forward Secrecy strong encryption (AEAD ciphers)  offered (OK)


 Testing server's cipher preferences

 Has server cipher order?     yes (OK) -- only for < TLS 1.3
 Negotiated protocol          TLSv1.3
 Negotiated cipher            TLS_AES_256_GCM_SHA384, 256 bit ECDH (P-256)
 Cipher per protocol

Hexcode  Cipher Suite Name (OpenSSL)       KeyExch.   Encryption  Bits     Cipher Suite Name (IANA/RFC)
-----------------------------------------------------------------------------------------------------------------------------
SSLv2
 -
SSLv3
 -
TLSv1
 -
TLSv1.1
 -
TLSv1.2 (server order)
 xc02f   ECDHE-RSA-AES128-GCM-SHA256       ECDH 256   AESGCM      128
 xc030   ECDHE-RSA-AES256-GCM-SHA384       ECDH 256   AESGCM      256
 x9c     AES128-GCM-SHA256                 RSA        AESGCM      128
 x9d     AES256-GCM-SHA384                 RSA        AESGCM      256
TLSv1.3 (no server order, thus listed by strength)
 x1302   TLS_AES_256_GCM_SHA384            ECDH 256   AESGCM      256
 x1303   TLS_CHACHA20_POLY1305_SHA256      ECDH 256   ChaCha20    256
 x1301   TLS_AES_128_GCM_SHA256            ECDH 256   AESGCM      128


 Testing robust forward secrecy (FS) -- omitting Null Authentication/Encryption, 3DES, RC4

 FS is offered (OK)           TLS_AES_256_GCM_SHA384
                              TLS_CHACHA20_POLY1305_SHA256
                              ECDHE-RSA-AES256-GCM-SHA384
                              TLS_AES_128_GCM_SHA256
                              ECDHE-RSA-AES128-GCM-SHA256
 Elliptic curves offered:     prime256v1
As you can see our FIPS compliant Istio build only allows connection with TLS1.2+ and only with FIPS compliant ciphers. Also, the only allowed elliptic curve algorithm is the FIPS compliant P-256/prime256v1.
Backyards provides gateway configuration features with Let's Encrypt support utilizing an FIPS 140-2 compliant build of cert-manager, so the same TLS settings apply not just in east-west mTLS communication, but for ingress TLS traffic as well.

FIPS 140-2 compliant Backyards TLS settings

Allowed TLS versions
  • TLS v1.2
  • TLS v1.3
Although FIPS 140-2 would allow lower TLS versions under some circumstances, we disabled them for security reasons. TLS 1.0 and 1.1 are out-of-date protocols that do not support modern cryptographic algorithms, and they contain security vulnerabilities that may be exploited by attackers. The IETF is also planning to officially deprecate both protocols. In addition, the vast majority of encrypted Internet traffic is now over TLS 1.2, which was introduced over a decade ago.
Allowed FIPS compatible ciphers
  • ECDHE-RSA-AES128-GCM-SHA256
  • ECDHE-RSA-AES256-GCM-SHA384
  • ECDHE-ECDSA-AES128-GCM-SHA256
  • ECDHE-ECDSA-AES256-GCM-SHA384
  • AES128-GCM-SHA256
  • AES256-GCM-SHA384
There are more ciphers allowed by FIPS 140-2. We only enable GCM ciphers, because only those ciphers can prevent a LUCKY13 timing attack
Allowed Elliptic-curve algorithm
  • P-256

Summary

As was mentioned in the beginning, ensuring that each of the components are using an FIPS validated cryptographic module is easily complicated if:
  • there might be no FIPS validated library (cryptographic module) available that is compatible with the component
  • third party components may not allow replacing the used cryptography with an FIPS-validated one
  • each component requires its own custom solution which increases the maintenance burden
Having an FIPS 140-2 compliant service mesh can help you a great deal. It gives you a unified layer in front of your applications and in this particular case, provides the encryption settings and enforcement that your environment requires. Istio features
Check out Backyards in action on your own clusters! Register for an evaluation version and run a simple install command! If you’d like to use a FIPS 140-2 compliant version of Istio/Backyards, get in touch with us. Or just take a look at some of the Istio features that Backyards automates and simplifies for you, and which we've already blogged about. .
Subscribe card background
Subscribe
Subscribe to
the Shift!

Get emerging insights on emerging technology straight to your inbox.

Unlocking Multi-Cloud Security: Panoptica's Graph-Based Approach

Discover why security teams rely on Panoptica's graph-based technology to navigate and prioritize risks across multi-cloud landscapes, enhancing accuracy and resilience in safeguarding diverse ecosystems.

thumbnail
Download
I

Related articles

Featured home blog
Icon
Inside Outshift

KubeCon + CloudNativeCon Europe 2024 wrap up: Two key themes and special editions of Cloud Unfiltered

Event Cloud Native
Featured home blog
Icon
Insights

Why integrating AI with graph-based technology is the future of cloud security

Cloud Native Panoptica
Featured home blog
Icon
Insights

Better secret management with Bank-Vaults Secret Sync

Cloud Native
Subscribe
Subscribe
 to
the Shift
!
Get
emerging insights
on emerging technology straight to your inbox.

The Shift keeps you at the forefront of cloud native modern applications, application security, generative AI, quantum computing, and other groundbreaking innovations that are shaping the future of technology.

Outshift Background