Certificate management on Istio
Recently, we blogged about certificate management on Kubernetes. Today, we'll be returning to that topic, but we'll be focusing on the differences an Istio service mesh makes.
The primary difference is the method of solving
Solving this challenge involves routing an HTTP request from
the ACME server (the Certificate Authority) to the
cert-manager challenge solver pod.
Cert management with Istio
As we saw in our previous blog post, you can route such a
challenge request by using a Kubernetes Ingress gateway.
When using Istio, that same method is still viable, provided
that Kubernetes Ingress support is enabled in your Istio
config. If it's enabled, and you're using Kubernetes
Ingress resources for routing the traffic entering your
cluster, everything should work just fine. However, you will
be limited to using one ingress gateway for all your ingress
traffic. This can be a security issue, and it also prevents
you from setting up separate ingress gateways for your
different teams or products. Additionally, this will cause
you to miss out on some Istio features.
Considering these limitations, it usually makes sense to use native Istio routing.
Cert management using Istio
Using native Istio features for ingress gives you more freedom and flexibility:
- you can direct traffic for different domains to separate ingress gateways
- you can set up routing for HTTP and HTTPS traffic on non-standard ports
- you also get to use the familiar Istio features (circuit breaker, metrics, etc.)
While technically it's possible to use Kubernetes
and Istio resources together to configure the same Istio
ingress gateway, conflicts between them can easily cause
difficult to diagnose problems; you should stick to using
one or the other for a given ingress gateway.
All in all, if you have Istio running in your cluster, it's a good idea to use it for your ingress gateways, too.
So, what does this mean in terms of cert management
automation? Well, cert-manager doesn't support Istio at the
moment. There are solutions, though. When using an ACME
issuer with challenge type
HTTP-01, you will end up with
Ingress resource for routing the ACME challenge
request. If you disabled Kubernetes
Ingress support in
Istio (or you specified a non-existent ingress class in the
Issuer resource), then no ingress controller will react to
Ingress resource, therefore, no HTTP request will be
routed to the ACME solver pod(s).
What can we do? Two possible solutions: translate the
Ingress resource into
resources, or add Istio support to cert-manager.
Obviously, adding Istio support to cert-manager would be the perfect solution, but it's a more involved process. Nonetheless, at Banzai Cloud, we're already working on just such an implementation. In fact, we have a functional solution, although it needs some more love before it's production ready (cert-manager#3011).
While we're waiting for direct Istio support in cert-manager
to be released, we can make do with another solution:
Ingress resource with Istio resources.
It's possible to do this by hand, but usually the reason we
use the ACME protocol is automation, so introducing an error
prone and repetitive manual step into that process sort of
defeats the point.
When properly automated, this can serve as an effective solution, though. The good news is, if you're a Backyards (now Cisco Service Mesh Manager) user, you can try it out right now, because Backyards 1.3 includes this feature. If not, then feel free to use the evaluation version!
Try it out!
open the Backyards dashboard:
$ backyards dashboard
Gatewaysfrom the menu on the left
select the gateway you want secured. You can select whichever you like, but make sure the "Service Type" is "LoadBalancer". This selection determines the load balancer and, therefore, the IP address(es) that will be used for the ACME
HTTP-01challenge. In our case, it's
point your domain name to the IP address or DNS name found in the
Ports & Hostssection, click on
create newin the upper right corner
fill in the fields:
you will probably want HTTPS protocol on the standard port
enter your domain name(s) for
hosts. You can enter several by hitting enter after each one.
You can also set up a domain under
.banzaicloud.iofor testing purposes if you don't want to use your own domain name. Just click on the checkbox to the right of
use .banzaicloud.ioand a new domain name will be generated for you, then add a subdomain under the generated domain name.
click on the checkbox
Use Let's Encrypt for TLSto get a certificate for your domain from Let's Encrypt
enter your email address. This address will be for ACME account management, so will be sent to Let's Encrypt.
As you can see, two more items have appeared in the
Ports & Hosts list, both for the host
frontpage.qahqab.backyards.banzaicloud.io. One of them on
port 443, and the other on port 80. The latter one is for
solving the ACME
HTTP-01 challenge created by translating
While the certificate is being acquired, you can look up and
see the contents of related
Ingressresource created by cert-manager:
apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: kubernetes.io/ingress.class: istio labels: acme.cert-manager.io/http-domain: "1270878147" acme.cert-manager.io/http-token: "1101532839" acme.cert-manager.io/http01-solver: "true" name: cm-acme-http-solver-nnpg2 namespace: istio-system spec: rules: - host: frontpage.qahqab.backyards.banzaicloud.io http: paths: - backend: serviceName: cm-acme-http-solver-rrg8c servicePort: 8089 path: /.well-known/acme-challenge/XMMsPcDx1KkqfG1DE6xsgjyoh9r697I9BXkgwnr_45s
VirtualServicecreated by Backyards from the above
apiVersion: networking.istio.io/v1beta1 kind: Gateway metadata: labels: acme.backyards.banzaicloud.io/http-domain: "1270878147" acme.backyards.banzaicloud.io/http-token: "1101532839" acme.backyards.banzaicloud.io/http01-solver: "true" name: cm-acme-http-solver-nnpg2-czvnv namespace: istio-system spec: selector: app: istio-ingressgateway istio: ingressgateway servers: - hosts: - istio-system/frontpage.qahqab.backyards.banzaicloud.io port: name: http number: 80 protocol: HTTP
apiVersion: networking.istio.io/v1beta1 kind: VirtualService metadata: labels: acme.backyards.banzaicloud.io/http-domain: "1270878147" acme.backyards.banzaicloud.io/http-token: "1101532839" acme.backyards.banzaicloud.io/http01-solver: "true" name: cm-acme-http-solver-nnpg2-tv22w namespace: istio-system spec: exportTo: - "*" gateways: - istio-system/cm-acme-http-solver-nnpg2-czvnv hosts: - frontpage.qahqab.backyards.banzaicloud.io http: - match: - uri: exact: /.well-known/acme-challenge/XMMsPcDx1KkqfG1DE6xsgjyoh9r697I9BXkgwnr_45s route: - destination: host: cm-acme-http-solver-rrg8c.istio-system.svc.cluster.local port: number: 8089
As you can see, these two resources describe the same
routing path as the
After a short while the item with port 80 and protocol HTTP will disappear, and a green checkmark will appear next to HTTPS. This signals that the certificate was issued and is being used for securing your domain:
Let's see if that's really the case:
$ curl -v https://frontpage.qahqab.backyards.banzaicloud.io ... * Server certificate: * subject: CN=frontpage.qahqab.backyards.banzaicloud.io * start date: Jul 14 10:42:43 2020 GMT * expire date: Oct 12 10:42:43 2020 GMT * subjectAltName: host "frontpage.qahqab.backyards.banzaicloud.io" matched cert's "frontpage.qahqab.backyards.banzaicloud.io" * issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3 * SSL certificate verify ok. ... < HTTP/2 404 < date: Tue, 14 Jul 2020 11:44:09 GMT < server: istio-envoy
As you can see, the connection is secured with a certificate
from Let's Encrypt. Did you notice that 404, though? That's
because we haven't set up any routing yet. We can do that
now, so head over to the
routes tab, click
create new in
the upper right corner and fill in the fields. At the very
least, the gateway, host and port number need to be filled
in, but, depending on your requirements, you might want to
specify other values and settings, too.
When you're ready, click
Create. Now that routing is set
up for our demo service, let's see if it works:
$ curl https://frontpage.qahqab.backyards.banzaicloud.io frontpage
You can also try it in your browser to inspect the certificate more easily.
As we have seen in this blog post, while cert-manager doesn't directly support Istio, it's possible to make it work. And with Backyards, it's not only possible but seamless, too.