Outshift Logo

PRODUCT

11 min read

Blog thumbnail

by Laszlo Bence Nagy

Published on 04/26/2021
Last updated on 03/21/2024

What's new in Istio 1.9, a quick walkthrough

Share

Istio 1.9 contains some exciting new features and numerous enhancements. The core focus of the release, however, is to increase operational stability for production users. In this post, we'll review what's new in Istio 1.9, and highlight a few potential snags to look out for when upgrading to the latest version.

Istio 1.9 changes

High impact changes (can cause issues when upgrading the mesh): Under the radar changes (smaller changes, but which we believe are interesting/useful): Security features: Experimental features: Other notable changes:
For the full list of changes please refer to the official change notes.

High impact changes

Supported Kubernetes versions changed

For Istio 1.8, the officially supported Kubernetes versions are 1.16, 1.17, 1.18 and 1.19. For Istio 1.9, these are 1.17, 1.18, 1.19 and 1.20.

In Istio 1.9, Kubernetes 1.16 is not officially supported, therefore it is recommended to upgrade to K8s 1.17 before upgrading to Istio 1.9!

EnvoyFilter XDS v2 removal

The XDS v2 API support was removed in the new Envoy version, hence all EnvoyFilters using the old v2 API need to be updated to v3 before upgrading to Istio 1.9.
More info can be found in Istio upgrade docs.

Service tags added to trace spansCanonical service tags are added to all trace spans by default now.

This feature increases the storage need per span. If that causes an issue, it can be disabled with the PILOT_ENABLE_ISTIO_TAGS=false env var in istiod.

Under the radar changes

Virtual Machine Integration promoted to Beta

In Istio 1.8, experimental features like smart DNS proxying and auto registration for VMs provided features to conveniently include workloads running in VMs as part of the Istio mesh. For Istio 1.9, the stability, testing and documentation were further improved for VM integration and the feature is now promoted to Beta. Virtual Machine Integration One specific area where improvements were delivered for Istio 1.9 is in the multi-primary multi-cluster setup. Previously, the WorkloadEntry resources needed to be manually copied to all the clusters so that the VMs could be reached from every cluster. Because of this, in the multi-cluster setup the auto registration feature did not work automatically, it needed these additional manual copying steps. The solution for this in Istio 1.9 is that all WorkloadEntry resources will be read from all clusters in multi-cluster installations. This way auto registration works for multi-primary multi-cluster scenarios as well. This feature is disabled by default and can be enabled with the PILOT_ENABLE_CROSS_CLUSTER_WORKLOAD_ENTRY environment variable in istiod.
One more aspect to note here is that VMs can be attached to a mesh in single-network and multi-network setups as well.
Here are a few links collected on the Istio webpage to get started with the VM integration feature.

Request/Response Classification promoted to Beta

There is always a need to more precisely monitor application traffic. That is why it's good to see Request Classification as a Beta feature in Istio 1.9.

The Request/Response classification feature makes it possible to easily collect telemetry data based on request parameters (e.g. URL paths) or response parameters (e.g. response codes). As an example, how many GET requests arrived for a specific URL path can be measured.

The feature can be used with EnvoyFilters using the AttributeGen plugin. The resulting attribute can be added as a dimension to a standard Istio metric (e.g. istio_requests_total).

Kubernetes Gateway API support

In Kubernetes, there is an ongoing effort to restructure the old ingress API and have a new well-defined API for service networking. This effort is called the Gateway API project.

If this effort rings a bell to you, but you remember it being named to Service APIs, don't worry, this one is not yet another project. These two are the same, only the Service APIs project was renamed to Gateway API in February 2021.

The goal of the project is to provide a good common API (which should cover all the common features of different ingress implementations), and built-in integration points for custom features in different implementations.

Implementing this common API opens up opportunity for out-of-the-box integrations between different tools implementing this API. Migrating between different ingress implementations using this API should be painless in the future. Kubernetes Gateway API support In Istio 1.9, Alpha support was added for this Gateway API. On the long run, the goal is to use this API in Istio by default. To do that, an incremental and slow migration process is aimed to be used in Istio. Right now the Gateway API can be extended by Istio resources either by:
  • explicit composition: concrete reference to Istio resources from the k8s Gateway resource
  • implicit composition: Istio can layer the resources together without concrete references
Here's a quick intro on the subject by John Howard presented at IstioCon and a getting started guide for the Gateway API on the Istio page.

Improved sidecar injector selection

The Istio sidecar injector webhook now better utilizes pod labels to determine if injection is required. There were two limitations with the previous implementation:
  • Opting out individual pods with the sidecar.istio.io/inject: false label from an auto injected namespace still triggered the webhook and the pod was only filtered out in istiod's mutating webhook implementation.
  • It was not possible to opt in a single pod with sidecar.istio.io/inject: true label if auto injection was not enabled for the whole namespace.
In the new solution, the mutating webhook selection logic relies more on native Kubernetes features, namely the ObjectSelector field, instead istiod's mutating webhook endpoint implementation. This way the aforementioned two issues are solved and overall a better sidecar injection experience is provided based on namespace and pod labels. The feature is not enabled by default in Istio 1.9, but can be tested using values.sidecarInjectorWebhook.useLegacySelectors=false setting. It is planned to be enabled by default in Istio 1.10.

Security features

Integration with external authorization systems

There are cases where Istio users would like to use their own external authorization system (e.g. OPA, OAuth2, etc.) for their applications and not the built-in authorization solutions provided by Istio. Prior to Istio 1.9, this could only be done with some EnvoyFilter magic, which was not straightforward, and as we know EnvoyFilters are not very reliable.
To enhance that, in Istio 1.9 experimental support was added to use external authorization systems in a more convenient way. This is a high level view of how the feature works (borrowed from this blog post): Security features

This feature can be enabled by using the CUSTOM action in Istio's Authorization Policy CR spec. Here's an example where the authorization of the requests arriving to the foo-gateway is delegated to the custom-authz authorization extension:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: ext-authz
  namespace: foo
spec:
  selector:
    matchLabels:
      app: foo-gateway
  action: CUSTOM
  provider:
    name: "custom-authz"
Here's how you can get started with the feature, there is a blog post and a video, which explain the feature in greater detail.

Enable (Security Token Service) STS token exchange for XDS flow

It is possible now to exchange a Kubernetes token for another token, which can be authenticated by XDS.

Add OIDC JWT authenticator

There's a new OIDC JWT authenticator added to Istio 1.9, which supports both JWKS-URI and OIDC discovery. The feature can be enabled with the JWT_RULE environment variable.

Experimental Features

Remote fetch and load of WebAssembly (Wasm) HTTP filters

Istio 1.9 introduced an experimental feature to fetch WASM modules from remote locations and dynamically load them without the need of restarting the proxies. This way custom C++ code can be injected to proxies easily. Allowing advanced use cases to be handled that are not possible with the existing Istio APIs. To use this feature a new EnvoyFilter.ApplyTo field called EXTENSION_CONFIG was introduced. This new field distributes the provided configuration as an Envoy Extension Configuration Discovery Service (ECDS) resource. When the istio-agent in an istio-proxy container receives this update, it downloads the remote WASM filter from the remote URL and stores its content in its local file system. Here's a (not complete!) example how this could be used: 
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: basic-auth-config
  namespace: istio-system
spec:
  configPatches:
    - applyTo: EXTENSION_CONFIG
      match:
        context: GATEWAY
      patch:
        operation: ADD
        value:
          name: istio.basic_auth
          typed_config:
            "@type": type.googleapis.com/udpa.type.v1.TypedStruct
            type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
            value:
              config:
                vm_config:
                  vm_id: basic-auth
                  runtime: envoy.wasm.runtime.v8
                  code:
                    remote:
                      http_uri:
                        uri: https://github.com/istio-ecosystem/wasm-extensions/releases/download/1.9.2/basic-auth.wasm
                # The configuration for the Wasm extension itself
                configuration:
                  "@type": type.googleapis.com/google.protobuf.StringValue
                  value: |
                    {
                      "basic_auth_rules": [
                        {
                          "prefix": "/productpage",
                          "request_methods":[ "GET", "POST" ],
                          "credentials":[ "ok:test", "YWRtaW4zOmFkbWluMw==" ]
                        }
                      ]
                    }
Here's the getting started doc for the WASM remote fetch and load feature.

Introduce DNS_AUTO_ALLOCATE option

Prior to Istio 1.9, the DNS_CAPTURE flag controlled both substituting for kube-dns for VMs and auto allocation of ServiceEntry addresses. In Istio 1.9, these features are now decoupled. The DNS_CAPTURE flag now only controls kube-dns DNS, while the new experimental DNS_AUTO_ALLOCATE flag enables the automatic ServiceEntry address allocation.
More info can be found in theIstio docs on DNS proxying.

List sidecar injectors

An experimental istioctl x injector list command was added to show which namespaces are using which Istio revision and how many proxies are up-to-date/need restart: 
$ istioctl x injector list
NAMESPACE  ISTIO-REVISION POD-REVISIONS
backend    18x            18x: 9
frontend   19x            18x: 4 NEEDS RESTART: 4
This is particularly useful during an Istio control plane canary upgrade.

Other notable changes

Enhance access logs

When debugging Istio, you often need to check the Envoy access logs to figure out what's wrong with your application. That's why it is welcome to see that the default access log format in Istio has been supplemented with two new fields, which might help debugging in some cases. These fields are RESPONSE_CODE_DETAILS and CONNECTION_TERMINATION_DETAILS, this is how they might look like in logs:
  • RESPONSE_CODE_DETAILS (via_upstream , filter_chain_not_found, http1.codec_error, rbac_access_denied_matched_policy[none]): 
    [2020-10-29T03:22:16.126Z] "GET /headers HTTP/1.1" 200 - via_upstream - "-" 0 547 5 4 "-" "curl/7.73.0-DEV" "24e58e43-e576-9048-829b-02e3f3da16f0" "httpbin:8000" "127.0.0.1:80" inbound|8000|| 127.0.0.1:51680 172.17.0.7:80 172.17.0.6:59022 outbound_.8000_._.httpbin.foo.svc.cluster.local default
[2020-10-29T03:23:33.415Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 172.17.0.7:80 172.17.0.6:59800 - -
[2020-10-29T03:24:13.520Z] "- - HTTP/1.1" 400 DPE http1.codec_error - "-" 0 11 0 - "-" "-" "-" "-" "-" - - 172.17.0.7:80 172.17.0.6:60214 - -
[2020-10-29T03:25:20.162Z] "GET /headers HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-" "curl/7.73.0-DEV" "07541777-6ee2-908f-a78b-867fe257ac80" "httpbin:8000" "-" - - 172.17.0.7:80 172.17.0.6:60898 outbound_.8000_._.httpbin.foo.svc.cluster.local -
[2020-10-29T03:26:11.561Z] "GET /headers HTTP/1.1" 403 - rbac_access_denied_matched_policy[ns[foo]-policy[httpbin]-rule[0]] - "-" 0 19 0 - "-" "curl/7.73.0-DEV" "090a1897-d696-99b1-90f0-2673113ba258" "httpbin:8000" "-" - - 172.17.0.7:80 172.17.0.6:33204 outbound_.8000_._.httpbin.foo.svc.cluster.local -
  • CONNECTION_TERMINATION_DETAILS (rbac_access_denied_matched_policy[ns[foo]-policy[httpbin]-rule[0]]): 
    [2020-10-29T03:22:16.126Z] "GET /headers HTTP/1.1" 200 - via_upstream - "-" 0 547 5 4 "-" "curl/7.73.0-DEV" "24e58e43-e576-9048-829b-02e3f3da16f0" "httpbin:8000" "127.0.0.1:80" inbound|8000|| 127.0.0.1:51680 172.17.0.7:80 172.17.0.6:59022 outbound_.8000_._.httpbin.foo.svc.cluster.local default
[2020-10-29T03:23:33.415Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 172.17.0.7:80 172.17.0.6:59800 - -
[2020-10-29T03:24:13.520Z] "- - HTTP/1.1" 400 DPE http1.codec_error - "-" 0 11 0 - "-" "-" "-" "-" "-" - - 172.17.0.7:80 172.17.0.6:60214 - -
[2020-10-29T03:25:20.162Z] "GET /headers HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-" "curl/7.73.0-DEV" "07541777-6ee2-908f-a78b-867fe257ac80" "httpbin:8000" "-" - - 172.17.0.7:80 172.17.0.6:60898 outbound_.8000_._.httpbin.foo.svc.cluster.local -
[2020-10-29T03:26:11.561Z] "GET /headers HTTP/1.1" 403 - rbac_access_denied_matched_policy[ns[foo]-policy[httpbin]-rule[0]] - "-" 0 19 0 - "-" "curl/7.73.0-DEV" "090a1897-d696-99b1-90f0-2673113ba258" "httpbin:8000" "-" - - 172.17.0.7:80 172.17.0.6:33204 outbound_.8000_._.httpbin.foo.svc.cluster.local -

Add pprof endpoint to pilot-agent

New pprof profiling debug endpoints were added to pilot-agent for easier debugging of performance issues, memory leaks, etc.

Allow gRPC logging for pilot

gRPC logging can be enabled for pilot with the --log_output_level=grpc:debug setting.

DestinationRule inheritance

If you are familiar with the PeerAuthentication and Sidecar Istio resources, you might know that there are mesh-level, namespace-level, and workload-level customizations for them. The settings are inherited from their "parents" by default, and the "child" CRs can override those settings. The same architecture now applies for DestinationRules as well, support for DestinationRule inheritance from mesh/namespace level were added. The feature is off by default and can be enabled with the PILOT_ENABLE_DESTINATION_RULE_INHERITANCE environment variable in istiod.

Mirroring of images on gcr.io

Last, but not least, to mitigate DockerHub's rate limiting restrictions, all Istio images are published to gcr.io/istio-release as well.

Takeaway

Istio 1.9 mostly improved operational stability, which will likely help increase production adoption in the future.
Subscribe card background
Subscribe
Subscribe to
the Shift!

Get emerging insights on emerging technology straight to your inbox.

Unlocking Multi-Cloud Security: Panoptica's Graph-Based Approach

Discover why security teams rely on Panoptica's graph-based technology to navigate and prioritize risks across multi-cloud landscapes, enhancing accuracy and resilience in safeguarding diverse ecosystems.

thumbnail
Download
I

Related articles

Featured home blog
Icon
Insights

Top 15 software supply chain attacks: Case studies

Cloud Native Kubernetes
Featured home blog
Icon
Insights

Platform engineering for cloud-native applications developments

Cloud Native Kubernetes
Featured home blog
Icon
Product

KubeClarity’s cloud security tools: Architecture deep dive

Kubernetes Cloud Native
Subscribe
Subscribe
 to
the Shift
!
Get
emerging insights
on emerging technology straight to your inbox.

The Shift keeps you at the forefront of cloud native modern applications, application security, generative AI, quantum computing, and other groundbreaking innovations that are shaping the future of technology.

Outshift Background