What's new in Istio 1.9, a quick walkthrough

Tuesday, April 27th, 2021
Istio 1.9 contains some exciting new features and numerous enhancements. The core focus of the release, however, is to increase operational stability for production users.
In this post, we'll review what's new in Istio 1.9, and highlight a few potential snags to look out for when upgrading to the latest version.
Istio 1.9 changes
High impact changes (can cause issues when upgrading the mesh):
- Supported Kubernetes versions changed
- EnvoyFilter XDS v2 removal
- Service tags added to trace spans
Under the radar changes (smaller changes, but which we believe are interesting/useful):
- Virtual Machine Integration promoted to Beta
- Request/Response Classification promoted to Beta
- Kubernetes Gateway API support
- Improved sidecar injector selection
- Integration with external authorization systems
- Enable Security Token Service (STS) token exchange for XDS flow
- Add OIDC JWT authenticator
- Remote fetch and load of WebAssembly (Wasm) HTTP filters
- Introduce DNS_AUTO_ALLOCATE option
- List sidecar injectors
Other notable changes:
- Enhance access logs
- Add pprof endpoint to pilot-agent
- Allow gRPC logging for pilot
- DestinationRule inheritance
- Mirroring of images on gcr.io
For the full list of changes please refer to the official change notes.
High impact changes
Supported Kubernetes versions changed
For Istio 1.8, the officially supported Kubernetes versions are 1.16, 1.17, 1.18 and 1.19. For Istio 1.9, these are 1.17, 1.18, 1.19 and 1.20.
In Istio 1.9, Kubernetes 1.16 is not officially supported, therefore it is recommended to upgrade to K8s 1.17 before upgrading to Istio 1.9!
EnvoyFilter XDS v2 removal
The XDS v2 API support was removed in the new Envoy version, hence all EnvoyFilters using the old v2 API need to be updated to v3 before upgrading to Istio 1.9.
More info can be found in Istio upgrade docs.
Service tags added to trace spans
Canonical service tags are added to all trace spans by default now.
This feature increases the storage need per span. If
that causes an issue, it can be disabled with the
PILOT_ENABLE_ISTIO_TAGS=false
env var in istiod.
Under the radar changes
Virtual Machine Integration promoted to Beta
In Istio 1.8, experimental features like smart DNS proxying and auto registration for VMs provided features to conveniently include workloads running in VMs as part of the Istio mesh. For Istio 1.9, the stability, testing and documentation were further improved for VM integration and the feature is now promoted to Beta.
One specific area where improvements were delivered for
Istio 1.9 is in the
multi-primary multi-cluster
setup. Previously, the WorkloadEntry
resources needed to
be manually copied to all the clusters so that the VMs could
be reached from every cluster. Because of this, in the
multi-cluster setup the
auto registration feature did not
work automatically, it needed these additional manual
copying steps.
The solution for this in Istio 1.9 is that all
WorkloadEntry resources will be read from all clusters
in multi-cluster installations. This way auto registration
works for multi-primary multi-cluster scenarios as well.
This feature is disabled by default and can be enabled with
the PILOT_ENABLE_CROSS_CLUSTER_WORKLOAD_ENTRY
environment
variable in istiod.
One more aspect to note here is that VMs can be attached to a mesh in single-network and multi-network setups as well.
Here are a few links collected on the Istio webpage to get started with the VM integration feature.
Request/Response Classification promoted to Beta
There is always a need to more precisely monitor application traffic. That is why it's good to see Request Classification as a Beta feature in Istio 1.9.
The Request/Response classification feature makes it possible to easily collect telemetry data based on request parameters (e.g. URL paths) or response parameters (e.g. response codes). As an example, how many GET requests arrived for a specific URL path can be measured.
The feature can be used with EnvoyFilters
using the
AttributeGen plugin.
The resulting attribute can be added as a dimension to a
standard Istio metric (e.g. istio_requests_total
).
Kubernetes Gateway API support
In Kubernetes, there is an ongoing effort to restructure the old ingress API and have a new well-defined API for service networking. This effort is called the Gateway API project.
If this effort rings a bell to you, but you remember it being named to
Service APIs
, don't worry, this one is not yet another project. These two are the same, only theService APIs
project was renamed toGateway API
in February 2021.
The goal of the project is to provide a good common API (which should cover all the common features of different ingress implementations), and built-in integration points for custom features in different implementations.
Implementing this common API opens up opportunity for out-of-the-box integrations between different tools implementing this API. Migrating between different ingress implementations using this API should be painless in the future.
In Istio 1.9, Alpha support was added for this Gateway API. On the long run, the goal is to use this API in Istio by default. To do that, an incremental and slow migration process is aimed to be used in Istio.
Right now the Gateway API can be extended by Istio resources either by:
- explicit composition: concrete reference to Istio resources from the k8s Gateway resource
- implicit composition: Istio can layer the resources together without concrete references
Here's a quick intro on the subject by John Howard presented at IstioCon and a getting started guide for the Gateway API on the Istio page.
Improved sidecar injector selection
The Istio sidecar injector webhook now better utilizes pod labels to determine if injection is required.
There were two limitations with the previous implementation:
- Opting out individual pods with the
sidecar.istio.io/inject: false
label from an auto injected namespace still triggered the webhook and the pod was only filtered out in istiod's mutating webhook implementation. - It was not possible to opt in a single pod with
sidecar.istio.io/inject: true
label if auto injection was not enabled for the whole namespace.
In the new solution, the mutating webhook selection logic
relies more on native Kubernetes features, namely the
ObjectSelector
field, instead istiod's mutating webhook
endpoint implementation. This way the aforementioned two
issues are solved and overall a better sidecar injection
experience is provided based on namespace and pod labels.
The feature is not enabled by default in Istio 1.9, but can
be tested using
values.sidecarInjectorWebhook.useLegacySelectors=false
setting. It is planned to be enabled by default in Istio
1.10.
Security features
Integration with external authorization systems
There are cases where Istio users would like to use their
own external authorization system (e.g. OPA, OAuth2, etc.)
for their applications and not the built-in authorization
solutions provided by Istio. Prior to Istio 1.9, this could
only be done with some EnvoyFilter
magic, which was not
straightforward, and as we know EnvoyFilters are not very
reliable.
To enhance that, in Istio 1.9 experimental support was added to use external authorization systems in a more convenient way. This is a high level view of how the feature works (borrowed from this blog post):
This feature can be enabled by using the CUSTOM
action in
Istio's
Authorization Policy CR spec.
Here's an example where the authorization of the requests
arriving to the foo-gateway
is delegated to the
custom-authz
authorization extension:
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: ext-authz
namespace: foo
spec:
selector:
matchLabels:
app: foo-gateway
action: CUSTOM
provider:
name: "custom-authz"
Here's how you can get started with the feature, there is a blog post and a video, which explain the feature in greater detail.
Enable (Security Token Service) STS token exchange for XDS flow
It is possible now to exchange a Kubernetes token for another token, which can be authenticated by XDS.
Add OIDC JWT authenticator
There's a new OIDC JWT authenticator
added to Istio 1.9,
which supports both JWKS-URI and OIDC discovery. The feature
can be enabled with the JWT_RULE
environment variable.
Experimental Features
Remote fetch and load of WebAssembly (Wasm) HTTP filters
Istio 1.9 introduced an experimental feature to fetch WASM modules from remote locations and dynamically load them without the need of restarting the proxies. This way custom C++ code can be injected to proxies easily. Allowing advanced use cases to be handled that are not possible with the existing Istio APIs.
To use this feature a new
EnvoyFilter.ApplyTo
field called EXTENSION_CONFIG
was introduced. This new
field distributes the provided configuration as an Envoy
Extension Configuration Discovery Service
(ECDS) resource. When the istio-agent in an istio-proxy
container receives this update, it downloads the remote WASM
filter from the remote URL and stores its content in its
local file system.
Here's a (not complete!) example how this could be used:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: basic-auth-config
namespace: istio-system
spec:
configPatches:
- applyTo: EXTENSION_CONFIG
match:
context: GATEWAY
patch:
operation: ADD
value:
name: istio.basic_auth
typed_config:
"@type": type.googleapis.com/udpa.type.v1.TypedStruct
type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
value:
config:
vm_config:
vm_id: basic-auth
runtime: envoy.wasm.runtime.v8
code:
remote:
http_uri:
uri: https://github.com/istio-ecosystem/wasm-extensions/releases/download/1.9.2/basic-auth.wasm
# The configuration for the Wasm extension itself
configuration:
"@type": type.googleapis.com/google.protobuf.StringValue
value: |
{
"basic_auth_rules": [
{
"prefix": "/productpage",
"request_methods":[ "GET", "POST" ],
"credentials":[ "ok:test", "YWRtaW4zOmFkbWluMw==" ]
}
]
}
Here's the getting started doc for the WASM remote fetch and load feature.
Introduce DNS_AUTO_ALLOCATE option
Prior to Istio 1.9, the DNS_CAPTURE
flag controlled both
substituting for kube-dns for VMs and auto allocation of
ServiceEntry
addresses. In Istio 1.9, these features are
now
decoupled.
The DNS_CAPTURE
flag now only controls kube-dns DNS, while
the new experimental DNS_AUTO_ALLOCATE
flag enables the
automatic ServiceEntry
address allocation.
More info can be found in the Istio docs on DNS proxying.
List sidecar injectors
An experimental istioctl x injector list
command was
added to show
which namespaces are using which Istio revision and how many
proxies are up-to-date/need restart:
$ istioctl x injector list
NAMESPACE ISTIO-REVISION POD-REVISIONS
backend 18x 18x: 9
frontend 19x 18x: 4 NEEDS RESTART: 4
This is particularly useful during an Istio control plane canary upgrade.
Other notable changes
Enhance access logs
When debugging Istio, you often need to check the Envoy access logs to figure out what's wrong with your application. That's why it is welcome to see that the default access log format in Istio has been supplemented with two new fields, which might help debugging in some cases.
These fields are RESPONSE_CODE_DETAILS
and
CONNECTION_TERMINATION_DETAILS
, this is how they might
look like in logs:
RESPONSE_CODE_DETAILS (
via_upstream
,filter_chain_not_found
,http1.codec_error
,rbac_access_denied_matched_policy[none]
):[2020-10-29T03:22:16.126Z] "GET /headers HTTP/1.1" 200 - via_upstream - "-" 0 547 5 4 "-" "curl/7.73.0-DEV" "24e58e43-e576-9048-829b-02e3f3da16f0" "httpbin:8000" "127.0.0.1:80" inbound|8000|| 127.0.0.1:51680 172.17.0.7:80 172.17.0.6:59022 outbound_.8000_._.httpbin.foo.svc.cluster.local default [2020-10-29T03:23:33.415Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 172.17.0.7:80 172.17.0.6:59800 - - [2020-10-29T03:24:13.520Z] "- - HTTP/1.1" 400 DPE http1.codec_error - "-" 0 11 0 - "-" "-" "-" "-" "-" - - 172.17.0.7:80 172.17.0.6:60214 - - [2020-10-29T03:25:20.162Z] "GET /headers HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-" "curl/7.73.0-DEV" "07541777-6ee2-908f-a78b-867fe257ac80" "httpbin:8000" "-" - - 172.17.0.7:80 172.17.0.6:60898 outbound_.8000_._.httpbin.foo.svc.cluster.local - [2020-10-29T03:26:11.561Z] "GET /headers HTTP/1.1" 403 - rbac_access_denied_matched_policy[ns[foo]-policy[httpbin]-rule[0]] - "-" 0 19 0 - "-" "curl/7.73.0-DEV" "090a1897-d696-99b1-90f0-2673113ba258" "httpbin:8000" "-" - - 172.17.0.7:80 172.17.0.6:33204 outbound_.8000_._.httpbin.foo.svc.cluster.local -
CONNECTION_TERMINATION_DETAILS (
rbac_access_denied_matched_policy[ns[foo]-policy[httpbin]-rule[0]]
):[2020-10-29T03:28:26.690Z] "- - -" 0 - - rbac_access_denied_matched_policy[ns[foo]-policy[httpbin]-rule[0]] "-" 761 701 5 - "-" "-" "-" "-" "127.0.0.1:80" inbound|8000|| 127.0.0.1:55464 172.17.0.7:80 172.17.0.6:34574 outbound_.8000_._.httpbin.foo.svc.cluster.local -
Add pprof endpoint to pilot-agent
New pprof profiling debug endpoints were added to pilot-agent for easier debugging of performance issues, memory leaks, etc.
Allow gRPC logging for pilot
gRPC logging can be enabled for pilot
with the --log_output_level=grpc:debug
setting.
DestinationRule inheritance
If you are familiar with the PeerAuthentication
and
Sidecar
Istio resources, you might know that there are
mesh-level, namespace-level, and workload-level
customizations for them. The settings are inherited from
their "parents" by default, and the "child" CRs can override
those settings.
The same architecture now applies for DestinationRules
as
well, support for
DestinationRule inheritance from mesh/namespace level
were added. The feature is off by default and can be enabled
with the PILOT_ENABLE_DESTINATION_RULE_INHERITANCE
environment variable in istiod.
Mirroring of images on gcr.io
Last, but not least, to
mitigate
DockerHub's rate limiting restrictions, all Istio images are
published to gcr.io/istio-release
as well.
Takeaway
Istio 1.9 mostly improved operational stability, which will likely help increase production adoption in the future.