INSIGHTS
12 min read
Published on 04/18/2020
Last updated on 04/04/2024
How to write WASM filters for Envoy and deploy it with Istio
Share
Envoy is a high performance, programmable L3/L4 and L7 proxy that many service mesh implementations, such as Istio, are based on. At the core of Envoy's connection and traffic handling are network filters, which, once mixed into filter chains, allow the implementation of higher-order functionalities for access control, transformation, data enrichment, auditing, and so on. You can add new filters to extend Envoy's current feature set with new functionalities. There are two ways to go about doing this:
Create a config map to hold the
Create a config map to hold the Inject the
- Integrate the additional filters into Envoy's source code and compile a new Envoy version. The drawback of this approach is that you need to maintain your own version of Envoy, and continuously keep it in sync with the official distribution. Moreover, since Envoy is implemented in C++, the filter has to be implemented in C++ as well.
- Dynamically load new filters into the Envoy Proxy at runtime.
Read more about running Kafka over Istio on our blog:Our Kafka ACL WASM filter for Envoy reads the client certificate information that comes with mTLS traffic, and extracts the subject field required by Kafka to identify the client. The filter enriches the stream that targets Kafka with the extracted client identity, which Kafka will map to Kafka users. This enables Kafka to automatically authenticate and authorize client applications without configuring SSL on each broker, while maintaining mTLS communication between all interacting services. Moreover, this solution also allows client applications to identify themselves as their Kubernetes service accounts, without the need for the user to create and configure certificates for the client application while running inside the same Istio mesh as the Kafka cluster. Now let's return to the topic of WASM filters, but in greater detail.
Why WASM filters
With WASM filter implementations we get:- Agility - filters can be dynamically loaded into the running Envoy process without the need to stop or re-compile.
- Maintainability - we don't have to change the Envoy's codebase to extend its functionality.
- Diversity - popular programming languages such as C/C++ and Rust can be compiled into WASM, thus developers can implement filters using their programming language of choice.
- Reliability and isolation - filters are deployed into a VM (sandbox), therefore are isolated from the hosting Envoy process itself (e.g. when the WASM filter crashes it will not impact the Envoy process).
- Security - since filters communicate with the host (Envoy Proxy) through a well-defined API, they have access to and can modify only a limited number of connection or request properties.
- Performance is ~70% as fast as native C++.
- Higher memory usage due to the need to start one or more WASM virtual machines.
Envoy Proxy WASM SDK
Envoy Proxy runs WASM filters inside a stack-based virtual machine, thus the filter's memory is isolated from the host environment. All interactions between the embedding host (Envoy Proxy) and the WASM filter are realized through functions and callbacks provided by the Envoy Proxy WASM SDK. The Envoy Proxy WASM SDK has implementations in various programming languages like:- C++
- Rust
- AssemblyScript
- Go - still experimental
class RootContext;
class Context;
When the WASM plugin (the WASM binary that contains the filter) is loaded, a root context is created. The root context has the same lifetime as the VM instance, which executes our filter and is used for:
- interactions at initial setup between your code and the Envoy Proxy
- interactions that outlive a request
class RegisterContextFactory;
the variable will expect the root context factory and the context factory in the form of constructor arguments.
Example filter
Below is a very simple example that shows the skeleton for a WASM filter using the CPP Envoy Proxy WASM SDK: example-filter.cc:#include "proxy_wasm_intrinsics.h"
class ExampleRootContext: public RootContext {
public:
explicit ExampleRootContext(uint32_t id, StringView root_id): RootContext(id, root_id) {}
bool onStart(size_t) override;
};
class ExampleContext: public Context {
public:
explicit ExampleContext(uint32_t id, RootContext* root) : Context(id, root) {}
FilterHeadersStatus onResponseHeaders(uint32_t) override;
FilterStatus onDownstreamData(size_t, bool) override;
};
// register factories for ExampleContext and ExampleRootContext
static RegisterContextFactory register_FilterContext(CONTEXT_FACTORY(ExampleContext),
ROOT_FACTORY(ExampleRootContext),
"my_root_id");
// invoked when the plugin initialised and is ready to process streams
bool ExampleRootContext::onStart(size_t n) {
LOG_DEBUG("ready to process streams");
return true;
}
// invoked when HTTP response header is decoded
FilterHeadersStatus ExampleContext::onResponseHeaders(uint32_t) {
addResponseHeader("resp-header-demo", "added by our filter");
return FilterHeadersStatus::Continue;
}
// invoked when downstream TCP data chunk is received
FilterStatus ExampleContext::onDownstreamData(size_t, bool) {
auto res = setBuffer(WasmBufferType::NetworkDownstreamData, 0, 0, "prepend payload to downstream data");
if (res != WasmResult::Ok) {
LOG_ERROR("Modifying downstream data failed: " + toString(res));
return FilterStatus::StopIteration;
}
return FilterStatus::Continue;
}
Build the filter
The easiest way to build a filter is using Docker as it won't require you to keep various libraries on your local machine.- First, create a docker image with the C++ Envoy Proxy WASM SDK as described, here
- Create Makefile for the WASM filter. Makefile:
.PHONY = all clean PROXY_WASM_CPP_SDK=/sdk all: example-filter.wasm include ${PROXY_WASM_CPP_SDK}/Makefile.base_lite
- Build the WASM filter:
docker run -v $PWD:/work -w /work wasmsdk:v2 /build_wasm.sh
Deploy the WASM filter with Istio
Let's see how you can deploy our Envoy WASM filter for an application running inside an Istio service mesh on Kubernetes. You can quickly spin up an Istio mesh, including a demo application on Kubernetes with Backyards (now Cisco Service Mesh Manager), the Banzai Cloud Istio distribution.backyards install -a --run-demo
With this single command, you get a production-ready and fully operational Istio service mesh and a demo application that consists of multiple microservices running inside the mesh.
Create a config map to hold the wasm
binary
Create a config map to hold the WASM
binary of your filter in the backyards-demo
namespace where the demo application is running.
kubectl create cm -n backyards-demo example-filter --from-file=example-filter.wasm
Inject the wasm
binary into the demo application using Istio
- Inject the
wasm
binary into the frontpage service of our demo application using the following two annotations:sidecar.istio.io/userVolume: '[{"name":"wasmfilters-dir","configMap": {"name": "example-filter"}}]' sidecar.istio.io/userVolumeMount: '[{"mountPath":"/var/local/lib/wasm-filters","name":"wasmfilters-dir"}]'
- Execute the following:
Your WASM filter binary should now be available atkubectl scale deployment -n backyards-demo frontpage-v1 --replicas=1 kubectl patch deployment -n backyards-demo frontpage-v1 -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/userVolume":"[{\"name\":\"wasmfilters-dir\",\"configMap\": {\"name\": \"example-filter\"}}]","sidecar.istio.io/userVolumeMount":"[{\"mountPath\":\"/var/local/lib/wasm-filters\",\"name\":\"wasmfilters-dir\"}]"}}}}}'
/var/local/lib/wasm-filters
in the istio-proxy container:kubectl exec -n backyards-demo -it deployment/frontpage-v1 -c istio-proxy -- ls /var/local/lib/wasm-filters/ example-filter.wasm
-
To enable WASM filters to log at DEBUG log-level when processing traffic which targets the
frontpage
service:kubectl port-forward -n backyards-demo deployment/frontpage-v1 15000 curl -XPOST "localhost:15000/logging?wasm=debug"
-
Insert our WASM filter into the HTTP-level filter chain hooked to the HTTP port 8080:
kubectl apply -f-<<EOF apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: frontpage-v1-examplefilter namespace: backyards-demo spec: configPatches: - applyTo: HTTP_FILTER match: context: SIDECAR_INBOUND proxy: proxyVersion: '^1\.5.*' listener: portNumber: 8080 filterChain: filter: name: envoy.http_connection_manager subFilter: name: envoy.router patch: operation: INSERT_BEFORE value: config: config: name: example-filter rootId: my_root_id vmConfig: code: local: filename: /var/local/lib/wasm-filters/example-filter.wasm runtime: envoy.wasm.runtime.v8 vmId: example-filter allow_precompiled: true name: envoy.filters.http.wasm workloadSelector: labels: app: frontpage version: v1 EOF
Note: in our testing we found that the
portNumber
filter specified for the listener match in theEnvoyFilter
custom resource wasn't handled properly by upstream Istio, resulting in hooks not being invoked on our filter. This issue has been remediated in our Istio distribution, Backyards (now Cisco Service Mesh Manager). - Send some traffic to HTTP port 8080 on the
frontpage
service:
In the response, we expect to see the header of our filter added to the response header:kubectl run curl --image=yauritux/busybox-curl --restart=Never -it --rm sh /home # curl -L -v http://frontpage.backyards-demo:8080
- About to connect() to frontpage.backyards-demo port 8080 (#0) - Trying 10.10.178.38... - Adding handle: conn: 0x10eadbd8 - Adding handle: send: 0 - Adding handle: recv: 0 - Curl_addHandleToPipeline: length: 1 - - Conn 0 (0x10eadbd8) send_pipe: 1, recv_pipe: 0 - Connected to frontpage.backyards-demo (10.10.178.38) port 8080 (#0) > GET / HTTP/1.1 User-Agent: curl/7.30.0 Host: > frontpage.backyards-demo:8080 Accept: _/_ < HTTP/1.1 200 OK < content-type: text/plain < date: Thu, 16 Apr 2020 16:32:20 GMT < content-length: 9 < x-envoy-upstream-service-time: 10 < resp-header-demo: added by our filter < x-envoy-peer-metadata: CjYKDElOU1RBTkNFX0lQUxImGiQxMC4yMC4xLjU3LGZlODA6OmQwNDM6NDdmZjpmZWYwOmVkMjkK2QEKBkxBQkVMUxLOASrLAQoSCgNhcHASCxoJZnJvbnRwYWdlCiEKEXBvZC10ZW1wbGF0ZS1oYXNoEgwaCjU3OGM2NTU0ZDQKJAoZc2VjdXJpdHkuaXN0aW8uaW8vdGxzTW9k ZRIHGgVpc3RpbwouCh9zZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1uYW1lEgsaCWZyb250cGFnZQorCiNzZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1yZXZpc2lvbhIEGgJ2MQoPCgd2ZXJzaW9uEgQaAnYxChoKB01FU0hfSUQSDxoNY2x1c3Rlci5sb2NhbAonCgROQU1FEh8aHWZyb250cGFnZS12MS01N zhjNjU1NGQ0LWxidnFrCh0KCU5BTBZkWWuUHNahSjQZtmeoQYjMvmHe1WYuCTpXCgVPV05FUhJOGkxrdWJlcm5ldGVzOi8vYXBpcy9hcHBzL3YxL25hbWVzcGFjZXMvYmFja3lhcmRzLWRlbW8vZGVwbG95bWVudHMvZnJvbnRwYWdlLXYxCi8KEVBMQVRGT1JNX01FVEFEQVRBEhoqGAoWCgpjbHVzdGVyX2lkEg gaBm1hc3RlcgocCg9TRVJWSUNFX0FDQ09VTBZkWWuUHNahSjQZtmeoQYjMvmHe1WYuCT9OQU1FEg4aDGZyb250cGFnZS12MQ== < x-envoy-peer-metadata-id: sidecar~10.20.1.57~frontpage-v1-578c6554d4-lbvqk.backyards-demo~backyards-demo.svc.cluster.local < x-by-metadata: CjYKDElOU1RBTkNFX0lQUxImGiQxMC4yMC4xLjU3LGZlODA6OmQwNDM6NDdmZjpmZWYwOmVkMjkK2QEKBkxBQkVMUxLOASrLAQoSCgNhcHASCxoJZnJvbnRwYWdlCiEKEXBvZC10ZW1wbGF0ZS1oYXNoEgwaCjU3OGM2NTU0ZDQKJAoZc2VjdXJpdHkuaXN0aW8uaW8vdGxzTW9kZRIHGgVp c3RpbwouCh9zZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1uYW1lEgsaCWZyb250cGFnZQorCiNzZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1yZXZpc2lvbhIEGgJ2MQoPCgd2ZXJzaW9uEgQaAnYxChoKB01FU0hfSUQSDxoNY2x1c3Rlci5sb2NhbAonCgROQU1FEh8aHWZyb250cGFnZS12MS01NzhjNjU1N GQ0LWxidnFrCh0KCU5BTBZkWWuUHNahSjQZtmeoQYjMvmHe1WYuCTpXCgVPV05FUhJOGkxrdWJlcm5ldGVzOi8vYXBpcy9hcHBzL3YxL25hbWVzcGFjZXMvYmFja3lhcmRzLWRlbW8vZGVwbG95bWVudHMvZnJvbnRwYWdlLXYxCi8KEVBMQVRGT1JNX01FVEFEQVRBEhoqGAoWCgpjbHVzdGVyX2lkEggaBm1hc3 RlcgocCg9TRVJWSUNFX0FDQ09VTBZkWWuUHNahSjQZtmeoQYjMvmHe1WYuCT9OQU1FEg4aDGZyb250cGFnZS12MQ== - Server istio-envoy is not blacklisted < server: istio-envoy < x-envoy-decorator-operation: frontpage.backyards-demo.svc.cluster.local:8080/* < - Connection #0 to host frontpage.backyards-demo left intact frontpage ```
-
If you want to register your WASM filter into a TCP filter chain for the
frontpage
service, which accepts TCP connections on port 8083, then theEnvoyFilter
custom resource would look like this:
When the filter is added to a TCP-level filter chain, only hooks specific to TCP traffic will be honoured by the SKD and invoked on the filter.kubectl apply -f-<<EOF apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: frontpage-v1-examplefilter namespace: backyards-demo spec: configPatches: - applyTo: NETWORK_FILTER match: context: SIDECAR_INBOUND proxy: proxyVersion: '^1\.5.*' listener: portNumber: 8083 filterChain: filter: name: "envoy.tcp_proxy" patch: operation: INSERT_BEFORE value: config: config: name: example-filter rootId: my_root_id vmConfig: code: local: filename: /var/local/lib/wasm-filters/example-filter.wasm runtime: envoy.wasm.runtime.v8 vmId: example-filter allow_precompiled: true name: envoy.filters.network.wasm workloadSelector: labels: app: frontpage version: v1 EOF
Write WASM filters for Envoy with WASME
solo.io has provided a solution for developing WASM filters for Envoy which is a WebAssembly hub where people can upload/download their WASM filter binaries. They provide a tool called WASME that helps you to scaffold WASM filters, building and pushing the filters to WebAssembly Hub. When a WASM filter is deployed,wasme
pulls the image that contains the WASM filter plugin from WebAssembly Hub, launches a daemonset to extract the WASM plugin binary from the pulled image and make it available to Envoy Proxies on each node through hostPath
volumes.
Note: the images pulled from WebAssembly Hub would not normally show up as standard Docker imagesSince this solution involves publishing and storing WASM filters to an external central location (WebAssembly Hub) it may not be an option for those enterprises that, due to stringent security policies, (or for any other reason) are unwilling to publish proprietary business logic, even in binary format outside the boundaries of the company network.
Wrap-up
With WASM filters for Envoy, developers can write their custom code, compile it to WASM plugins, and configure Envoy to execute it. These plugins can hold arbitrary logic, so they're useful for all kinds of message integrations and mutations, which makes WASM filters for Envoy Proxy the perfect way for us to integrate Kafka on Kubernetes with Istio. In the next blog post, we'll talk about integrating Kafka's ACL mechanism with Istio mTLS in more detail. If you need new Envoy filters and need help in writing, building, self hosting and delivering them in an automated way, contact us. We are happy to help.Subscribe to
the Shift!
Get emerging insights on emerging technology straight to your inbox.
Unlocking Multi-Cloud Security: Panoptica's Graph-Based Approach
Discover why security teams rely on Panoptica's graph-based technology to navigate and prioritize risks across multi-cloud landscapes, enhancing accuracy and resilience in safeguarding diverse ecosystems.
Subscribe
to
the Shift
!Get on emerging technology straight to your inbox.
emerging insights
The Shift keeps you at the forefront of cloud native modern applications, application security, generative AI, quantum computing, and other groundbreaking innovations that are shaping the future of technology.