Today we are happy to announce the 1.1 release of Backyards (now Cisco Service Mesh Manager), Banzai Cloud's automated and operationalized service mesh product built on Istio.
This is an announcement post describing the new features of Backyards (now Cisco Service Mesh Manager). If you're not familiar with Backyards (now Cisco Service Mesh Manager) yet, and want to know why we decided to build this product, we suggest reading the blog post about the first major release.
In Backyards 1.1 we've added the following major features:
- Multi-cluster meshes can be managed from the CLI: use
backyards istio cluster attachto add a new cluster to a mesh.
- Outbound traffic from the Envoy proxies can be restricted through the Sidecar resource to reduce resource consumption
- The traffic tap feature helps debugging HTTP traffic that flows through an Envoy sidecar proxy
- mTLS settings can be configured for individual namespaces or services through the dashboard or the CLI
- Traffic to external services can be monitored through the dashboard's topology view
- TCP throughput metrics are now properly displayed on the dashboard
- The Overview page shows a summary of the service mesh with global metrics, global settings, and validations
- Multiple ingress and egress gateways can be added through Kubernetes custom resources
We have been working hard over the previous two and a half months to push as many things into this release as possible, and the length of the changelog shows it. Previously we had already published previews of two major features in this release: support for sidecar resources, and building multi cluster service meshes from the CLI. We'll recap these first, then we'll continue with the remaining major features.
Multi-cluster meshes from the CLI
We noticed increased interest in hybrid- and multi-cloud solutions at most of the companies we had discussions with. Use-cases can range from scaling out to the public cloud for peak workloads, through cost effectiveness to cloud migration. But regardless of the reason, many installations need to be able to span across multiple clusters.
The service mesh can be a good fit for some of these use-cases by connecting clusters and handling the complexity of inter-cluster communication. Backyards - and the underlying Banzai Cloud Istio operator - have been built with these scenarios in mind from day one. Backyards (now Cisco Service Mesh Manager) collects and displays telemetry data properly from these clusters, and makes routing and policy configuration as easy as in a single cluster. There had been one major feature still missing: the ability to manage (attach, detach, display) clusters easily using Backyards.
From the 1.1 release, the CLI can be used to build a
multi-cluster mesh with two commands: a
backyards install on one cluster to install the control
plane, and a
backyards attach with the
Kubeconfig of the
second cluster to add in.
We've already published a blog post about this topic, check it out if you want to read the details.
Fine tune sidecar proxy configuration
Istio's Envoy proxies can consume a lot of memory if not configured properly. By default Pilot automatically pushes the same configuration to all proxies in the mesh. That configuration contains information about every workload in the mesh.
Sidecar resource can be used to fine tune the
configuration received by the sidecar proxies. For example,
you can restrict the set of services that the proxy can
reach to specific workloads or namespaces. It means a
smaller configuration to handle for Envoy, and therefore
reduced memory usage.
From the 1.1 release, Backyards can be used to easily set up outbound traffic restrictions using the dashboard or the CLI. It even offers recommendations by monitoring and analyzing the current traffic between services.
To learn more about the sidecar resource, read the corresponding blog post.
It is (in)famously hard to debug connection errors in Istio when something goes wrong. One of the main value propositions of Backyards is to help diagnosing and debugging issues within the service mesh. The new traffic tap feature is a further step in this direction.
This feature brings you an option to easily trace all the HTTP requests that flow through a specific Envoy proxy. Just select a workload from the dashboard, and you'll see the inbound and outbound requests from that proxy. This can help you gain some insight into whether your requests are heading to the right destination, or what the failure cause might be.
We'll write a post about the tap feature soon, stay tuned for the details and examples.
Advanced mTLS policy settings
One of the exciting new features of Istio 1.4 was automatic
mutual TLS support, which brought some long-awaited
convenience to Istio users configuring mTLS for their
applications. In previous Istio releases, it was necessary
to create Destination Rules to use mTLS. The new automatic
mTLS feature allows you to adopt mutual TLS without creating
The Banzai Cloud
Istio operator -
underlying Backyards - have had this feature
configurable since the 1.4 release. But with the new
release, Backyards further simplifies the mTLS concept by
providing easy to understand CLI commands, and an intuitive
UI to configure mutual TLS for specific services,
namespaces, or for the whole mesh. On one hand it makes it
easier to decide if a specific service is using mTLS or not.
It's not always that as easy as it seems: there are
MeshPolicies, namespace-wide and workload-wide
and who knows without looking at the docs if the CR is valid
in a different namespace or not. On the other hand,
Backyards makes it easy to configure mTLS. No more YAML
syntax errors, hard to debug typos, or mis-specified
Displaying external services
An Istio service mesh has a few different ways of reaching
services that are external to the mesh. External services
are everything that are not defined in Istio’s internal
service registry. The default setting allows all outbound
traffic through an Envoy
Passthrough listener. In a
production environment it is recommended to only allow
outbound traffic that's registered with Istio through
Service Entries. Otherwise traffic is trapped with an
The above scenarios are now properly displayed in the
Backyards (now Cisco Service Mesh Manager)
topology view, external traffic is shown either through
Service Entry nodes.
Outbound traffic policy mode is configurable from the CLI
backyards istio outbound-traffic-policy command.
In this release,
Service Entries are display only,
configuration through Backyards is on our roadmap.
Handle TCP metrics
In previous Backyards (now Cisco Service Mesh Manager) releases, HTTP and TCP traffic had not been distinguished properly. Simple TCP traffic was visible on the topology view, but throughput metrics were not present. In this release we've made a lot of improvements around how TCP traffic is handled, so it's corresponding metrics - like throughput (MiB/s) - are now accurately displayed on the graph view of the Backyards UI.
The overview page is the best place to get a summary of the current state of your service mesh. It displays some global statistics, configs and metrics of the mesh, like the overall error rate and latency values, or the number of pods and services available. Mesh-wide mTLS configs can also be changed here, and you'll also see the current status of the attached clusters in a multicluster setup. Last but not least, the overview page shows all validation warnings and errors of the service mesh configuration.
Manage multiple ingress and egress gateways
It is a common use-case to have multiple ingress gateways in an environment (we also have one for Backyards), and we've had a feature request about it for a long time. This is more of a Banzai Cloud Istio operator feature (for now), but the new release of Backyards brings a new version of the underlying operator as well. And because we suggest using Backyards to install the Istio operator and therefore Istio as well, we'll highlight this addition here.
got a new CRD, called the
MeshGateway that can be used to
add and configure a new Istio ingress or egress gateway to
the mesh. When you create a new
MeshGateway CR, the
operator will take care of configuring and reconciling the
necessary resources, including the Envoy deployment and its
We'll have a blog post about this feature soon, stay tuned if you're interested in the details.
Along with the new features, we also did some "battle hardening" of Backyards. Some of the larger, notable bug fixes include:
- better handling of workloads with custom controllers
- mirrored HTTP traffic is now properly displayed
- fixed some latency metric values on the topology view
- fixed error rates in the workloads list view
- handling cases when owners are missing for workloads
While this release contains a ton of new features, we won't stop on our journey to bring you the best service mesh experience. Our next release will focus on ingress and egress gateway management, stay tuned!