Kubernetes Observability - One Tool to Rule Them All
It's been a while since we posted about our observability solution. We did not abandon observability, we were just busy with the development. First things first: One Eye is now called Cisco MCOM (Multi-Cloud Observability Manager). As you may know, Cisco has acquired Banzai Cloud. With the new environment come some changes as well. For those who are not familiar with One Eye, let's quickly review the basics. We will cover the current state of our Kubernetes observability solution.
What is MCOM?
Cisco MCOM brings observability to Kubernetes clusters. The three main pillars of observability are Logging, Monitoring, and Tracing. Several battle-tested open-source components provide building blocks for your observability system. On the CNCF landscape, you can pick your preferred tools and start from scratch if you want. I can assure you it will cost you significant time and effort to polish the components to fit to your needs. So how does Cisco MCOM help? It applies best practices and integration points to these components. With Cisco MCOM you get the benefit of open-source with batteries included. Moreover, the latest trends show that companies tend to go with a multi-cluster approach with Kubernetes. So automating those tasks is a must. That is where Cisco MCOM shines!
The Kubernetes environment provides different kinds of deployments. Most of these tools (Helm, kpt, Kustomize) have their strong and weak side as well. We wanted to make deployment and operation as seamless as possible. As a result, Cisco MCOM can work as a standard operator, maybe installed from a Helm chart, or as a CLI tool for doing a quick evaluation and configuration. If you want to dig deeper into our decision, read our Declarative installer post. We believe that both methods have their place in the operational lifecycle.
Logging is a simple yet complex part of Kubernetes. You can
kubectl logs command to grab the logs for your
Pod, simple as that. However, if you want to collect,
centralize, and analyze them, you have to invest in your
All containers' logs are accessible on the Kubernetes Nodes as simple text files. There is no access control nor separation whatsoever. Logging Operator solves these problems by configuring Fluent Bit and Fluentd instances through custom resources. These custom resources enable you to control what logs go where and what transformations you want to apply to them. The Cisco MCOM logging sub-system is based on the Logging Operator, but extends it with a UI and extra tools. If you are not familiar with the Logging Operator, you can catch up by watching my presentation at Fluentcon.
If you don't find something in the Logging Operator, there is a chance that it is included in the Logging Extensions. As the name suggests, Logging Extensions provide additional features for the Operator, including:
- Webhook support - Tailing logs from inside a container with a sidecar container
- Event tailer - Tail Kubernetes events as logs
- Host log tailer - Tail logs from your Host filesystem or systemd
Last but not least, I want to highlight the log restoration feature. Without getting into details, this powerful component helps you to load logs from cold storage for on-demand analysis.
The next item on the list is monitoring. Under the hood Cisco MCOM provisions Prometheus via the Prometheus Operator and Thanos via the Thanos Operator. Prometheus Operator is an open-source project and I'm sure most of you are familiar with it. On the other hand, Thanos Operator is yet another tool of our ecosystem. If you are new to Thanos and Thanos Operator, please read our introduction blog post.
Thanos Operator supports both simple and complex Thanos deployments. That includes long-term storage and connecting remote metric endpoints. The latest release contains helpers for automated multi-cluster setup, but that's alone worth its own blog post. So stay tuned, we will cover that in the following weeks!
If we are talking about plotting metrics, the obvious choice is Grafana. Cisco MCOM uses the Grafana Operator to deploy and provision dashboards and data sources. If you need more freedom when digging into metrics, you are just one click away to do that. Grafana is seamlessly integrated into Cisco MCOM.
There is always room for improvement. For us, tracing is one of them. We are at the planning table to provide a smooth way of managing traces like logs and metrics with Cisco MCOM. One of the promising paths can be the Open Telemetry way of handling traces, but we still have this question open.
A good observability tool gives you instant feedback on your system. Yes, I'm talking about the fancy dashboards and graphs. It does not matter if you are just checking the health of your system or you have to debug issues. A good visual representation of your system helps you fix things much faster.
Cisco MCOM doesn't want to become another Grafana. Instead, it fills the gap where you are missing visibility. It enables browsing resources by representing logical and physical connections between components. And whatever an operator would need to keep everything healthy.
As you can see the Cisco MCOM observability stack is rather heterogeneous. We use different tools for different tasks. That is why a common authentication interface saves you a lot of trouble, especially if you can integrate your company SSO solution in it. This is the reason we chose Pomerium and Dex for this task. Earlier we had a guest blog post from Bobby DeSimone, core Pomerium maintainer. Since then it has become one of the best authentication proxies that I know of. Cisco MCOM configures Pomerium and Dex in front of all its components. Moreover, you can configure Read or Write access to different groups.
Of course observing the clusters does not solve your problems. It just helps you identify them. You need alerts and events to react to incidents and issues. Cisco MCOM already provides some insight on the active alerts, but we are working on to make them easier to manage. It is important to reach the right person with the corresponding problem.
I hope this "short" post gives you a good overview of the several capabilities Cisco MCOM possesses. In the future, we will cover further topics in more detail with examples. I hope those will help to find the synergies between your actual stack and Cisco MCOM.