In today’s complex and ever-changing IT environments, observability is essential for ensuring the health and performance of your systems. By understanding what is happening in your system, you can identify and troubleshoot problems quickly, improve performance, prevent problems, and make better decisions.
Here are some of the key components of observability:
By collecting and analyzing these data, users can gain a deep understanding of their system and its performance. This helps in identifying and troubleshooting problems, improving performance, and preventing problems.
This requires a significant amount of work, from choosing the right tools, instrumenting the code, collecting and storing the data for analysis, and generating insights from it. Having a technology that simplifies most of these tasks and also provides integration options with well-known monitoring providers will simplify implementing observability for your infrastructure and application performance monitoring. In this article we demonstrate using eBPF and OpenTelemetry to achieve observability without any code instrumentation.
eBPF is a technology that can run sandboxed programs in a privileged context such as the operating system kernel. It extends the capabilities of the kernel safely and efficiently without requiring changing kernel source code or load kernel modules.
eBPF observability is a new technology that allows for greater visibility and control over systems and networks. It does this by providing a way to collect and analyze data from the kernel without having to modify the kernel itself also without any instrumentation of the application code. This makes it a powerful tool for troubleshooting performance problems, identifying security vulnerabilities, and optimizing system resources.
OpenTelemetry is an Observability framework and toolkit designed to create and manage telemetry data such as traces, metrics, and logs. Crucially, OpenTelemetry is vendor- and tool-agnostic, meaning that it can be used with a broad variety of Observability backends, including open source tools like Jaeger and Prometheus, as well as commercial offerings. OpenTelemetry is a Cloud Native Computing Foundation (CNCF) project.
eBPF provides a powerful mechanism for dynamic tracing and analysis within the Linux kernel, while OpenTelemetry is a set of open standards and tools for collecting, exporting, and visualizing telemetry data. By combining eBPF and OpenTelemetry, you can gain deep visibility into your application’s internals, as well as the underlying system, with minimal overhead. This combination enables you to collect fine-grained telemetry data from the kernel and various application layers, providing a comprehensive understanding of your system’s behavior and performance.
Developers can use Pixie to view the high-level state of their cluster (service maps, cluster resources, application traffic) and also drill-down into more detailed views (pod state, flame graphs, individual full body application requests). It offers features like:
Pixie automatically collects the following data:
For the eBPF capabilities demonstration, we picked the Cisco Nexus Dashboard (a platform that transforms Data Center, Cloud Network operations with simplicity, automation and analytics) and deployed eBPF/Pixie in k8s environment to demonstrate its capabilities with zero code instrumentation. We were able to collect the metrics from the infrastructure, application, protocol, cpu/memory/network and JVM stats.
eBPF automatically collect telemetry data such as full-body requests, resource and network metrics, application profiles, and more. The following Grafana dashboard shows the CPU usage, Network RX bytes, and Connection stats generated by eBPF, collected using Pixie and streamed these metrics to Prometheus using OpenTelemetry plugin.
eBPF collects individual processes stack traces (java, Golang, etc.) to display kernel and user level performance metrics and summarizes overall CPU consumption. If we take a look at the frame-graph of this mond pod, it helps us understand the reason for this spike in CPU utilization. Every ~10ms, the Pixie profiler samples the current stack trace on each CPU. The stack trace includes the function that was executing at the time of the sample, along with the ancestor functions that were called to get to this point in the code.
The collected samples are aggregated across a larger 30 second window that includes thousands of stack traces. These stack traces are then grouped by their common ancestors. At any level, the wider the stack, the more often that function appeared in the stack traces. Wider stack traces are typically of more interest as it indicates a significant amount of the application time being spent in that function.
eBPF allows dynamic trace injection to effectively troubleshoot system level issues such as TCP drops, log level changes etc. The following diagram shows the map of TCP drops and retransmits across the Nexus Dashboard cluster. To see the number of drops between the pod pairs, you can hover over an edge in the map view. The color and thickness of the edges indicate an increase in the number of TCP drops.
The following diagram shows the traffic flow view of the micro-services along with incoming, outgoing traffic with request/bytes throughput and error rate metrics.
eBPF along with Pixie can help navigate end to end metrics from node level to all the way to the function level. Lets take a look at how to navigate this flow.
Here, we are looking at the cluster view, that shows the list of nodes in the cluster and its CPU/memory usage along with no.of K8S pods running in each node.
We can select a node from the cluster view and drill down for the detailed view of the node. This view provides individual node’s CPU/Memory/Network resources consumption.
The namespace view, which shows all the K8S namespaces, along with different metrics like, pod/service counts, disk throughput etc.
The pod lifetime resources view shows the total resource usage of a pod over its lifetime.
Pod lifetime resource usage
We can also look at the JVM stats for JVM memory management metrics for java applications.
In this demo, we demonstrated the observability of infrastructure and applications with eBPF by collecting metrics from the Cisco Nexus Dashboard platform without any code instrumentation. We then used the OpenTelemetry plugins available from Pixie to export these metrics to the Prometheus monitoring system to analyze the data using Grafana dashboards.
We have demonstrated how eBPF would help achieve observability for the Cisco Nexus Dashboard platform. eBPF is a powerful tool that can be used to collect data from a wide variety of systems. It is a promising technology for achieving observability in modern cloud-native systems.
We would like to express our thanks to Kalyan Ghosh for providing guidance throughout the project and James Chen for his valuable suggestions.