Software tracing is the specialized use of logging to capture the operational behavior of code down to the code execution path. It’s indispensable for developers for troubleshooting network software in production to catch bugs, errors, misconfigurations, or other problems throughout the birth and lifecycle of products. A more efficient and effective way to use network software traces at scale is unified tracing. With unified tracing, all traces deployed in software running anywhere in a system (e.g., in line cards or other field replaceable units [ FRUs ]), are streamed to the main processor of a device and collected in a single set of files. This integration of trace messages provides time-ordered, real-time visibility into what a router, switch, wireless controller, or Internet of Things (IoT) device is doing across its approximately 100 processes. Here’s how we’ve ramped up tracing in a big way and what it means for enterprise networking.

Tracing Gets Binary in Cisco IOS XE 16.0

Release 16.x introduced binary tracing to the Cisco IOS XE code base. It’s now widely used. Binary tracing relies on compiler technology to assist in the encoding of each trace point, allowing for the storage of non-ASCII text objects, like packets and software-generated objects in the trace stream. These binary objects provide richer operational information about how network platforms are performing. Fully distributed, binary tracing also enables tracing on steroids, with some Cisco enterprise platforms able to exceed peak trace rates of two million traces per second per process. It also separates the encoding of high-performance traces from the decoding of traces, which can be displayed to users later.

Unified Tracing introduced in Cisco IOS XE Release 17.7

With binary tracing alone, each process writes traces independently to separate files. If you have 100 processes, you have 100 separate sets of files, which slows down troubleshooting. With unified tracing, all trace files for a system are merged into one trace file set of messages with all of the information about their origins (Figure 1). Each trace event uniquely identifies the calling site information and carries context to know where in the system and the source code it was produced. [caption id="attachment_1933" align="aligncenter" width="722"] Figure 1. From Per-Process Binary Trace Files (BTF) to Unified Trace Files (UTF) and Messages[/caption] Users can filter the time-ordered unified trace messages to make intelligent filtering decisions and see and understand what’s going on in the 100 or so processes at work in each device with greater detail and in real-time. You might notice a large number of errors coming from a single process with unified tracing. Then you can inspect the code and quickly understand what’s going on. Before, reviewing individual trace files one at a time made this process much slower and not scalable. Developers don’t have to change a single line of code to enjoy the improved logging performance of unified tracing. They can also continue to use the Buginf API as their Cisco IOS XE debug trace command. The goal was to introduce a uniform way to log information throughout a system regardless of the source, avoiding expensive data transformations or duplicate information logged in different ways for different customers.

Features and Benefits of Unified Tracing

• Automated traces from 100+ processes are streamed in real-time, in temporal order, across FRUs to a centralized set of unified trace files
• Centralized trace inspection based on high-rate filtering in real time is now possible―an industry first
• Richer information is collected on bugs, errors, misconfigurations, etc., across all system processes
• Identification of software issues during development and for post-release troubleshooting is accelerated

Additionally, in a coming version of unified tracing, trace files can be exported for use by analytical frameworks to provide further trace inspection and improve troubleshooting and insights. It’s a brand-new feature that also will allow for more efficient use of device disc space because of current CPU limits on the number of traces that can be created. It will enable more traces to be created and the ability to retain trace files elsewhere for the life of a system.

More Meaningful, Scalable Traces

With unified tracing IT administrators get a lot more information about what’s happening in their network devices than ever before. Developers can use unified tracing to finetune performance. Systems administrators and tech support agents can use it for more detailed, faster, and more scalable troubleshooting.