It used to be a challenge to upgrade an enterprise networking software image with 300 million lines of code, like Cisco IOS XE. But a team of developers at Cisco has developed upgrade and patching solutions that have made the upgrade process simpler and faster.

An Evolving Software Stack

Cisco IOS XE is huge and getting bigger. It powers enterprise networks around the world and is in a continual state of transformation, with 700 new features added per year, plus bug and security vulnerability fixes. It takes time and careful planning to download an image from www.cisco.com and distribute it to multiple network devices without causing much disruption. That might not be a problem with a smaller company, but for an enterprise with thousands of devices, it’s an ongoing challenge. It requires planning ahead, creating a maintenance window with scheduled downtime, and the time it takes to evaluate and qualify the new content that comes with the new version.

Upgrade Automation at Scale

With a few clicks, Cisco customers can use network management and automation solutions like Cisco DNA Center, Cisco vManage for SD-WAN, the Cisco IoT Field Network Director (FND) and Industrial Network Director (IND) for industrial switches and routers to easily upgrade their networks either at scale or for designated platforms or sections of their infrastructure (Figure 1). Upgrades can be scheduled for the middle of the night or for other times of low demand. For example, using the Software Image Management (SWIM) application in Cisco DNA Center, customers can download Cisco recommended images automatically, designate a section of the network running older images, and notify the administrator to upgrade the devices all at the same time. Before starting the upgrade, SWIM runs diagnostics on the devices to see if they’re ready for an upgrade. If there’s a problem (e.g., not enough space for the new image) the admin is notified. After the upgrade, Cisco DNA Center runs diagnostics again to make sure the upgrade is functioning properly. Using SWIM and Cisco DNA Center, Cisco enterprise customers can do image upgrades of up to 1000 devices per hour. The devices expose a variety of management interfaces—like CLI, WebUI, NETCONF, RESTCONF, and Google Remote Procedure Calls (gRPC) with YANG data models to invoke the upgrade operation. The interfaces are exactly the same across the entire Cisco enterprise product portfolio, which has over 40 different platforms. Each device protects itself from attackers by implementing strong security. All Cisco images are digitally signed at the time of build and verified for authenticity at the time of upgrade. The devices also have a graceful recovery mechanism if anything goes wrong during the upgrade process, allowing them to fall back to the previous version. Since the upgrade is a time-consuming process, a granular status update is exposed to the admin to provide full visibility into the process. Simplicity is provided for complex systems. Instead of upgrading each member switch one by one, the stack can be upgraded in one go. A wireless LAN (WLAN) controller along with thousands of APs connected to it can be upgraded at the same time like a single system. Innovations like rolling AP upgrade are in place to ensure minimal end user connectivity issues during upgrade.

In-Service Software Upgrade

With the in-service software upgrade (ISSU) feature, Cisco customers using platforms offering redundancy can avoid the disruptions from image upgrades altogether. ISSU orchestrates the upgrade on standby and active processors one after the other and the switches between them so that there is zero effective downtime and zero traffic loss. The IOS XE software stack has the capability to do ISSU between any–to–any releases and the development team has an elaborate feature development testing and governance process to ensure this happens without failures. Cisco defines policies for a smooth ISSU experience based on platform and release combinations. Customers using Cisco DNA center can use these policies for a smooth and non-disruptive ISSU experience.

Hot Patching Micro Images

The Cisco IOS XE development team has traditionally delivered bug fixes with new rebuild releases. This requires complete image upgrades. But aside from the actual image upgrades, customers often have validation processes in IT labs that must be completed before software is certified for use. For some customers this can take months during each year. For critical bug or security fixes, that timeframe has become unacceptable. To speed up the process and lower the complexity, Cisco is issuing small micro images containing only the code necessary for a critical bug or security fix. Customers can install it on devices in a fraction of a second using hot patching without any network disruption. Hot patching doesn’t result in a device reload and the fix takes effect immediately. Additionally, because of the small size of the patches, they are easy to distribute. Because of their limited content, customers can have much higher confidence in installing these micro patches in their production network without going through the complete validation process. The Cisco IOS XE hot patching feature is a toolchain of integrated technology and is expected to provide a default hitless defect fix. The focus here is on consistency and automation of the patch process with tooling that guides developers to produce consistent results. The depth of the functionality does not compromise correctness in the presence of compiler optimizations and inlining, while producing the minimal patch code that is distributed. This approach is unique with Cisco IOS XE as opposed to the traditional approach of updating shared libraries and updating processes with stateful restarts. Patch deployment at scale is further simplified using Cisco DNA Center or Cisco vManage. Hot patching is supported for most of the Cisco IOS XE code, which provides the ability to hot patch YANG data models. For the rest, cold patching involving a system restart is supported, which still alleviates the need for customers to go through lengthy validation cycles. To avoid network impacts, a cold patch can be applied using the ISSU workflow.

Software Version and Patch Recommendation by Cisco DNA Center

With Cisco IOS XE on so many devices, in so many releases, combined with all of the bug fixes and security vulnerability fixes that are available, it’s easy for customers to become confused. What version should they be on? What patches should be deployed? Cisco DNA Center takes the pressure off by recommending what release each customer should be on and what benefit they will get by upgrading to it. Cisco DNA Center also learns the security vulnerabilities that have been fixed in the new releases or released as patches and alerts admins to upgrade or install patches. With the threat of business losses due to network downtime; over-stretched IT departments; time-consuming software maintenance, certification, and qualification; Cisco enterprise customers have a lot to celebrate with automated, intelligent, controller-based software image and patch management for Cisco IOS XE. Image upgrade automation using Cisco DNA Center, combined with ISSU and hot patching, are all examples of Cisco innovation. Our goal is to simplify and accelerate the work of IT. These innovations provide more control, flexibility, and security to the image upgrade process to ensure that the right software is always running.