Warm Reboot for AI Data Center Switches: Technical Overview

In modern cloud data center and AI computing center networks, high availability and business continuity are key indicators of network infrastructure capability. As network scale continues to grow, switch operating system maintenance, software upgrades, security patching, and fault recovery are becoming increasingly frequent. Performing these operations without affecting critical tenant services or distributed computing workloads has become a major challenge in large-scale data center network operations.

Traditional Cold Reboot operations reload the software stack and interrupt both the switch control plane and data plane. This can cause protocol neighbors to go down and trigger route reconvergence. As a result, traffic may be interrupted for several minutes. This has a significant impact on cloud computing, AI training, and inference workloads that require continuous network connectivity. To reduce the impact of control plane maintenance, upgrades, and fault recovery on packet forwarding, Warm Reboot was introduced. Its primary goal is to keep the data plane forwarding traffic during a control plane software restart and minimize service disruption.

AsterNOS supports Warm Reboot through the decoupled, modular, and containerized architecture provided by SONiC. Unlike traditional closed network operating systems, where software processes are tightly coupled with underlying hardware states, AsterNOS runs core networking functions and system services in a modular and containerized architecture. It also uses the Redis database to centrally manage system configuration and runtime states across modules. This architecture provides a foundation for state persistence and rapid recovery.

With this control plane and data plane separation, AsterNOS Warm Reboot minimizes the impact on traffic forwarding during a control plane software restart. The key mechanism is to persist critical system configuration and runtime state while preserving the existing forwarding state in the underlying ASIC. The data plane can therefore continue forwarding traffic using existing hardware forwarding entries. After the control plane software restarts and resumes operation, state synchronization restores normal control of the data plane. This enables rapid control plane recovery while maintaining service continuity.

The goal of Warm Reboot is to restart or upgrade AsterNOS software with minimal impact on the data plane. Warm restarting an individual process or container is also within the scope of this mechanism.

The Warm Reboot process can be divided into three layers:

1. Network Applications and Orchagent: Each network application and its corresponding Orchagent submodule must work together to restore the original data and generate the incremental data required to initiate Warm Reboot.
2. Syncd: Before the restart, Syncd dumps the data in ASIC_DB. When the Syncd container is warm-started, it reads the dumped data and reconstructs its internal software state from before the restart in memory. Syncd recovery itself must not affect the ASIC state. Syncd receives configuration changes from Orchagent and passes them to LibSAI/ASIC after performing the necessary transformations.
3. LibSAI/ASIC: The ASIC must continue running its existing hardware forwarding entries throughout the software reload. This ensures uninterrupted data plane forwarding. As the hardware abstraction layer, LibSAI must support warm takeover. During startup after the restart, it reads the serialized file preserved before the restart and takes over the ASIC in its current operational state.

The following figure shows the overall Warm Reboot architecture:

The core of the system-level Warm Reboot shutdown path is to preserve the existing ASIC forwarding state while safely suspending the control plane containers. At the same time, SAI and ASIC-related software states are serialized and saved to external persistent storage. This ensures that the hardware state is preserved during the control plane restart. Using Docker containers such as BGP, teamd, and SWSS as examples, the Warm Reboot shutdown path consists of the following steps:

1. Graceful protocol shutdown: Trigger graceful restart mechanisms for network protocols, such as BGP Graceful Restart. This notifies peers to retain forwarding entries for a predefined period. The teamd link aggregation process is also gracefully stopped, with the neighbor timeout extended to prevent physical links from immediately transitioning to Down.
2. Control plane freeze: Disable aging mechanisms in the underlying hardware, such as MAC address aging, to prevent normal entries from aging out while the CPU is offline. Freeze the Orchagent programming threads and write a Warm Reboot flag to the system database.
3. State persistence: Rapidly dump the complete database state currently held in memory to an externally mounted persistent directory. This ensures that runtime data is preserved.
4. Hardware state preservation: Syncd instructs the underlying hardware to enter the Warm Reboot preparation stage. Based on this instruction, the LibSAI layer serializes and exports the current ASIC and SAI hardware mappings and states to separate files. Syncd then safely exits without physically resetting the ASIC.
5. Fast reboot: Use kexec to quickly boot into Warm Reboot mode and skip time-consuming hardware initialization and self-test procedures.

The core of the system-level Warm Reboot startup path is to rapidly restore the control plane state, reconstruct the logical views, and reconcile them with the underlying hardware, which has remained operational throughout the restart.

1. Data reload: After detecting the Warm Reboot boot parameters, the system starts the database container first and automatically restores the complete runtime data from the persistent directory.
2. Seamless hardware takeover: Syncd starts and reads the serialized files saved before shutdown. It reconstructs the internal driver state and takes over the ASIC without affecting physical packet forwarding.
3. View reconstruction: The Orchagent container starts and instructs the underlying system to enter Temporary View construction mode. Each service module replays its complete dataset based on the restored database. After the replay is complete, the system triggers the State Reconciliation
4. Upper-layer protocol recovery: Protocol containers such as BGP and teamd start together with SWSS. Before the internal control plane state is fully synchronized with the underlying hardware, the system temporarily ignores change notifications generated by external protocols. After synchronization is complete, each module performs its final convergence.

To connect the reconstructed control plane state with the underlying hardware state that remains operational throughout the Warm Reboot, AsterNOS uses a view reconciliation mechanism and matching algorithm. This avoids destructive full rewrites of the hardware state. The main steps are as follows:

1. Dual-view construction:

• a. Current View: The actual state programmed into the ASIC before the restart and continuously used by the hardware during the restart.
• b. Temporary View: The target state recalculated from the database after the control plane Warm Reboot and expected to be programmed into the underlying hardware.

2. Fast matching stage:

• a. The algorithm first identifies invariant objects that do not change during Warm Reboot, such as physical ports, hardware queues, and schedulers. These objects are used as anchors.
• b. The algorithm compares the VIDs of these objects in the Temporary View and Current View. If the VIDs are identical, their corresponding physical RIDs are also guaranteed to be identical. The algorithm therefore marks them directly as MATCHED, avoiding further deep traversal and significantly reducing processing overhead.

3. Deep transition and reconciliation stage:

• a. Using the invariants as anchors, the algorithm recursively compares each object in the Temporary View.
• b. For each temporary object, the algorithm searches the Current View for the most suitable object to reuse. If no matching object is found, a creation command is sent to the hardware. If the best match is found but its attributes differ, a differential update command is issued. If an exact match with identical attributes is found, the existing object is reused without any hardware write operation.

4. Stale entry cleanup:

After the comparison is complete, the system scans all objects in the Current View that are not marked as MATCHED and have a reference count of zero. These objects are safely removed from the hardware. A consistency-first strategy ensures that the control plane and data plane eventually reach a consistent state.

When building AI distributed training clusters with thousands or tens of thousands of GPUs, large model training jobs typically run for weeks or even months. During these long-running, high-load workloads, switch software upgrades and security patching are unavoidable operational tasks. Traditional software upgrades require a full cold reboot, which can cause widespread network interruptions across the cluster. This can directly interrupt running distributed workloads and leave GPU resources idle.

With Warm Reboot, network operators can perform seamless software upgrades on switch operating systems. During a system-level Warm Reboot, the underlying switching ASIC continues forwarding traffic. This enables smooth maintenance of the data center network and improves the overall availability of the AI computing cluster.

In multi-tenant public and private cloud data centers, the network carries a large volume of business-critical workloads that are highly sensitive to packet loss and latency, including online financial transactions, real-time audio and video, and cloud storage data synchronization. Traditional switch upgrades often require complex traffic drain procedures. A direct cold reboot can create traffic black holes and service interruptions lasting several minutes on Spine or Leaf nodes.

With Warm Reboot, network operations teams can perform smooth online software upgrades on production switches while business traffic continues to flow. During the control plane OS reload, routing protocols such as BGP perform graceful restart and state reconstruction. Meanwhile, the underlying ASIC continues forwarding traffic at line rate using the forwarding entries preserved and locked before the restart. This mechanism enables transparent service upgrades and smooth data plane transitions when the cloud data center performs network software upgrades, introduces new features, or applies emergency security patches. It allows the underlying network infrastructure to evolve without affecting cloud tenants.