Skip to main content

AI Checkpoint Traffic: How 400G Ethernet Prevents Network Bottlenecks in GPU Training

written by Asterfusion

September 8, 2026

Introduction

During the training of LLMs with hundreds of billions or trillions of parameters, such as Llama 3 and DeepSeek, a single job depends on the coordinated operation of GPUs, CPUs, memory, NICs, optical transceivers, switch ports, storage nodes, drivers, training frameworks, and schedulers. A failure in any of these components can interrupt the job. To reduce the compute loss caused by failures, training systems need to save Checkpoints more frequently and resume training from the most recent Checkpoint after recovery.

When thousands of GPUs simultaneously write hundreds of GB to TB of state data to shared storage, the resulting short-lived throughput bursts can easily cause network congestion. This can leave GPUs idle while waiting for I/O, directly reducing MFU (Model FLOPs Utilization) and increasing overall TCO.

The AI Checkpoint bottleneck may appear to be a storage media issue. In practice, it is also a test of the network architecture, observability, and congestion control capabilities of the shared Fabric.

What Is AI Checkpoint and Why Can It Become a Bottleneck?

In AI training, a AI Checkpoint is a snapshot of the complete state of a model at a specific point in the training process. It allows training to resume from that state later. A Checkpoint typically includes model weights, optimizer states, and training context such as the learning-rate scheduler state. The table below lists the main data stored in an AI Checkpoint and their roles.

ContentPurpose
Model weightsParameters learned by the model at the current training stage
Optimizer statesInternal optimizer states, such as momentum and variance in Adam
Learning-rate scheduler stateCurrent progress of the learning-rate schedule
Training stepThe current training step
Random statesHelps maintain consistent training behavior after recovery
Data loader stateData loading state and progress
inter-gpu traffic in storage networking

Compared with regular training traffic, AI Checkpoint traffic has four distinct network characteristics:

  • High Concurrency (Fan-In): Hundreds or thousands of GPU compute nodes concurrently initiate write operations toward a relatively small number of storage nodes or storage gateways.
  • High Burstiness: Traffic occurs in periodic bursts. A burst typically appears when a predefined AI Checkpoint Interval is reached, such as saving the training state after a fixed number of training steps.
  • High Volume: A single Checkpoint can involve hundreds of GB to several TB of data transfers.
  • Highly Synchronized Traffic: Distributed training introduces a high degree of temporal correlation between training nodes. Training processes typically follow the same training progress and Checkpoint policy. As a result, many nodes may enter the Checkpoint phase at the same time or within a short interval. Their write traffic overlaps heavily in time, creating synchronized traffic peaks.

These characteristics make Checkpoint traffic fundamentally different from regular training traffic. It does not consume network bandwidth continuously and evenly. Instead, large volumes of write traffic are concentrated within a short time window.

When a large number of training nodes enter the Checkpoint phase simultaneously, their write traffic converges at uplinks, Leaf-Spine links, and the egress points of the storage network. If the instantaneous traffic exceeds the available bandwidth of a link or storage system, a significant Traffic Burst occurs. This causes queues to grow rapidly and may further trigger congestion control mechanisms.

Therefore, the Checkpoint bottleneck is not simply caused by the large amount of data per Checkpoint. More importantly, its high concurrency, burstiness, and synchronization concentrate bandwidth demand on limited network and storage resources. This can expose the network to peak loads far above its average utilization within a short period.

For GPU training, this bottleneck can create a chain of effects:

Checkpoint traffic burst → Network congestion → Increased write latency → Longer Checkpoint completion time → Training job waits → Idle GPU compute resources.

Therefore, in large-scale AI clusters, Checkpoint is not just a storage concern. It can also become a significant bottleneck for network performance, training efficiency, and GPU utilization.

The AI Checkpoint Data Path: Key Components of a 400G Network Fabric

In large-scale AI training, Checkpoint data is typically written from GPU compute nodes to a shared storage system. In Ethernet-based architectures such as Parallel File Systems, NVMe-oF, and distributed storage, the data traverses the compute node NICs, Compute Leaf switches, Spine Fabric, and Storage Leaf switches before reaching the storage nodes. The role of a 400G Ethernet Fabric is to provide sufficient aggregate bandwidth for this front-end storage path and reduce the risk of the network Fabric becoming a bottleneck for Checkpoint writes.

This section focuses on the data path from the compute cluster to the storage cluster, as shown in the figure below:

AI Checkpoint data path in Storage Network
Path SegmentPotential BottleneckWhat a 400G Ethernet Fabric Can Mitigate
GPU/Host → NICCPU, PCIe, or NUMA bandwidth limitationsPrimarily a host-side bottleneck that cannot be directly addressed by the network Fabric
Compute Cluster → LeafInsufficient server-facing link bandwidth and port aggregationProvides high-bandwidth server-facing ports to increase per-node connectivity and port density
Leaf → SpineUplink oversubscription and uneven ECMP traffic distributionUses high-bandwidth 400G uplinks to increase Fabric capacity and reduce the risk of link oversubscription
Spine → Storage LeafInsufficient storage-side link bandwidth and link congestion caused by concentrated Checkpoint trafficUses high-bandwidth Spine-to-Storage links to increase aggregate throughput on the storage side
Storage Leaf → Storage ClusterInsufficient storage-facing link bandwidth and port contentionProvides high-bandwidth connectivity to storage nodes and increases storage-side throughput

Building a Lossless Network Fabric for AI Checkpoint Traffic

Network Architecture Isolation (Storage vs. Compute Fabric):

Adopt an architecture that physically or logically isolates the Storage Network from the Compute Network using technologies such as VLAN, VRF, or Overlay. This prevents storage traffic from interfering with GPU communication.

Fine-Grained PFC and ECN Congestion Control Tuning:

Configure dynamic ECN thresholds and PFC trigger thresholds in the Storage Network for large-packet and bursty traffic. This helps maintain lossless RoCEv2 transmission while avoiding excessive PFC activation and PFC storms.

Adaptive Routing (Adaptive Routing / ARS / Flowlet Switching):

Traditional ECMP distributes traffic based on hashing. Large Checkpoint flows (Elephant Flows) can easily cause hash collisions and congestion on individual links. Adaptive routing can dynamically distribute Checkpoint bursts across multiple available paths.

Open Networking and High-Performance Switching Hardware:

400G open networking switches based on the AsterNOS operating system provide large packet buffers, automated RoCEv2 tuning, and lossless Ethernet capabilities. These capabilities help absorb bursty Checkpoint write traffic.

How Asterfusion 400G Switches Address Checkpoint Traffic

High-Density 400G Fabric for Checkpoint Traffic

Checkpoint traffic creates a distinctive many-to-few, bursty and highly synchronized workload. When hundreds or thousands of GPU nodes write training state to shared storage, the network must provide sufficient aggregate bandwidth across the compute-facing, spine and storage-facing links.

Asterfusion offers a portfolio of 400G platforms covering 32-port 400G and 64-port 400G configurations, with switching capacities from 12.8 Tbps to 25.6 Tbps. This allows the same 400G Ethernet architecture to scale from smaller Leaf/Spine deployments to high-density AI and storage fabrics.

32-Port 400G Platforms for Leaf and Spine Roles

The CX732Q-N provides 32 × 400G QSFP-DD ports and 12.8 Tbps line-rate switching based on Marvell Teralynx 7. With latency around 500 ns and a 70 MB packet buffer, it is positioned for AI/ML, HPC and CLOS spine deployments. AsterNOS adds RoCEv2, PFC/ECN, ECMP, in-band telemetry and AI-oriented traffic-management capabilities.

The CX732Q-N-V2 uses the Marvell Falcon ASIC and provides 32 × 400G QSFP-DD ports with 12.8 Tbps switching capacity. It supports RoCEv2, PFC/ECN, ECMP and rich telemetry, making it suitable for 400G Leaf/Spine fabrics where Enterprise SONiC features and operational visibility are important.

For OCP-based deployments, the CX732Q-N-ORv3 adopts the OCP ORv3 form factor and provides 32 × 400G QSFP-DD ports. It is designed for high-density AI/HPC fabrics and supports non-blocking Leaf-Spine and AI backend network deployments. Its ORv3 design also uses a 48V DC power architecture for rack-level power integration.

64-Port 400G Platforms for High-Density AI and Storage Fabrics

For larger fabrics, the CX764QH-N provides 64 × 400G QSFP112 ports and 25.6 Tbps switching capacity. Its high port density allows more 400G links to be concentrated in a single switch, which is particularly relevant when Checkpoint traffic creates high aggregate demand on the storage-facing side of the Fabric. The platform also supports WCMP, Packet Spray and INT-driven routing for AI/ML/HPC traffic management.

The CX764QD-N provides 64 × 400G QSFP-DD ports, 25.6 Tbps switching capacity, approximately 560 ns port-to-port latency and more than 200 MB of packet buffer. For Checkpoint-related traffic, the large buffer and congestion-management capabilities are particularly relevant because bursty storage writes can temporarily increase queue occupancy before traffic is drained by downstream storage links. The platform also supports RoCEv2, PFC/ECN, WCMP, Packet Spray and INT-driven routing.

The CX764QO-N provides a similar 25.6 Tbps / 64 × 400G architecture using OSFP interfaces. It combines 560 ns-class latency, more than 200 MB of packet buffer and AI-oriented traffic-management capabilities including WCMP, INT-driven routing, Flowlet and Packet Spray. This makes it another high-density option for AI, HPC and distributed-storage fabrics.

Conclusion and Outlook: Open Lossless Networking for Efficient AI Clusters

As cluster scale grows from hundreds to tens of thousands of GPUs, AI Checkpoint is no longer simply a file-saving operation. It has become a critical stress point for the data center network. Minimizing GPU wait time caused by Checkpoint operations is a direct way to improve overall cluster ROI and GPU MFU.

The focus is shifting from a compute-centric approach to deeper coordination across compute, storage, and high-performance networking. A 400G lossless Fabric, fine-grained congestion control, and full-stack network observability are essential to ensure that compute resources are used efficiently.

Explore More in Our 400G Technical Series

Building a resilient, high-performance 400G infrastructure requires a unified understanding of standards, interconnects, and hardware choices. Whether you are scaling an AI data center or upgrading campus backbones, explore our comprehensive 400G guide series to master every layer of the architecture:

Latest Posts