Skip to main content

Why GPU Utilization Drops in AI Clusters: The Role of Networking

written by Asterfusion

September 18, 2026

Introdcution

As AI factories continue to evolve and scale, enterprises are making significant investments in AI infrastructure. GPUs account for a large share of this investment because they are expensive and are typically deployed in large numbers. As a result, improving GPU utilization has become a key concern for most enterprises.

There are several ways to improve GPU utilization, from optimizing GPU performance to improving the efficiency of the network that connects GPUs. This article focuses on the network layer and explains how network design and optimization can help maximize GPU utilization.

What Is GPU Utilization

GPU utilization refers to the proportion of time that a GPU is actively processing workloads within a given period. However, in AI infrastructure, GPU utilization is a broad topic. It covers at least four different aspects: whether the GPU is active, whether it is performing matrix computations efficiently, whether the GPU cluster is allocated and scheduled effectively, and how efficiently GPU resources are converted into business output. We explain these aspects one by one.

1. GPU Activity

The first level is simply whether the GPU is doing work. A key metric is GPU-Util. For example, if GPU-Util is 90%, the GPU is active for about 54 seconds in a one-minute period.

However, this does not mean that 90% of the GPU’s compute capacity is being used. It only means that the GPU is busy for 90% of the time. The workload may still use only a small portion of the available compute resources. For example, a workload may keep the GPU busy for 90% of the time while using only 10% of its compute capacity.

This is why we need a second level of measurement.

2. Compute Efficiency

The second level looks at how much of the GPU’s actual compute capability is being used while it is active. Key metrics include MFU, HFU, achieved TFLOPS, and Tensor Core utilization.

For example, assume a GPU has a theoretical peak performance of 100 TFLOPS and achieves 60 TFLOPS when running a workload. This can be roughly understood as using 60% of its theoretical compute capability.

If compute efficiency is low at this level, further analysis is needed to determine whether the GPU is affected by memory access bottlenecks, inefficient kernels, insufficient data preparation, or communication wait time.

3. Cluster Resource Efficiency

The first two levels focus on an individual GPU. The third level looks at the GPU cluster as a whole: whether the available GPUs are allocated and used efficiently.

For example, a job may require 8 GPUs, but resource fragmentation may prevent the scheduler from allocating the required resources. The job has to remain in the queue even though some GPUs in the cluster are idle. These GPUs are not being used because they cannot form the required resource group. In this case, the issue is not necessarily insufficient GPU performance. It may instead be caused by inefficient scheduling and resource allocation.

The focus at this level therefore expands from “Is each GPU being used effectively?” to “Is the entire GPU cluster being scheduled and utilized efficiently?”

4. Business Efficiency

The first three levels focus on the GPU and GPU cluster. The fourth level looks at a more practical question: after investing so many GPU resources, how much useful business output is being produced?

For example, two training jobs may each use 100 GPUs. Job A takes 10 hours to complete, while Job B takes 6 hours. If both jobs achieve the same training objective, Job A consumes 1,000 GPU-hours, while Job B consumes only 600 GPU-hours.

From a business perspective, both jobs may maintain high GPU-Util. However, they consume different amounts of GPU resources to achieve the same training objective.

At this level, the key question is therefore: How many GPU resources are required to achieve the same business objective

How the Network Reduces GPU Utilization

A study presented in GPU-Efficient Communication Scheduling for Deep Learning Training (SIGCOMM 2024) examines GPU utilization in large-scale deep learning training. It shows that actual GPU utilization is not determined only by the GPU, framework, or batch size.

During training, each iteration repeatedly alternates between computation and communication: Compute → Communication → Compute → Communication → …. After completing part of the computation, GPUs need to exchange gradients or model states through Collective Communication operations such as AllReduce, AllGather, and ReduceScatter. If communication is not completed in time, the next stage of computation cannot start. The GPU then transitions from a compute state to a waiting state.

This means the network does not need to reach 100% utilization to reduce GPU utilization. The key question is whether communication becomes part of the critical path for GPU computation.

In multi-job and distributed training environments, this process is mainly affected by three factors.

Inter-job Communication Contention

Figure 1 communication contention resulting low GPU utilization
Based on GPU-Efficient Communication Scheduling for Deep Learning Training

The figure shows two types of contention: contention along the network forwarding path and contention on the host through PCIe/NVLink.

In production environments, most communication contention occurs along network forwarding paths.

Figure 2 two job co-executing gpt vs bert

The figure shows the results of running GPT and BERT jobs together for comparison.

The experiment shows that when the GPT job runs alone (Standalone), its Iteration Time is 1.53 seconds. When it runs together with the BERT job (Co-execute), the two jobs compete for network resources between the ToR (Top-of-Rack) and aggregation switches. GPT’s Iteration Time increases by 11.0%, from 1.53 seconds to 1.70 seconds. The throughput of GPT and BERT decreases by 9.9% and 7.7%, respectively. Overall cluster GPU utilization decreases by 9.5%.

To put this into perspective, in a cluster with 1,000 GPUs, a 9.5% drop in utilization is equivalent to leaving the compute capacity of 95 GPUs unused. This can result in additional costs of hundreds of thousands or even millions of dollars per day.

Collective Communication Scheduling

If communication contention between different jobs is unavoidable, which job should be given priority?

Figure 3 Collective Communication scheduling

In a single-job environment, network scheduling typically aims to reduce the average Job Completion Time (JCT). However, in multi-job distributed training, simply optimizing JCT can actually reduce GPU utilization.

As shown in the figure, when two jobs compete for the same link, different communication priority schemes can result in very different GPU waiting states. One scheduling scheme may cause 10 GPUs to wait, while another may cause only one GPU to wait. As a result, the two schemes can produce different levels of GPU utilization.

This leads to an important observation: jobs with higher compute demand have a greater impact on overall cluster utilization and should be given higher communication priority. This may involve a small trade-off in fairness for smaller jobs, but it helps preserve overall cluster compute throughput.

Topology-Unaware Job Placement

For distributed training, allocating 64 GPUs is not simply a matter of selecting 64 available GPUs. The location of those GPUs matters. Which hosts they are located on, which ToR switches those hosts connect to, and how many layers of the network separate the GPUs can all affect the communication paths used by Collective Communication.

If the GPUs assigned to a training job are distributed across different network locations, its communication may need to traverse more switches and share the same network links with other jobs.

Therefore, GPU placement also determines where communication contention occurs. Communication across different topology domains can introduce orders-of-magnitude increases in latency, directly disrupting the timing alignment between computation and communication. If the scheduling algorithm is unaware of the physical topology, adding more GPUs can still result in lower GPU utilization.

How 400G Switches Improve GPU Utilization

The value of 400G switches is not limited to higher peak bandwidth. They can reduce waiting time for Collective Communication in the network, allowing more communication to be hidden behind computation and reducing GPU idle time.

Higher Network Capacity Reduces Inter-job Communication Contention

In a real AI cluster, the issue is not simply the transmission speed of an individual link.

As shown in the previous experiment, when GPT and BERT run concurrently, the GPUs themselves remain unchanged. However, communication contention between the two jobs across the ToR and aggregation switches increases GPT’s Iteration Time from 1.53 seconds to 1.70 seconds. GPU utilization decreases as a result.

Upgrading a 100G link to 400G essentially increases the capacity of the shared network resource. When communication traffic from multiple jobs follows the same path, higher link capacity reduces the likelihood of link saturation and helps reduce queuing delay.

This is particularly important for Collective Communication. A GPU does not necessarily need to wait for all network communication to finish. It may only need to wait for a critical communication operation to complete. Once this operation encounters queuing on the critical path, the additional network latency can directly appear as GPU idle time.

Therefore, the value of a 400G network is not just higher peak bandwidth. It also provides more communication headroom for concurrent workloads.

Low Latency and Congestion Control Reduce Communication Waiting

Higher bandwidth does not guarantee that communication will never wait. In addition to bandwidth, effective congestion control is needed to reduce communication delays.

When multiple Collective Communication operations reach the same link at the same time, queues can still become congested. The network therefore needs to minimize the additional latency caused by queuing and congestion.

For a RoCEv2 AI Fabric, this typically involves congestion control mechanisms such as ECN, PFC, and DCQCN, as well as QoS, queue management, and low-latency forwarding.

These mechanisms address different aspects of network congestion.

PFC and ECN do not directly increase GPU compute performance, nor do they determine when NCCL initiates a Collective Communication operation.

Their role is to control congestion after communication traffic enters the network. They help reduce the additional latency caused by queuing, pause events, and packet retransmissions. This is important for GPU utilization.

For example, assume a Collective Communication operation normally takes 8 μs, and the GPU’s computation window can fully overlap with this communication. The communication can then be hidden behind computation.

If congestion increases the communication time to 15 μs, the communication exceeds the portion that can be hidden by computation. The remaining time becomes GPU waiting time.

The goal of tuning PFC, ECN, and related parameters on a 400G switch is therefore to reduce this communication time as much as possible. This allows more Collective Communication operations to remain within the computation overlap window.

Higher Radix and Port Density Improve GPU and Job Placement

The topology analysis above shows that where GPUs are placed directly affects the paths used by Collective Communication.

A switch with higher radix and port density can connect more GPUs, servers, or downstream switches. This can reduce the number of network tiers required as the AI cluster scales.

Therefore, high radix is not simply a switch specification. It can affect the topology of the entire AI Fabric. Higher port density allows more GPUs to remain on shorter network paths and can reduce the need for additional aggregation layers and hops.

This helps reduce the path length of Collective Communication. It can also reduce the probability that different jobs will share the same intermediate links.

Asterfusion 400G RoCE AI Switches to Improve GPU Utilization

For large-scale AI training environments, Asterfusion provides six 400G RoCE AI switches for AI Fabric deployments with different scales and port densities. The switches support RoCEv2 and provide PFC, ECN, QoS, and other capabilities for lossless Ethernet and congestion management. They can be used to build high-bandwidth, low-latency networks for GPU clusters.

On the hardware side, 400G high-speed ports provide high-bandwidth connectivity for GPU servers. High port density also supports the scaling requirements of Spine-Leaf AI Fabrics. Switches with different port configurations and port densities can be combined based on the GPU cluster size, server access bandwidth, and Spine-Leaf topology. This helps avoid adding excessive network tiers and forwarding hops as more GPU nodes are added.

For the network architecture, Asterfusion 400G RoCE AI switches can be deployed in a Leaf-Spine architecture to build an AI Fabric. GPU servers connect to Leaf switches, while the Leaf and Spine switches are interconnected through high-speed 400G links. Uplink bandwidth and port counts can be configured based on the cluster size. For larger GPU clusters, the Spine layer can be expanded further, or a multi-tier Clos architecture can be used to scale network capacity horizontally.

ProductThroughputChipPortsHeight
CX764QO-N25.6 TbpsMarvell Teralynx 1064 × 400G OSFP2x 10G SFP+2U
CX764QD-N25.6 TbpsMarvell Teralynx 1064 × 400G QSFP-DD2x 10G SFP+2U
CX764QH-N25.6 TbpsClounix64 × 400G QSFP1121x 25G SFP282U
CX732Q-N12.8 TbpsMarvell Teralynx 732 × 400G QSFP-DD2 x 10G SFP+1U
CX732Q-N-V212.8 TbpsMarvell Falcon32 × 400G QSFP-DD2 x 10G SFP+1U
CX732Q-N-ORv312.8 TbpsMarvell Teralynx 732 × 400G QSFP-DD2 x 10G SFP+1UORv3

Conclusion

GPU utilization depends not only on GPU compute capability, but also on network communication, job contention, GPU placement, and cluster scheduling. In distributed AI training, when Collective Communication becomes part of the critical path for computation, additional network waiting time directly translates into GPU idle time.

The value of a 400G RoCE AI network therefore extends beyond higher bandwidth. Higher network capacity, low-latency forwarding, congestion control, and higher port density can help reduce communication waiting and network contention. Combined with appropriate GPU placement and AI Fabric design, these capabilities can improve computation and communication overlap and increase overall GPU cluster utilization from the network layer.

Explore More in Our 400G Technical Series

Building a resilient, high-performance 400G infrastructure requires a unified understanding of standards, interconnects, and hardware choices. Whether you are scaling an AI data center or upgrading campus backbones, explore our comprehensive 400G guide series to master every layer of the architecture:

Latest Posts