How AsterFabric Matrix Optimizes PD-Disaggregated AI Inference Networks

Traditional large-model compute clusters generally adopt symmetric topologies such as Clos/Fat-Tree and ROFT (Rail-Optimized Fat-Tree). As cluster scale expands and inference workloads grow more complex, these topologies have gradually exposed problems such as high hardware cost, load-balancing failure under asymmetric traffic, and large tail-latency jitter. The AsterFabric Matrix (Asterfusion orthogonal dual-plane network) topology removes the Spine layer and fully interconnects grouped Leaf switches to form a complete bipartite graph. Combined with mixed single-rail and multi-rail access, a constant two-hop forwarding path across the entire network, and deterministic routing, it eliminates avoidable congestion at the architectural level.

This document focuses on the design of the AsterFabric Matrix inference network for AI computing. Using Asterfusion data center 800G high-density interface switches as the core hardware platform, it adopts the Matrix topology to provide a standardized deployment solution.

This manual is intended primarily for solution planning and design personnel and on-site implementation engineers, who should have the following capabilities:

• Familiarity with Asterfusion data center network switch products
• An understanding of RoCE, RDMA, and the prefill-decode (PD) disaggregated architecture for LLM inference

Large-model inference is currently evolving from the traditional “shared training-inference cluster” model toward dedicated inference architectures based on Prefill-Decode (PD) disaggregation. PD disaggregation decouples compute-intensive prompt processing (Prefill) from memory-access-intensive token generation (Decode) into separate instance pools, significantly improving inference throughput and resource utilization. However, this architectural change places new and stringent requirements on the underlying network:

• Cross-node KV Cache transfer becomes the core traffic: After a Prefill node generates the KV Cache, it must be transferred to a Decode node with high bandwidth and low latency, with transfer sizes varying dynamically from the MB level to the GB level;
• Highly asymmetric traffic: The relationship between the KV Cache source node (Prefill) and destination node (Decode) is non-deterministic and NIC load is highly unbalanced, so the load-balancing mechanisms of traditional symmetric topologies fail;
• Extreme sensitivity to tail latency: Time to first token (TTFT) and time per output token (TPOT) directly determine user experience, and any congestion or jitter at the network layer is amplified in a cascading manner.

Traditional large-model compute clusters generally adopt symmetric topologies such as Clos/Fat-Tree and ROFT, which were originally designed to serve symmetric collective communication such as All-Reduce and All-to-All in training scenarios. When reused directly for PD-disaggregated inference, they gradually expose problems such as hash polarization, structural congestion, large tail-latency jitter, and high hardware cost. The AsterFabric Matrix topology proposed in this solution (hereinafter referred to as Matrix) is a network architecture designed specifically for PD-disaggregated inference scenarios. By removing the Spine layer, fully interconnecting grouped Leaf switches to form a complete bipartite graph, combining mixed single-rail and multi-rail access, providing a constant two-hop forwarding path across the entire network, and using deterministic routing, it eliminates avoidable congestion in PD-disaggregated scenarios at the architectural level, providing an optimal network foundation for large-scale GPU inference clusters.

In large-model inference, PD disaggregation has become the core architecture for improving system efficiency:

• Prefill stage: Receives the complete user prompt and performs a forward pass to generate the KV Cache. This stage is compute-intensive, with a large amount of computation per execution but a relatively low execution frequency;
• Decode stage: Generates output autoregressively, token by token, based on the KV Cache produced by Prefill. This stage is memory-access-intensive, with a small amount of computation but an extremely high execution frequency, and is extremely latency-sensitive.

PD disaggregation deploys the two stages in independent GPU instance pools, and a scheduler transfers the KV Cache generated by a Prefill instance to a designated Decode instance, achieving the following:

• Compute specialization: The Prefill pool can be configured with high-compute GPUs, while the Decode pool can be configured with high-memory-capacity/high-bandwidth GPUs;
• Elastic scaling: The number of Prefill or Decode instances can be scaled independently according to service load, with a typical P:D ratio of 1:3;
• Batch optimization: The Prefill stage can make full use of large batches to improve computational efficiency, while the Decode stage reduces idle time through Continuous Batching.

After PD disaggregation decouples the two stages, the network traffic model changes fundamentally and differs markedly from that of traditional training scenarios:

Table 1. Comparison of Traffic Models: Training vs. PD-Disaggregated Inference

This combination of “source-destination asymmetry, dynamically varying traffic size, and nearly irregular traffic direction” renders load-balancing mechanisms based on ECMP hashing or static Rail mapping completely ineffective.

2.3.1 Hash Polarization

Traditional Clos/Fat-Tree architectures rely on ECMP per-flow load balancing. In PD-disaggregated scenarios, when multiple high-volume KV Cache transfers (elephant flows) are processed by the hash calculation, they are very likely to be assigned to the same member link, congesting that link while other links remain idle. This hash polarization phenomenon—“ample total bandwidth but severe local congestion”—is the primary cause of surging tail latency in PD-disaggregated inference.

2.3.2 Structural Congestion

The ROFT architecture improves All-Reduce efficiency in training scenarios through Rail optimization, but its design assumes that traffic is symmetric and predictable.

In PD disaggregation, the source-destination relationship of KV Cache transfers is determined entirely and dynamically by the scheduler, and Prefill nodes may be concentrated under a small number of Leaf switches, causing the uplinks of those Leaf switches to be hit by concentrated traffic and forming architecture-level hotspots. This congestion is not caused by transient traffic bursts, but is structural congestion resulting from a mismatch between the network topology and the PD-disaggregated traffic model.

2.3.3 Multi-Hop Forwarding and the Spine Bottleneck

In a traditional three-tier Clos architecture, inter-Leaf traffic must traverse the Spine layer (Leaf → Spine → Leaf, three hops in total). In PD-disaggregated scenarios, more than 90% of KV Cache transfers are inter-Leaf traffic, and all such traffic must converge at the Spine layer. Even if the Spine layer is provisioned with oversubscription, the Spine-Leaf links still become an unavoidable bottleneck, and multi-hop forwarding further amplifies transmission latency.

The design goal of the Matrix architecture is to eliminate structural congestion in PD-disaggregated scenarios at the network topology level, rather than relying on upper-layer scheduling or more complex transport-layer protocols. Its core design principles include:

• Removing the Spine layer: Completely eliminates the aggregation bottleneck for inter-Leaf traffic and reduces the network diameter from 3 hops to 2 hops;
• Complete bipartite interconnection: Leaf switches are divided into two groups forming two planes, with Full-Mesh interconnection between the Leaf switches of the two planes, providing the shortest and a deterministic path for KV Cache transfers;
• Mixed single-rail + multi-rail access: Through orthogonal dual partitioning, structured Prefill/Decode node blocks are evenly spread across the underlying links, achieving inherent load balancing;

Deterministic routing: There is only one optimal path between any pair of GPUs, eliminating ECMP hash collisions.

The Matrix architecture removes the Spine switching layer of the traditional Fat-Tree architecture and divides all Leaf switches evenly into two groups of equal size:

• Plane A (multi-rail plane, odd-numbered Leaf group): numbered 1, 3, 5, …
• Plane B (single-rail plane, even-numbered Leaf group): numbered 2, 4, 6, …

A complete bipartite full-mesh is built between the two groups of switches: every Plane A switch has a direct link to every Plane B switch, and switches within the same plane are not connected to each other.

The dual-port NIC of each GPU connects to the two groups of switches respectively:

• First port (multi-rail port): Connects to the odd-numbered Leaf switches in Plane A according to the multi-rail rule
• Second port (single-rail port): Connects to the even-numbered Leaf switches in Plane B according to the single-rail rule

The topology and cabling relationships are as follows, using a minimal network of 4 GPU servers and 4 Leaf switches per group as an example:

The server’s multi-rail ports connect respectively to the 4 Leaf switches in Plane A, and the server’s single-rail ports connect respectively to the 4 Leaf switches in Plane B; the two groups of Leaf switches are fully interconnected. Under this network architecture, the forwarding path between any two GPUs across the entire network is always two hops: GPU → local-group Leaf → opposite-group Leaf → GPU. For PD-disaggregated scenarios, this means that a KV Cache transfer from any Prefill node to any Decode node passes through only 2 switches, and the path is the shortest and unique.

The GPU-to-GPU traffic forwarding paths in different cases are as follows:

• Same-Rail Communication

GPU 1 on Server 1 accesses GPU 1 on Server 2:

① Server 1-1-A → Leaf A1 → Leaf B1 → Server 2-1-B, forwarded in two hops;

② Server 1-1-B → Leaf B1 → Leaf A1 → Server 2-1-A, forwarded in two hops.

• Cross-Rail Communication

GPU 1 on Server 1 accesses GPU 4 on Server 2:

① Server 1-1-A → Leaf A1 → Leaf B1 → Server 2-4-B, forwarded in two hops;

② Server 1-1-B → Leaf B1 → Leaf A4 → Server 2-4-A, forwarded in two hops.

3.4.1 Dual-Port NIC Access

The server NIC uses bond mode with dual uplinks, connecting to the Leaf switches of the two planes respectively. To ensure that the two Leaf switches maintain consistent ARP requests and MAC address learning for the server, the following mechanisms must be enabled:

• Dual ARP transmission from the NIC: The server NIC must be able to send ARP requests simultaneously on both uplink ports, so that both Leaf switches can learn the server’s MAC address and IP address;
• Switch ARP-to-Host route function: The Leaf switch enables the ARP-to-Host route function, converts the learned ARP entries into 32-bit host routes, and advertises them to the other plane via the upstream BGP protocol.

3.4.2 Routing Scheme

To achieve the constant two-hop forwarding described above, one-hop routes (traffic must not be forwarded via local routes) and multi-hop routes (to prevent Plane A traffic from passing through Plane B and then returning to Plane A) must be strictly filtered:

• Leaf switches in the same group are configured with the same AS: The BGP AS-path attribute is used to filter routes of more than two hops, ensuring that traffic is not forwarded within the same plane or detoured across planes multiple times;
• IPv6 link-local addresses establish unnumbered BGP neighbors: No IP addresses need to be planned for the interconnect interfaces between Planes A and B; BGP neighbors are established automatically using IPv6 link-local addresses, simplifying deployment;
• Only host routes are advertised: Through the ARP-to-Host route mechanism, each Leaf advertises only the host routes of its directly connected GPUs, keeping the routing table compact and convergence fast.

3.4.3 Load-Balancing Technology

Traditional ECMP per-flow load balancing has a critical flaw in PD-disaggregated scenarios: when traffic characteristics are homogeneous (for example, a large number of KV Cache elephant flows), the hash algorithm tends to map multiple large flows onto the same member link, causing congestion.

Matrix’s load-balancing capability comes from the topology itself, not from ECMP hashing:

• Inherently uniform distribution of the complete bipartite graph: Every Plane A Leaf is fully interconnected with all Plane B Leaf switches, so traffic spreads naturally across the inter-group links and there are no inherent structural hotspot links;
• Mixed single-rail + multi-rail access: The multi-rail ports spread GPUs in the same slot across different Plane A Leaf switches, while the single-rail ports concentrate consecutively numbered GPUs on a Plane B Leaf. This orthogonal dual partitioning ensures a uniform path distribution whether traffic is Rail-aligned (training pattern) or highly unbalanced between source and destination (PD-disaggregated pattern).

Taking a 51.2 Tbps switch and 400 Gbps of access bandwidth per GPU as an example, with the Matrix architecture a single Pod can connect 128 GPU servers (1K GPUs), and the network can scale to a maximum of 16 Pods, i.e., a maximum cluster size of 16K GPUs.

Using an 8K GPU cluster as the baseline, the hardware configurations of three network topologies—AsterFabric ROFT, AsterFabric Dual-Mesh (rail-optimized dual-plane network), and AsterFabric Matrix—are compared as follows:

Table 2. Hardware Configuration Comparison for an 8K GPU Cluster

As the table shows, while supporting a GPU cluster of the same scale, the Matrix topology reduces the number of switches by 33% and hardware cost by approximately 30%–40%, and also provides better transmission efficiency for the asymmetric traffic model of PD disaggregation.

4.1.1 Network Topology Solution

The figure below shows an AI computing Matrix network topology for 128 Prefill/Decode nodes (1,024 GPUs). Each server contains 8 GPU cards, and each GPU corresponds to a dual-port 200G network interface. A total of 16 CX864E-N switches are deployed, of which 8 are in Plane A (odd-numbered Leaf nodes) and 8 are in Plane B (even-numbered Leaf nodes). The CX864E-N is equipped with 64 800GE interfaces; through interface breakout, it can provide 128 200G interfaces (between Leaf and Server) and 64 400G interfaces (between Leaf switches).

The core design principles of the Matrix architecture are as follows:

• Each GPU connects to a dedicated 400G NIC, and each 400G NIC provides two 200G ports that connect to the two planes respectively. Server ports must be connected strictly according to the following rules: the first port of each server’s NIC connects to the Leaf switches in Plane A (odd-numbered nodes) according to the rule “NIC1-1 to Leaf1, NIC2-1 to Leaf3, NIC3-1 to Leaf5…”, and the second port connects in sequence to the Leaf switches in Plane B (even-numbered nodes);
• Each plane has only a single tier of Leaf switches, with no interconnect links between Leaf switches within the same plane; the Leaf switches of Planes A and B are fully interconnected in a Full-Mesh, using the IPv6 link-local feature to establish unnumbered BGP neighbors and advertise the host routes within each plane to achieve route exchange, without the need to plan IP addresses for the interconnect interfaces between Planes A and B;
• The ratio of Leaf downlink to uplink capacity must strictly follow a 1:1 convergence ratio to guarantee non-blocking transmission;
• All switches enable one-click RoCE and load-balancing features to build a lossless network.

4.1.2 Device Selection

For building medium-to-large-scale RoCEv2 networks, it is recommended to use the Asterfusion data center switch CX864E-N, leveraging its ultra-low forwarding latency: the CX864E-N offers end-to-end forwarding latency as low as 560 ns, enabling ultra-low-latency data transmission and fully meeting the stringent low-latency requirements of RoCEv2 networks. The Matrix topology removes the Spine layer, so Leaf switches serve both access and interconnect roles; all nodes within a single Pod are deployed with the same model, so a single spare-part model suffices for operations and maintenance.

In a Matrix network, taking the NVIDIA DGX H100 GPU server (with 8 GPUs per server) as an example, a Pod requires 16 Leaf nodes, with 8 Leaf switches in each of the two planes. The maximum number of servers that can connect to each Pod is related to the interface configuration of the Leaf nodes. To maintain a 1:1 convergence ratio, half of each Leaf node’s interfaces are used to connect GPU servers, and the other half are used to connect to the Leaf nodes of the other plane.

The table below shows the node configuration requirements for deploying Matrix networks with different GPU counts using the CX864E-N:

Table 3. Node Configuration Requirements for Deploying Matrix Networks with Different GPU Counts Using the CX864E-N

As large-model inference evolves from a mixed “training-inference” model toward dedicated PD-disaggregated architectures, the underlying network is undergoing a paradigm shift from “general-purpose symmetric topologies” to “inference-dedicated asymmetric topologies.” The AsterFabric Matrix architecture proposed in this solution eliminates structural congestion in PD-disaggregated scenarios at the network topology level through designs such as removing the Spine layer, building a complete bipartite graph, mixed single-rail/multi-rail access, and deterministic routing, providing a complete solution for dedicated large-scale GPU inference clusters.