Middle-to-Large Scale AI Compute Backend Fabric Configuration Guide

This guide provides a detailed standardized networking solution, configuration guidance, and maintenance manual for building medium-to-large scale AI compute backend fabric. The solution implements a 2-tier Clos network using Asterfusion CX864E-N switches, based on Rail-optimized architecture.

Intended for solution planners, designers, and on-site implementation engineers who are familiar with:

• Asterfusion data center switches
• RoCE, PFC, ECN, and related technologies

The Rail-optimized architecture is recommended for the deployment of backend fabric in medium-to-large scale AI clusters.

As shown above, the key design of the **Rail-optimized** architecture is to connect the same-indexed NICs of every server to the same Leaf switch, ensuring that multi-node GPU communication completes in the fewest possible hops. In this design, communication between GPU nodes can utilize internal NVSwitch [1] paths, requiring only one network hop to reach the destination without crossing multiple switches, thus avoiding additional latency. The details are as follows:

1. Intra-server: 8 GPUs connect to the NVSwitch via the NVLink bus, achieving low-latency intra-server communication and reducing Scale-Out network transmission pressure.
2. Server-to-Leaf: All servers follow a uniform cabling rule: NICs are connected to multiple Leaf switches according to the “NIC1-Leaf1, NIC2-Leaf2…”.
3. Network Layer: Leaf and Spine switches are fully meshed in a 2-tier Clos architecture.

This example illustrates an AI cluster consisting of 64 compute nodes (256 GPUs total, 4 per server). The deployment includes 6 CX864E-N: 2 Spine nodes and 4 Leaf nodes. Key design principles include:

• Each GPU connects to a dedicated NIC; NICs follow the “NIC  N  to Leaf  N ” rule. Independent subnets per Rail.
• 2-Tier Clos Fabric: Leaf and Spine switches are fully meshed. Leveraging IPv6 Link-Local, unnumbered BGP neighbors are established to exchange Rail subnet routes, eliminating the need for IP planning on interconnect interfaces.
• 1:1 Oversubscription: To ensure non-blocking transport, the oversubscription ratio on Leaf switches is strictly maintained at 1:1.
• Unified Lossless Fabric: Easy RoCE and advanced load balancing features are enabled on both Leaf and Spine nodes.

Note: For deployment convenience, it is recommended to connect the upper half of the Leaf interfaces to servers and the lower half to Spines.

The AS numbers, Loopback, and Gateway VLAN IP planning for each node are as follows:

When connecting 400G NICs to CX864E-N switches, split each of the downlink 800G port into two 400G interfaces.

After completing the configuration, verify the interface status using the `show interface summary` command.

Verify VLAN configuration using the `show vlan summary` command.

Enable the IPv6 link-local feature on Leaf-Spine interfaces to establish unnumbered BGP neighbors.

Verify BGP configuration and status using the `show bgp summary` command.

The CX-N series switches support queues 0-7 (8 queues in total). Queue 3 and queue 4 are lossless (supporting up to two lossless queues), while others are lossy.

The default template uses system-default DSCP mapping. PFC and ECN are enabled for queue 3 and queue 4, and Strict Priority (SP) scheduling is set for queues 6 and 7.

When creating a template, you can specify three parameters:

• cable-length: Specifies the cable length, affecting PFC and ECN parameter calculations. Options: 5m/40m/100m/300m. If the exact length is unavailable, choose the closest value (e.g., choose 5m for a 10m cable).
• incast-level: Specifies the traffic Incast model, affecting PFC parameters calculation. Options: low (e.g. 1:1) / medium (e.g. 3:1) / high (e.g. 10:1). Low is typically used for GPU backend fabric.
• traffic-model: Specifies the business type: throughput-sensitive, latency-sensitive, or balanced. This affects ECN parameters calculations. Options: throughput/latency/balance. balance and throughput are typically used for GPU backend fabric.

If the provided lossless RoCE configuration does not fully suit your scenario, refer to RoCE Parameter Adjustment/Optimization for fine-tuning.

Verify RoCE configuration using the `show qos roce` command.

The deployment logic for ARS follows these three phases: Create ARS Instances -> Bind Next-Hop Groups -> Fine-tune Idle-time
The following provides an explanation for each step:

A. Architectural Relationship

It is essential to understand that ARS instances and Next-Hop Groups (ECMP groups) maintain a one-to-one mapping.

At the Spine Layer: Each Leaf switch advertises unique routes. For example, the ECMP group for routes advertised by Leaf1 consists of all physical links connecting the Spine to Leaf1. Consequently, the Spine requires a dedicated Next-Hop Group for each Leaf. The number of ARS instances on a Spine switch must match the total number of Leaf switches.
At the Leaf Layer: All routes advertised by other Leafs share the same ECMP members (the uplink paths to Spine1 and Spine2). Therefore, **a Leaf switch only requires a single ARS instance** to manage all northbound traffic.

B. Binding Destination Networks

After creating the instances, it is necessary to associate the destination network segments with their corresponding ARS instances.

For Spine1: The Next-Hop Group targets the links to Leaf1; therefore, you only need to specify the Loopback 0 IP of Leaf1 as the destination.
For Leaf1: The Next-Hop Group targets the uplinks to both Spines; therefore, specifying the Loopback 0 IP of any other Leaf in the cluster will bind the traffic to the corresponding ARS instance.

C. Idle-time Calibration

Idle-time determines the granularity at which a flow is split into a series of flowlets. A flow-split is triggered whenever the inter-frame gap exceeds this defined interval.

It is recommended to set the idle-time to RTT[2]/2.  Start with the system default and fine-tune based on real-time traffic load:
Increase idle-time if significant packet reordering is detected at the endpoints.
Decrease idle-time if load distribution between the Leaf and Spine layers appears unbalanced.

Verify ARS configuration using the `show ars instance` command.

The NextHop Group Members and Member Count will reflect the actual next-hop group members and the member quantity after the route is reachable.

Verify BGP configuration and status using the `show bgp summary` command.

Verify RoCE configuration using the `show qos roce` command.

As previously described, the Spine node requires a dedicated ARS instance for each Leaf node. Each instance is then bound to its corresponding next-hop group by specifying the Loopback 0 IP of each Leaf.

The purpose of configuring Hash Seed is to mitigate Hash Polarization (also known as hash imbalance). This phenomenon occurs when traffic remains unevenly distributed across available paths after undergoing multiple stages of hashing.

Hash polarization is most prevalent in Clos topology. It typically arises when multi-tier switches utilize identical ASIC chips for ECMP, as they often employ the same hashing algorithms by default. Consequently, the second-tier switches fail to effectively redistribute traffic that was already hashed by the first tier, leading to sub-optimal bandwidth utilization and “hot spots” on certain links. This issue can be effectively resolved by adjusting the hash factors or the Hash Seed on devices at different network layers to ensure distinct hashing results at each stage.

Verify ARS configuration using the `show ars instance` command.

When default configurations are insufficient, use the following commands to optimize performance.

Note: The COS value represents the Queue ID the packet is mapped to.

If the interface has been bound to a lossless RoCE policy, unbind it before modifying.

Note: The COS value represents the Queue ID the packet is mapped to.

ECN thresholds are adjusted via min_th, max_th, and probability:

• min_th sets the lower absolute value for ECN marking (Bytes).
• max_th sets the upper absolute value for ECN marking (Bytes).
• probability sets the maximum marking probability [1-100].

PFC thresholds are adjusted via the dynamic threshold coefficient dynamic_th:
PFC threshold =  2^dynamic_th× remaining available buffer. Other parameters can remain unchanged during modification.

Recommended values for CX864E-N:

• PFC dynamic_th: 1, 2, 3
• WRED min (Bytes): 1,000,000, 2,000,000, 3,000,000
• WRED max (Bytes): 8,000,000, 10,000,000, 12,000,000
• WRED probability (%): 10, 30, 50, 70, 90

Note: Try ECN adjustment first, then PFC. Follow the principle: WRED Min < WRED Max < PFC xON < PFC xOFF. This ensures ECN triggers rate adjustment early during congestion to avoid unnecessary PFC, while still allowing PFC to trigger promptly when necessary to prevent packet loss.

The specific command lines to adjust the PFC and ECN thresholds are as follows: