1. Overview
AI backend networks can be deployed using a Rail-Optimized architecture. The AI backend network scenario built into the Asterfusion AIDC Network Operations Platform is based on this architecture.
As shown in the preceding figure, the core design principle is to connect the NICs with the same index on every server to the same Leaf switch. This minimizes the hop count for inter-server GPU communication. With this design, GPUs within a server communicate through the local NVSwitch[1] fabric. Traffic requires only a single network hop to reach the destination server, eliminating unnecessary switch hops and reducing latency.
① Intra-server connectivity: Eight GPUs are connected to the NVSwitch through the NVLink interconnect. This provides low-latency communication between GPUs within the same server and reduces traffic on the scale-out network.
② Server-to-network connectivity: All servers follow the same cabling model. NICs are connected to Leaf switches in a fixed pattern, such as NIC1 → Leaf1, NIC2 → Leaf2, and so on.
③ Network connectivity: Leaf and Spine switches are deployed in a full-mesh topology using a two-tier Clos architecture.
2. Deployment of AI Backend Network
The following example describes an AI cluster with 16 compute nodes (each server has 4 GPUs, 64 GPUs in total). The deployment uses six CX532-N switches, including two Spine nodes and four Leaf nodes. The key configuration design is as follows:
- Each GPU connects to a dedicated NIC. NICs on each server follow a fixed mapping pattern, such as NIC1 → Leaf1, NIC2 → Leaf2, to connect to Leaf switches. Each Rail is assigned a dedicated subnet. Leaf switches act as the default gateway within each Rail.
- The network uses a two-tier Clos architecture. Spine and Leaf switches are fully interconnected. Unnumbered BGP sessions are established using IPv6 link-local addresses. Route advertisement is used to distribute Rail subnet routes. No IP addressing is required on Leaf-Spine interconnect interfaces.
- The oversubscription ratio between Leaf downlink and uplink capacity must be strictly 1:1 to ensure non-blocking forwarding.
- Leaf and Spine switches enable one-click RoCE optimization and load balancing features to build a lossless network.
Figure 2.1-1 shows the AI backend network topology. The following sections describe how to deploy this topology and enable services using the AIDC Network Operations Platform.
The AS numbers, Loopback addresses, and gateway VLAN IP addressing plan for each node is shown below:
Table 2.1-1 AS Numbers and IP Addressing Plan for Each Node
2.1 Create Venue
A venue is a logical container used by administrators to monitor, manage, and configure all network devices.
First, log in to the Network Operations Platform. Navigate to Navigation > Map. Create a new venue for isolated management of the AI backend network to be deployed. All subsequent inventory management and network configuration tasks are performed within this site.
Figure 2.1-1 Create Venue
Figure 2.1-2 Display Site Venue
2.2 Inventory Provisioning
Devices can be created or batch-imported into a specified site or organization. When devices added to the inventory come online and connect to the Network Operations Platform, the platform automatically assigns them to the corresponding organization or site based on their MAC address.
Two methods are available for inventory configuration:
- In the Organization view, you can assign inventory devices to a specific organization or site.
- In the Site view, newly added inventory devices are directly assigned to the current site.
In this example, the second method is used. Inventory devices are configured directly in the created AIDC-BE Venue.
In the AIDC-BE venue view, navigate to Configuration > Inventory Information. In this section, CSV-based batch import is used to quickly provision inventory devices.
Figure 2.2-1 Import Inventory Devices
Click Select File to import the pre-planned inventory CSV file. In this example, the CSV content is shown in Table 2.2-1. After clicking Start Test Upload Data, the platform displays the validation result of the CSV file, as shown in Figure 2.2-2.
After validation is completed, click Start Upload Devices to finish the inventory import. Figure 2.2-3 shows the device inventory after the import is completed.
Table 2.2-1 Inventory Device Information Table
Figure 2.2-2 Uploading Inventory Devices
Figure 2.2-3 Device Inventory Information
You can also manually add inventory devices. Click the 【Create】 button, enter the required information, and complete the inventory device creation.
Figure 2.2-4 Manual Inventory Entry
2.3 Device Automatic Onboarding
The devices must have Layer 3 connectivity with the network management platform (in this example, all devices communicate with the platform through the out-of-band management network). After powering on, the devices need to be configured with the platform IP address to establish a connection. It is recommended to use an Ansible playbook to execute the following commands in batches to enable the uCentral feature and specify the platform server IP address.
sonic# configure terminal
sonic(config)# feature ucentral state enable
sonic(config)# feature ucentral autorestart enable
sonic(config)# ucentral-client server 10.240.3.201
After the configuration is completed, the platform automatically discovers the devices and assigns them to the AIDC-BE site according to the imported inventory information. In the AIDC-BE site view, click 【Devices】 to view the information and status of all devices.
Figure 2.3-1 Device Information
2.4 Topology Planning
2.4.1 Importing Planned Topology
The AIDC Network Operations Platform has built-in support for three typical AI data center scenarios. In this example, the AI Data Center Backend Network Deployment scenario is selected. In the AIDC-BE site view, go to 【Configuration】 – 【Topology Planning】. When accessing topology planning for the first time, the platform will directly open the scenario selection page. Select 【AI Data Center Backend Network Deployment】.
Since the topology will be imported using a CSV file, click the 【Generate Blank Topology】 button to clear the default Spine/Leaf types and quantities.
Figure 2.4-1 Scenario Selection
After completion, a blank topology will be generated. Click 【Import Configuration】 and select the prepared planned topology CSV file. The content of the CSV file is shown in Figure 2.4-2. In this topology, interfaces 1-8 on each Leaf are connected to Spine-1, and interfaces 9-16 are connected to Spine-2. The platform automatically validates the CSV content and displays the validation results, as shown in Figure 2.4-3.
2.4-2 Planned Topology CSV Information
Figure 2.4-3 Import Topology Configuration
After the validation passes, click 【Next】 to enter the configuration analysis page. The page displays detailed information about the topology configuration to be imported. After confirmation, click 【Apply Configuration】 to automatically complete the topology planning. Figure 2.4-5 shows the planned topology generated after applying the configuration.
Figure 2.4-4 Apply Configuration
Figure 2.4-5 Generated Topology
The platform also supports manually configuring the planned topology. For manual topology planning, refer to Sections 2.4.2 and 2.4.3.
2.4.2 (Optional) Scenario Selection
In the AIDC-BE venue view, go to 【Configuration】 – 【Topology Planning】. When accessing topology planning for the first time, the platform will directly open the scenario selection page. Select 【AI Data Center Backend Network Deployment】. According to the network topology shown in Figure 2.1-1, configure two Spine nodes and four Leaf nodes (in this example, the device type is set to SONiC-VM). Click the 【Complete】 button to automatically create the topology template, as shown in Figure 2.4-7.
Figure 2.4-6 Scenario Selection and Device Quantity Configuration
Figure 2.4-7 AIDC Backend Network Topology Template
2.4.3 (Optional) Topology Editing
According to the topology design, the topology template needs to be edited. Click the 【Edit】 button on each device in the planned topology to enter the editing page. On the editing page, select a device from the inventory to complete the device role assignment.
Figure 2.4-8 Device Role Configuration
Figure 2.4-9 Device Role Configuration Results
After completing the device role assignment, configure the links between devices according to the network topology. Interfaces 1-8 on each Leaf are connected to Spine-1, while interfaces 9-16 are connected to Spine-2. The remaining 16 interfaces on each Leaf are used to connect GPU servers.
Using the interface connections on Spine-1 as an example, click the 【Edit】 button to enter the editing page. After selecting the local interface and the peer interface, click 【Save】 to complete the link configuration. The completed topology after all link configurations are applied is shown in Figure 2.4-10.
Figure 2.4-10 Inter-Device Link Configuration
Figure 2.4-11 Complete Planned Topology
2.4.4 Topology Validation
Click the 【Topology Consistency Validation】 button to verify the planned topology against the actual topology. If any inconsistencies are detected, the platform will display the specific reasons for the discrepancies and the actual topology information. Ensure that the validation passes before proceeding with subsequent configuration operations to avoid potential service issues.
Figure 2.4-12 Topology Consistency Validation
Figure 2.4-13 Topology Validation Results View
2.5 Basic Network Configuration
The basic network configuration for the AI Data Center Backend Network scenario consists of interface configuration and BGP configuration. Its main purpose is to establish link connectivity and routing protocol communication between Spine and Leaf devices.
Click the 【Basic Network Configuration】 button to enter the configuration page.
Figure 2.5-1 Basic Network Configuration Button
Figure 2.5-2 Basic Network Configuration Page
2.5.1 Interface Configuration
In this example, Leaf and Spine devices use unnumbered BGP neighbors to exchange routes. Since IP addresses do not need to be manually assigned to the interconnection interfaces, the interface configuration step is skipped.
2.5.2 BGP Configuration
Click Step 2 【AIDC Backend Network BGP Deployment】 to configure BGP. Refer to the AS numbers planned in Table 2.1-1 and configure BGP on each device.
Taking Spine-1 as an example, enable BGP and configure the local AS number in the Spine-1 configuration section.
Figure 2.5-3 Enable BGP and Configure Local AS
Click the 【Create BGP PEER-GROUP】 button to enter the BGP peer configuration page. All interfaces on Spine-1 establish BGP neighbors with Leaf-1 to Leaf-4. The final configuration is shown in Figure 2.5-4.
Figure 2.5-4 Configure BGP Peers
Click the 【Complete】 button to finish the peer configuration. Figure 2.5-5 shows the BGP configuration information after BGP is configured on Spine-1. Configure BGP on the remaining devices in the same way according to the planned topology.
Figure 2.5-5 Spine-1 BGP Configuration Information
2.5.3 Configuration Deployment
After completing the basic network configuration, click the 【Push Configuration】 button to push the configuration to the devices and establish connectivity between Spine and Leaf devices.
Figure 2.5-6 Push Basic Network Configuration Button
Select the devices for configuration deployment and click 【Next】 to start the deployment. Wait until the deployment is completed successfully before proceeding with wired service configuration.
Figure 2.5-7 Select Devices for Configuration Deployment
Figure 2.5-8 Start Configuration Deployment
2.6 Wired Service Configuration
After completing the basic network configuration, Spine and Leaf devices have established connectivity. Click the 【Wired Service Configuration】 button to configure network services.
Figure 2.6-1 Wired Service Configuration Button
After entering the wired service configuration page, click the 【Create】 button to create a wired service configuration template.
Figure 2.6-2 Wired Service Configuration Template
To meet the requirements of lossless networking in AI backend networks, the wired service configuration module of the network operations platform supports RoCE, service VLANs, intelligent path selection, and ARS features. In this example, RoCE and ARS are selected to work together to build a lossless network.
2.6.1 Leaf Node Service Configuration
a. Configure One-Click RoCE
CX-N series switches support eight queues (Queue 0-7). Among them, Queue 3 and Queue 4 are lossless queues (supporting a maximum of two lossless queues), while the remaining queues are lossy queues.
The default template uses the system default DSCP mapping. Queue 3 and Queue 4 have PFC and ECN enabled, while Queue 6 and Queue 7 use strict priority scheduling.
When creating a template, you can specify the following three parameters:
- cable-length: Specifies the cable length, which affects the calculation of PFC and ECN parameters. Available options are 5m/40m/100m/300m. If the actual cable length is not available, select the closest value. For example, if the actual cable length is 10 meters, select 5m.
- incast-level: Specifies the traffic incast model, which affects the calculation of PFC parameters. Available options are low (for example, 1:1), medium (for example, 3:1), and high (for example, 10:1). In GPU backend networks, low is generally recommended.
- traffic-model: Specifies the traffic type, including throughput-sensitive, latency-sensitive, and balanced traffic models. This parameter affects the calculation of ECN parameters. Available options are throughput/latency/balance. In GPU backend networks, the balance or throughput mode is generally selected.
If the provided lossless RoCE configuration does not fully match your service scenario, refer to Section 3.1 RoCE Parameter Tuning and Optimization to adjust the configuration and tune parameters for optimal service performance.
Taking Leaf-1 RoCE configuration as an example, after entering the template name and selecting the device on the wired service configuration page, click 【RoCE Feature】 – 【EasyRoCE】 – 【Create RoCE】 to apply the RoCE configuration to all interfaces.
Figure 2.6-3 Configure RoCE
Click 【Add】 to complete the RoCE configuration.
Figure 2.6-4 RoCE Configuration Information
b. VLAN Configure Service VLANs
According to the network topology and the gateway VLAN IP address plan in Table 2.1-2, configure service VLANs on the Leaf devices.
Taking Leaf-1 as an example, click 【Service VLAN】 – 【Create】 to enter the service VLAN configuration page. The planned service VLAN for Leaf-1 is 101, with the gateway IP address 10.10.1.1/25. According to the planned topology, the downlink interfaces are configured as interfaces 17-32. Configure the settings based on the above information.
Figure 2.6-5 Create Service VLAN
Click 【Add】 to complete the service VLAN creation.
Figure 2.6-6 Service VLAN Configuration Information
c. Configure ARS
The ARS configuration workflow is as follows: Enable ARS → Create an ARS instance → Bind a next-hop group → Adjust idle-time.
Each step is described below:
(1) First, enable the ARS feature. Subsequent ARS instance configurations are based on ARS being enabled.
(2) Second, it should be noted that an ARS instance and a next-hop group (also known as an ECMP group) have a one-to-one relationship. For Spine devices, the routes advertised by each Leaf are different. For example, the ECMP group members for routes advertised by Leaf-1 are the multiple links between Spine and Leaf-1. Therefore, each Leaf has a corresponding next-hop group. The number of ARS instances that need to be created is equal to the number of Leaf devices.
For Leaf-1, however, the ECMP group members for all routes advertised by other Leaf devices are the links connected to Spine-1 and Spine-2. Therefore, only one ARS instance needs to be created on Leaf-1.
(3) Next, specify the destination network of the route and bind the desired next-hop group to the corresponding ARS instance. For Spine-1, the next-hop group consists of the ECMP group formed by the links connected to Leaf-1. Therefore, only the Loopback0 IP address of Leaf-1 needs to be specified. For Leaf-1, the next-hop group consists of the ECMP group formed by the links connected to the two Spine devices. Therefore, only the Loopback0 IP address of any other Leaf device needs to be specified.
(4) Finally, configure idle-time. Idle-time determines the granularity at which a flow is split into a series of Flowlets (sub-flows). Flow splitting is triggered when the inter-frame gap exceeds this time value. In theory, idle-time is recommended to be configured as RTT[2]/2. In practice, the default value can be used initially and adjusted according to the actual traffic load.
If significant packet reordering is observed at the endpoint, increase the idle-time value. If load balancing between Leaf and Spine devices is uneven, decrease the idle-time value.
Based on the above description, configure ARS using Leaf-1 as an example. Click 【ARS】 – 【Create ARS Profile】 to enter the ARS Profile page. This section is used to enable the ARS feature.
Figure 2.6-7 Enable ARS Feature
Click 【Save】 to save the configuration. After enabling the ARS feature, the 【Create ARS Instance】 button will appear. Click it to enter the ARS instance configuration page.
Figure 2.6-8 Configure ARS Instance
Click 【Save】 to complete the ARS instance creation.
Figure 2.6-9 ARS Configuration Information
At this point, all service configurations on Leaf-1 are complete.
2.6.2 Spine Node Service Configuration
The service configuration for Spine devices includes RoCE and ARS configuration. The following uses Spine-1 as an example.
a. Configure One-Click RoCE
Create a new template on the wired service configuration page. After entering the template name and selecting the device, click 【RoCE Feature】 – 【EasyRoCE】 – 【Create RoCE】 to apply the RoCE configuration to all interfaces. After completing the configuration, click 【Add】 to finish the RoCE configuration.
Figure 2.6-10 RoCE Configuration Information
b. Configure ARS
As described above, for Spine devices, an ARS instance should be created for each Leaf device. The Loopback0 IP address of each Leaf should be specified to bind the corresponding next-hop group.
Click 【ARS】 – 【Create ARS Profile】 to enter the ARS Profile page. This section is used to enable the ARS feature. The configuration steps are the same as those for Leaf-1 and are not repeated here.
Click 【Create ARS Instance】 to enter the ARS instance configuration page. Four instances need to be configured, corresponding to the four Leaf devices.
Figure 2.6-11 Spine-1 ARS Instance Configuration
At this point, the wired service configuration for Spine-1 is complete. Configure the remaining devices in the same way according to the service plan.
2.6.3 Configuration Deployment
After completing the service configuration for all devices, the configured service template information can be viewed on the wired service configuration page.
Figure 2.6-12 Service Configuration Template Information
Click the 【Push Configuration】 button to deploy a single template to a device. You can also click 【Filter Configuration Templates】 to enter the batch deployment page. In this example, batch configuration deployment is selected. On the configuration template filtering page, click 【Select All】 to select all configuration templates.
Figure 2.6-13 Configuration Template Filtering Page
Click 【Batch Configuration Deployment】 – 【Next】 to start deploying the configuration to devices. After the service deployment is completed, the entire network service deployment is complete.
Figure 2.6-14 Batch Service Configuration Deployment
Figure 2.6-15 Service Configuration Deployment Completed
3. RoCE and Daily Maintenance
3.1 RoCE Parameter Tuning and Optimization
When the provided lossless RoCE configuration does not fully match your service scenario, you can adjust the PFC/ECN thresholds in the corresponding service configuration template on the wired service configuration page to optimize service performance.
The ECN threshold is adjusted through min_th, max_th, and probability:
- min_th: Sets the lower threshold for Explicit Congestion Notification (ECN), in bytes. When the packet length in the queue reaches this value, the interface starts marking the ECN field of packets as CE based on the configured probability.
- max_th: Sets the upper threshold for ECN, in bytes. When the packet length in the queue reaches this value, the interface marks the ECN field of all packets as CE.
- probability: Sets the maximum marking probability. The value is an integer ranging from [1,100].
The PFC threshold is adjusted by modifying the dynamic threshold coefficient dynamic_th:
PFC threshold = 2^dynamic_th × remaining available buffer
Taking the Spine-1 service configuration template as an example, select the Spine-1 service configuration template and click 【Edit】 – 【RoCE Feature】 – 【PFC/ECN】. The PFC/ECN policy information automatically generated by Easy RoCE will be displayed.
Figure 3.1-1 PFC/ECN Information
Select the ECN entry and click the 【Edit】 button to adjust the ECN threshold. After making the adjustment, click 【Save】 to complete the ECN threshold update.
Figure 3.1-2 ECN Threshold Adjustment
Select the PFC entry and click the 【Edit】 button to adjust the PFC threshold. After making the adjustment, click 【Save】 to complete the PFC threshold update.
Figure 3.1-3 PFC Threshold Adjustment
After completing the adjustment, save the Spine-1 service configuration template and redeploy it to the device to apply the updated PFC/ECN thresholds.
3.2 Device Daily Maintenance
Click 【Devices】 and select the device you want to view to enter the device page. This page provides a wide range of device-level maintenance functions.
Figure 3.2-1 Device Information
【Overview】 Page:
Displays basic device information, interface status, real-time alarm information, and associated links.
Figure 3.2-2 Device Overview
【Details】 Page:
【System Details】 displays the device overall health score, hardware temperature, fan status, power supply status, and other information.
Figure 3.2-3 Device System Details
【Service Details】 displays the main service status information of the device.
Figure 3.2-4 Device Service Details
【Statistics】 Page:
【Overview】 provides device CPU usage, memory usage, interface rates, throughput, and LLDP information.
Figure 3.2-5 Device Statistics Overview
【Interfaces】 provides more comprehensive interface statistics.
Figure 3.2-6 Device Interface Statistics Details
【Optical Transceivers】 provides detailed optical transceiver information.
Figure 3.2-7 Device Optical Transceiver Information
【Buffer】 provides device buffer statistics.
Figure 3.2-8 Device Buffer Statistics Information
【RoCE】 provides detailed RoCE statistics.
Figure 3.2-9 Device RoCE Statistics
【Configuration】 Page:
Displays the basic configuration information of the device.
Figure 3.2-10 Device Configuration Information