In early deployments of AI and ML, NVIDIA’s InfiniBand networking solution was the preferred choice because of its excellent low network latency. Over time, however, networking vendors and hyperscalers are actively exploring Ethernet alternatives to cater to the demands of rapidly expanding AI workloads. Currently, RoCEv2 replacing InfiniBand to carry RDMA traffic has become a common solution in the industry. Let’s explore the following article！

What is InfiniBand？

InfiniBand is a network communication protocol that facilitates the movement of data and messages by creating a dedicated, protected channel directly between nodes through a switch. RDMA and send/receive offloads managed and performed by InfiniBand adapters. One end of the adapter connects to the CPU via PCIe, and the other end connects to the InfiniBand subnet via an InfiniBand network port. This provides significant advantages over other network communication protocols, including higher bandwidth, lower latency, and enhanced scalability.

Challenges of InfiniBand Network Deployments

• Vendor lock-in: Only one vendor has mature IB products & solutions, which are expensive.
• Compatibility: InfiniBand uses a unique protocol, not TCP/IP; Specialized network, separate cabling and switches
• Availability: IB switches generally have a long delivery time
• Service: O&M depends on the manufacturer, which makes it difficult to locate faults and takes a long time to solve problems
• Expansion and upgrading: Depends on the progress of the vendor’s product releases

What is RoCE and RoCEv2?

Among these alternatives, RDMA (Remote Direct Memory Access) emerges as a groundbreaking technology that enhances data throughput and reduces latency significantly. It initially deployed on the InfiniBand, but has now been successfully integrated into Ethernet infrastructure.

Currently, the prevailing solution for high-performance networks involves constructing an RDMA-supported network based on the RoCEv2 (RDMA over Converged Ethernet version 2) protocol. This innovative approach makes use of two crucial technologies, Priority Flow Control (PFC) and Explicit Congestion Notification (ECN). The utilization of this technology offers several advantages, including cost-effectiveness, impeccable scalability, and the elimination of vendor lock-in.

For more about ROCE please read: https://cloudswit.ch/blogs/roce-rdma-over-converged-ethernet-for-high-efficiency-network-performance/#what-is-ro-ce%EF%BC%9F

Next, we’ll take a look at how Asterfusion ROCEv2 low latency SONiC switches perform against InfiniBand switches or other Ethernet switches in AIGC, distributed storage and HPC network.

RoCEv2 Switches Test Result in AIGC Network

We built an AIGC network test environment using the Asterfusion CX664D-N Ultra Low Latency Switch and another brand RoCEv2 supported switch（Switch A） to compare NCCL performance.

NCCL is short for NVIDIA Collective Communications Library. It is an open source tool from NVIDIA for testing collection communication. It can be used to test whether the collection communication is normal, and to stress test the collection communication rate.

https://github.com/NVIDIA/nccl-tests

https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/overview.html

NCCL-Test（Bandwidth & Latency）

Layer 2 network, RoCEv2 enabled on CX664P-N Switch, GPU server NICs optimized for RDMA.

64-Port 200G QSFP56 Low Latency Data Center Switch, Enterprise SONiC Ready, Marvell Teralynx

More about Asterfusion AIGC Networking Solutions: Cost Effective AIGC Network Solution by Asterfusion ROCEv2 Ready Switch

Asterfusion RoCEv2 AIGC network Solution

Your AI/ML computing cluster, hungering for speed and reliability, fueled by the power of 800G/400G/200G Ethernet, RoCEv2 technology, and intelligent load balancing. This isn’t just any connection – it’s a high-performance, low-TCO network powerhouse, courtesy of Asterfusion. No matter the size of your cluster, we’ve got you covered with our one-stop solution.

For more about Asterfusion AI network solution :  https://cloudswit.ch/solution/rocev2-ai-solution-with-dgx-superpod/

RoCEv2 Switches Test Result in HPC Network

We build an HPC test network using an Asterfusion CX564P-N ultra-low latency switch and a Mellanox InifiBand switch to compare their performance.

Test Result

Layer 2 network, RoCEv2 enabled on CX532P-N, HPC node (Mellanox NICs) optimized for RDMA

E2E forwarding performance test

HPC application performance test (WRF)

64-Port 100G QSFP28 Data Center Switch, Enterprise SONiC Ready,Low Latency: CX564P-N

RoCEv2 Switches Test Result in Distributed Storage Network

We built a storage network using Asterfusion CX532P-N ultra-low-latency switches and performed stress tests. The tests use FIO tools to complete 4K/1M read/write tests and record storage latency, bandwidth, IOPS, and other data.

Test Result

32-Port QSFP28 Low Latency Data Center Switch, Enterprise SONiC Ready, Marvell Teralynx: CX532P-N

More about Asterfusion Distributed Storage Network Solutions（NetAPP MetroCluster IP Compatible）：https://cloudswit.ch/blogs/boost-netapp-metrocluster-ip-storage-with-asterfusion-data-center-switch/

Conclusion: Asterfusion RoCEv2 low latency SONiC switches are fully capable of replacing IB switches

Asterfusion low latency network solutions offer the following advantages:

• Low cost: Less than a half of the cost of an IB switch
• Availability: Asterfusion has hundreds of low-latency switches with a lead time of 2-4 weeks.
• After-sales service: Professional, patient, reliable team, to provide 24-hour remote online technical support.
• Expansion and upgrading: Based on AsterNOS (Asterfusion enterprise ready SONiC) to support flexible functionality expansion and online upgrade

We have summarised more test data from our customers in different industries, so if you are interested, please visit our website and download it yourself. https://help.cloudswit.ch/portal/en/home