Skip to main content

Asterfusion Easy RoCE: Enabling Lossless Ethernet with a Single Command Line

written by Asterfuison

May 9, 2024

RDMA (Remote Direct Memory Access) technology is a technology that allows memory data to be transferred directly between computers, bypassing the CPU or operating system. It frees up memory bandwidth and CPU and enables communication between nodes with lower latency and higher throughput. Currently, RDMA technology is widely used in HPC, AI workloads, storage and many other scenarios.

RoCEv1 is implemented based on the Ethernet link layer and ensures reliable transmission at the physical layer through flow control techniques on the switch. RoCEv2, which is implemented on top of the UDP layer, compensates for some of the limitations of InfiniBand and enables a wider range of RDMA applications.

Why RoCEv2 Needs Lossless Ethernet?

Compared to TCP protocol, UDP is faster and consumes less resources, but unlike TCP there are mechanisms such as sliding windows and acknowledgement responses to ensure reliable transmission. In a RoCEv2 network, if there is a packet loss, the NIC will discard any packets received, so the sender needs to retransmit all subsequent packets. Therefore, we usually use PFC (Priority Flow Control) and ECN (Explicit Congestion Notification).

Configuring those functions on Ethernet switches requires familiarity with QoS mechanisms, configuration logic, and relevant command lines. It may not be difficult for an engineer who has been configuring RoCEv2 networks for customers for a long time.

But for the technicians I know who are engaged in high-performance computing and storage, they usually concentrate on servers, and this “must-have” network configurations has caused them a lot of trouble. Even the engineers who run the IB network need to spend time learning about it.

Routine Deployment Steps on a SONiC Switch

When deploying a RoCEv2 network, be sure to confirm your network hardware conditions first (e.g., low-latency network switches that support PFC and ECN, NICs that support RoCEv2).Then, start configuring and debugging.

  • Enabling and disabling: Configure the PFC and ECN separately.
  • Troubleshooting or status checking: Usually, it is necessary to go to different command line views and execute the “show” command several times. To determine the current queue mapping, buffer, enabled queues, threshold, queue throughput, Pause and CNP triggers.

So, let’s take a quick look at the basics of configuring a RoCEv2 network with Asterfusion SONiC switches, in the normal way. First step, ensure that the server NIC is working in RoCEv2 mode, and configure PCP or DSCP and enable ECN for RDMA traffic.

Then configure the switch ports to enable the PFC and ECN features and specify queues. You need to enable PFC and ECN on a designated queue (matching the server) on the Ethernet switch and adjust buffer and threshold.

Easy RoCE on AsterNOS: Multi-command Line Encapsulation and Templating

Asterfusion has launched the “EasyRoCE” feature on AsterNOS, which is optimized for the configuration and deployment of RoCE networks.

Enabling Lossless Ethernet with a Single Command Line

Troubleshooting or Status Checking

The Easy RoCE feature of AsterNOS supports the show roce command line for a one-stop view (global or interface view) of RoCE configurations and counts, as well as clearing all of them.

Automation and Visibility

The above commands can help you quickly configure Lossless Ethernet, you can also change the default template provided by the device if you need to fine-tune the parameters. Additionally, this RoCE template supports downstreaming to devices via cloud OS platforms or AI platforms.

Based on the open architecture of AsterNOS, we have also developed a containerized roce_exporter that extracts device RoCE-related information and seamlessly interfaces with Prometheus to enhance network visibility.

Related Products (All CX-N series switch products from 100G-800G support RoCE and EasyRoCE)

Latest Posts