SRv6 vs VXLAN: SRv6 Is Changing Data Center Network Fabric

Over the past decade of cloud computing evolution, EVPN-VXLAN has become the de facto foundation for data center virtualization and multi-tenant networking. It removes the 4096-VLAN scaling limit and builds a virtual Layer 2 network (Overlay) over a Layer 3 physical network (Underlay), enabling seamless workload mobility.

However, the rise of distributed LLM training, high-density GPU clusters, and cross-data-center collaborative computing has fundamentally changed network traffic patterns. Traditional VXLAN architectures are increasingly showing structural limitations when handling massive elephant-flow scheduling, cross-domain protocol integration, and packet encapsulation overhead. SRv6 (Segment Routing over IPv6), based on native IPv6, is rapidly moving from WAN deployments into data centers and becoming a core foundation for next-generation compute fabrics.

In traditional general-purpose cloud environments, VXLAN effectively meets the multi-tenant isolation requirements of virtual machines and containers. However, as infrastructure evolves toward heterogeneous computing and multi-data-center collaboration, the limitations of VXLAN are becoming increasingly apparent:

1. Cross-Domain “Protocol Stitching” and Gateway Bottlenecks

In multi-data-center interconnect (DCI) or hybrid cloud scenarios, data centers typically run EVPN-VXLAN, while WAN backbones often use MPLS, SR-MPLS, or pure IP routing.

• State Overhead and Single-Point Bottlenecks: Border gateways (Border Leaf / DCI Gateway) must handle complex protocol translation and state maintenance. They need to decapsulate VXLAN, remove internal labels, perform route lookups, and re-encapsulate packets for MPLS/WAN transport.
• Operational Silos: DC and cross-domain backbone networks use different OAM and monitoring mechanisms. This makes end-to-end fault isolation and performance telemetry difficult.

2. AI “Elephant Flows” Facing ECMP Polarization and Congestion

AI compute clusters, such as those used for LLM pre-training and fine-tuning, rely heavily on large-scale collective communications such as AllReduce and All-to-All. These workloads typically generate a small number of flows with high burst throughput per flow, resulting in elephant flows.

• Hash Collisions: Traditional data centers rely on Equal-Cost Multi-Path (ECMP) routing to distribute traffic. VXLAN uses the outer UDP source port for ECMP hashing. A small number of bursty elephant flows can easily hash onto the same physical link, causing severe localized congestion.
• PFC Storms and Lower MFU: In lossless RoCEv2 networks, link congestion can frequently trigger PFC (Priority Flow Control) backpressure. In severe cases, this can lead to PFC deadlocks or significant queuing latency, ultimately reducing cluster-wide compute utilization (MFU, Model FLOPs Utilization). VXLAN itself provides limited traffic engineering (TE) capabilities and cannot dynamically steer traffic based on real-time network congestion.

3. Encapsulation Overhead and Packet Parsing Costs in the Chip Pipeline

Standard VXLAN adds at least 50 bytes of protocol header overhead, including the outer MAC, IPv4, UDP, and VXLAN headers. During LLM gradient synchronization and frequent exchanges of small parameter packets, this overhead reduces effective payload utilization. It also adds processing overhead to the switch ASIC’s parsing depth and lookup pipeline.

SRv6 (Segment Routing over IPv6) is not simply a tunneling protocol. It is a programmable instruction set for networking, built on the Source Routing concept and native IPv6 extension headers.

SRv6 abstracts network nodes and service behaviors into 128-bit IPv6 addresses called Segment IDs (SIDs). A SID consists of three logical parts:

• Locator: Provides Underlay reachability across the network. Intermediate nodes only need to perform standard IPv6 routing, such as BGP or OSPFv3, using longest-prefix matching. They do not need to maintain complex tunnel entries.
• Function: Specifies the network action to be performed by the destination node.
  • End.DT4 / End.DT6: Decapsulate the packet and perform a route lookup in the specified VRF for IPv4/IPv6 forwarding, enabling L3 multi-tenant isolation.
  • End.DX2: Decapsulate the packet and forward the inner Layer 2 frame directly through the specified egress interface, supporting L2 EVPN services.
• Args: Carries flow-table matching information, security tags, or network telemetry metadata.

The control plane still uses the mature BGP EVPN protocol, but the forwarding mechanism is significantly simplified.

During route advertisement, the Egress Leaf advertises the service SID to the entire network through the BGP Prefix-SID attribute, such as 2001:db8:1::End.DT4. When the Ingress Leaf receives traffic from a tenant, it uses the corresponding IPv6 SID as the destination IP address in the outer packet. The packet is then forwarded transparently across the Fabric, without requiring Spine nodes to maintain tenant-specific forwarding state.

For elephant-flow scheduling in RoCEv2 AI clusters, SRv6 provides deterministic traffic steering:

• Explicit Path Selection: The Ingress Leaf or a centralized controller uses real-time link telemetry to select an uncongested path and inserts a list of SIDs (Segment List) into the packet.
• Stateless Transit Forwarding: Intermediate Spine and switch nodes only need to read the current SID, forward the packet accordingly, and decrement the Segments Left value by 1. They do not need fine-grained flow entries to be configured or installed on core switches.

To address packet expansion caused by the long standard 128-bit SRH, RFC 9800 standardizes two compression mechanisms: NEXT-C-SID and REPLACE-C-SID. By packing short operation codes, such as 16-bit or 32-bit values, for multiple nodes into a single 128-bit IPv6 destination address, these mechanisms significantly reduce packet overhead while maintaining short-packet throughput and ASIC hardware lookup performance.

The MRC architecture (Resilient AI Supercomputer Networking using MRC and SRv6), developed by OpenAI, Microsoft, NVIDIA, AMD, Intel, and Broadcom, is a good example. A core principle of SRv6 is to let applications control their network experience. Implementing MRC at the transport layer enables a programmable network where the transport stack selects the path for each packet. By distributing packets across multiple stateless paths and planes, MRC avoids flow collisions caused by low-entropy traffic, a common issue in traditional ECMP-based deployments. This is a practical implementation of application-defined networking, enabled by the programmability introduced by SRv6.

Alibaba Cloud’s NetPila demonstrates how SRv6 can take container networking beyond traditional VXLAN tunnels. By embedding tenant and interface identifiers directly into the IPv6/SRv6 addressing model, it enables direct Pod-to-Pod connectivity without tunnels and provides tighter integration between endpoints and the network.

Alibaba Cloud’s eCore is an IPv6/SRv6-based backbone and inter-data center network architecture. It uses a unified IPv6 Underlay and SRv6 Traffic Engineering to shift path selection from traditional hop-by-hop network decisions to programmable end-to-end path control. In AI scenarios, eCore further enables host-network coordination. Servers select a Flow Label based on factors such as application type and QoS requirements. The network then maps the Flow Label to different SRv6 SID Lists, enabling fine-grained path selection for RDMA, high-bandwidth, low-cost, and other traffic requirements.

Eliminating DCI Performance Bottlenecks and Protocol Gaps: The complex EVPN-to-MPLS translation logic on the DCI is removed. The DCI can operate as a high-capacity IPv6 router, eliminating protocol compatibility issues between multi-vendor DCI devices.

Simplified NFV Instance and Service Chain Orchestration: NFV instances such as cloud gateways and firewalls can run natively on general-purpose Linux/x86 hosts with native IPv6/SRv6 kernel support. Service chain traffic steering shifts from hard-coded configurations to flexible instruction-based orchestration.

Highly Consolidated Service Provisioning and Operations: Network planning shifts from three independently managed domains—DC, DCI, and WAN—to a business-centric, end-to-end, flat L3VPN architecture.

1. Turning Services into SIDs: This is the key difference. In a VXLAN network, a service node such as a firewall is effectively a “black box” in the topology. Traffic must be steered to it through complex routing policies. In SRv6, each service node or specific service interface is assigned a globally unique SID. A service chain is essentially a Segment List containing multiple service SIDs attached to the packet, allowing the network to perform the required service steering.
2. Unified Control and Forwarding Planes: A VXLAN Overlay network typically requires a separate control plane, such as EVPN, while service chain policies also rely on controller-based provisioning. SRv6 integrates SID advertisement through IGP/BGP extensions with SID-based packet forwarding. This simplifies the protocol stack and allows the network to execute service chain policies more directly.
3. Stateless Service Chaining: Traditional service chains, such as those built through VRF-based service insertion, often require service nodes to maintain significant state. SRv6 service chaining is stateless. The packet’s SID list determines the path and service sequence. Service nodes only need to process the current SID and perform the corresponding forwarding action. They do not need to maintain per-flow state, which significantly simplifies the design of service nodes, especially third-party appliances.

To support the evolution toward an end-to-end SRv6 Fabric, AsterNOS has defined a clear SRv6 feature roadmap. AsterNOS currently supports standard SRv6 and REPLACE-C-SID compression. Full support for NEXT-C-SID (uSID) is planned for Q4, helping enterprises build a simpler, flatter network architecture for AI computing workloads.