SRv6 vs VXLAN: SRv6 Is Changing Data Center Network Fabric

Over the past decade of cloud computing evolution, EVPN-VXLAN has become the de facto foundation for data center virtualization and multi-tenant networking. It removes the 4096-VLAN scaling limit and builds a virtual Layer 2 network (Overlay) over a Layer 3 physical network (Underlay), enabling seamless workload mobility.

However, the rise of distributed LLM training, high-density GPU clusters, and cross-data-center collaborative computing has fundamentally changed network traffic patterns. Traditional VXLAN architectures are increasingly showing structural limitations when handling massive elephant-flow scheduling, cross-domain protocol integration, and packet encapsulation overhead. SRv6 (Segment Routing over IPv6), based on native IPv6, is rapidly moving from WAN deployments into data centers and becoming a core foundation for next-generation compute fabrics.

In traditional general-purpose cloud environments, VXLAN effectively meets the multi-tenant isolation requirements of virtual machines and containers. However, as infrastructure evolves toward heterogeneous computing and multi-data-center collaboration, the limitations of VXLAN are becoming increasingly apparent:

1. Cross-Domain “Protocol Stitching” and Gateway Bottlenecks

In multi-data-center interconnect (DCI) or hybrid cloud scenarios, data centers typically run EVPN-VXLAN, while WAN backbones often use MPLS, SR-MPLS, or pure IP routing.

• State Overhead and Single-Point Bottlenecks: Border gateways (Border Leaf / DCI Gateway) must handle complex protocol translation and state maintenance. They need to decapsulate VXLAN, remove internal labels, perform route lookups, and re-encapsulate packets for MPLS/WAN transport.
• Operational Silos: DC and cross-domain backbone networks use different OAM and monitoring mechanisms. This makes end-to-end fault isolation and performance telemetry difficult.

2. AI “Elephant Flows” Facing ECMP Polarization and Congestion

AI compute clusters, such as those used for LLM pre-training and fine-tuning, rely heavily on large-scale collective communications such as AllReduce and All-to-All. These workloads typically generate a small number of flows with high burst throughput per flow, resulting in elephant flows.

• Hash Collisions: Traditional data centers rely on Equal-Cost Multi-Path (ECMP) routing to distribute traffic. VXLAN uses the outer UDP source port for ECMP hashing. A small number of bursty elephant flows can easily hash onto the same physical link, causing severe localized congestion.
• PFC Storms and Lower MFU: In lossless RoCEv2 networks, link congestion can frequently trigger PFC (Priority Flow Control) backpressure. In severe cases, this can lead to PFC deadlocks or significant queuing latency, ultimately reducing cluster-wide compute utilization (MFU, Model FLOPs Utilization). VXLAN itself provides limited traffic engineering (TE) capabilities and cannot dynamically steer traffic based on real-time network congestion.

3. Encapsulation Overhead and Packet Parsing Costs in the Chip Pipeline

Standard VXLAN adds at least 50 bytes of protocol header overhead, including the outer MAC, IPv4, UDP, and VXLAN headers. During LLM gradient synchronization and frequent exchanges of small parameter packets, this overhead reduces effective payload utilization. It also adds processing overhead to the switch ASIC’s parsing depth and lookup pipeline.

SRv6 (Segment Routing over IPv6) is not simply a tunneling protocol. It is a programmable instruction set for networking, built on the Source Routing concept and native IPv6 extension headers.

Traditional Hop-by-Hop Routing: It is like asking for directions. When a packet reaches each router, the router looks up its routing table and determines where to forward the packet next.

SRv6 Source Routing: It is like following a predefined itinerary. The entire journey, including all intermediate destinations (Segment List), is defined at the starting point (Ingress Leaf) and carried in the packet header. Intermediate routers (Spine/P nodes) simply follow the list and execute the specified instructions without making independent routing decisions.

The Essence of SRv6 Network Programming: SRv6 uses 128-bit IPv6 addresses as more than just addresses. They can also represent instructions. The Locator defines where to go through routing and addressing, while the Function defines what to do, such as decapsulation or entering a VRF.

SRv6 abstracts network nodes and service behaviors into 128-bit IPv6 addresses called Segment IDs (SIDs). A SID consists of three logical parts:

• Locator: Provides Underlay reachability across the network. Intermediate nodes only need to perform standard IPv6 routing, such as BGP or OSPFv3, using longest-prefix matching. They do not need to maintain complex tunnel entries.
• Function: Specifies the network action to be performed by the destination node.
  • End.DT4 / End.DT6: Decapsulate the packet and perform a route lookup in the specified VRF for IPv4/IPv6 forwarding, enabling L3 multi-tenant isolation.
  • End.DX2: Decapsulate the packet and forward the inner Layer 2 frame directly through the specified egress interface, supporting L2 EVPN services.
• Args: Carries flow-table matching information, security tags, or network telemetry metadata.

The control plane still uses mature BGP-EVPN or BGP L3VPN, while the data-plane mechanism is significantly simplified.

During route advertisement, the Egress Leaf (Leaf1) advertises the service SID to the network through the BGP Prefix-SID attribute, such as 3167::1:0:0:0.

Step 1 (Ingress Leaf): Leaf2 acts as the ingress node and receives tenant TG2 traffic. After matching the destination SID, the Leaf forwards the packet using SRv6. For pure Layer 3 services, such as L3VPN or EVPN Type-5, the Leaf removes the original Layer 2 MAC header and encapsulates the inner IP packet directly with an outer SRv6 IPv6 header (DA=3167::1:0:0:0). The outer source address (SA) must be set to the local fixed Loopback address, such as SA=9177::1, to ensure proper PMTU discovery and OAM troubleshooting.

Step 2 (Spine — Pure Transit / Naked Transit): The intermediate Spine nodes only inspect the first 64 bits of the destination address (DA), which identify the Locator. They forward the packet using standard Longest Prefix Match (LPM) in hardware. They do not decapsulate the packet or process the Function portion.

Step 3 (Egress Leaf): When the packet reaches Leaf1, Leaf1 identifies the destination address as its own Locator and processes the Function field, such as End.DT4. It removes the outer IPv6 header and forwards the original packet into the specified tenant VRF, where it is delivered to the destination TG1.

SRv6 BE (Best Effort, widely used in data center scenarios):

• Characteristics: No SRH extension header is required. SRv6 uses only the standard 40-byte outer IPv6 header.
• Principle: The outer IPv6 DA directly serves as the SID. Transit nodes forward packets along the default IGP/ECMP shortest path. This minimizes overhead and provides an alternative to traditional VXLAN for basic overlay connectivity.

SRv6 TE (Traffic Engineering, for AI and large-scale computing scenarios):

• Characteristics: To address congestion caused by RoCEv2 elephant flows, SRv6 can insert an SRH (Segment Routing Header) extension header containing a list of SIDs that define the required transit nodes.
• Principle: SRv6 TE explicitly defines the forwarding path. It can steer traffic around congested links or force traffic through specific network functions, such as a firewall in an SFC (Service Function Chaining) path.

For RoCEv2 elephant-flow scheduling in AI computing clusters, SRv6 provides deterministic traffic engineering capabilities:

• Explicit Path Selection: The Ingress Leaf or centralized controller uses real-time Telemetry data to select an uncongested path and inserts a corresponding list of SIDs (Segment List) into the packet.
• Stateless Transit Forwarding: Intermediate Spine and transit nodes only need to read the current SID and update the segment pointer (Segments Left decremented by 1). They do not need to maintain or install fine-grained flow entries on core switches.

First, it is important to understand why SID compression is needed. Imagine that five transit nodes must be specified to avoid congested links. Standard SRv6 would need to carry five complete 128-bit addresses in the packet header. This can make the header larger than the payload and significantly increase the parsing workload on the switch ASIC.

To address the packet expansion caused by standard 128-bit SRH entries, RFC 9800 standardizes two compression mechanisms: NEXT-C-SID and REPLACE-C-SID. These mechanisms pack multiple short node instructions, such as 16-bit or 32-bit values, into a single 128-bit IPv6 destination address. This significantly reduces packet overhead while maintaining short-packet throughput and efficient hardware table lookups.

• NEXT-C-SID (Shift and Forward): Uses a Shift-and-Forward mechanism. Multiple 16-bit short instructions are arranged sequentially in the IPv6 destination address. At each hop, the hardware shifts the address content to expose the instruction for the next node. This approach can eliminate the need for an additional SRH extension header.
• REPLACE-C-SID (Replace in Place): Uses a Replace-in-Place mechanism. It still uses an SRH extension header, but compresses the SID list into a smaller format. For example, four 32-bit short instructions can be packed into a single 128-bit container. At each hop, the router reads the next short instruction from the container and directly replaces the dynamic field in the outer destination IP address. This guides the packet toward the next hop.

The control-plane workflow is based on a four-phase process. Across the network, ISIS or eBGP advertises Locator prefixes and Loopback addresses to establish Underlay connectivity between the physical network nodes (Phase 1). The Leaf nodes then establish a multi-hop MP-BGP control channel directly between their Loopback addresses, such as 9167::1 and 9177::1, across the intermediate Spine nodes (Phase 2).

During service onboarding, each Leaf imports locally connected tenant private routes into the corresponding VRF and assigns an SRv6 service instruction, such as End.DT4. It then adds a Route Target (RT) to identify the VPN route and prepare it for advertisement (Phase 3). Finally, the Leaf generates a BGP Update message containing the private prefix, RT, and SID, and sends it to the remote Leaf. After receiving the update, the remote Leaf imports the route into the local VRF based on the RT and automatically builds a mapping between the inner private IP prefix and the outer SRv6 SID (Phase 4).

This example uses the BGP VPNv4/v6 address families to advertise SRv6 L3VPN routes. The underlying logic is the same as BGP-EVPN, which advertises MAC/IP routes. In both cases, the control plane distributes SID mappings, while the data plane uses these mappings for packet encapsulation and forwarding.

The MRC architecture (Resilient AI Supercomputer Networking using MRC and SRv6), developed by OpenAI, Microsoft, NVIDIA, AMD, Intel, and Broadcom, is a good example. A core principle of SRv6 is to let applications control their network experience. Implementing MRC at the transport layer enables a programmable network where the transport stack selects the path for each packet. By distributing packets across multiple stateless paths and planes, MRC avoids flow collisions caused by low-entropy traffic, a common issue in traditional ECMP-based deployments. This is a practical implementation of application-defined networking, enabled by the programmability introduced by SRv6.

Alibaba Cloud’s NetPila demonstrates how SRv6 can take container networking beyond traditional VXLAN tunnels. By embedding tenant and interface identifiers directly into the IPv6/SRv6 addressing model, it enables direct Pod-to-Pod connectivity without tunnels and provides tighter integration between endpoints and the network.

Alibaba Cloud’s eCore is an IPv6/SRv6-based backbone and inter-data center network architecture. It uses a unified IPv6 Underlay and SRv6 Traffic Engineering to shift path selection from traditional hop-by-hop network decisions to programmable end-to-end path control. In AI scenarios, eCore further enables host-network coordination. Servers select a Flow Label based on factors such as application type and QoS requirements. The network then maps the Flow Label to different SRv6 SID Lists, enabling fine-grained path selection for RDMA, high-bandwidth, low-cost, and other traffic requirements.

Eliminating DCI Performance Bottlenecks and Protocol Gaps: The complex EVPN-to-MPLS translation logic on the DCI is removed. The DCI can operate as a high-capacity IPv6 router, eliminating protocol compatibility issues between multi-vendor DCI devices.

Simplified NFV Instance and Service Chain Orchestration: NFV instances such as cloud gateways and firewalls can run natively on general-purpose Linux/x86 hosts with native IPv6/SRv6 kernel support. Service chain traffic steering shifts from hard-coded configurations to flexible instruction-based orchestration.

Highly Consolidated Service Provisioning and Operations: Network planning shifts from three independently managed domains—DC, DCI, and WAN—to a business-centric, end-to-end, flat L3VPN architecture.

1. Turning Services into SIDs: This is the key difference. In a VXLAN network, a service node such as a firewall is effectively a “black box” in the topology. Traffic must be steered to it through complex routing policies. In SRv6, each service node or specific service interface is assigned a globally unique SID. A service chain is essentially a Segment List containing multiple service SIDs attached to the packet, allowing the network to perform the required service steering.
2. Unified Control and Forwarding Planes: A VXLAN Overlay network typically requires a separate control plane, such as EVPN, while service chain policies also rely on controller-based provisioning. SRv6 integrates SID advertisement through IGP/BGP extensions with SID-based packet forwarding. This simplifies the protocol stack and allows the network to execute service chain policies more directly.
3. Stateless Service Chaining: Traditional service chains, such as those built through VRF-based service insertion, often require service nodes to maintain significant state. SRv6 service chaining is stateless. The packet’s SID list determines the path and service sequence. Service nodes only need to process the current SID and perform the corresponding forwarding action. They do not need to maintain per-flow state, which significantly simplifies the design of service nodes, especially third-party appliances.

To support the evolution toward an end-to-end SRv6 Fabric, AsterNOS has defined a clear SRv6 feature roadmap. AsterNOS currently supports standard SRv6 and REPLACE-C-SID compression. Full support for NEXT-C-SID (uSID) is planned for Q4, helping enterprises build a simpler, flatter network architecture for AI computing workloads.