Table of Contents
Introduction
When it comes to BGP EVPN in SONiC, VXLAN is an essential concept. In a data center (DC), the first thing to understand is that using VXLAN EVPN means that the data plane utilizes VXLAN technology, while the control plane relies on BGP EVPN technology.
This article will introduce why EVPN is used in VXLAN networks, the five types of BGP EVPN messages in SONiC, and how to configure them on the Asterfusion platform.
Abbreviations used in this article:
- NLRI: Network Layer Reachability Information
- AFI: Address Family Identifier
- SAFI: Subsequent Address Family Identifier
Why use EVPN in VXLAN and What is EVPN ?
In data center (DC) networks, we use the VXLAN protocol (RFC 7348) for data transmission. But if VXLAN is already working well, why introduce EVPN?
VXLAN is essentially an Overlay data plane protocol. It solves the problem of how Layer 2 frames can be transmitted over a Layer 3 IP network. In simple terms, VXLAN is responsible for encapsulating and forwarding data, but it doesn’t determine the destination.
In a pure VXLAN network, since VTEPs only know local MAC addresses and those of remote devices they’ve communicated with, when trying to forward a frame to an unknown destination MAC address, it results in a flooding operation to discover the destination MAC. This causes widespread broadcast traffic throughout the network, consuming a significant amount of bandwidth.
According to RFC 7348, VXLAN does not define a control plane mechanism. Therefore, we need to find a good control plane protocol for VXLAN, which leads us to EVPN.
EVPN (Ethernet VPN) is essentially a set of Layer 2 VPN technical specifications defined by the IETF. Its primary design goal is to use the control plane to learn MAC/IP address and avoid flooding behavior, thereby providing deterministic forwarding for overlay technologies such as VXLAN. As a result, EVPN requires a control plane protocol carrier responsible for transmitting various types of route information defined by EVPN, though it is not tied to any specific protocol.
So why was BGP chosen as the control plane protocol? The answer lies in BGP-4 (BGP version 4), which has been extended by MP-BGP (Multi-Protocol BGP). MP-BGP supports VPNv4/VPNv6, EVPN, large-scale route distribution, AFI/SAFI multi-address family support, NLRI extension capabilities, and more.
What is BGP EVPN in SONiC ?
From the above, we know that BGP EVPN in SONiC uses MP-BGP as the control plane protocol for EVPN, which learns the MAC and IP information of the destination host and provides deterministic forwarding behavior.
BGP EVPN in SONiC is not a new protocol; rather, it is an extension of EVPN within the BGP message framework. It relies on the MP-BGP protocol. Before understanding the five types of BGP EVPN messages, let’s first take a look at MP-BGP.
MP-BGP exists as an extension of BGP-4. BGP-4 refers to the fourth version of the BGP protocol, which defines the core session mechanism, path attributes, and scalable framework, and originally supported only IPv4 Unicast. On top of this, IETF RFC 4760 introduced the multi-address family capability (AFI/SAFI) via MP-BGP, allowing BGP to carry multiple types of routing information, including IPv6, VPNv4/VPNv6, and EVPN.
The BGP EVPN in SONiC we use today still operates within the BGP-4 protocol framework. Below is the Update message format in BGP-4.
In BGP-4, NLRI is used to carry IPv4 unicast prefixes. It serves as the mechanism within the UPDATE message to announce “which IPv4 networks are reachable.”
Why is the BGP Update message mentioned separately? This is because, in RFC 4760, MP-BGP specifically modifies the BGP Update message. It introduces two new attributes, MP_REACH_NLRI and MP_UNREACH_NLRI, in the Path Attribute section. These new attributes encapsulate multi-protocol NLRI within the path attributes, enabling BGP to support multiple address families.
When the path attribute is Multiprotocol Reachable NLRI (MP_REACH_NLRI), it is used in a BGP UPDATE message to declare “these multi-protocol routes are now reachable, and what the next-hop is”. The message format is shown below.
In VXLAN EVPN, MP_REACH_NLRI may represent:
- Using AFI = 25 and SAFI = 70 to indicate it’s an EVPN route.
- NLRI section: Using Route Type 2 (MAC/IP route) to indicate the MAC and IP information.
- Next-hop: VTEP IP.
This means: “I have these MAC/IP addresses behind this VTEP, and they can be reached via my VTEP IP.”
When the path attribute is Multiprotocol Unreachable NLRI (MP_UNREACH_NLRI), it is used in the BGP UPDATE message to declare “these multi-protocol routes are now unreachable and need to be withdrawn.” The message format is shown below.
In the VXLAN EVPN example:
- AFI / SAFI: AFI = 25, SAFI = 70, indicating that the route being withdrawn is an EVPN route.
- Withdrawn routes: The specific NLRI that needs to be withdrawn.
5 Route Types of BGP EVPN in SONiC
In the previous section, we learned about the message structure of BGP EVPN in SONiC based on the MP-BGP mechanism. Essentially, it extends the path attributes and NLRI fields of the BGP-4 message. The types of BGP EVPN messages are defined through the router type within the NLRI field.
In BGP EVPN in SONiC, when the path attribute is Multiprotocol Reachable NLRI, it supports five types of messages. Combined with the next-hop information, these Route Types define “all the key states that need to be advertised” in Layer 2 / Layer 2-3 integrated networks, as shown below.
- Route Type 1 – Ethernet Auto-Discovery (EAD) Route: Used for EVPN Multihoming.
- Route Type 2 – MAC/IP Advertisement Route: Advertises MAC addresses and optional IP addresses, specifying the associated EVI (VNI) and VTEP IP.
- Route Type 3 – Inclusive Multicast Ethernet Tag (IMET) Route: Advertises the VXLAN Flood List for BUM (Broadcast, Unknown Unicast, and Multicast) traffic.
- Route Type 4 – Ethernet Segment (ES) Route: Advertises the member relationship of an Ethernet Segment Identifier (ESI).
- Route Type 5 – IP Prefix Route: Advertises Layer 3 prefixes (L3 VNI).
Route Type 1 and Route Type 4 are required in EVPN multihoming scenarios. These are supported on our campus equipment, CX-M, but will not be explained in detail in this article.
Below, we will explain the fields of Route Type 2, Route Type 3, and Route Type 5:
Route Type 2 – MAC/IP Advertisement Route
Definition and Purpose
Route Type 2 in the EVPN control plane is used to advertise Layer 2 MAC addresses of end devices, along with optional Layer 3 IP address information (host IP). This allows the control plane to learn remote MAC/IP addresses, enabling MAC learning and ARP/ND resolution (such as ARP suppression).
For example, when a device is powered on and its interface or VLAN is configured with corresponding MAC/IP addresses, the device will automatically generate a Route Type 2 message. It will send an MP-BGP Update message to notify the peer about its own MAC/IP.
NLRI Field Structure
| Fields | Length | Meaning | Function |
| RD (Route Distinguisher) | 8 octets | Uniquely identifies the EVPN instance | Differentiates between different EVPN instances, ensuring global uniqueness. |
| Ethernet Segment Identifier (ESI) | 10 octets | Identifies an Ethernet segment | Represents the physical/logical segment where the MAC originates. A value of 0 indicates a single-homed host. |
| Ethernet Tag ID | 4 octets | Identifies the VLAN or broadcast domain | Associates the route with a specific broadcast domain (e.g., VLAN). |
| MAC Address Length | 1 octet | MAC address length (in bits) | Typically 48 bits for MAC addresses. |
| MAC Address | 6 octets | MAC address of the end device | The destination address learned by the control plane. |
| IP Address Length | 1 octet | IP address length (in bits) | Indicates IPv4/IPv6 if an IP is present. |
| IP Address | Variable (0, 4, or 16 octets) | Host’s IP address | Facilitates ARP/ND Suppression or IRB information advertisement. |
| (Optional) MPLS Label1 / MPLS Label2 | 3 octets (x1 or x2) | Labels associated with data plane forwarding | Indicates the VNI/VPN label that the MAC belongs to (as allowed by RFC 7432 ). |
Route Type 3 – Inclusive Multicast Ethernet Tag (IMET) Route
Definition and Purpose
Route Type 3 is primarily used to control the forwarding of BUM (Broadcast, Unknown-unicast, Multicast) traffic in the EVPN network. It allows nodes in the same EVPN instance to know how to forward multicast/flooding traffic within a broadcast domain.
NLRI Field Structure
| Fields | Length | Meaning |
| RD | 8 octets | Unique identifier for the EVPN instance. |
| Ethernet Tag ID | 4 octets | Specifies the broadcast domain / service VLAN. |
| IP Address Length | 1 octet | Length (in bits) of the Originating Router’s IP address. |
| Originating Router’s IP Address | 4 or 16 octets | IP address (Loopback) of the router that generated this Route. |
PMSI Attribute: Provider Multicast Service Interface Attribute. This attribute indicates the method of transporting BUM traffic. Since the standard fields above do not carry detailed configuration information on how to transmit BUM data, this section supplements the above fields. Its specific functions are as follows:
- Indicates the transmission mechanism for BUM traffic:
- Ingress Replication (unicast replication to each VTEP)
- Multicast/Broadcast LSP (using P2MP LSP or multicast tree)
- For more details on these two transmission mechanisms, please refer to VXLAN Multicast BUM Traffic Forwarding: Best Practices for IR and Multicast Underlay
- Provides data plane encapsulation information:
- Label/VNI and Tunnel Type inform the receiver how to encapsulate and decapsulate BUM traffic.
- Supports automatic construction of the BUM forwarding table in EVPN:
- PMSI + Type 3 NLRI = Control plane indicates the forwarding path for BUM data traffic.
Its fields are as follows:
| Fields | Length | Meaning | Function |
| Attribute Type | 1 octet | 0x0D (PMSI) | BGP Extended Community type identifier. |
| Flags | 1 octet | Indicates whether Leaf Info, Ingress Replication, etc., are included. | Controls data plane behavior. |
| Tunnel Type | 1 octet | Specifies the encapsulation type: e.g., MPLS P2MP, VXLAN, Ingress Replication. | Guides BUM traffic encapsulation. |
| Label / VNI | 3 octets | MPLS Label or VXLAN VNI. | Data plane forwarding identifier. |
| Optional Fields | Variable | Leaf Address List or other information. | Indicates the list of PE/VTEP devices receiving BUM traffic. |
Typical Applications
- Distribution of BUM traffic (Broadcast/Multicast/Unknown-Unicast).
- Endpoint discovery in Ingress Replication or multicast tree establishment.
- Control plane informs which PE/VTEP devices participate in forwarding for a broadcast domain.
Route Type 5 – IP Prefix Route
Definition and Purpose
EVPN Route Type 5 (abbreviated as RT-5) is a new route type registered with IANA under the EVPN Route Types. It is used to advertise standard IP prefix route information in the EVPN control plane. The goal is to address the issue where using only Route Type 2 to advertise host IPs (/32 or /128) does not meet the needs for inter-subnet prefix aggregation and scalability.
NLRI Field Structure
| Fields | Length | Meaning | Purpose |
| RD (Route Distinguisher) | 8 octets | Globally unique identifier for the EVPN instance (MAC‑VRF or IP‑VRF) | Differentiates between different tenants/route instances. |
| Ethernet Segment Identifier (ESI) | 10 octets | Can be used as Overlay Index or all zeros | Specifies the entity associated with the prefix (optional). |
| Ethernet Tag ID | 4 octets | VLAN or EVPN broadcast domain identifier | Defines the EVPN logical domain to which the prefix belongs. |
| IP Prefix Length | 1 octet | Prefix length (in bits) | Specifies the network mask length (e.g., /24 or /64). |
| IP Prefix | 4 / 16 octets | IPv4 or IPv6 prefix address | The advertised Layer 3 network prefix. |
| GW IP Address | 4 / 16 octets | Gateway IP (Overlay Index) | Specifies the recursive next-hop index. |
| MPLS Label | 3 octets | Label value or VNI | Associated with the data plane label or VXLAN VNI. |
The following is a summary of 5 packet types:
| EVPN Route Types | Core Function | Problem Solved |
| Type 1 (Ethernet Segment Route) | Negotiates the multi-active role of dual-homed access (DF) | Prevents MAC flapping and enables link load balancing. |
| Type 2 (MAC/IP Route) | Synchronizes MAC-IP-VTEP-VNI mapping | Replaces ARP flooding and enables precise unicast forwarding. |
| Type 3 (Inclusive Ethernet Segment Route) | Optimizes broadcast/unknown unicast traffic forwarding | Prevents network-wide flooding and reduces bandwidth consumption. |
| Type 4 (ES Import Route) | Synchronizes host redundancy information in multi-active scenarios | Prevents route black holes and ensures traffic consistency. |
| Type 5 (IP Prefix Route) | Transmits Layer 3 IP prefix information | Enables inter-VNI Layer 3 communication in VXLAN. |
Regarding Type 2 and Type 4, we will provide a detailed introduction in the upcoming section on EVPN Multihoming.
Next, we will focus on Route Type 2, Route Type 3, and Route Type 5, using a topology to demonstrate how these messages interact and work.
Typical Topology and Message Interaction
In the context of BGP EVPN in SONiC with VXLAN, let’s take the scenario where Host A (a VM under Leaf1) communicates with Host B (a VM under Leaf2). We’ll break down the complete message interaction flow so you can clearly see the interaction from Control Plane layer:
Preconditions and IP Division:
- A VXLAN tunnel has been established between Leaf 1 and Leaf 2, and both are VTEP devices.
- BGP EVPN is running across the network.
- Leaf 1: 1.1.1.1/32
- Leaf 2: 2.2.2.2/32
- Host A: 192.168.1.11/24, MAC = MAC-A
- Host B: 192.168.2.22/24, MAC = MAC-B
Layer 2 Frame:
- BUM transmission method: Ingress Replication
- Underlay: IP reachable
- Overlay: VXLAN
- Leaf = VTEP + EVPN PE
- Host A and Host B are in different subnets (L3 communication).
Taking Symmetric IRB as an example:
Route Type 3 (BUM Flooding List Synchronization)
- Leaf1 publishes: Sends Route Type 3 to Leaf2/Leaf3 with core fields:
- EVI (VNI) + Originating Router’s IP (VTEP IP) = 1.1.1.1
- PMSI attribute: Tunnel Type = Ingress Replication, Tunnel Endpoint = 1.1.1.1
- Leaf2 publishes: Sends Route Type 3 to Leaf1/Leaf3 with core fields:
- EVI (VNI) + Originating Router’s IP (VTEP IP) = 2.2.2.2
- PMSI attribute: Tunnel Type = Ingress Replication, Tunnel Endpoint = 2.2.2.2
- Interaction result:
- Leaf1 & Leaf2 add each other’s VTEP to their local Ingress Replication List (IRL) based on the Route Type 3 messages.
- For subsequent BUM traffic (e.g., ARP requests, unknown unicast, broadcast frames), the source Leaf will multicast replicate and send the traffic via the VXLAN tunnel to all remote VTEPs in the IRL.
Route Type 2 (Host MAC/IP Mapping Synchronization)
- Leaf1 publishes: Sends Route Type 2 to Leaf2/Leaf3 with core fields:
- MAC Address = MAC-A
- IP Address = 192.168.1.11 (/32, host route)
- EVI (VNI)
- Next Hop = 1.1.1.1 (Leaf1 VTEP IP)
- Leaf2 publishes: Sends Route Type 2 to Leaf1/Leaf3 with core fields:
- MAC Address = MAC-B
- IP Address = 192.168.2.22 (/32, host route)
- EVI (VNI)
- Next Hop = 2.2.2.2 (Leaf2 VTEP IP)
- Interaction result:
- Leaf1 generates a mapping table: MAC-B / 192.168.2.22 → Next Hop = 2.2.2.2
- Leaf2 generates a mapping table: MAC-A / 192.168.1.11 → Next Hop = 1.1.1.1
- Through the EVPN control plane, host reachability information is distributed, replacing traditional VXLAN flooding learning, enabling precise Layer 2 forwarding and ARP suppression.
Route Type 5 (Inter-Subnet IP Prefix Synchronization)
Since Host A (192.168.1.0/24) and Host B (192.168.2.0/24) are in different subnets, Leaf publishes Route Type 5 for Layer 3 reachability:
- Leaf1 publishes: Sends Route Type 5 to Leaf2/Leaf3 with core fields:
- IP Prefix = 192.168.1.0/24
- EVI (L3 VNI)
- Next Hop = 1.1.1.1 (Leaf1 VTEP IP, carried via MP_REACH_NLRI)
- Leaf2 publishes: Sends Route Type 5 to Leaf1/Leaf3 with core fields:
- IP Prefix = 192.168.2.0/24
- EVI (L3 VNI)
- Next Hop = 2.2.2.2 (Leaf2 VTEP IP, carried via MP_REACH_NLRI)
- Router MAC Extended Community = Leaf 2’s Router MAC
- Interaction result:
- Leaf1 generates a Layer 3 routing table: 192.168.2.0/24 → Next Hop = 2.2.2.2, and associates the outgoing interface’s MAC with Leaf 2’s Router MAC.
- Leaf2 generates a Layer 3 routing table: 192.168.1.0/24 → Next Hop = 1.1.1.1
- Supports distributed Layer 3 gateway forwarding based on the VXLAN Overlay.
How can Configure VXLAN BGP EVPN in SONiC?
The above content is intended to help understand how Route Types 2, 3, and 5 interact in a topology, without distinguishing between L2 VXLAN and L3 VXLAN. Asterfusion provides professional configuration documentation to guide you in configuring BGP EVPN in SONiC, offering typical configurations.
For details about the commands and parameters for configuring BGP EVPN in SONiC on Asterfusion Data Center and Cloud switches, see VXLAN Configuration Guide.
Contact US !
- To request a proposal, send an E-Mail to bd@cloudswit.ch
- To receive timely and relevant information from Asterfusion, sign up at AsterNOS Community Portal
- To submit a case, visit Support Portal
- To find user manuals for a specific command or scenario, access AsterNOS Documentation
- To find a product or product family, visit Asterfusion-cloudswit.ch
