Building a Cisco NX-OS VXLAN EVPN Multi-Site Lab with RoCEv2

A hands-on walkthrough for configuring a VXLAN EVPN Multi-Site Lab, achieving lossless RDMA over Ethernet and troubleshooting the pieces that tend to break along the way.

A hands-on walkthrough · Cisco NX-OS 9000v · ~15 min read · Intermediate Difficulty

By Klaus Mucelli, CCIE DC No. 69110 · klausmucelli15@gmail.com

1. Introduction to Multi-Site

Most modern data centers run on VXLAN with BGP EVPN. VXLAN gives you a MAC-in-UDP overlay, so a Layer 2 segment can ride across a fully routed Layer 3 network, and BGP EVPN is the control plane that tells every VTEP (VXLAN Tunnel End Point) where all the MACs and IPs actually live. That split is powerful: tenants get their own address space, and the physical fabric underneath stays simple leaf-and-spine. The catch is that one big fabric is also one big blast radius. Broadcast, unknown-unicast and multicast (BUM) flooding, control-plane churn, and the occasional fat-fingered config all spread freely because nothing stops them at a boundary.

That is the problem VXLAN EVPN Multi-Site is built to solve. Instead of one sprawling fabric, you run several smaller, independent ones and join them at a single, well-controlled hand-off. Each site keeps its own underlay IGP, its own BGP autonomous system, and its own overlay, and it handles its BUM flooding locally. The piece doing the stitching is the Border Gateway (BGW). It sits at the edge of a site, terminates the local overlay, re-originates EVPN routes toward the far side, and rewrites the VXLAN next-hop so that failures, flooding, and route churn stop at the site edge instead of leaking across the whole network. You end up scaling out by adding sites, not by adding risk, and tenants still see stretched Layer 2 and Layer 3 as if it were one fabric.

Put plainly, a data center multi-site architecture is needed to provide:

Disaster recovery and business continuity: by stretching Layer 2/Layer 3 services across two or more geographically separate sites, workloads and their IP identity can fail over (or be actively load-shared) between locations. If one site is lost to a power, cooling, fiber, or natural-disaster event, applications keep running from the surviving site with minimal disruption, protecting RPO/RTO targets and keeping the business online.
Fault isolation: each site runs its own independent underlay, BGP AS, and overlay, handling BUM flooding locally. The Border Gateway contains broadcast storms, control-plane churn, and configuration errors at the site edge, so a failure in one fabric does not propagate into the other.
Centralized policy orchestration: tenants, VRFs, and segmentation policy are defined consistently and pushed across all sites from a single point of control, giving uniform security and connectivity everywhere without per-site drift.

We are going to build this from scratch. First we stand up two independent fabrics, DC1 (AS 65001) and DC2 (AS 65002), then stitch them together back-to-back over a Data Center Interconnect (DCI) using two eBGP sessions: one IPv4 unicast session for the underlay and one L2VPN EVPN session for the overlay. Once traffic flows between sites, we push things further and build a RoCEv2 (DCQCN) lossless class on top of the same fabric, mixing PFC (Priority Flow Control, a hop-by-hop pause that stops an upstream device from overrunning a full buffer) with ECN (Explicit Congestion Notification, which marks packets to signal congestion instead of dropping them) and WRED (Weighted Random Early Detection, the queue mechanism that does the ECN marking as a queue starts to fill) so that latency-sensitive RDMA (Remote Direct Memory Access, where one host writes directly into another host's memory with no CPU involvement) traffic never has to drop a packet. Everything here is real Cisco NX-OS, copied verbatim, so you can follow along on Nexus 9000 gear or a lab and land on the exact same result.

2. The Border Gateway Explained

Before diving into the CLI, it's crucial to understand the linchpin of this architecture: the Border Gateway (BGW). The BGW terminates the overlay at the site edge, re-originates EVPN routes between the local fabric and the remote site, and rewrites the next-hop so remote VTEPs point at the local site's BGW instead of the real originating VTEP.

It leverages two distinct Loopbacks: the PIP (Primary IP), which is its own VTEP loopback used for locally-originated routes and BUM, and the MS-VIP (Multi-Site Virtual IP), an anycast VTEP loopback shared by all BGWs of a site. By configuring evpn multisite fabric-tracking on links facing the local spine and evpn multisite dci-tracking on links facing the DCI, the BGW intelligently manages its VIP advertisement.

We also use per-L2VNI multisite ingress-replication to make the BGW head-end-replicate BUM to the remote site's VIP. Crucially, the peer-type fabric-external command on the inter-site eBGP neighbor enables the Multi-Site next-hop rewrite, while rewrite-evpn-rt-asn dynamically adjusts the Route Target ASN mismatch between our two distinct BGP domains.

3. Topology and Roles

Our architecture consists of two symmetric data centers. DC1 acts as Site 1 (AS 65001) with SPINE1, LEAF1, LEAF2, and BGW1. DC2 acts as Site 2 (AS 65002) with SPINE2, LEAF3, and BGW2.

The two sites are stitched back-to-back over a Data Center Interconnect (DCI). Two eBGP sessions ride this DCI: an IPv4 unicast session for underlay reachability, and an L2VPN EVPN session for the overlay.

VXLAN EVPN Multi-Site: two independent fabrics (DC1 / AS 65001 and DC2 / AS 65002) stitched back-to-back over the DCI via BGW1 (E1/3) ↔ BGW2 (E1/3). Each site runs its own underlay, spine route-reflector, and leaf VTEPs.

4. Addressing Plan

To keep our lab clean and scalable, we define a strict addressing plan. Loopbacks handle VTEP termination and BGP peering, while point-to-point /31 links build the OSPF underlay.

Node	lo0 (RID/BGP)	lo1 (PIP/VTEP)	lo100 (MS-VIP)
SPINE1	10.1.0.1	n/a	n/a
LEAF1	10.1.0.11	10.1.1.11	n/a
LEAF2	10.1.0.12	10.1.1.12	n/a
BGW1	10.1.0.21	10.1.1.21	10.1.100.1
SPINE2	10.2.0.1	n/a	n/a
LEAF3	10.2.0.13	10.2.1.13	n/a
BGW2	10.2.0.22	10.2.1.22	10.2.100.1

Link	Subnet	Notes
SPINE1 to LEAF1	10.1.10.0/31	DC1 OSPF area 0
SPINE1 to LEAF2	10.1.10.2/31	DC1 OSPF area 0
SPINE1 to BGW1	10.1.10.4/31	DC1 OSPF area 0 (BGW fabric link)
SPINE2 to LEAF3	10.2.10.0/31	DC2 OSPF area 0
SPINE2 to BGW2	10.2.10.2/31	DC2 OSPF area 0 (BGW fabric link)
BGW1 to BGW2 (DCI)	172.16.1.0/31	eBGP, NOT in OSPF

Tenant	VRF	L3VNI	VLAN	L2VNI	Subnet (anycast GW)
RED	RED	50001	10	10010	192.168.10.1/24
RED	RED	50001	20	10020	192.168.20.1/24
BLUE	BLUE	50002	30	10030	192.168.30.1/24
BLUE	BLUE	50002	40	10040	192.168.40.1/24

Our tenant model uses L3VNI core VLANs (VLAN 3900 for RED vn-segment 50001, and VLAN 3901 for BLUE vn-segment 50002). We configure the Anycast-gateway-mac 0000.00aa.aaaa identically across the entire fabric to enable seamless virtual machine mobility.

5. Step 1: Spine Configuration

With our blueprint in hand, we start by provisioning the Spines in each site. The spine acts as an OSPF underlay transit node and the EVPN Route-Reflector for the local fabric. It does not run VXLAN/VTEP configurations itself.

Here is the baseline configuration for SPINE1 in DC1 (AS 65001):

feature ospf
feature bgp
feature nv overlay
nv overlay evpn

interface loopback0
  ip address 10.1.0.1/32
  ip router ospf UNDERLAY area 0.0.0.0

! to LEAF1
interface Ethernet1/1
  no switchport
  ip address 10.1.10.0/31
  ip router ospf UNDERLAY area 0.0.0.0
  mtu 9216
  no shutdown
! to LEAF2
interface Ethernet1/2
  no switchport
  ip address 10.1.10.2/31
  ip router ospf UNDERLAY area 0.0.0.0
  mtu 9216
  no shutdown
! to BGW1
interface Ethernet1/3
  no switchport
  ip address 10.1.10.4/31
  ip router ospf UNDERLAY area 0.0.0.0
  mtu 9216
  no shutdown

router ospf UNDERLAY
  router-id 10.1.0.1

router bgp 65001
  router-id 10.1.0.1
  address-family l2vpn evpn
    retain route-target all
  template peer FABRIC-RR
    remote-as 65001
    update-source loopback0
    address-family l2vpn evpn
      send-community extended
      route-reflector-client
  neighbor 10.1.0.11
    inherit peer FABRIC-RR
  neighbor 10.1.0.12
    inherit peer FABRIC-RR
  neighbor 10.1.0.21
    inherit peer FABRIC-RR

SPINE2 (DC2, AS 65002) delta: same pattern with lo0 10.2.0.1; links Eth1/1 10.2.10.0/31 (to LEAF3), Eth1/2 10.2.10.2/31 (to BGW2); router-id 10.2.0.1; router bgp 65002; RR clients 10.2.0.13 (LEAF3) and 10.2.0.22 (BGW2), template remote-as 65002.

6. Step 2: Tenant Provisioning

Next, we provision our tenant layer. This exact VLAN, VNI, VRF, and EVPN configuration block must be applied on every leaf and every BGW to ensure the Layer 2 and Layer 3 VNIs exist end-to-end.

! ---- L2VNI VLANs ----
vlan 10
  vn-segment 10010
vlan 20
  vn-segment 10020
vlan 30
  vn-segment 10030
vlan 40
  vn-segment 10040

! ---- L3VNI "core" VLANs (one per VRF) ----
vlan 3900
  vn-segment 50001
vlan 3901
  vn-segment 50002

! ---- VRFs ----
vrf context RED
  vni 50001
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn
vrf context BLUE
  vni 50002
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn

! ---- L3VNI core SVIs (routed, no IP) ----
interface Vlan3900
  no shutdown
  vrf member RED
  ip forward
  no ip redirects
interface Vlan3901
  no shutdown
  vrf member BLUE
  ip forward
  no ip redirects

! ---- EVPN L2VNI definitions ----
evpn
  vni 10010 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10020 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10030 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10040 l2
    rd auto
    route-target import auto
    route-target export auto

For the Leaf switches only, we additionally configure the Anycast-GW SVIs. The Border Gateways do not need these SVIs unless they directly host endpoints.

interface Vlan10
  no shutdown
  vrf member RED
  ip address 192.168.10.1/24
  fabric forwarding mode anycast-gateway
interface Vlan20
  no shutdown
  vrf member RED
  ip address 192.168.20.1/24
  fabric forwarding mode anycast-gateway
interface Vlan30
  no shutdown
  vrf member BLUE
  ip address 192.168.30.1/24
  fabric forwarding mode anycast-gateway
interface Vlan40
  no shutdown
  vrf member BLUE
  ip address 192.168.40.1/24
  fabric forwarding mode anycast-gateway

7. Step 3: Leaf (VTEP) Configuration

Our leaves are the workhorses of the VXLAN fabric, mapping local access VLANs to their respective VNIs and enforcing routing in the overlay. Below is the comprehensive configuration for LEAF1.

feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
nv overlay evpn
fabric forwarding anycast-gateway-mac 0000.00aa.aaaa

interface loopback0
  ip address 10.1.0.11/32
  ip router ospf UNDERLAY area 0.0.0.0
interface loopback1
  ip address 10.1.1.11/32
  ip router ospf UNDERLAY area 0.0.0.0

interface Ethernet1/1
  no switchport
  ip address 10.1.10.1/31
  ip router ospf UNDERLAY area 0.0.0.0
  mtu 9216
  no shutdown

router ospf UNDERLAY
  router-id 10.1.0.11

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 10010
    ingress-replication protocol bgp
  member vni 10020
    ingress-replication protocol bgp
  member vni 10030
    ingress-replication protocol bgp
  member vni 10040
    ingress-replication protocol bgp
  member vni 50001 associate-vrf
  member vni 50002 associate-vrf

route-map REDIST-CONN permit 10

router bgp 65001
  router-id 10.1.0.11
  template peer SPINE-EVPN
    remote-as 65001
    update-source loopback0
    address-family l2vpn evpn
      send-community extended
  neighbor 10.1.0.1
    inherit peer SPINE-EVPN
  vrf RED
    address-family ipv4 unicast
      advertise l2vpn evpn
      redistribute direct route-map REDIST-CONN
  vrf BLUE
    address-family ipv4 unicast
      advertise l2vpn evpn
      redistribute direct route-map REDIST-CONN

Leaf deltas (apply the Tenant provisioning block from STEP 2 on each leaf, plus):

LEAF2 (DC1, AS 65001): loopback0 10.1.0.12, loopback1 10.1.1.12, Eth1/1 to SPINE1 10.1.10.3/31, router-id 10.1.0.12, BGP neighbor 10.1.0.1 remote-as 65001.
LEAF3 (DC2, AS 65002): loopback0 10.2.0.13, loopback1 10.2.1.13, Eth1/1 to SPINE2 10.2.10.1/31, router-id 10.2.0.13, template/neighbor remote-as 65002, BGP neighbor 10.2.0.1 (SPINE2), router bgp 65002.

8. Step 4: Border Gateway Configuration

Now for the critical multi-site configuration. The BGW must orchestrate local fabric peering while exchanging Multi-Site updates over the DCI. Remember to apply the Step 2 Tenant provisioning block on the BGW first so the VNIs exist.

feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
nv overlay evpn
fabric forwarding anycast-gateway-mac 0000.00aa.aaaa

evpn multisite border-gateway 1

interface loopback0
  ip address 10.1.0.21/32 tag 54321
  ip router ospf UNDERLAY area 0.0.0.0
interface loopback1
  ip address 10.1.1.21/32 tag 54321
  ip router ospf UNDERLAY area 0.0.0.0
interface loopback100
  ip address 10.1.100.1/32 tag 54321
  ip router ospf UNDERLAY area 0.0.0.0

interface Ethernet1/1
  no switchport
  ip address 10.1.10.5/31
  ip router ospf UNDERLAY area 0.0.0.0
  mtu 9216
  evpn multisite fabric-tracking
  no shutdown

interface Ethernet1/3
  no switchport
  ip address 172.16.1.0/31 tag 54321
  mtu 9216
  evpn multisite dci-tracking
  no shutdown

router ospf UNDERLAY
  router-id 10.1.0.21

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  multisite border-gateway interface loopback100
  member vni 10010
    multisite ingress-replication
    ingress-replication protocol bgp
  member vni 10020
    multisite ingress-replication
    ingress-replication protocol bgp
  member vni 10030
    multisite ingress-replication
    ingress-replication protocol bgp
  member vni 10040
    multisite ingress-replication
    ingress-replication protocol bgp
  member vni 50001 associate-vrf
  member vni 50002 associate-vrf

route-map RMAP-LOOPBACKS permit 10
  match tag 54321

router bgp 65001
  router-id 10.1.0.21
  address-family ipv4 unicast
    redistribute direct route-map RMAP-LOOPBACKS
    maximum-paths 4
  address-family l2vpn evpn
    advertise-pip
  ! ---- iBGP EVPN to local spine (RR) ----
  neighbor 10.1.0.1
    remote-as 65001
    update-source loopback0
    address-family l2vpn evpn
      send-community extended
  ! ---- DCI underlay: eBGP IPv4 to remote BGW (exchange loopbacks) ----
  neighbor 172.16.1.1
    remote-as 65002
    update-source Ethernet1/3
    address-family ipv4 unicast
  ! ---- DCI overlay: eBGP L2VPN EVPN to remote BGW ----
  neighbor 10.2.0.22
    remote-as 65002
    update-source loopback0
    ebgp-multihop 5
    peer-type fabric-external
    address-family l2vpn evpn
      send-community
      send-community extended
      rewrite-evpn-rt-asn
  vrf RED
    address-family ipv4 unicast
      advertise l2vpn evpn
  vrf BLUE
    address-family ipv4 unicast
      advertise l2vpn evpn

BGW2 (DC2, AS 65002, site-id 2) delta:

evpn multisite border-gateway 2
loopback0 10.2.0.22, loopback1 10.2.1.22 tag 54321, loopback100 10.2.100.1 tag 54321
Fabric link Eth1/1 -> SPINE2: 10.2.10.3/31, evpn multisite fabric-tracking
DCI link Eth1/3 -> BGW1: 172.16.1.1/31 tag 54321, evpn multisite dci-tracking
router ospf router-id 10.2.0.22
router bgp 65002, router-id 10.2.0.22
iBGP EVPN neighbor 10.2.0.1 (SPINE2) remote-as 65002
DCI underlay eBGP IPv4 neighbor 172.16.1.0 remote-as 65001, update-source Ethernet1/3
DCI overlay eBGP EVPN neighbor 10.1.0.21 remote-as 65001, update-source loopback0, ebgp-multihop 5, peer-type fabric-external, rewrite-evpn-rt-asn

9. Step 5: Attach Hosts & Test

With the fabrics bridged, let's provision access ports on the leaves to connect our user endpoints. This will map raw traffic into our VRFs.

! ---- LEAF1 (RED-User-DC1) ----
interface Ethernet1/5
  switchport
  switchport access vlan 10
  spanning-tree port type edge
  no shutdown

! ---- LEAF2 (BLUE-User-DC1) ----
interface Ethernet1/5
  switchport
  switchport access vlan 30
  spanning-tree port type edge
  no shutdown

! ---- LEAF3 (RED-User-DC2 + BLUE-User-DC2) ----
interface Ethernet1/5
  switchport
  switchport access vlan 20
  spanning-tree port type edge
  no shutdown
interface Ethernet1/4
  switchport
  switchport access vlan 40
  spanning-tree port type edge
  no shutdown

Host	Leaf / Port	VLAN	VRF	Host IP/Mask	Default Gateway
RED-User-DC1	LEAF1 Eth1/5	10	RED	192.168.10.10/24	192.168.10.1
BLUE-User-DC1	LEAF2 Eth1/5	30	BLUE	192.168.30.10/24	192.168.30.1
RED-User-DC2	LEAF3 Eth1/5	20	RED	192.168.20.10/24	192.168.20.1
BLUE-User-DC2	LEAF3 Eth1/4	40	BLUE	192.168.40.10/24	192.168.40.1

For your endpoint machines (using an Alpine Linux /etc/network/interfaces example for RED-User-DC1), configure static addressing. Ensure to run rc-service networking restart after updating.

auto eth0
iface eth0 inet static
    address 192.168.10.10
    netmask 255.255.255.0
    gateway 192.168.10.1

We can now run our test matrix to prove reachability and isolation across our sites.

Test	Path	Proves	Result
RED-User-DC1 (192.168.10.10) <-> RED-User-DC2 (192.168.20.10)	same VRF RED, inter-site inter-subnet via L3VNI 50001 (symmetric IRB, Type-5 + RT-ASN rewrite)	Inter-site routed reachability within a tenant	PASS
BLUE-User-DC1 (192.168.30.10) <-> BLUE-User-DC2 (192.168.40.10)	same VRF BLUE via L3VNI 50002	Inter-site routed reachability, second tenant	PASS
RED-User-DC1 <-> BLUE-User-DC1 (intra-DC1, cross-VRF)	no route-leak between RED/BLUE	Tenant isolation intra-site	FAIL (expected)
RED-User-DC1 <-> BLUE-User-DC2 (inter-site, cross-VRF)		Tenant isolation across sites	FAIL (expected)

10. Verification Cheat-Sheet

Deploying is only half the battle. When validating a Multi-Site deployment, order matters: always verify the local underlay, then local EVPN, followed by the DCI IPv4 session, and finally the cross-site EVPN exchange.

Interactive: click any command to reveal its live output (captured from the running lab) and a quick explanation of what to look for.

On the BGW · Multi-Site health

What to look for: the DCI-facing link (Eth1/3, tagged evpn multisite dci-tracking) must be Up. If this list is empty, tracking is missing and the VIP is withheld.

Interface      State
---------      -----
Ethernet1/3    Up

What to look for: the spine-facing link (Eth1/1, tagged evpn multisite fabric-tracking) must be Up. Fabric-tracking is what tells the BGW the local site is reachable so it can advertise the VIP.

Interface      State
---------      -----
Ethernet1/1    Up

What to look for: local VTEPs (10.1.1.11/12) plus the remote side reachable via the remote MS-VIP 10.2.100.1 and the remote BGW PIP 10.2.1.22, all Up, LearnType CP (control-plane / BGP-learned). Seeing the remote ...100.1 anycast VIP as a peer is the multi-site tunnel working.

Interface Peer-IP        State LearnType Uptime  Router-Mac
--------- -------------  ----- --------- ------  ----------------
nve1      10.1.1.11      Up    CP        2d20h   529e.b837.1b08
nve1      10.1.1.12      Up    CP        2d20h   521f.1d54.1b08
nve1      10.2.1.22      Up    CP        2d19h   52f8.3617.1b08
nve1      10.2.100.1     Up    CP        2d19h   0200.0a02.6401

What to look for: two EVPN peers: the local spine RR 10.1.0.1 (type I / iBGP, AS 65001) and the remote BGW 10.2.0.22 (type E / eBGP, AS 65002). A numeric PfxRcd (not Idle/Active) means the session is Established and exchanging routes. The per-type breakdown shows Type-2 (MAC/IP), Type-3 (IMET) and Type-5 (prefix) routes flowing.

BGP router identifier 10.1.0.21, local AS number 65001
BGP table version is 1171, L2VPN EVPN config peers 2, capable peers 2

Neighbor    V    AS   MsgRcvd MsgSent TblVer InQ OutQ Up/Down  State/PfxRcd
10.1.0.1    4 65001    4410    4231    1171   0    0   2d20h   20
10.2.0.22   4 65002    4367    4216    1171   0    0   2d19h   18

Neighbor    T    AS Type-1 Type-2 Type-3 Type-4 Type-5
10.1.0.1    I 65001   0      4      8      0      8
10.2.0.22   E 65002   0     10      4      0      4

Control plane · re-origination + next-hop rewrite

What to look for: in the L2VNI 10010 table the same MAC set appears three ways: i (iBGP, from a local leaf 10.1.1.11), l (locally originated, 10.1.1.21), and e (eBGP, from remote BGW 10.2.1.22). Remote Type-5 prefixes (e.g. 192.168.20.0) arrive with next-hop 10.2.100.1 (the remote MS-VIP), proving the next-hop rewrite worked. Output trimmed to the illustrative RDs.

Route Distinguisher: 10.1.0.21:32777    (L2VNI 10010)
*>i[2]:...[5254.00c0.1890]:[0.0.0.0]/216
                      10.1.1.11        100      0 i        <- local leaf (iBGP)
*>l[2]:...[52be.8479.1b08]:[0.0.0.0]/216
                      10.1.1.21        100  32768 i        <- local BGW (self)
*>e[2]:...[52f8.3617.1b08]:[0.0.0.0]/216
                      10.2.1.22               0 65002 i    <- remote BGW (eBGP)

Route Distinguisher: 10.1.0.21:4    (L3VNI 50001)
*>e[2]:...[192.168.20.10]/272
                      10.2.100.1  2000      0 65002 i      <- rewritten to remote VIP
*>i[2]:...[192.168.10.10]/272
                      10.1.1.11        100      0 i
*>l[5]:[0]:[0]:[24]:[192.168.10.0]/224
                      10.1.1.21    0   100      0 ?
*>l[5]:[0]:[0]:[24]:[192.168.20.0]/224
                      10.1.1.21    0   100      0 ?

What to look for: the same view as above but scoped to a single L2VNI, the fastest way to confirm a specific segment is stitched. You want all three path types present: local leaf (i), self (l), and the remote BGW (e, AS 65002). Swap 10010 for 10020/10030/10040 to check the other segments.

Route Distinguisher: 10.1.0.21:32777    (L2VNI 10010)
*>i[2]:...[5254.00c0.1890]:[0.0.0.0]/216
                      10.1.1.11        100      0 i
*>l[2]:...[52be.8479.1b08]:[0.0.0.0]/216
                      10.1.1.21        100  32768 i
*>e[2]:...[52f8.3617.1b08]:[0.0.0.0]/216
                      10.2.1.22               0 65002 i
*>i[2]:...[5254.00c0.1890]:[192.168.10.10]/272
                      10.1.1.11        100      0 i
*>i[3]:[0]:[32]:[10.1.1.11]/88   10.1.1.11    100  0 i
*>i[3]:[0]:[32]:[10.1.1.12]/88   10.1.1.12    100  0 i
*>l[3]:[0]:[32]:[10.1.1.21]/88   10.1.1.21    100  32768 i
*>e[3]:[0]:[32]:[10.2.1.22]/88   10.2.1.22          0 65002 i

What to look for: the local L2RIB view of learned MACs per topology (L2VNI), independent of BGP. Locally attached leaf MACs show Prod BGP via a local VTEP (e.g. 10.1.1.11 in VNI 10 with Label 10010). Crucially, the remote-site hosts (VNI 20 & 40) are learned via the remote MS-VIP 10.2.100.1 and carry the (Flags: RS) = Remote Site tag, and that RS flag is the fingerprint of a Multi-Site-stitched MAC. The Rmac entries in the L3VNI topologies (3900/3901) are the per-VTEP router MACs used for symmetric IRB.

Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote (Dup):Duplicate
(Spl):Split (Rcv):Recv (Nho):NH-Override (Dum):Dummy ...
NH Flags- (RS): Remote Site Flag  (Asy): Asymmetric VNI

Topology  Mac Address    Prod  Flags        Seq  Next-Hops
--------- -------------- ----- ------------ ---- --------------------------
10        5254.00c0.1890 BGP   SplRcv       0    10.1.1.11 (Label: 10010)
10        52be.8479.1b08 VXLAN Stt,Nho,Dum  0    10.1.1.21
20        5254.002a.d74b BGP   SplRcv       0    10.2.100.1 (Label: 10020)(Flags: RS)
20        52be.8479.1b08 VXLAN Stt,Nho,Dum  0    10.1.1.21
30        5254.00d8.8862 BGP   SplRcv       0    10.1.1.12 (Label: 10030)
30        52be.8479.1b08 VXLAN Stt,Nho,Dum  0    10.1.1.21
40        5254.0020.1f2e BGP   SplRcv       0    10.2.100.1 (Label: 10040)(Flags: RS)
40        52be.8479.1b08 VXLAN Stt,Nho,Dum  0    10.1.1.21
3900      0200.0a02.6401 VXLAN Rmac         0    10.2.100.1
3900      521f.1d54.1b08 VXLAN Rmac         0    10.1.1.12
3900      529e.b837.1b08 VXLAN Rmac         0    10.1.1.11
3900      52be.8479.1b08 VXLAN Stt,Nho,Dum  0    10.1.1.21
3900      52f8.3617.1b08 VXLAN Rmac         0    10.2.1.22
3901      0200.0a02.6401 VXLAN Rmac         0    10.2.100.1
3901      521f.1d54.1b08 VXLAN Rmac         0    10.1.1.12
3901      529e.b837.1b08 VXLAN Rmac         0    10.1.1.11
3901      52be.8479.1b08 VXLAN Stt,Nho,Dum  0    10.1.1.21
3901      52f8.3617.1b08 VXLAN Rmac         0    10.2.1.22

Underlay across DCI · eBGP IPv4

What to look for: the DCI underlay eBGP IPv4 session to neighbor 172.16.1.1 (AS 65002) is Established, the State/PfxRcd column shows 4 (a number, not Idle/Active/Connect), meaning it received 4 prefixes: the remote BGW's loopbacks and its MS-VIP. This session must be up before the overlay EVPN session can form over those loopbacks.

BGP summary information for VRF default, address family IPv4 Unicast
BGP router identifier 10.1.0.21, local AS number 65001
BGP table version is 14, IPv4 Unicast config peers 1, capable peers 1
7 network entries and 8 paths using 2652 bytes of memory

Neighbor     V    AS  MsgRcvd MsgSent TblVer InQ OutQ Up/Down  State/PfxRcd
172.16.1.1   4 65002    4215    4213     14   0    0   2d22h   4

What to look for: the single most important underlay check: does the local BGW have a route to the remote MS-VIP? Here 10.2.100.1/32 resolves via 172.16.1.1 (the DCI link), learned by bgp-65001, external, tag 65002, i.e. across the eBGP DCI session, exactly as intended. If this route is missing, the cross-site NVE tunnel can never form; nine times out of ten that traces back to a loopback that lost its tag 54321.

IP Route Table for VRF "default"
'*' denotes best ucast next-hop

10.2.100.1/32, ubest/mbest: 1/0
    *via 172.16.1.1, [20/0], 2d22h, bgp-65001, external, tag 65002

Tenant routing · symmetric IRB

What to look for: local subnets (192.168.10.0, .20.0) resolve via 10.1.1.11 (the local leaf VTEP), while the remote host 192.168.20.10/32 resolves via 10.2.100.1 (the remote MS-VIP), tag 65002, encap: VXLAN, segid 50001. That VIP next-hop on a remote host route is symmetric-IRB inter-site routing working.

IP Route Table for VRF "RED"
'*' denotes best ucast next-hop

192.168.10.0/24, ubest/mbest: 1/0
    *via 10.1.1.11%default, [200/0], 2d20h, bgp-65001, internal,
         tag 65001, segid: 50001 encap: VXLAN
192.168.20.0/24, ubest/mbest: 1/0
    *via 10.1.1.11%default, [200/0], 2d20h, bgp-65001, internal,
         tag 65001, segid: 50001 encap: VXLAN
192.168.20.10/32, ubest/mbest: 1/0
    *via 10.2.100.1%default, [20/2000], 2d19h, bgp-65001, external,
         tag 65002, segid: 50001 encap: VXLAN   <- remote host via VIP

What to look for: run on the leaf that owns the endpoint. It shows the locally-attached host (192.168.10.10), its MAC, the SVI (Vlan10) and the physical port (Eth1/5). This is the source of the Type-2 route the leaf advertises into EVPN, and if a host is missing here, it will never appear on remote VTEPs.

HMM host IPv4 routing table information for VRF RED
Status: *-valid, x-deleted, D-Duplicate, ...

    Host              MAC Address     SVI     Flags     Physical Interface
*   192.168.10.10/32  5254.00c0.1890  Vlan10  0x400201  Ethernet1/5

Troubleshooting order: per-site underlay (OSPF) up → intra-site EVPN (leaf<->spine) up → DCI IPv4 up (remote loopbacks/VIP learned) → eBGP EVPN (BGW<->BGW) up with peer-type fabric-external → remote routes show VIP next-hop → RT-ASN rewrite working (routes imported into local VRF) → data plane (MTU 9216 on fabric AND DCI). Most Multi-Site failures: missing dci-tracking, missing rewrite-evpn-rt-asn, or VIP not reachable across the DCI.

11. RoCEv2 Lossless Fabric: DCQCN (PFC + ECN/WRED)

A modern data center must handle storage and high-performance computing workloads elegantly. RoCEv2 (RDMA over Converged Ethernet v2) encapsulates RDMA into UDP/IP (UDP destination port 4791), making it routable across our VXLAN fabric. However, RDMA collapses catastrophically if packets drop, mandating a lossless network class.

To achieve this without drops, we implement DCQCN (Data Center Quantized Congestion Notification), a triad of features:

PFC (Priority Flow Control): A hop-by-hop pause mechanism for the no-drop class when ingress buffers fill. It must be enabled on every hop.
ECN (Explicit Congestion Notification): WRED on the egress queue marks packets (CE bits) instead of dropping them, signaling congestion end-to-end.
CNP (Congestion Notification Packet): The receiver returns a high-priority CNP (DSCP 48 / CoS 7) to throttle the sender.

Nexus 9000v Caveat The Nexus 9000v (virtual) accepts these QoS/PFC/ECN commands but does NOT enforce them in the data plane, since it has no switching ASIC. The configuration is correct and deployable for physical hardware (Nexus 9200 / 9300-EX/FX/GX / 9500); on the 9000v CML lab it validates syntax and control-plane only.

Traffic	DSCP	CoS	qos-group	Queue	Treatment
RoCEv2 data	24	3	3	c-out-8q-q3	No-drop (PFC) + ECN/WRED
CNP (congestion notification)	48	7	7	c-out-8q-q7	Strict priority (level 1)
Everything else	n/a	0	0	c-out-8q-q-default	Best-effort (bandwidth remaining)

The end-to-end rule dictates that PFC must be enabled on every single hop of the RoCE path. We start by applying global QoS classification and policies to all devices (Spines, Leaves, and BGWs).

class-map type qos match-any ROCEv2
  match dscp 24
class-map type qos match-any CNP
  match dscp 48
policy-map type qos QOS_MARKING
  class ROCEv2
    set qos-group 3
  class CNP
    set qos-group 7
  class class-default
    set qos-group 0
policy-map type network-qos qos_network
  class type network-qos c-8q-nq3
    pause pfc-cos 3
    mtu 9216
  class type network-qos c-8q-nq7
    mtu 9216
  class type network-qos c-8q-nq-default
    mtu 9216
policy-map type queuing QOS_EGRESS_PORT
  class type queuing c-out-8q-q7
    priority level 1
  class type queuing c-out-8q-q6
    bandwidth remaining percent 0
  class type queuing c-out-8q-q5
    bandwidth remaining percent 0
  class type queuing c-out-8q-q4
    bandwidth remaining percent 0
  class type queuing c-out-8q-q3
    bandwidth remaining percent 80
    random-detect minimum-threshold 150 kbytes maximum-threshold 3000 kbytes drop-probability 7 weight 0 ecn
  class type queuing c-out-8q-q2
    bandwidth remaining percent 0
  class type queuing c-out-8q-q1
    bandwidth remaining percent 0
  class type queuing c-out-8q-q-default
    bandwidth remaining percent 20
system qos
  service-policy type network-qos qos_network
  service-policy type queuing output QOS_EGRESS_PORT

Two rules the egress queuing policy must obey Note that the Nexus 9000 defaults to 4q mode (queues q0–q3); this design needs qos-groups up to 7, so the fabric must first be switched to 8q mode (the network-qos and queuing policies above do exactly that). Once in 8q mode, NX-OS enforces both of these rules, and violating either throws an error when you attach the policy (as per the Cisco QoS whitepaper):

List all eight queues. The queuing policy must reference every system class-map (c-out-8q-q1 through c-out-8q-q7 plus c-out-8q-q-default), you cannot define just the queues you care about. The unused queues (q1, q2, q4, q5, q6 here) are still present with bandwidth remaining percent 0. Omit them and you get: "Queuing policy map must have exactly 4 or 8 internally created class-maps when applied to a target."
A priority queue takes no bandwidth statement. The strict-priority class (c-out-8q-q7, our CNP queue) carries priority level 1 only. Adding bandwidth remaining percent to it collides with the priority scheduler and throws: "This Policy-map class has another action that cannot coexist." The bandwidth remaining percent values on the non-priority queues must sum to 100 (here 80 on q3 + 20 on default).

interface Ethernet1/1
  mtu 9216
  service-policy type qos input QOS_MARKING
  priority-flow-control mode on
  priority-flow-control watch-dog-interval on

Device	Interfaces	Role
SPINE1	Eth1/1, Eth1/2, Eth1/3	Fabric (to LEAF1, LEAF2, BGW1)
SPINE2	Eth1/1, Eth1/2	Fabric (to LEAF3, BGW2)
LEAF1	Eth1/1, Eth1/5	Uplink + RED host (VLAN 10)
LEAF2	Eth1/1, Eth1/5	Uplink + BLUE host (VLAN 30)
LEAF3	Eth1/1, Eth1/4, Eth1/5	Uplink + BLUE host (VLAN 40) + RED host (VLAN 20)
BGW1	Eth1/1, Eth1/3	Fabric uplink + DCI to DC2
BGW2	Eth1/1, Eth1/3	Fabric uplink + DCI to DC1

With classification, no-drop network-qos, ECN/WRED queuing, and per-interface PFC all in place across every hop, the RoCE path is complete. The final step is to prove the deployment programmed correctly.

Step 6: Verify the RoCEv2 / QoS Deployment

Before trusting the fabric with lossless traffic, confirm every layer of DCQCN actually programmed into the box. Walk the verification in the same direction traffic is treated: classification → marking (qos-group) → no-drop network-qos → egress queuing with ECN/WRED → PFC per interface. Run these on every device in the RoCE path (Spines, Leaves, and BGWs).

! --- 1. Classification + marking: is traffic landing in qos-group 3 / 7? ---
show class-map type qos ROCEv2
show class-map type qos CNP
show policy-map type qos QOS_MARKING
show policy-map interface ethernet 1/1 type qos

! --- 2. No-drop class: PFC pause bound to CoS 3, jumbo MTU ---
show policy-map system type network-qos
show queuing interface ethernet 1/1 | include -i "qos-group|pause|no-drop"

! --- 3. Egress queuing + ECN/WRED on the RoCE queue (q3) ---
show policy-map system type queuing
show queuing interface ethernet 1/1
show queuing interface ethernet 1/1 | include -i "ECN|WRED|random-detect|Minimum|Maximum"

! --- 4. PFC operational per interface (must be ON, every hop) ---
show interface priority-flow-control
show interface ethernet 1/1 priority-flow-control detail
show queuing pfc-queue interface ethernet 1/1 detail

! --- 5. Prove it is working: ECN marks rising, PFC pause frames, ZERO tail-drops on q3 ---
show queuing interface ethernet 1/1 | include -i "ECN Marked|WRED Dropped|Tail Drop"
show interface ethernet 1/1 priority-flow-control | include -i "RxPPP|TxPPP"

What a healthy deployment looks like: the QoS input policy shows non-zero match counters for the ROCEv2 class; system type network-qos lists pause pfc-cos 3 and mtu 9216 on c-8q-nq3; queue c-out-8q-q3 reports WRED enabled with min-threshold 150 KB / max-threshold 3000 KB and ECN on; PFC Oper state is On with the CoS 3 priority-vector set on every hop. Under load you should see ECN-marked counters and PFC pause (PPP) frames incrementing while the tail-drop counter on q3 stays at zero, and that zero is the entire point of a lossless class.

12. Troubleshooting

Multi-Site fabrics tend to fail in a handful of predictable ways, and almost all of them come back to one rule: a small piece of config is missing on exactly one box, and the whole cross-site path quietly stops working while the local fabric looks perfectly healthy. Work the path in order, from the underlay up to the tenant VRF, and each of these gets easy to spot. Here are the ones you will actually hit.

What you see	Most likely cause	Quick fix
Remote MS-VIP not in the routing table, DCI NVE peer stuck Down	Loopback missing `tag 54321`	Re-tag the loopback
`show nve multisite dci-links` is empty	Missing `dci-tracking` / `fabric-tracking`	Add tracking on the right links
Remote routes point at a remote leaf, traffic blackholes	Missing `peer-type fabric-external`	Set it on the inter-site EVPN neighbor
EVPN routes present but not in `vrf RED`	Missing `rewrite-evpn-rt-asn`	Add it under the EVPN address-family
Control plane up, big pings drop	MTU not 9216 on a hop	Fix MTU on fabric and DCI
RoCE drops under load	PFC not on every hop	Enable PFC on the whole path

Missing tag on a loopback interface

The classic one. The BGW matches tag 54321 in route-map RMAP-LOOPBACKS to redistribute its loopbacks into the DCI BGP session. Drop the tag on one loopback (especially the MS-VIP lo100) and that address is never advertised, so the remote BGW cannot reach your VIP, the DCI NVE peer stays Down, and no cross-site tunnel forms.

! ===== Detect: is the VIP actually tagged and advertised? =====
! On BGW1, confirm the loopback carries the tag
show running-config interface loopback100
show route-map RMAP-LOOPBACKS
! Is it redistributed into the DCI IPv4 table?
show ip bgp 10.1.100.1
! On BGW2 (remote side): did we learn it, and is the peer up?
show ip route 10.1.100.1
show nve peers
show nve multisite dci-links

! ===== Fix: put the tag back and let it re-advertise =====
interface loopback100
  ip address 10.1.100.1/32 tag 54321

Rule of thumb: if a cross-site tunnel will not form, the very first thing to check is whether the remote BGW has a route to your lo100 VIP. Nine times out of ten it traces back to a loopback that lost its tag 54321, or a redistribute route-map that is not matching it.

Missing multisite tracking on an interface

The BGW advertises its VIP based on link tracking: fabric links need evpn multisite fabric-tracking, DCI links need evpn multisite dci-tracking. Miss it and the VIP is withheld and cross-site BUM breaks, even with the interfaces physically up.

! ===== Detect =====
show nve multisite dci-links
show nve multisite fabric-links
show running-config interface Ethernet1/3

! ===== Fix =====
interface Ethernet1/3
  evpn multisite dci-tracking
interface Ethernet1/1
  evpn multisite fabric-tracking

Next-hop not rewritten (peer-type fabric-external)

The BGW rewrites the overlay next-hop to its own VIP before handing routes to the remote site, and peer-type fabric-external on the inter-site EVPN neighbor is what turns that on. Leave it off and the remote site learns routes pointing at the original leaf VTEP, which it cannot reach, so traffic is accepted and then silently dropped.

! ===== Detect: next-hop should be the remote MS-VIP, not a remote leaf =====
show bgp l2vpn evpn vni-id 10010
show bgp l2vpn evpn

! ===== Fix =====
router bgp 65001
  neighbor 10.2.0.22
    peer-type fabric-external

Routes learned but not imported into the VRF (rewrite-evpn-rt-asn)

With the two sites in different AS numbers, the ASN portion of the Route Targets does not match on import. Routes show up in show bgp l2vpn evpn but never land in the tenant VRF, so hosts cannot talk across sites. rewrite-evpn-rt-asn rewrites that ASN on the fly so the import works.

! ===== Detect: present in EVPN, missing from the VRF =====
show bgp l2vpn evpn route-type 2
show ip route vrf RED

! ===== Fix =====
router bgp 65001
  neighbor 10.2.0.22
    address-family l2vpn evpn
      rewrite-evpn-rt-asn

MTU mismatch on the path

VXLAN adds about 50 to 54 bytes of outer headers, so every underlay and DCI link needs MTU 9216. Leave one hop at the default and BGP and small pings still pass, so the problem only surfaces when full-size frames start dropping.

! ===== Detect: control plane fine, large DF-bit ping fails =====
show interface Ethernet1/1 | include -i MTU
ping 10.2.100.1 df-bit packet-size 9000 source loopback0

! ===== Fix =====
interface Ethernet1/1
  mtu 9216

RoCEv2 drops because PFC is not on every hop

A lossless class is only lossless if every hop honors it. Miss priority-flow-control mode on or the input QoS policy on even one transit interface and pause is not propagated there, so that hop drops RoCE traffic under congestion no matter how well the rest of the fabric is tuned.

! ===== Detect: find the hop where PFC is not On =====
show interface priority-flow-control
show queuing interface ethernet 1/1

! ===== Fix: apply on every RoCE-path interface, on every device =====
interface Ethernet1/1
  service-policy type qos input QOS_MARKING
  priority-flow-control mode on

13. Wrap-up

Over the course of this lab we built two independent VXLAN EVPN fabrics and stitched them into a single stretched Layer 2/Layer 3 service with Multi-Site, the architecture that lets you scale a data center by adding sites instead of enlarging one fragile blast radius, while the Border Gateway keeps flooding, route churn, and failures contained at the site edge. On top of that fabric we layered a RoCEv2 lossless class with DCQCN (PFC, ECN/WRED, and CNP), the same traffic-engineering discipline that makes a network trustworthy for storage, AI, and HPC workloads where a single dropped packet is catastrophic.

The BGP and VXLAN control plane you configured here transfers verbatim to physical Nexus 9300-EX/FX3/GX hardware; the one real difference is that a true switching ASIC will enforce your no-drop queuing and let the DCQCN control loops engage at line rate. From here, look toward scaling out to more than two sites, adding vPC or ESI multihoming for host redundancy, and validating your lossless class under real congestion, the point where multi-site reach and QoS rigor combine to build the resilient, high-performance data centers the industry now demands.

Full Configuration

The complete, copy-ready running configuration for every switch, grouped by site. Use the menu on the left to jump straight to a device. Each block is the full config for that box, exactly as applied.

DC1 · LEAF1 AS 65001

feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
nv overlay evpn
fabric forwarding anycast-gateway-mac 0000.00aa.aaaa

vlan 10
  vn-segment 10010
vlan 20
  vn-segment 10020
vlan 30
  vn-segment 10030
vlan 40
  vn-segment 10040
vlan 3900
  vn-segment 50001
vlan 3901
  vn-segment 50002

vrf context RED
  vni 50001
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn
vrf context BLUE
  vni 50002
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn

interface Vlan3900
  no shutdown
  vrf member RED
  ip forward
  no ip redirects
interface Vlan3901
  no shutdown
  vrf member BLUE
  ip forward
  no ip redirects

interface Vlan10
  no shutdown
  vrf member RED
  ip address 192.168.10.1/24
  fabric forwarding mode anycast-gateway
interface Vlan20
  no shutdown
  vrf member RED
  ip address 192.168.20.1/24
  fabric forwarding mode anycast-gateway
interface Vlan30
  no shutdown
  vrf member BLUE
  ip address 192.168.30.1/24
  fabric forwarding mode anycast-gateway
interface Vlan40
  no shutdown
  vrf member BLUE
  ip address 192.168.40.1/24
  fabric forwarding mode anycast-gateway

evpn
  vni 10010 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10020 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10030 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10040 l2
    rd auto
    route-target import auto
    route-target export auto

interface loopback0
  ip address 10.1.0.11/32
  ip router ospf UNDERLAY area 0.0.0.0
interface loopback1
  ip address 10.1.1.11/32
  ip router ospf UNDERLAY area 0.0.0.0

interface Ethernet1/1
  no switchport
  ip address 10.1.10.1/31
  ip router ospf UNDERLAY area 0.0.0.0
  mtu 9216
  no shutdown

interface Ethernet1/5
  switchport
  switchport access vlan 10
  spanning-tree port type edge
  no shutdown

router ospf UNDERLAY
  router-id 10.1.0.11

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 10010
    ingress-replication protocol bgp
  member vni 10020
    ingress-replication protocol bgp
  member vni 10030
    ingress-replication protocol bgp
  member vni 10040
    ingress-replication protocol bgp
  member vni 50001 associate-vrf
  member vni 50002 associate-vrf

route-map REDIST-CONN permit 10

router bgp 65001
  router-id 10.1.0.11
  template peer SPINE-EVPN
    remote-as 65001
    update-source loopback0
    address-family l2vpn evpn
      send-community extended
  neighbor 10.1.0.1
    inherit peer SPINE-EVPN
  vrf RED
    address-family ipv4 unicast
      advertise l2vpn evpn
      redistribute direct route-map REDIST-CONN
  vrf BLUE
    address-family ipv4 unicast
      advertise l2vpn evpn
      redistribute direct route-map REDIST-CONN

! ===== RoCEv2 lossless fabric - DCQCN (PFC + ECN/WRED) =====
class-map type qos match-any ROCEv2
  match dscp 24
class-map type qos match-any CNP
  match dscp 48
policy-map type qos QOS_MARKING
  class ROCEv2
    set qos-group 3
  class CNP
    set qos-group 7
  class class-default
    set qos-group 0
policy-map type network-qos qos_network
  class type network-qos c-8q-nq3
    pause pfc-cos 3
    mtu 9216
  class type network-qos c-8q-nq7
    mtu 9216
  class type network-qos c-8q-nq-default
    mtu 9216
policy-map type queuing QOS_EGRESS_PORT
  class type queuing c-out-8q-q7
    priority level 1
  class type queuing c-out-8q-q6
    bandwidth remaining percent 0
  class type queuing c-out-8q-q5
    bandwidth remaining percent 0
  class type queuing c-out-8q-q4
    bandwidth remaining percent 0
  class type queuing c-out-8q-q3
    bandwidth remaining percent 80
    random-detect minimum-threshold 150 kbytes maximum-threshold 3000 kbytes drop-probability 7 weight 0 ecn
  class type queuing c-out-8q-q2
    bandwidth remaining percent 0
  class type queuing c-out-8q-q1
    bandwidth remaining percent 0
  class type queuing c-out-8q-q-default
    bandwidth remaining percent 20
system qos
  service-policy type network-qos qos_network
  service-policy type queuing output QOS_EGRESS_PORT

interface Ethernet1/1
  mtu 9216
  service-policy type qos input QOS_MARKING
  priority-flow-control mode on
  priority-flow-control watch-dog-interval on
interface Ethernet1/5
  mtu 9216
  service-policy type qos input QOS_MARKING
  priority-flow-control mode on
  priority-flow-control watch-dog-interval on

DC1 · LEAF2 AS 65001

feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
nv overlay evpn
fabric forwarding anycast-gateway-mac 0000.00aa.aaaa

vlan 10
  vn-segment 10010
vlan 20
  vn-segment 10020
vlan 30
  vn-segment 10030
vlan 40
  vn-segment 10040
vlan 3900
  vn-segment 50001
vlan 3901
  vn-segment 50002

vrf context RED
  vni 50001
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn
vrf context BLUE
  vni 50002
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn

interface Vlan3900
  no shutdown
  vrf member RED
  ip forward
  no ip redirects
interface Vlan3901
  no shutdown
  vrf member BLUE
  ip forward
  no ip redirects

interface Vlan10
  no shutdown
  vrf member RED
  ip address 192.168.10.1/24
  fabric forwarding mode anycast-gateway
interface Vlan20
  no shutdown
  vrf member RED
  ip address 192.168.20.1/24
  fabric forwarding mode anycast-gateway
interface Vlan30
  no shutdown
  vrf member BLUE
  ip address 192.168.30.1/24
  fabric forwarding mode anycast-gateway
interface Vlan40
  no shutdown
  vrf member BLUE
  ip address 192.168.40.1/24
  fabric forwarding mode anycast-gateway

evpn
  vni 10010 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10020 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10030 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10040 l2
    rd auto
    route-target import auto
    route-target export auto

interface loopback0
  ip address 10.1.0.12/32
  ip router ospf UNDERLAY area 0.0.0.0
interface loopback1
  ip address 10.1.1.12/32
  ip router ospf UNDERLAY area 0.0.0.0

interface Ethernet1/1
  no switchport
  ip address 10.1.10.3/31
  ip router ospf UNDERLAY area 0.0.0.0
  mtu 9216
  no shutdown

interface Ethernet1/5
  switchport
  switchport access vlan 30
  spanning-tree port type edge
  no shutdown

router ospf UNDERLAY
  router-id 10.1.0.12

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 10010
    ingress-replication protocol bgp
  member vni 10020
    ingress-replication protocol bgp
  member vni 10030
    ingress-replication protocol bgp
  member vni 10040
    ingress-replication protocol bgp
  member vni 50001 associate-vrf
  member vni 50002 associate-vrf

route-map REDIST-CONN permit 10

router bgp 65001
  router-id 10.1.0.12
  template peer SPINE-EVPN
    remote-as 65001
    update-source loopback0
    address-family l2vpn evpn
      send-community extended
  neighbor 10.1.0.1
    inherit peer SPINE-EVPN
  vrf RED
    address-family ipv4 unicast
      advertise l2vpn evpn
      redistribute direct route-map REDIST-CONN
  vrf BLUE
    address-family ipv4 unicast
      advertise l2vpn evpn
      redistribute direct route-map REDIST-CONN

! ===== RoCEv2 lossless fabric - DCQCN (PFC + ECN/WRED) =====
class-map type qos match-any ROCEv2
  match dscp 24
class-map type qos match-any CNP
  match dscp 48
policy-map type qos QOS_MARKING
  class ROCEv2
    set qos-group 3
  class CNP
    set qos-group 7
  class class-default
    set qos-group 0
policy-map type network-qos qos_network
  class type network-qos c-8q-nq3
    pause pfc-cos 3
    mtu 9216
  class type network-qos c-8q-nq7
    mtu 9216
  class type network-qos c-8q-nq-default
    mtu 9216
policy-map type queuing QOS_EGRESS_PORT
  class type queuing c-out-8q-q7
    priority level 1
  class type queuing c-out-8q-q6
    bandwidth remaining percent 0
  class type queuing c-out-8q-q5
    bandwidth remaining percent 0
  class type queuing c-out-8q-q4
    bandwidth remaining percent 0
  class type queuing c-out-8q-q3
    bandwidth remaining percent 80
    random-detect minimum-threshold 150 kbytes maximum-threshold 3000 kbytes drop-probability 7 weight 0 ecn
  class type queuing c-out-8q-q2
    bandwidth remaining percent 0
  class type queuing c-out-8q-q1
    bandwidth remaining percent 0
  class type queuing c-out-8q-q-default
    bandwidth remaining percent 20
system qos
  service-policy type network-qos qos_network
  service-policy type queuing output QOS_EGRESS_PORT

interface Ethernet1/1
  mtu 9216
  service-policy type qos input QOS_MARKING
  priority-flow-control mode on
  priority-flow-control watch-dog-interval on
interface Ethernet1/5
  mtu 9216
  service-policy type qos input QOS_MARKING
  priority-flow-control mode on
  priority-flow-control watch-dog-interval on

DC1 · SPINE1 AS 65001

feature ospf
feature bgp
feature nv overlay
nv overlay evpn

interface loopback0
  ip address 10.1.0.1/32
  ip router ospf UNDERLAY area 0.0.0.0

interface Ethernet1/1
  no switchport
  ip address 10.1.10.0/31
  ip router ospf UNDERLAY area 0.0.0.0
  mtu 9216
  no shutdown
interface Ethernet1/2
  no switchport
  ip address 10.1.10.2/31
  ip router ospf UNDERLAY area 0.0.0.0
  mtu 9216
  no shutdown
interface Ethernet1/3
  no switchport
  ip address 10.1.10.4/31
  ip router ospf UNDERLAY area 0.0.0.0
  mtu 9216
  no shutdown

router ospf UNDERLAY
  router-id 10.1.0.1

router bgp 65001
  router-id 10.1.0.1
  address-family l2vpn evpn
    retain route-target all
  template peer FABRIC-RR
    remote-as 65001
    update-source loopback0
    address-family l2vpn evpn
      send-community extended
      route-reflector-client
  neighbor 10.1.0.11
    inherit peer FABRIC-RR
  neighbor 10.1.0.12
    inherit peer FABRIC-RR
  neighbor 10.1.0.21
    inherit peer FABRIC-RR

! ===== RoCEv2 lossless fabric - DCQCN (PFC + ECN/WRED) =====
class-map type qos match-any ROCEv2
  match dscp 24
class-map type qos match-any CNP
  match dscp 48
policy-map type qos QOS_MARKING
  class ROCEv2
    set qos-group 3
  class CNP
    set qos-group 7
  class class-default
    set qos-group 0
policy-map type network-qos qos_network
  class type network-qos c-8q-nq3
    pause pfc-cos 3
    mtu 9216
  class type network-qos c-8q-nq7
    mtu 9216
  class type network-qos c-8q-nq-default
    mtu 9216
policy-map type queuing QOS_EGRESS_PORT
  class type queuing c-out-8q-q7
    priority level 1
  class type queuing c-out-8q-q6
    bandwidth remaining percent 0
  class type queuing c-out-8q-q5
    bandwidth remaining percent 0
  class type queuing c-out-8q-q4
    bandwidth remaining percent 0
  class type queuing c-out-8q-q3
    bandwidth remaining percent 80
    random-detect minimum-threshold 150 kbytes maximum-threshold 3000 kbytes drop-probability 7 weight 0 ecn
  class type queuing c-out-8q-q2
    bandwidth remaining percent 0
  class type queuing c-out-8q-q1
    bandwidth remaining percent 0
  class type queuing c-out-8q-q-default
    bandwidth remaining percent 20
system qos
  service-policy type network-qos qos_network
  service-policy type queuing output QOS_EGRESS_PORT

interface Ethernet1/1
  mtu 9216
  service-policy type qos input QOS_MARKING
  priority-flow-control mode on
  priority-flow-control watch-dog-interval on
interface Ethernet1/2
  mtu 9216
  service-policy type qos input QOS_MARKING
  priority-flow-control mode on
  priority-flow-control watch-dog-interval on
interface Ethernet1/3
  mtu 9216
  service-policy type qos input QOS_MARKING
  priority-flow-control mode on
  priority-flow-control watch-dog-interval on

DC1 · BGW1 AS 65001

feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
nv overlay evpn
fabric forwarding anycast-gateway-mac 0000.00aa.aaaa

evpn multisite border-gateway 1

vlan 10
  vn-segment 10010
vlan 20
  vn-segment 10020
vlan 30
  vn-segment 10030
vlan 40
  vn-segment 10040
vlan 3900
  vn-segment 50001
vlan 3901
  vn-segment 50002

vrf context RED
  vni 50001
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn
vrf context BLUE
  vni 50002
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn

interface Vlan3900
  no shutdown
  vrf member RED
  ip forward
  no ip redirects
interface Vlan3901
  no shutdown
  vrf member BLUE
  ip forward
  no ip redirects

evpn
  vni 10010 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10020 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10030 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10040 l2
    rd auto
    route-target import auto
    route-target export auto

interface loopback0
  ip address 10.1.0.21/32 tag 54321
  ip router ospf UNDERLAY area 0.0.0.0
interface loopback1
  ip address 10.1.1.21/32 tag 54321
  ip router ospf UNDERLAY area 0.0.0.0
interface loopback100
  ip address 10.1.100.1/32 tag 54321
  ip router ospf UNDERLAY area 0.0.0.0

interface Ethernet1/1
  no switchport
  ip address 10.1.10.5/31
  ip router ospf UNDERLAY area 0.0.0.0
  mtu 9216
  evpn multisite fabric-tracking
  no shutdown

interface Ethernet1/3
  no switchport
  ip address 172.16.1.0/31 tag 54321
  mtu 9216
  evpn multisite dci-tracking
  no shutdown

router ospf UNDERLAY
  router-id 10.1.0.21

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  multisite border-gateway interface loopback100
  member vni 10010
    multisite ingress-replication
    ingress-replication protocol bgp
  member vni 10020
    multisite ingress-replication
    ingress-replication protocol bgp
  member vni 10030
    multisite ingress-replication
    ingress-replication protocol bgp
  member vni 10040
    multisite ingress-replication
    ingress-replication protocol bgp
  member vni 50001 associate-vrf
  member vni 50002 associate-vrf

route-map RMAP-LOOPBACKS permit 10
  match tag 54321

router bgp 65001
  router-id 10.1.0.21
  address-family ipv4 unicast
    redistribute direct route-map RMAP-LOOPBACKS
    maximum-paths 4
  address-family l2vpn evpn
    advertise-pip
  neighbor 10.1.0.1
    remote-as 65001
    update-source loopback0
    address-family l2vpn evpn
      send-community extended
  neighbor 172.16.1.1
    remote-as 65002
    update-source Ethernet1/3
    address-family ipv4 unicast
  neighbor 10.2.0.22
    remote-as 65002
    update-source loopback0
    ebgp-multihop 5
    peer-type fabric-external
    address-family l2vpn evpn
      send-community
      send-community extended
      rewrite-evpn-rt-asn
  vrf RED
    address-family ipv4 unicast
      advertise l2vpn evpn
  vrf BLUE
    address-family ipv4 unicast
      advertise l2vpn evpn

! ===== RoCEv2 lossless fabric - DCQCN (PFC + ECN/WRED) =====
class-map type qos match-any ROCEv2
  match dscp 24
class-map type qos match-any CNP
  match dscp 48
policy-map type qos QOS_MARKING
  class ROCEv2
    set qos-group 3
  class CNP
    set qos-group 7
  class class-default
    set qos-group 0
policy-map type network-qos qos_network
  class type network-qos c-8q-nq3
    pause pfc-cos 3
    mtu 9216
  class type network-qos c-8q-nq7
    mtu 9216
  class type network-qos c-8q-nq-default
    mtu 9216
policy-map type queuing QOS_EGRESS_PORT
  class type queuing c-out-8q-q7
    priority level 1
  class type queuing c-out-8q-q6
    bandwidth remaining percent 0
  class type queuing c-out-8q-q5
    bandwidth remaining percent 0
  class type queuing c-out-8q-q4
    bandwidth remaining percent 0
  class type queuing c-out-8q-q3
    bandwidth remaining percent 80
    random-detect minimum-threshold 150 kbytes maximum-threshold 3000 kbytes drop-probability 7 weight 0 ecn
  class type queuing c-out-8q-q2
    bandwidth remaining percent 0
  class type queuing c-out-8q-q1
    bandwidth remaining percent 0
  class type queuing c-out-8q-q-default
    bandwidth remaining percent 20
system qos
  service-policy type network-qos qos_network
  service-policy type queuing output QOS_EGRESS_PORT

interface Ethernet1/1
  mtu 9216
  service-policy type qos input QOS_MARKING
  priority-flow-control mode on
  priority-flow-control watch-dog-interval on
interface Ethernet1/3
  mtu 9216
  service-policy type qos input QOS_MARKING
  priority-flow-control mode on
  priority-flow-control watch-dog-interval on

DC2 · LEAF3 AS 65002

feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
nv overlay evpn
fabric forwarding anycast-gateway-mac 0000.00aa.aaaa

vlan 10
  vn-segment 10010
vlan 20
  vn-segment 10020
vlan 30
  vn-segment 10030
vlan 40
  vn-segment 10040
vlan 3900
  vn-segment 50001
vlan 3901
  vn-segment 50002

vrf context RED
  vni 50001
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn
vrf context BLUE
  vni 50002
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn

interface Vlan3900
  no shutdown
  vrf member RED
  ip forward
  no ip redirects
interface Vlan3901
  no shutdown
  vrf member BLUE
  ip forward
  no ip redirects

interface Vlan10
  no shutdown
  vrf member RED
  ip address 192.168.10.1/24
  fabric forwarding mode anycast-gateway
interface Vlan20
  no shutdown
  vrf member RED
  ip address 192.168.20.1/24
  fabric forwarding mode anycast-gateway
interface Vlan30
  no shutdown
  vrf member BLUE
  ip address 192.168.30.1/24
  fabric forwarding mode anycast-gateway
interface Vlan40
  no shutdown
  vrf member BLUE
  ip address 192.168.40.1/24
  fabric forwarding mode anycast-gateway

evpn
  vni 10010 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10020 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10030 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10040 l2
    rd auto
    route-target import auto
    route-target export auto

interface loopback0
  ip address 10.2.0.13/32
  ip router ospf UNDERLAY area 0.0.0.0
interface loopback1
  ip address 10.2.1.13/32
  ip router ospf UNDERLAY area 0.0.0.0

interface Ethernet1/1
  no switchport
  ip address 10.2.10.1/31
  ip router ospf UNDERLAY area 0.0.0.0
  mtu 9216
  no shutdown

interface Ethernet1/5
  switchport
  switchport access vlan 20
  spanning-tree port type edge
  no shutdown
interface Ethernet1/4
  switchport
  switchport access vlan 40
  spanning-tree port type edge
  no shutdown

router ospf UNDERLAY
  router-id 10.2.0.13

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 10010
    ingress-replication protocol bgp
  member vni 10020
    ingress-replication protocol bgp
  member vni 10030
    ingress-replication protocol bgp
  member vni 10040
    ingress-replication protocol bgp
  member vni 50001 associate-vrf
  member vni 50002 associate-vrf

route-map REDIST-CONN permit 10

router bgp 65002
  router-id 10.2.0.13
  template peer SPINE-EVPN
    remote-as 65002
    update-source loopback0
    address-family l2vpn evpn
      send-community extended
  neighbor 10.2.0.1
    inherit peer SPINE-EVPN
  vrf RED
    address-family ipv4 unicast
      advertise l2vpn evpn
      redistribute direct route-map REDIST-CONN
  vrf BLUE
    address-family ipv4 unicast
      advertise l2vpn evpn
      redistribute direct route-map REDIST-CONN

! ===== RoCEv2 lossless fabric - DCQCN (PFC + ECN/WRED) =====
class-map type qos match-any ROCEv2
  match dscp 24
class-map type qos match-any CNP
  match dscp 48
policy-map type qos QOS_MARKING
  class ROCEv2
    set qos-group 3
  class CNP
    set qos-group 7
  class class-default
    set qos-group 0
policy-map type network-qos qos_network
  class type network-qos c-8q-nq3
    pause pfc-cos 3
    mtu 9216
  class type network-qos c-8q-nq7
    mtu 9216
  class type network-qos c-8q-nq-default
    mtu 9216
policy-map type queuing QOS_EGRESS_PORT
  class type queuing c-out-8q-q7
    priority level 1
  class type queuing c-out-8q-q6
    bandwidth remaining percent 0
  class type queuing c-out-8q-q5
    bandwidth remaining percent 0
  class type queuing c-out-8q-q4
    bandwidth remaining percent 0
  class type queuing c-out-8q-q3
    bandwidth remaining percent 80
    random-detect minimum-threshold 150 kbytes maximum-threshold 3000 kbytes drop-probability 7 weight 0 ecn
  class type queuing c-out-8q-q2
    bandwidth remaining percent 0
  class type queuing c-out-8q-q1
    bandwidth remaining percent 0
  class type queuing c-out-8q-q-default
    bandwidth remaining percent 20
system qos
  service-policy type network-qos qos_network
  service-policy type queuing output QOS_EGRESS_PORT

interface Ethernet1/1
  mtu 9216
  service-policy type qos input QOS_MARKING
  priority-flow-control mode on
  priority-flow-control watch-dog-interval on
interface Ethernet1/4
  mtu 9216
  service-policy type qos input QOS_MARKING
  priority-flow-control mode on
  priority-flow-control watch-dog-interval on
interface Ethernet1/5
  mtu 9216
  service-policy type qos input QOS_MARKING
  priority-flow-control mode on
  priority-flow-control watch-dog-interval on

DC2 · SPINE2 AS 65002

feature ospf
feature bgp
feature nv overlay
nv overlay evpn

interface loopback0
  ip address 10.2.0.1/32
  ip router ospf UNDERLAY area 0.0.0.0

interface Ethernet1/1
  no switchport
  ip address 10.2.10.0/31
  ip router ospf UNDERLAY area 0.0.0.0
  mtu 9216
  no shutdown
interface Ethernet1/2
  no switchport
  ip address 10.2.10.2/31
  ip router ospf UNDERLAY area 0.0.0.0
  mtu 9216
  no shutdown

router ospf UNDERLAY
  router-id 10.2.0.1

router bgp 65002
  router-id 10.2.0.1
  address-family l2vpn evpn
    retain route-target all
  template peer FABRIC-RR
    remote-as 65002
    update-source loopback0
    address-family l2vpn evpn
      send-community extended
      route-reflector-client
  neighbor 10.2.0.13
    inherit peer FABRIC-RR
  neighbor 10.2.0.22
    inherit peer FABRIC-RR

! ===== RoCEv2 lossless fabric - DCQCN (PFC + ECN/WRED) =====
class-map type qos match-any ROCEv2
  match dscp 24
class-map type qos match-any CNP
  match dscp 48
policy-map type qos QOS_MARKING
  class ROCEv2
    set qos-group 3
  class CNP
    set qos-group 7
  class class-default
    set qos-group 0
policy-map type network-qos qos_network
  class type network-qos c-8q-nq3
    pause pfc-cos 3
    mtu 9216
  class type network-qos c-8q-nq7
    mtu 9216
  class type network-qos c-8q-nq-default
    mtu 9216
policy-map type queuing QOS_EGRESS_PORT
  class type queuing c-out-8q-q7
    priority level 1
  class type queuing c-out-8q-q6
    bandwidth remaining percent 0
  class type queuing c-out-8q-q5
    bandwidth remaining percent 0
  class type queuing c-out-8q-q4
    bandwidth remaining percent 0
  class type queuing c-out-8q-q3
    bandwidth remaining percent 80
    random-detect minimum-threshold 150 kbytes maximum-threshold 3000 kbytes drop-probability 7 weight 0 ecn
  class type queuing c-out-8q-q2
    bandwidth remaining percent 0
  class type queuing c-out-8q-q1
    bandwidth remaining percent 0
  class type queuing c-out-8q-q-default
    bandwidth remaining percent 20
system qos
  service-policy type network-qos qos_network
  service-policy type queuing output QOS_EGRESS_PORT

interface Ethernet1/1
  mtu 9216
  service-policy type qos input QOS_MARKING
  priority-flow-control mode on
  priority-flow-control watch-dog-interval on
interface Ethernet1/2
  mtu 9216
  service-policy type qos input QOS_MARKING
  priority-flow-control mode on
  priority-flow-control watch-dog-interval on

DC2 · BGW2 AS 65002

feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
nv overlay evpn
fabric forwarding anycast-gateway-mac 0000.00aa.aaaa

evpn multisite border-gateway 2

vlan 10
  vn-segment 10010
vlan 20
  vn-segment 10020
vlan 30
  vn-segment 10030
vlan 40
  vn-segment 10040
vlan 3900
  vn-segment 50001
vlan 3901
  vn-segment 50002

vrf context RED
  vni 50001
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn
vrf context BLUE
  vni 50002
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn

interface Vlan3900
  no shutdown
  vrf member RED
  ip forward
  no ip redirects
interface Vlan3901
  no shutdown
  vrf member BLUE
  ip forward
  no ip redirects

evpn
  vni 10010 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10020 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10030 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10040 l2
    rd auto
    route-target import auto
    route-target export auto

interface loopback0
  ip address 10.2.0.22/32 tag 54321
  ip router ospf UNDERLAY area 0.0.0.0
interface loopback1
  ip address 10.2.1.22/32 tag 54321
  ip router ospf UNDERLAY area 0.0.0.0
interface loopback100
  ip address 10.2.100.1/32 tag 54321
  ip router ospf UNDERLAY area 0.0.0.0

interface Ethernet1/1
  no switchport
  ip address 10.2.10.3/31
  ip router ospf UNDERLAY area 0.0.0.0
  mtu 9216
  evpn multisite fabric-tracking
  no shutdown

interface Ethernet1/3
  no switchport
  ip address 172.16.1.1/31 tag 54321
  mtu 9216
  evpn multisite dci-tracking
  no shutdown

router ospf UNDERLAY
  router-id 10.2.0.22

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  multisite border-gateway interface loopback100
  member vni 10010
    multisite ingress-replication
    ingress-replication protocol bgp
  member vni 10020
    multisite ingress-replication
    ingress-replication protocol bgp
  member vni 10030
    multisite ingress-replication
    ingress-replication protocol bgp
  member vni 10040
    multisite ingress-replication
    ingress-replication protocol bgp
  member vni 50001 associate-vrf
  member vni 50002 associate-vrf

route-map RMAP-LOOPBACKS permit 10
  match tag 54321

router bgp 65002
  router-id 10.2.0.22
  address-family ipv4 unicast
    redistribute direct route-map RMAP-LOOPBACKS
    maximum-paths 4
  address-family l2vpn evpn
    advertise-pip
  neighbor 10.2.0.1
    remote-as 65002
    update-source loopback0
    address-family l2vpn evpn
      send-community extended
  neighbor 172.16.1.0
    remote-as 65001
    update-source Ethernet1/3
    address-family ipv4 unicast
  neighbor 10.1.0.21
    remote-as 65001
    update-source loopback0
    ebgp-multihop 5
    peer-type fabric-external
    address-family l2vpn evpn
      send-community
      send-community extended
      rewrite-evpn-rt-asn
  vrf RED
    address-family ipv4 unicast
      advertise l2vpn evpn
  vrf BLUE
    address-family ipv4 unicast
      advertise l2vpn evpn

! ===== RoCEv2 lossless fabric - DCQCN (PFC + ECN/WRED) =====
class-map type qos match-any ROCEv2
  match dscp 24
class-map type qos match-any CNP
  match dscp 48
policy-map type qos QOS_MARKING
  class ROCEv2
    set qos-group 3
  class CNP
    set qos-group 7
  class class-default
    set qos-group 0
policy-map type network-qos qos_network
  class type network-qos c-8q-nq3
    pause pfc-cos 3
    mtu 9216
  class type network-qos c-8q-nq7
    mtu 9216
  class type network-qos c-8q-nq-default
    mtu 9216
policy-map type queuing QOS_EGRESS_PORT
  class type queuing c-out-8q-q7
    priority level 1
  class type queuing c-out-8q-q6
    bandwidth remaining percent 0
  class type queuing c-out-8q-q5
    bandwidth remaining percent 0
  class type queuing c-out-8q-q4
    bandwidth remaining percent 0
  class type queuing c-out-8q-q3
    bandwidth remaining percent 80
    random-detect minimum-threshold 150 kbytes maximum-threshold 3000 kbytes drop-probability 7 weight 0 ecn
  class type queuing c-out-8q-q2
    bandwidth remaining percent 0
  class type queuing c-out-8q-q1
    bandwidth remaining percent 0
  class type queuing c-out-8q-q-default
    bandwidth remaining percent 20
system qos
  service-policy type network-qos qos_network
  service-policy type queuing output QOS_EGRESS_PORT

interface Ethernet1/1
  mtu 9216
  service-policy type qos input QOS_MARKING
  priority-flow-control mode on
  priority-flow-control watch-dog-interval on
interface Ethernet1/3
  mtu 9216
  service-policy type qos input QOS_MARKING
  priority-flow-control mode on
  priority-flow-control watch-dog-interval on