VXLAN BGP EVPN

based Multi-Pod, Multi-

Fabric, Multi-Site
Max Ardica – Principal Engineer
Lukas Krattiger – Principal Engineer
BRKDCN-2035
Cisco Spark
Questions?
Use Cisco Spark to chat with the
speaker after the session

How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space

Cisco Spark spaces will be cs.co/ciscolivebot#BRKDCN-2035

available until July 3, 2017.

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Who Are the Presenters?

Max Ardica Lukas Krattiger

Principal Engineer - INSBU Principal Engineer – INSBU
@ccie21921

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Session Objectives
At the end of the session, the participants should be able to:
Articulate the different deployment options to interconnect
VXLAN EVPN Networks (Multi-Pod vs. Multi-Fabric vs. Multi-
Site)
Understand the functionalities and specific design
considerations associated to the new VXLAN Multi-Site
architecture
Initial assumption:
The audience already has a good knowledge of the VXLAN
EVPN technology and its use to deploy modern Data Center
Fabrics

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Session Reference
• BRKDCN-2304
• L4-L7 Service Integration in Multi-Tenant VXLAN EVPN Data Center Fabrics
• BRKDCN-3378
• Building DataCenter Networks with VXLAN BGP-EVPN
• Wednesday, Jun 28, 1:30 pm

• BRKDCN-2125
• Overlay Management and Visibility with VXLAN
• Thursday, Jun 29, 10:30 am

• BRKDCN-2342
• Programmable Fabric Automation and Management with DCNM 10
• Thursday, Jun 29, 1:00 p.m.

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Agenda
Introduction
VXLAN EVPN Interconnect Evolution
• Multi-Pod
• Multi-Fabric
• Multi-Site

VXLAN EVPN Multi-Site Deep Dive

• Walkthrough
• Control- and Data-Plane
• Deployment Considerations

Conclusions and Q&A

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Introduction
 Data Center Interconnect – DCI Model
Connecting Virtualized Data Centers

Layer 2 Domain Elasticity

Local LAN Fabric
Extended LAN fabric IP Mobility
Multi-tenancy/Segmentation Optimal Ingress and Egress
Segment-IDs in VXLAN, LISP, FabricPath, OTV
Routing
and OTV
OTV

Fabric Consolidation
Unified Fabric & I/O Network Service Localization
Device Virtualization Any service anywhere
Segmentation
OTV

VN-link
Storage Elasticity OTV notifications
SAN Extensions

Storage Solutions & Partners:

FCIP, I/O Acceleration
EMC, NetApp
VM-awareness
VXLAN, DFA, ACI, VN-link

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
 VXLAN Evolves as the Control Plane Evolves!
Back Then
Yet Another Encapsulation
Flood & Learn (Multicast-based)
Data-Plane only Yesterday
VXLAN for the Data Center – Intra-DC
Control-Plane
Active VTEP Discovery Now!
Multicast and Unicast
VXLAN for DCI – Inter-DC
DCI Ready
ARP/ND caching/suppress
Multi-Homing
Failure Domain Isolation
Loop Protection
BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Back Then
VXLAN for Interconnecting Networks

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
 Inter-X Connectivity
Multi-Pod Multi-Fabric Multi-Site

BGP EVPN EVPNFabric

Control-Plane EVPNFabric
Control-Plane EVPN Control-Plane BGP EVPN EVPN Control-Plane
EVPN Control-
Fabric #1 EVPN Control-
Fabric #2 #1 #2 Fabric #1 Fabric #2
Plane Domain 1 Plane Domain 2 Domain 1 Domain 2 Domain 1 Domain 2

Overlay Overlay Overlay Overlay Overlay Overlay

VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE
P P P P P P P P P P P P P P P P P P P P P P P P

Bar Bar Bar Bar

em em em em Bar Bar Bar Bar Bar Bar Bar Bar
eta eta eta eta em em em em em em em em
l l l l etal etal etal etal etal etal etal etal

DCI Data-Plane DCI

Single Data-Plane – End-to-End Data-Plane Domain 1 Data-Plane Domain 2 Data-Plane Domain 2
Data-Plane Domain 1 Data-Plane

• Multiple Fabrics – • Multiple Fabrics with

• Single Fabric with End-
Normalized through Integrated DCI
to-End Encapsulation
Ethernet • Integrated DCI –
• Build Hierarchy in the
• Multiple Fabrics Scaling within and
Underlay – Flatten it in
Interconnect using DCI between Fabrics
the Overlay
(Layer 2 and Layer 3) • The Happy Place

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
VXLAN EVPN Interconnect
Evolution
Multi-Pod
VXLAN EVPN – Single Pod / Single Fabric

External Network
VTEP VTEP

Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Pod 1

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
VXLAN EVPN – Multi-Pod

Underlay Extension
VTEP VTEP VTEP VTEP

Spine Spine Spine Spine Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Pod 1 Pod n

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Multi-Pod Characteristics – ”The Single”

Single Overlay Domain – End-to-End Encapsulation

Single Overlay Control-Plane Domain – End-to-End EVPN Updates
Single Underlay Domain End-to-End
Single Replication Domain for BUM
Single VNI Administrative Domain

Building Underlay Hierarchies – Non Hierarchical Overlay

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
 Multi-Pod – End-to-End Encapsulation

Underlay Extension
VTEP VTEP VTEP VTEP

Spine Spine Spine Spine Overlay Spine Spine Spine Spine

VTEP VTEP
10.1.1.1 VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
10.2.2.7
VTEP

Unicast

Pod 1 Pod n
Baremetal Baremetal

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Multi-Pod – BUM Replication

Underlay Extension
VTEP VTEP VTEP VTEP

Spine Spine Spine Spine Overlay Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

BUM

Pod 1 Pod 2
Baremetal

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Multi-Pod Challenges – ”The Single”

Single Overlay Domain – End-to-End Encapsulation

• Scaling the VXLAN EVPN Network
Single Overlay Control-Plane Domain – End-to-End EVPN Updates
• Overlay Control-Plane Update Propagation
Single Underlay Domain End-to-End
• Network must be extended in Underlay (VTEP to VTEP reachability)
Single Replication Domain for BUM
• One BUM flooding domain through out all connected Pods

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Multi-Fabric
VXLAN EVPN – Multi-Fabric

L2 DCI L2 DCI L2 DCI L2 DCI

L3 DCI Underlay No Extension L3 DCI
VTEP VTEP VTEP VTEP

Spine Spine Spine Spine Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Fabric 1 Fabric 2

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
 Multi-Fabric Characteristics – ”The Separate”

• Separate Overlay Domains –Independent L2 and L3 DCI (complexity)

• Separate Overlay Control-Plane Domains – Manual Configuration
• Separate Underlay Domains - Isolated
• Separate Replication Domains for BUM – Independent BUM transport/DCI
• Dedicated Border Leaf – no local End-Point Attachment

Underlay Isolation – Separate DC Interconnection

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
 Multi-Fabric – End-to-End Encapsulation
L2 DCI

L2 DCI L2 DCI L2 DCI L2 DCI

L3 DCI Underlay No Extension L3 DCI
VTEP VTEP VTEP VTEP
VLAN Hand-Off VRF-Lite Hand-Off

Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Unicast

Fabric 1 Fabric n
Baremetal Baremetal

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
VXLAN EVPN Multi-Site
Multi-Site
Walkthrough
VXLAN EVPN – Multi-Site

No Underlay Extension
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Spine Spine Spine Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Site 1 Site n

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Multi-Site Characteristics – ”The Multiple”

Multiple Overlay Domains – Interconnected & Controlled

Multiple Overlay Control-Plane Domains – Interconnected & Controlled
Multiple Underlay Domains - Isolated
Multiple Replication Domains for BUM – Interconnected & Controlled
Multiple VNI Administrative Domains – Phase 2

Underlay Isolation – Overlay Hierarchies

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
 Multi-Site – Hierarchical Overlay Domains
Overlay Multi-Site

VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Unicast

Site 1 Site n
Baremetal Baremetal

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Multi-Site – Underlay Isolation

Border (VIP) Border (VIP)

10.1.1.111 No Underlay Extension 10.2.2.222
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Border (PIP) Border (PIP) Border (PIP) Border (PIP)
10.1.1.101 10.1.1.102 10.2.2.101 10.2.2.102
Spine Spine Spine Spine Spine Spine Spine Spine

Site 1 Underlay Site n Underlay

Routing Table Routing Table
Border: Leaf: Border: Leaf:
VTEP 10.1.1.101 10.1.1.1 VTEP
10.2.2.101 10.2.2.1
10.1.1.1
VTEP VTEP VTEP VTEP VTEP
10.1.1.102 10.1.1.2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP
10.2.2.7
10.2.2.102 10.2.2.2
VTEP VTEP

10.1.1.111 10.1.1.3 10.2.2.222 10.2.2.3

10.1.1.4 10.2.2.4
10.1.1.5 10.2.2.5
Site 1 10.1.1.6 Site n 10.2.2.6
10.1.1.7 10.2.2.7

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Multi-Site – Inter Site Network
Inter-Site Network
Routing Table
Border Site1: Border Site2:
10.1.1.101 10.2.2.101
10.1.1.102 10.2.2.102
10.1.1.111 10.2.2.222

Border (VIP) Border (VIP)

10.1.1.111 Inter Site Network 10.2.2.222
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Border (PIP) Border (PIP) Border (PIP) Border (PIP)
10.1.1.101 10.1.1.102 10.2.2.101 10.2.2.102
Spine Spine Spine Spine Spine Spine Spine Spine

VTEP VTEP
10.1.1.1
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
10.2.2.7
VTEP

Site 1 Site n

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Border Gateways
Deployment
Considerations
Border Gateways Deployment Considerations
Anycast Border Gateways

BGW BGW BGW BGW

VTEP VTEP VTEP VTEP

Border Gateways used for two main functions:

1. Interconnecting each site to the Inter-Site network (for
East-West traffic flows)
2. Connecting each site to the external Layer 3 domain
(for North-South traffic flows) Site 1

May also be used to connect End-Points and/or

network service nodes (FWs, ADCs) VPC Border Gateways
BGW BGW

Two deployment models supported:

VTEP VTEP

1. Anycast Border Gateways

2. VPC Border Gateways

Site 1

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Anycast Border
Gateways
Anycast Border Gateway (1)
Anycast Border Gateway
Up to 4 Border Gateways
Border Gateway
BGW BGW BGW BGW • Deploying at Leaf – 7.0(3)I7(1)
VTEP VTEP VTEP VTEP
• Deploying at Spine – 7.0(3)I7(2)

Site 1

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Anycast Border Gateway (2)
Anycast Border Gateway
Common Virtual IP (VIP) across BGW
Border VIP
10.1.1.111 • VIP is used for Intra- and Inter-Site
BGW BGW BGW BGW
Communication
VTEP
PIP-BGW1
VTEP
PIP-BGW2
VTEP
PIP-BGW3
VTEP
PIP-BGW4 • VIP for communication between the Border
10.1.1.101 10.1.1.102 10.1.1.103 10.1.1.104
Gateways in different Sites

Border VIP
• VIP for communication between Border
10.1.1.111 Gateway and Leaf within a Site
Individual Primary IP (PIP) per BGW
• Used for Broadcast, Unknown Unicast and
Multicast (BUM) replication
• PIP for communication with Single-Homed
End-Points (routed only), intra- and inter-Site
Site 1

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Anycast Border Gateway (3)
Anycast Border Gateway
Type: 00 IP: 10.1.1.101
4 System MAC: 00:00:00:00:00:01
Ethernet Segment: 00:00:07 VNI: 30010 Per-VNI Designated Forwarder (DF) election
• Each BGW can serve as DF for a single or a
BGW BGW BGW BGW
set of Layer-2 VNI
VTEP VTEP VTEP VTEP
• DF election and assignment is automatic
DF
30010
DF
30011
DF
30012
DF
30099
Using BGP EVPN Route Type 4 for DF election
• Operator Managed Assignment (Type: 00)
BGP EVPN
• Six Octet Site Identifier (System MAC:
00:00:00:00:00:01)
RR RR
Spine Spine
• Multi-Site Discriminator (Ethernet-Segment:
00:00:07)
• Originators IP Address (PIP): 10.1.1.101
• Layer-2 VNI: 30010
Site 1

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
VPC Border
Gateways
VPC Border Gateway (1)
VPC Border Gateway
2 Border Gateways
Border Gateway
BGW BGW • Using a Leaf – 7.0(3)I7(2)
VTEP VTEP

Site 1

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
VPC Border Gateway (2)
VPC Border Gateway
Common Virtual IP (VIP) across BGW
Border VIP
10.1.1.111 • VIP is used for Intra- and Inter-Site
BGW BGW
Communication
VTEP
PIP-BGW1
VTEP
PIP-BGW2 • VIP for communication between the Border
10.1.1.101 10.1.1.102
Gateways in different Sites

Border VIP
• VIP for communication between Border
10.1.1.111 Gateway and Leaf within a Site
Individual Primary IP (PIP) per BGW
• Used for Broadcast, Unknown Unicast and
Multicast (BUM) replication
• PIP for communication with Single-Homed
End-Points, intra- and inter-Site
Site 1

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
VPC Border Gateway (3)
VPC Border Gateway
VPC-based Designated Forwarder Election
Per-Site Designated Forwarder (DF) election
BGW BGW • Using same approach as in VPC
VTEP VTEP
• Best Path to Rendezvous-Point or VPC
DF Primary Node

Site 1

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
VPC Border Gateway (4)
VPC Border Gateway
Single- or Dual-Homed End-Points
Border VIP
10.1.1.111 • Services Appliance (i.e. Firewall, ADC etc.)
BGW BGW • Physical or Virtual Servers
VTEP VTEP
Advertised and Reachable through Virtual IP
Address (VIP)
Border VIP • Intra-Site: Leaf nodes use VIP to reach End-
10.1.1.111
Points connected to Border Gateways
ADC ADC

• Inter-Site: Remote Border Gateways use VIP

ADC ADC
0000.3010.1102 0000.3010.1101 to reach End-Points connected to Border
192.168.10.102 192.168.10.101
• Traffic potentially traverses VPC Peer-Link
VTEP
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.

2 0000.3010.1101/48 30010, 65599:30010 192.168.10.101/32 50001, 65599:50001 10.1.1.111

Site 1
2 0000.3010.1102/48 30010, 65599:30010 192.168.10.102/32 50001, 65599:50001 10.1.1.111

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Multi-Site Control
Plane
Control Plane Deployment Considerations

Both MP-eBGP or MP-iBGP peering supported intra-Site between leaf nodes

Only MP-eBGP EVPN sessions supported inter-Sites mandates that each
site is part of a separate AS
Full mesh of MP-eBGP EVPN adjacencies only currently supported across sites
• Recommended to deploy a couple of Route-Servers in the Inter-Site network when 3 or
more sites are deployed
• Route-Servers only perform control plane functions (“eBGP Route-Reflectors”)
• Need to ensure that Route-Servers offer support for Route Type 4 EVPN routes,
required for DF election

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
 Multi-Site – Overlay Control-Plane (L3Core)

DC Core
(Layer-3 Unicast)

DCI

…. ….
VTEP VTEP VTEP VTEP

Fabric BGW BGW BGW BGW

VXLAN EVPN
iBGP-EVPN VXLAN EVPN
iBGP-EVPN
Spine Spine Spine Spine
RR RR
Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
 Multi-Site – Overlay Control-Plane (L3Core)
RS RS Route Server (eBGP ”Route Reflector”)

DC Core
(Layer-3 Unicast)

DCI

…. ….
VTEP VTEP VTEP VTEP

Fabric BGW BGW BGW BGW

VXLAN EVPN
iBGP-EVPN VXLAN EVPN
iBGP-EVPN
Spine Spine Spine Spine
RR RR
Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
 Multi-Site – Overlay Control-Plane (L3Core)
RS

DC Core
(Layer-3 Unicast)

DCI

…. ….
VTEP VTEP VTEP VTEP

Fabric BGW BGW BGW BGW

VXLAN EVPN
iBGP-EVPN VXLAN EVPN
iBGP-EVPN
Spine Spine Spine Spine
RR RR
Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

RS – Route Server (eBGP ”Route Reflector”) BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
 Multi-Site – Overlay Control-Plane (L3Core, no RS)

eBGP-EVPN
DC Core
(Layer-3 Unicast)

DCI

…. ….
VTEP VTEP VTEP VTEP

Fabric BGW BGW BGW BGW

VXLAN EVPN
iBGP-EVPN VXLAN EVPN
iBGP-EVPN
Spine Spine Spine Spine
RR RR
Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

RS – Route Server (eBGP ”Route Reflector”) BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
 Multi-Site – Overlay Control-Plane
RS

L3VNI: 50001 L3VNI: 50001

Route-Target: 65501:50001 DC Core Route-Target: 65502:50001
(Layer-3 Unicast)

VRF VRF
Tenant1 Tenant1
DCI
VIP1 VIP2

…. ….
VTEP VTEP VTEP VTEP
10.1.1.111 10.2.2.222
Fabric BGW BGW BGW BGW

VXLAN EVPN
iBGP-EVPN VXLAN EVPN
iBGP-EVPN
Spine Spine Spine Spine
RR RR
L2VNI: 30010 (VLAN 10)
Site1 L2VNI: 30020 (VLAN 20)
Site2 L2VNI: 30010 (VLAN 10)
L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1)
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102

RS – Route Server (eBGP ”Route Reflector”) BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
 Multi-Site – Overlay Control-Plane (Site1)
RS

L3VNI: 50001 L3VNI: 50001

Route-Target: 65501:50001 DC Core Route-Target: 65502:50001
(Layer-3 Unicast)
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.

2 0000.3010.1101/48 30010, 65501:30010 192.168.10.101/32 50001, 65501:50001 10.1.1.1

VRF VRF
Tenant1 2 0000.3020.2101/48 30020, 65501:30020 192.168.20.101/32 Tenant1
50001, 65501:50001 10.1.1.111
DCI 2 0000.3010.1102/48 30010, 65501:30010 192.168.10.102/32 50001, 65501:50001 10.1.1.111
VIP1 VIP2

…. ….
VTEP VTEP VTEP VTEP
10.1.1.111 10.2.2.222
Fabric BGW BGW BGW BGW

VXLAN EVPN VXLAN EVPN

Spine Spine Spine Spine
RR RR
L2VNI: 30010 (VLAN 10)
Site1 L2VNI: 30020 (VLAN 20)
Site2 L2VNI: 30010 (VLAN 10)
L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1)
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
 Multi-Site – Overlay Control-Plane (Site2)
RS

L3VNI: 50001 L3VNI: 50001

Route-Target: 65502:50001 DC Core Route-Target: 65502:50001
(Layer-3 Unicast)
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.

2 VRF 30010, 65502:30010

0000.3010.1101/48 192.168.10.101/32 50001, 65502:50001 10.2.2.222 VRF
2 Tenant1 30020, 65502:30020
0000.3020.2101/48 192.168.20.101/32 50001, 65502:50001 10.2.2.1 Tenant1
DCI
2 0000.3010.1102/48 30010, 65502:30010 192.168.10.102/32 50001, 65502:50001 10.2.2.3
VIP1 VIP2

…. ….
VTEP VTEP VTEP VTEP
10.1.1.111 10.2.2.222
Fabric BGW BGW BGW BGW

VXLAN EVPN VXLAN EVPN

Spine Spine Spine Spine
RR RR
L2VNI: 30010 (VLAN 10)
Site1 L2VNI: 30020 (VLAN 20)
Site2 L2VNI: 30010 (VLAN 10)
L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1)
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
 Multi-Site – Overlay Control-Plane (DCI)
RS

L3VNI: 50001 L3VNI: 50001

Route-Target: 65501:50001 DC Core Route-Target: 65502:50001
(Layer-3 Unicast)

VRF VRF
Tenant1 Tenant1
DCI
VIP1 VIP2

…. ….
VTEP VTEP VTEP VTEP
Type MAC / Length
10.1.1.111 L2VNI / RT IP / Length L3VNI / RT Next-Hop
10.2.2.222 Seq.
Fabric BGW 2 0000.3010.1101/48BGW30010, 65599:30010 192.168.10.101/32 BGW
50001, 65599:50001 10.1.1.111 BGW

2 0000.3020.2101/48 30020, 65599:30020 192.168.20.101/32 50001, 65599:50001 10.2.2.222

VXLAN
2
EVPN
0000.3010.1102/48 30010, 65599:30010 192.168.10.102/32
VXLAN
50001, 65599:50001
EVPN
10.2.2.222
Spine Spine Spine Spine
RR RR
L2VNI: 30010 (VLAN 10)
Site1 L2VNI: 30020 (VLAN 20)
Site2 L2VNI: 30010 (VLAN 10)
L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1)
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102

RS – Route Server (eBGP ”Route Reflector”) BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Multi-Site –
Selective
Advertisements
Multi-Site – Selective Advertisements

The Multi-Site architecture provides granular control on how Layer-2

and Layer-3 communication is extended across sites
Layer-2 and/or Layer-3 VNIs configured on the Border Gateways
(BGW) control the Control-Plane advertisement towards DCI
Enhances the overall scalability of the solution
• Scale up the total number of End-Points supported across sites

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
 Multi-Site – Selective Advertisements (DCI)
RS Only prefixes of VRF ”Tenant1” and L2VNI 30010 are
advertised from Site1 towards DCI. In this example this is
L3VNI: 50001
Route-Target: 65501:50001 DC Core Host1.
(Layer-3 Unicast)
All prefixes of VRF ”Tenant2” and L2VNI 30020 are not
advertised from Site2 towards DCI. These prefixes are not
VRF
seen within the DCI
Tenant1
DCI
VIP1 VIP2

…. ….
VTEP VTEP VTEP VTEP
Type MAC / Length
10.1.1.111 L2VNI / RT IP / Length L3VNI / RT Next-Hop
10.2.2.222 Seq.
Fabric BGW BGW BGW BGW
2 0000.3010.1101/48 30010, 65599:30010 192.168.10.101/32 50001, 65599:50001 10.1.1.111

VXLAN EVPN VXLAN EVPN

Spine Spine Spine Spine
RR RR
L2VNI: 30010 (VLAN 10)
Site1 L2VNI: 30020 (VLAN 20)
Site2
L3VNI: 50001 (Tenant1) L3VNI: 50002 (Tenant2)
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host2
0000.3010.1101 0000.3020.2101
192.168.10.101 192.168.20.101

RS – Route Server (eBGP ”Route Reflector”) BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Multi-Site Data Plane
 Multi-Site – Overlay Data Plane
Inter-site VXLAN
Data Plane

De-capsulation and DC Core

Re-encapsulation on
BGW
(Layer-3 Unicast) De-capsulation and
Re-encapsulation on
BGW

DCI
VIP1 VIP2

…. ….
VTEP VTEP VTEP VTEP
10.1.1.111 10.2.2.222
Fabric BGW BGW BGW BGW

VXLAN EVPN VXLAN EVPN

Spine Spine Spine Spine
Intra-site VXLAN
Data Plane
Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102
BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Multi-Site Packet
Walk (BUM)
 Packet Walk – Layer-2 (BUM) – Site1
Bridge

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
L10 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255

DF
Leaf10 replicates VTEP 30010 VTEP

traffic intra-Site BGW11 BGW21

2
VTEP VTEP
VXLAN EVPN VXLAN EVPN VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

DF
VTEP VTEP
1 Host 1 sends a L2
30010

BGW12 BGW22
BUM frame

Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
 Packet Walk – Layer-2 (DF & Split Horizon) – Site1
Bridge

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
L10 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255

DF
VTEP 30010 VTEP

BGW11 BGW21

VTEP BUM Forward VTEP

VXLAN EVPN VXLAN EVPN VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

DF
VTEP 30010 VTEP

BGW12 BGW22

Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 Drop due to Designated Forwarder (DF) rule 0000.3010.1102
192.168.10.101 192.168.10.102

Drop due to Split-Horizon rule

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
 Packet Walk – Layer-2 (BUM) – DCI
Bridge
SIP DIP VXLAN SMAC DMAC SIP DIP

BGW-VIP1 BGW21 30010 H1-MAC ALL-F H1-IP ALL-255

Payload
BGW-VIP1 BGW22 30010 H1-MAC ALL-F H1-IP ALL-255

BGW-VIP1 BGW12 30010 H1-MAC ALL-F H1-IP ALL-255

DF
VTEP 30010 VTEP
BGW11 replicates traffic inter-
BGW11 BGW21
Sites toward BGW nodes

VTEP BUM Forward 3 VTEP

VXLAN EVPN VXLAN EVPN VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

DF
VTEP 30010 VTEP

BGW12 BGW22

Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
 Packet Walk – Layer-2 (DF & Split Horizon) – DCI
Bridge
SIP DIP VXLAN SMAC DMAC SIP DIP

BGW-VIP1 BGW21 30010 H1-MAC ALL-F H1-IP ALL-255

Payload
BGW-VIP1 BGW22 30010 H1-MAC ALL-F H1-IP ALL-255

BGW-VIP1 BGW12 30010 H1-MAC ALL-F H1-IP ALL-255

DF
VTEP 30010 VTEP

BGW11 BGW21

VTEP BUM Forward VTEP

VXLAN EVPN VXLAN EVPN VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

DF
VTEP 30010 VTEP

BGW12 BGW22

BUM Forward
Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 Drop due to Designated Forwarder (DF) rule 0000.3010.1102
192.168.10.101 192.168.10.102

Drop due to Split-Horizon rule

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
 Packet Walk – Layer-2 (BUM) – Site2
Bridge

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
BGW-VIP2 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255

DF
VTEP 30010 VTEP

BGW11 BGW21 BGW22 replicates traffic

intra-Site
VTEP
VXLAN EVPN VXLAN EVPN VXLAN EVPN
4 VTEP

Leaf10 Site1 DCI Site2 Leaf20

DF
VTEP 30010 VTEP

BGW12 BGW22

BUM Forward
Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
 Packet Walk – Layer-2 (DF & Split Horizon) – Site2
Bridge

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
BGW-VIP2 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255

DF
VTEP 30010 VTEP

BGW11 BGW21

VTEP VTEP
VXLAN EVPN VXLAN EVPN VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

DF
VTEP 30010 VTEP

BGW12 BGW22

BUM Forward
Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 Drop due to Designated Forwarder (DF) rule 0000.3010.1102
192.168.10.101 192.168.10.102

Drop due to Split-Horizon rule

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
 Packet Walk – Layer-2 (BUM) – Site2
Bridge

DF
VTEP 30010 VTEP

BGW11 BGW21

VTEP VTEP
VXLAN EVPN VXLAN EVPN VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

DF
VTEP 30010 VTEP

BGW12 BGW22
5
Leaf20 sends traffic to
local Host 2
Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
Multi-Site Packet
Walk (Bridging)
 Packet Walk – Layer-2 (Host 1 to Host 2) – Site1
Bridge

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP

Leaf10 performs L2 lookup

and encapsulates toward VTEP VTEP

local BGW VIP1 address BGW11 BGW21

2
VTEP VTEP
VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

VTEP VTEP
1
Host 1 sends traffic BGW12 BGW22

destined to remote Host 2

Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
 Packet Walk – Layer-2 (Host 1 to Host 2) – DCI
Bridge

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
BGW-VIP1 BGW-VIP2 30010 H1-MAC H2-MAC H1-IP H2-IP

VTEP
BGW11 performs L2 lookup VTEP
and encapsulates toward
BGW11 BGW21
remote BGW VIP2 address
VTEP
3 VTEP
VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

VTEP VTEP

BGW12 BGW22

Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
 Packet Walk – Layer-2 (Host 1 to Host 2) – Site2
Bridge

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
BGW-VIP2 L20 30010 H1-MAC H2-MAC H1-IP H2-IP

VTEP VTEP BGW22 performs L2 lookup

BGW11 BGW21 and encapsulates toward
destination L20 node
VTEP
VXLAN EVPN VXLAN EVPN VXLAN EVPN
4 VTEP
VIP1 VIP2
Leaf10 Site1 DCI Site2 Leaf20

VTEP VTEP

BGW12 BGW22
5
Leaf20 bridges traffic to
local Host 2
Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
 Packet Walk – Layer-2 (Host 2 to Host 1) – Site2
Bridge

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
L20 BGW-VIP2 30010 H2-MAC H1-MAC H2-IP H1-IP

VTEP VTEP Leaf20 performs L2 lookup

BGW11 BGW21 and encapsulates toward
local BGW VIP2 address
VTEP
VXLAN EVPN VXLAN EVPN VXLAN EVPN
7 VTEP
VIP1 VIP2
Leaf10 Site1 DCI Site2 Leaf20

VTEP VTEP

BGW12 BGW22
6
Host 2 replies to remote
Host 1
Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
 Packet Walk – Layer-2 (Host 2 to Host 1) – DCI
Bridge

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
BGW-VIP2 BGW-VIP1 30010 H2-MAC H1-MAC H2-IP H1-IP

BGW21 performs L2 lookup

VTEP
and encapsulates toward VTEP

BGW11 remote BGW VIP1 address BGW21

VTEP
8 VTEP
VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

VTEP VTEP

BGW12 BGW22

Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
 Packet Walk – Layer-2 (Host 2 to Host 1) – Site1
Bridge

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
BGW-VIP1 L10 30010 H2-MAC H1-MAC H2-IP H1-IP

BGW12 performs L2 lookup

and encapsulates toward VTEP VTEP

destination L10 node BGW11 BGW21

9
VTEP VTEP
VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

VTEP VTEP
10 Leaf10 bridges traffic BGW12 BGW22
toward Host 1

Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
Multi-Site Packet
Walk (Routing)
 Packet Walk – Layer-3 (Host 1 to Host 3) – Site1
Route

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
L10 BGW-VIP1 50001 L10-MAC BGW-VMAC1 H1-IP H3-IP

Leaf10 performs a L3 lookup

and encapsulates toward VTEP VTEP

local BGW VIP1 address BGW11 BGW21

2
VTEP VTEP
VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN
Site1 VMAC1 DCI VMAC2 Site2
Leaf10 Leaf20

VTEP VTEP
1 Host 1 sends a data
packet to the remote BGW12 BGW22

Host 3

Baremetal Baremetal

Host 1 Host 3
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.20.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
 Packet Walk – Layer-3 (Host 1 to Host 3) – DCI
Route

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
BGW-VIP1 BGW-VIP2 50001 BGW-VMAC1 BGW-VMAC2 H1-IP H3-IP

BGW11 performs a L3 lookup

VTEP
and encapsulates toward VTEP

BGW11 remote BGW VIP2 address BGW21

3
VTEP VTEP
VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN
Site1 VMAC1 DCI VMAC2 Site2
Leaf10 Leaf20

VTEP VTEP

BGW12 BGW22

Baremetal Baremetal

Host 1 Host 3
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.20.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
 Packet Walk – Layer-3 (Host 1 to Host 3) – Site2
Route

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
BGW-VIP2 L20 50001 BGW-VMAC1 L20-MAC H1-IP H3-IP

VTEP VTEP BGW21 performs a L3

BGW11 BGW21 lookup and encapsulates
toward destination L20 node
VTEP
VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN
4 VTEP

Site1 VMAC1 DCI VMAC2 Site2

Leaf10 Leaf20

VTEP VTEP

BGW12 BGW22
5
Leaf20 routes traffic to
local Host 3
Baremetal Baremetal

Host 1 Host 3
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.20.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Multi-Site and
Failure Detection
on BGW
 Steady State Traffic – Site1
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP

VTEP VTEP

BGW11 BGW21

VTEP VTEP
VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

VTEP VTEP

BGW12 BGW22

Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
 DCI Link Failure BGW12 – Site1
On DCI Link Failure (i.e. BGW12)
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload • Virtual IP (VIP) on BGW is disabled
L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP
• BGW will stop participating in DF election
• BGW acts like a Leaf (Layer-3 only)
• Traffic towards others Sites is served by remaining
Intra-site VXLAN traffic re-
routing
BGWs (i.e. BGW11)
VTEP VTEP

BGW11 BGW21

VTEP VTEP
VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

VTEP VTEP

BGW12 BGW22
BGW12 gets isolated from
the DCI Core Network
Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
 Fabric Link Failure BGW12 – Site1
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload
L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP

VTEP VTEP

BGW11 BGW21

VTEP VTEP
VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

VTEP VTEP

BGW12 BGW22

Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
 Fabric Link Failure BGW12 – Site1
On Fabric Link Failure (i.e. BGW12)
SIP DIP VXLAN SMAC DMAC SIP DIP
Payload • Virtual IP (VIP) on BGW is disabled
L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP
• Primary IP (PIP) on BGW is disabled
• BGW will stop participating in the Overlay

Intra-site VXLAN traffic re-

routing VTEP VTEP

BGW11 BGW21

VTEP VTEP
VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

VTEP VTEP

BGW12 BGW22

Baremetal
BGW12 gets isolated from Baremetal

the Spine nodes

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
Multi-Site Setup
Walkthrough
 Site 1 Setup – Enable Border Gateway
Multi-Site Commands are marked in red
Various options do exist but the recommended design
choices are:
• Fabric Internal
IGP Underlay, iBGP Overlay
• DCI (primary choice)

….
VTEP VTEP
eBGP Underlay, eBGP Overlay
Fabric BGW1 BGW2
Route Server for DCI Overlay peerings
DC Core for reachability across n Sites
Spine Spine
• DCI (alternative option)
Any Routing Protocol Underlay, eBGP Overlay
Full-Mesh for DCI Overlay peerings
VTEP VTEP VTEP VTEP
Back-to-Back Site Reachability (physical, full-
mesh)

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
 Site 1 Setup – Enable Border Gateway
BGW1 BGW2
feature nv overlay
nv overlay evpn

feature bgp
feature interface-vlan
feature vn-segment-vlan-based

evpn multisite border-gateway

….
VTEP VTEP

Fabric BGW1 BGW2

Spine Spine

VTEP VTEP VTEP VTEP

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
 Site 1 Setup – BGW 1 Loopback & VTEP
BGW1
interface loopback0
description RID
ip address 10.10.10.101/32 tag 12345
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode

interface loopback1
description PIP VTEP
ip address 10.1.1.101/32 tag 12345
….
VTEP VTEP

BGW1
ip router ospf UNDERLAY area 0.0.0.0
Fabric
ip pim sparse-mode

Spine Spine interface loopback100

description VIP Multi-Site 1
ip address 10.1.1.111/32 tag 12345
ip router ospf UNDERLAY area 0.0.0.0
VTEP VTEP VTEP VTEP
ip pim sparse-mode

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
 Site 1 Setup – BGW 2 Loopback & VTEP
BGW2
interface loopback0
description RID
ip address 10.10.10.102/32 tag 12345
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode

interface loopback1
description PIP VTEP
ip address 10.1.1.102/32 tag 12345
….
VTEP VTEP

BGW2
ip router ospf UNDERLAY area 0.0.0.0
Fabric
ip pim sparse-mode

Spine Spine interface loopback100

description VIP Multi-Site 1
ip address 10.1.1.111/32 tag 12345
ip router ospf UNDERLAY area 0.0.0.0
VTEP VTEP VTEP VTEP
ip pim sparse-mode

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
 Site 1 Setup – Fabric Link Tracking BGW 1
BGW1
interface Ethernet1/53
description TO-SPINE1
ip address 10.0.1.1/30
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
evpn multisite fabric-tracking

interface Ethernet1/54
description TO-SPINE2

….
VTEP VTEP ip address 10.0.2.1/30
Fabric BGW1 ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
evpn multisite fabric-tracking
Spine Spine

Allows to bring down the PIP/VIP

VTEP VTEP VTEP VTEP loopback interfaces when the
BGW is isolated from the spines

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
 Site 1 Setup – Fabric Link Tracking BGW 2
BGW2
interface Ethernet1/53
description TO-SPINE1
ip address 10.0.1.5/30
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
evpn multisite fabric-tracking

interface Ethernet1/54
description TO-SPINE2

….
VTEP VTEP ip address 10.0.2.5/30
Fabric BGW2 ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
evpn multisite fabric-tracking
Spine Spine

VTEP VTEP VTEP VTEP

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
 Site 1 Setup – Multi-Site Underlay Interface
BGW1
interface Ethernet1/1
description TO-DC-CORE1
DC Coreip address 10.111.111.1/30 tag 12345
evpn multisite dci-tracking
(Layer-3 Unicast)
interface Ethernet1/2
description TO-DC-CORE2
ip address 10.111.222.1/30 tag 12345
DCI evpn multisite dci-tracking

….
VTEP VTEP

Fabric BGW1 BGW2

BGW2
interface Ethernet1/1
description TO-DC-CORE1
Spine Spine ip address 10.222.111.1/30 tag 12345
evpn multisite dci-tracking

interface Ethernet1/2
description TO-DC-CORE2
VTEP VTEP VTEP VTEP
ip address 10.222.222.1/30 tag 12345
evpn multisite dci-tracking

Allows to bring down the PIP/VIP loopback interfaces

when the BGW is isolated from the DC core BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
 Site 1 BGW 1 Setup – Multi-Site Overlay Peering
BGW1
RS router bgp 65501
router-id 10.10.10.101
DC Coreaddress-family ipv4 unicast
redistribute direct route-map REDIST-LOCAL
(Layer-3 Unicast)
neighbor 10.111.111.2
remote-as 65599
update-source ethernet1/1
address-family ipv4 unicast
DCI neighbor 10.111.222.2

….
VTEP VTEP remote-as 65599
Fabric BGW1 update-source ethernet1/2
address-family ipv4 unicast
neighbor 10.99.99.201
Spine Spine
remote-as 65599
update-source loopback0
ebgp-multihop 5
peer-type fabric-external
address-family l2vpn evpn
VTEP VTEP VTEP VTEP rewrite-evpn-rt-asn
send-community
send-community both

RS – Route Server (eBGP ”Route Reflector”) BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
 Site 1 BGW 2 Setup – Multi-Site Overlay Peering
BGW1
RS router bgp 65501
router-id 10.10.10.102
DC Coreaddress-family ipv4 unicast
redistribute direct route-map REDIST-LOCAL
(Layer-3 Unicast)
neighbor 10.222.111.2
remote-as 65599
update-source ethernet1/1
address-family ipv4 unicast
DCI neighbor 10.222.222.2

….
VTEP VTEP remote-as 65599
Fabric BGW2 update-source ethernet1/2
address-family ipv4 unicast
neighbor 10.99.99.201
Spine Spine
remote-as 65599
update-source loopback0
ebgp-multihop 5
peer-type fabric-external
address-family l2vpn evpn
VTEP VTEP VTEP VTEP rewrite-evpn-rt-asn
send-community
send-community both

RS – Route Server (eBGP ”Route Reflector”) BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
 Site 1 Setup – Multi-Site Overlay Peering
RS

DC Core
(Layer-3 Unicast)

DCI peer-type fabric-external

• Enables Next-Hop Rewrite for Multi-Site
….
VTEP VTEP

BGW1 BGW2
Fabric • Defines Site External BGP neighbors for EVPN
exchange
Spine Spine
rewrite-evpn-rt-asn
• Rewrites Route-Target Auto information to simplify
MAC-VRF and IP-VRF configuration
• Normalizes outgoing Route-Targets AS number to
VTEP VTEP VTEP VTEP
match remote AS number
• Uses BGP configured Neighbors Remote AS

RS – Route Server (eBGP ”Route Reflector”) BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
 Site 1 Setup – Multi-Site Overlay Peering

BGP Update:
MAC: DC Core
0000.3010.1101 (L2VNI 30001)
IP: 192.168.20.101 (L3VNI(Layer-3
50001) Unicast)
NH: 10.1.1.111
RMAC: BGW-VMAC1
Rewrite Next-Hop IP and Next-
Hop MAC (RMAC) based on
Neighbor Site BGW
DCI
peer-type fabric-external peer-type fabric-external
VIP1 VIP2

…. ….
VTEP VTEP VTEP VTEP
10.1.1.111 10.2.2.222
Fabric BGW BGW BGW BGW
BGP Update:
BGP Update: MAC: 0000.3010.1101 (L2VNI 30001)
MAC: 0000.3010.1101 (L2VNI 30001) IP: 192.168.20.101 (L3VNI 50001)
IP: VXLAN EVPN
192.168.20.101 (L3VNI 50001) Rewrite Next-Hop IP and Next- VXLAN EVPN
NH: 10.2.2.222
NH: 10.1.1.1
Spine Spine Hop MAC (RMAC) based on Spine RMAC: BGW-VMAC2 Spine
RMAC: Leaf1 Neighbor Site BGW

Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1
0000.3010.1101
192.168.10.101
BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
 Site 1 Setup – Multi-Site Overlay Peering

BGP Update: DC Core

Rewrite Route-Target based on
Remote AS: 65502 (Layer-3
BGP Unicast)
Neighbors Remote ASN
VNI: 50001
Route-Target: 65502:50001

DCI
rewrite-evpn-rt-asn rewrite-evpn-rt-asn
VIP1 VIP2

…. ….
VTEP VTEP VTEP VTEP
10.1.1.111 10.2.2.222
Fabric BGW BGW BGW BGW
BGP Update:
Remote AS: 65502
BGP Update:VXLAN EVPN VXLAN
VNI: EVPN 50001
Remote AS : 65501
Spine Spine Spine
Route-Target:Spine
65502:50001
VNI: 50001
Site1
Route-Target: 65501:50001 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host2
0000.3010.1101 0000.3020.2101
192.168.10.101 192.168.20.101
BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
 Site 1 Setup – Anycast BGW VTEP Configuration
BGW1 BGW2
interface nve1
no shutdown
host-reachability protocol bgp
DC Coremultisite ethernet-segment 7
(Layer-3 Unicast)
system-mac 0000.0000.0001
source-interface loopback1
multisite border-gateway interface loopback100
member vni 30010
DCI multisite ingress-replication

….
VTEP VTEP mcast-group 239.1.1.1
Fabric BGW1 BGW2 member vni 30011-30020
mcast-group 239.1.1.2
member vni 50001 associate-vrf
Spine Spine

VTEP VTEP VTEP VTEP

RS – Route Server (eBGP ”Route Reflector”) BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
 Site 1 Setup – Anycast BGW VTEP Configuration

DC Core
(Layer-3 Unicast)

DCI multisite ethernet-segment

• Defines the discriminator for Sites in a common Domain
….
VTEP VTEP

Fabric BGW1 BGW2

system-mac
• Defines the Multi-Site Site-Id (6 octets hex)
Spine Spine multisite border-gateway interface loopback#
• Defines the Loopback Interface used for the Border
Gateway Virtual IP Address (VIP)
VTEP VTEP VTEP VTEP multisite ingress-replication
• Per-VNI knob for extending Layer-2 VNI
• Defines the Multi-Site BUM Replication method
BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
 Site 1 Setup – Multi-Site Overlay Traffic Policy
BGW1 BGW2

evpn storm-control broadcast level 10

evpn storm-control unicast level 10
DC Core
evpn storm-control multicast level 10
(Layer-3 Unicast)

DCI • BUM Traffic Policing

• Limits Broadcast, Unknown Unicast and Layer-2
….
VTEP VTEP

Fabric BGW1 BGW2 Multicast Traffic across Multi-Site

• Level 0 = No B/U/M Forwarding
Spine Spine • Level 100 = All B/U/M Forwarding Forwarding
• Enforced on Encapsulation towards remote Sites

VTEP VTEP VTEP VTEP

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 97
Site 1 Setup – Multi-Site Overlay Traffic Policy
Overlay Multi-Site

VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

BUM

Site 1 Site n
Baremetal

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
Site 1 Setup – Multi-Site Overlay Traffic Policy
Overlay Multi-Site

Storm Control
VTEP VTEP Broadcast 0-100% VTEP VTEP

BGW BGW
Unknown Unicast 0-100% BGW BGW

Multicast 0-100%
Spine Overlay Site 1
Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

BUM

Site 1 Site n
Baremetal

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 99
Site 1 Setup – Multi-Site BUM Replication Modes
Overlay Multi-Site

Ingress Replication

VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

Multicast Multicast

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Site 1 Site n

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 100
Site 1 Setup – Multi-Site BUM Replication Modes
Overlay Multi-Site

Ingress Replication

VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

Ingress Replication Ingress Replication

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Site 1 Site n

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 101
Site 1 Setup – Multi-Site BUM Replication Modes
Overlay Multi-Site

Ingress Replication

VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

Ingress Replication Multicast

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Site 1 Site n

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 102
Connectivity to the
External Layer 3
Domain
Connectivity to the External Layer 3 Domain

The BGW nodes can be used to provide Layer-3 external

connectivity to each site
Different connectivity models are supported
• VRF-Lite peering with an external pair of WAN Edge routers
• MP-BGP EVPN peering with the external WAN Edge routers (GOLF)
• Dedicated or shared pair of WAN Edge routers across sites

External Layer-3 network may be different from the DCI network

used for inter-site communication

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 104
 Multi-Site – Border Gateway and VRF-Lite
Separate routing VRF-A VRF-B VRF-C
peering for each VRF
(IGP or eBGP)
Dedicated interface
(logical or physical) for
each VRF

DCI
VIP1 VIP2

…. ….
VTEP VTEP VTEP VTEP
10.1.1.111 10.2.2.222
Fabric BGW BGW BGW BGW

VXLAN EVPN VXLAN EVPN

Spine Spine Spine Spine

Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102
BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 105
 Multi-Site – Border Gateway and GOLF
Single MP-BGP EVPN VXLAN Data Plane
instance to exchange routes VRF-A VRF-B VRF-C between BGW and WAN
for all VRFs Edge Router

DCI
VIP1 VIP2

…. ….
VTEP VTEP VTEP VTEP
10.1.1.111 10.2.2.222
Fabric BGW BGW BGW BGW

VXLAN EVPN VXLAN EVPN

Spine Spine Spine Spine

Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102
BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 106
 Multi-Site – Shared Internet/WAN Gateways
MPLS
Internet/WAN L3VPN
BorderPE BorderPE

DC Core
(Layer-3 Unicast)
DCI
VIP1 VIP2

…. ….
VTEP VTEP VTEP VTEP
10.1.1.111 10.2.2.222
Fabric BGW BGW BGW BGW
Inter-Site VXLAN
Communication between
Border Gateways
VXLAN EVPN VXLAN EVPN
Spine Spine Spine Spine

Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102
BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 107
 Multi-Site – Per Site Internet/WAN Gateway
MPLS
Internet/WAN
BorderPE BorderPE L3VPN BorderPE BorderPE

DC Core
(Layer-3 Unicast)
DCI
VIP1 VIP2

…. ….
VTEP VTEP VTEP VTEP
10.1.1.111 10.2.2.222
Fabric BGW BGW BGW BGW
Inter-Site VXLAN
Communication between
Border Gateways
VXLAN EVPN VXLAN EVPN
Spine Spine Spine Spine

Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102
BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 108
 Multi-Site – Consolidated WAN and DCI Network Perform simple routing for
inter-site flows, VXLAN (or
VRF-Lite) to MPLS VPN
hand-off for north-south
MPLS communication
L3VPN
Internet/WAN BorderPE BorderPE BorderPE BorderPE

Inter-Site VXLAN
Communication between
DCI Border Gateways
VIP1 VIP2

…. ….
VTEP VTEP VTEP VTEP
10.1.1.111 10.2.2.222
Fabric BGW BGW BGW BGW

VXLAN EVPN VXLAN EVPN

Spine Spine Spine Spine

Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102
BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 109
Ingress and Egress
Traffic Optimization
 Ingress and Egress Traffic Optimization
The Issue of Extending Layer 2 Domains

The stretching of Layer-2 domains

WAN across separate sites may lead to
the creation of asymmetric traffic
Active FW DC Core Active FW
paths
(Layer-3 Unicast)
VTEP

BGW
VTEP

BGW
VTEP

BGW
VTEP

BGW
Deploying independent stateful
services (like FWs) across sites
VXLAN EVPN VXLAN EVPN
Spine Spine Spine Spine would result in traffic drops
Site1 Site2 In this case it is required to ensure
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP the symmetry of ingress and egress
communication paths

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 111
 Ingress and Egress Traffic Optimization
Maintaining Traffic Symmetry over Optimal Paths

Guarantee routing symmetry with the

outside of the Data Center
WAN
• Egress Always prefer the local BGW
• Ingress Steer traffic to the specific destination
DC Core End-Point’s location
(Layer-3 Unicast)
VTEP VTEP VTEP VTEP Maintain optimal routing over the
BGW BGW BGW BGW
dedicated DCI network (if existing) for
VXLAN EVPN VXLAN EVPN
Server-to-Server traffic
Spine Spine Spine Spine • The DC fabric must discriminate between DC
and WAN destinations
Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
If required provide a fallback path via
DCI for WAN isolation situations

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 112
 Multi-Site – Egress Path Optimization
172.16.1.10

MPLS
BorderPE BorderPE L3VPN BorderPE BorderPE

172.16.1.0/24 Border-PEs 1-2 eBGP-EVPN 172.16.1.0/24 Border-PEs 3-4

DC Core
(Layer-3 Unicast)
VTEP VIP1 VTEP VTEP VIP2 VTEP
10.1.1.111 10.2.2.222
BGW BGW BGW BGW
Less preferred
VXLAN EVPN advertisement of
172.16.1.0 because
VXLAN EVPN
Spine Spine Spine Spine
of longer AS-Path

Site1
172.16.1.0/24 VIP1 Site2
172.16.1.0/24 VIP2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host3
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 113
Multi-Site – Egress Path Optimization
172.16.1.10

MPLS
BorderPE BorderPE L3VPN BorderPE BorderPE

Optimized Egress Optimized Egress

Traffic Path eBGP-EVPN Traffic Path
DC Core
(Layer-3 Unicast)
VTEP VIP1 VTEP VTEP VIP2 VTEP
10.1.1.111 10.2.2.222
BGW BGW BGW BGW

VXLAN EVPN VXLAN EVPN

Spine Spine Spine Spine

Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host3
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 114
Multi-Site – Egress Path Optimization
WAN Isolation Scenario
172.16.1.10

WAN Isolation
Scenario MPLS
BorderPE BorderPE L3VPN BorderPE BorderPE

eBGP-EVPN 172.16.1.0/24 Border-PEs 3-4

DC Core
(Layer-3 Unicast)
VIP1 VIP2
VTEP
10.1.1.111
VTEP 172.16.1.0/24 VIP2 VTEP
10.2.2.222
VTEP

BGW BGW BGW BGW

VXLAN EVPN VXLAN EVPN

Spine Spine Spine Spine

Site1
172.16.1.0/24 VIP1 Site2
172.16.1.0/24 VIP2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host3
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 115
 Multi-Site – Ingress Path Optimization
192.168.10.0/24 Border-PE 1-4
192.168.10.101/32 Border-PE 1-2
Host routes 192.168.10.102/32 Border-PE 3-4
Deploying LISP on the
advertisement in Border-PEs is a viable
the WAN alternative to host routes
MPLS advertisement
BorderPE BorderPE L3VPN BorderPE BorderPE

eBGP-EVPN
192.168.10.0/24 BGW 1-2 DC Core 192.168.10.0/24 BGW 3-4
192.168.10.101/32 BGW-1-2 192.168.10.102/32 BGW 3-4
(Layer-3 Unicast)
VIP1 VIP2
VTEP
10.1.1.111
VTEP VTEP
10.2.2.222
VTEP Filter out host routes
BGW BGW
Host routes BGW BGW received from remote
advertised across sites. Only announce
VXLAN EVPN sites but NOT re- VXLAN EVPN local host route
Spine Spine advertised toward the Spine Spine information
local Border-PEs

Site1
192.168.10.101/32 Leaf1 Site2
192.168.10.102/32 -> Leaf3
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host3
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 116
Multi-Site – Ingress Path Optimization
192.168.10.0/24 Border-PE 1-4
192.168.10.101/32 Border-PE 1-2
192.168.10.102/32 Border-PE 3-4

MPLS
BorderPE BorderPE L3VPN BorderPE BorderPE

Optimized Ingress Optimized Ingress

Traffic Path eBGP-EVPN Traffic Path
DC Core
(Layer-3 Unicast)
VTEP VIP1 VTEP VTEP VIP2 VTEP
10.1.1.111 10.2.2.222
BGW BGW BGW BGW

VXLAN EVPN VXLAN EVPN

Spine Spine Spine Spine

Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host3
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 117
Multi-Site – Ingress Path Optimization
WAN Isolation Scenario
192.168.10.0/24 Border-PE 3-4
192.168.10.101/32 Border-PE 1-2
192.168.10.102/32 Border-PE 3-4

WAN Isolation
Scenario MPLS
BorderPE BorderPE L3VPN BorderPE BorderPE

eBGP-EVPN
DC Core
(Layer-3 Unicast)
VTEP VIP1 VTEP VTEP VIP2 VTEP
10.1.1.111 10.2.2.222
BGW BGW BGW BGW

VXLAN EVPN VXLAN EVPN

Spine Spine Spine Spine

Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host3
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 118
Network Services
Integration
Network Services Integration

Couple of different options where to connect network services:

1. Service Leaf nodes: recommended to connect devices used for east-
west communication
2. Border Gateway Nodes: used to connect network services for north-
south traffic flows
Depending on the specifics of the Multi-Site deployment, the
following deployment models would be possible:
• Active/Standby Service Nodes pair connected to different sites
• Active/Active cluster of Service Nodes deployed across sites
• Independent Active/Standby Service nodes pairs deployed in separate
sites

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 120
 Network Services Integration
Active/Standby Pair Deployed across Sites

North-South traffic
Requirement to extend Layer 2 communication
flows between Active/Standby nodes for keep-alives
WAN North-South traffic and state information exchange
flows
Perimeter service nodes connected to VPC Border
Gateways
Active FW DC Core Standby FW
(Layer-3 Unicast) Ingress and egress traffic always traversing the
VTEP VTEP VTEP VTEP
Active node in Site 1
BGW BGW BGW BGW
No issues related to the creation of asymmetric
VXLAN EVPN East-West traffic VXLAN EVPN traffic paths
Spine Spine
flows Spine Spine
East-West flows must be hair-pinned to the
Site1 Site2 active FW connected to the Service leaf nodes
in Site 1
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

• Need to properly dimension bandwidth in the DC

Core to accommodate for this extra traffic
Baremetal Baremetal Baremetal

Active FW Standby FW

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 121
 Network Services Integration
Active/Active Cluster of Service Nodes Deployed across Sites

North-South traffic
flows

WAN
Requirement to extend Layer 2 communication
Logical Intra- between Active/Active nodes for intra-cluster
Cluster Link (ICL)
communication and traffic redirection
DC Core Active/Active Perimeter service nodes connected to VPC Border
FW Cluster
(Layer-3 Unicast) Gateways
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW Asymmetric traffic issues taken care by native
intra-cluster traffic redirection
VXLAN EVPN VXLAN EVPN
Spine Spine Spine Spine Option to deploy ingress/egress optimization
technique to avoid inter-site traffic hair-pinning
Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
Service node cluster integration not supported
at FCS and planned for a future SW release
Active/Active
FW Cluster
Baremetal

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 122
 Network Services Integration
Independent Active/Standby Pair Deployed in Separate Sites

North-South traffic
North-South traffic Mandates the deployment of Ingress/Egress
flows
flows
traffic optimization to avoid creation of
WAN asymmetric traffic path for north-south
communication
Active/Standby DC Core Active/Standby Active/Standby nodes can use direct links to
FW (Layer-3 Unicast) FW sync state
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW Perimeter service nodes can connected to Anycast
Border Gateways
VXLAN EVPN VXLAN EVPN
Spine Spine Spine Spine Active/Standby pair (or cluster) still required for
service nodes used for east-west traffic flows
Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Baremetal Baremetal

Active FW Standby FW

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 123
Legacy Site
Integration
Multi-Site and Legacy Site Integration
Pair of VPC
Border Gateways
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Greenfield Site Legacy Site

Extend Layer-2 and Layer-3 connectivity between sites

• Coexistence and/or application migration use cases
Proposed approach is to deploy a pair of ‘remote’ VPC Border Gateways
in the legacy site
• Offers native Multi-Site functionalities (BUM containment, etc) to the legacy site
BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 125
 Multi-Site and Legacy Site Integration
Layer-2 Connectivity with the ‘Remote’ BGW
Legacy Aggregation Layer Legacy Aggregation Layer
devices support MLAG devices do not support MLAG
Single logical link to extend
Recommended to
VLANs mapped to L2VNIs move the STP root to VLANs mapped to L2VNIs
VLANs toward the Greenfield on the Border Gateways on the Border Gateways
VXLAN EVPN site
the BGW devices
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Single port-channel
from each aggregation
layer device

Legacy Site Legacy Site

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 126
Multi-Site and Legacy Site Integration
Layer-2 Control Plane Exchange across Sites
eBGP-EVPN
MAC NH MAC NH

0000.3010.1101 Leaf1 0000.3010.1101 VIP1

0000.3010.1102 VIP2 VIP1 VIP2 0000.3010.1102 Po1

10.1.1.111 10.2.2.222
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

All End-Points in the legacy
site are learned as directly
Po1
connected to the BGW
Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Baremetal Baremetal

Greenfield Site Legacy Site

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102
BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 127
 Multi-Site and Legacy Site Integration
Default Gateway Deployment – Option 1
Greenfield VXLAN
EVPN Fabric only offers VTEP VTEP VTEP VTEP Default Gateway
L2 services for the BGW BGW BGW BGW deployed on the
stretched IP subnets legacy aggregation
devices
Spine Spine Spine Spine
L3
L2

VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Greenfield Site Legacy Site

Integration between Anycast Gateway and legacy default gateway

(HSRP, VRRP, etc.) not initially supported with VXLAN Multi-Site
First option is to keep on the legacy network the active default gateway
for the stretched IP subnets

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 128
 Multi-Site and Legacy Site Integration
Default Gateway Deployment – Option 2 Default Gateway
migrated to the Border
Gateways (VXLAN
Greenfield VXLAN EVPN Anycast
EVPN Fabric offers L2 VTEP VTEP L3 VTEP VTEP Gateway)
and L3 services for the BGW BGW BGW BGW
stretched IP subnets L2

Spine Spine Spine Spine

Legacy infrastructure
offers only L2
services
L3 VTEP VTEP VTEP VTEP VTEP VTEP VTEP

L2

Distributed Anycast Greenfield Site Legacy Site

Gateway function

Recommended approach is to migrate the default gateway from the

legacy aggregation devices to the Border Gateways (VXLAN EVPN
Anycast Gateway)
Optimize routing between End-Points deployed across sites

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 129
Multi-Site and Legacy Site Integration
Layer-3 Control Plane Exchange across Sites
eBGP-EVPN
IP NH IP L3VNI

192.168.10.101 Leaf1 192.168.10.101 VIP1

192.168.20.101 VIP1 VIP1 VIP2 192.168.20.101 Po1

10.1.1.111 10.2.2.222
VTEP VTEP L3 VTEP VTEP

BGW BGW BGW BGW

L2 All End-Points in the legacy
site are learned as directly
Po1
connected to the BGW
Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Baremetal Baremetal

Greenfield Site Legacy Site

Host 1 Host 3
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.20.101
BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 130
Migration to Multi-
Site
Migration to Multi-Site
Use Cases

1. Site addition: need to connect a Greenfield VXLAN EVPN

Fabric to an existing VXLAN EVPN Fabric built with 1st
generation Nexus 9000
2. Migrating a VXLAN Multi-Pod Fabric to Multi-Site
3. Migrating a VXLAN Multi-Fabric design to Multi-Site

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 132
 Migration to Multi-Site
Site Addition
eBGP-EVPN

VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Spine Spine Spine

Spine Spine Spine Spine Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Existing VXLAN Existing VXLAN Greenfield

EVPN Fabric EVPN Fabric Fabric

Step 1: add a pair of Border Gateways to the existing VXLAN EVPN Fabric, running the
proper SW release supporting Multi-Site
Note: no requirement to change the HW/SW version on existing leaf nodes
Step 2: connect the BGW to the inter-site network and establish control plane peering with
the BGW in the Greenfield Fabric
Step 3: configure on the BGW the L2VNIs and L3VNIs to be extended

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 133
 Migration to Multi-Site
Multi-Fabric to Multi-Site

VTEP VTEP VTEP VTEP

Step 1: add a pair of Border Gateways to
each Pod (if needed) and connect them
to the spines and to the inter-site
Spine Spine Spine Spine Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
network
Fabric 1 Fabric 2 Step 2: upgrade the SW on both Fabrics
BGW to be able to support Multi-Site
Step 3: establish control plane
No Underlay Extension adjacencies across sites
Step 4: disconnect the previously used
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Spine Spine Spine Spine Spine Spine Spine

DCI and extend Layer-2 and Layer-3
across Multi-Site
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Site 1 Site 2
BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 134
Conclusion
Multi-Site Advantages – ”The Multiple”

Multiple Overlay Domains – Interconnected & Controlled

• Scaling and Segregating VXLAN EVPN Networks
Multiple Overlay Control-Plane Domains – Interconnected & Controlled
• Limited Overlay Control-Plane Update Propagation
Multiple Underlay Domains - Isolated
• Isolated Underlay Domains – No need for Extension
Multiple Replication Domains for BUM – Interconnected & Controlled
• Individual BUM flooding domain with Traffic control

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 136
VXLAN EVPN – Multi-Site

• New IETF Draft for Multi-Site Design

• Multi-site EVPN based VXLAN using Border Gateways
• https://tools.ietf.org/html/draft-sharma-multi-site-evpn

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 137
 Complete Your Online
Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 gift card.
• Complete your session surveys
through the Cisco Live mobile
app or on www.CiscoLive.com/us.

Don’t forget: Cisco Live sessions will be

available for viewing on demand after the
event at www.CiscoLive.com/Online.

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions

BRKDCN-2035 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 139
Thank you