INF-NET2162
VXLAN Deep Dive
Arun Goel, VMware, Inc.
Kenneth Duda, Arista Networks
#vmworld (hashtag here)
Disclaimer
This session may contain product features that are
currently under development.
This session/overview of the new technology represents
no commitment from VMware to deliver these features in
any generally available product.
Features are subject to change, and must not be included in
contracts, purchase orders, or sales agreements of any kind.
Technical feasibility and market demand will affect final delivery.
Pricing and packaging for any new technologies or features
discussed or presented have not been determined.
2
Why Overlays (VXLAN)?
Web Tier needs to expand?
Web Capacity ?
IP Address ?
Network ?
App
DB
Compute Cluster A Compute Cluster B
3
Why Overlays (VXLAN)?
VXLAN Fabric
Web Web
App
DB
Compute Cluster A Compute Cluster B
VXLAN disassociates workloads from physical networks, allowing for
possible transition to cloud based providers
4
Use cases
Increase compute utilization by pooling clusters
Allow non contiguous cluster expansion
Allow availability domains within a DC
Leverage capacity across multiple L2 POD’s
Overcome IP addressing challenges
VLAN sprawl
VLAN scale
5
Physical Topology
6
Deployment Model : Existing L2 or New L2 Access Layer
Optionally
Enable PIM
Aggregation Aggregation
vPC/MLAG
STP
Setup a single Subnet
Enable IGMP
Access Access
vSphere Distributed Switch Deploy VDS vSphere Distributed Switch
Edge Edge Edge Edge
Deploy VXLANs
VXLAN VXLAN VXLAN VXLAN
7
Deployment Model : Existing L3 or New L3 Access Layer
Aggregation
ECMP Enable PIM
Access
vSphere Distributed Switch Deploy VDS
Edge Edge Deploy VXLANs
VXLAN VXLAN
8
Deployment Model : Existing L3 or New L3 Access Layer
Aggregation
Deploying VXLANs is Very Easy
Enable PIM ECMP
Irrespective of Underlying Physical
Access
Network Topology
vSphere Distributed Switch Deploy VDS
Edge Edge Deploy VXLANs
VXLAN VXLAN
9
Logical Topology
10
Use Case Enabler #1 : Logical Layer 2 across L2
VM VM
VM VM
VDS
Host VLAN 20 Host VLAN 20
Switch
L2 Pod
11
Use Case Enabler #2 : Logical Layer 2 across L3
VM VM
VM VM
VDS VDS
Physical Physical
Switch Switch
L2 Pod L2 Pod
DC
Router
12
VXLAN Details
13
Virtual Extensible Local Area Network (VXLAN)
Ethernet in IP overlay Tunnel between ESX hosts
network • VMs do NOT see VXLAN ID
• Entire L2 frame encapsulated in IP multicast used for L2
UDP broadcast/multicast, unknown
• 50 bytes of overhead unicast
Include 24 bit VXLAN Identifier Technology submitted to IETF
• 16 M logical networks for standardization
• With Cisco, Citrix, Red Hat, Broadcom,
VXLAN can cross Layer 3 Arista, and Others
Outer Outer Inner Inner Optional Original
Outer Outer Outer Outer VXLAN ID
MAC MAC MAC MAC Inner Ethernet CRC
802.1Q IP DA IP SA UDP (24 bits)
DA SA DA SA 802.1Q Payload
VXLAN Encapsulation Original Ethernet Frame
14
VXLAN HOW – Initial Setup
VM1 VM2
MAC1 MAC2
ESX1 ESX 2
VTEP1 VTEP2
IGMP report for joining
IGMP report for joining
[Link]
[Link]
L2/L3 network
infra
VM1 and VM2 on VXLAN ID100, and VXLAN form the topology by joining mcast group [Link]
VTEP – Virtual Tunneling End Point
15
VXLAN protocol – ARP Query
VM1 VM2
MAC1 MAC2
1 BCAST MAC1 ARP Req 5 BCAST MAC1 ARP
Req
4 Learn inner src MAC/ outer src IP
mapping
ESX1 NetID MAC IP
NetID MAC IP ESX 2
VTEP1 VTEP2
100 MAC1 IP1_vtep1
2 For original bcast, encap w/ ip 3 Encapped pkt comes in via mcast
mcast
MAC Hdr IP Hdr UDP Hdr VXLAN Hdr BCAS MAC1 ARP MAC Hdr IP Hdr UDP Hdr VXLAN Hdr BCAS MAC1 ARP
DA:[Link] VXLAN ID: T Req DA:[Link] VXLAN ID: T Req
SA:IP_vtep1 100 SA:IP_vtep1 100
L2/L3 network
infra
VM1 sends ARP request (BCAST) in VXLAN ID 100 to get VM2’s MAC
VTEP – Virtual Tunneling End Point
16
VXLAN Protocol – ARP Response
VM1 VM2
MAC1 MAC2
4 MAC1 MAC2 ARP 1 MAC1 MAC2 ARP
Resp Resp
3 Learn inner src MAC/ outer src IP
mapping
ESX1 NetID MAC IP
NetID MAC IP ESX 2
VTEP1 100 MAC1 IP_vtep1 VTEP2
100 MAC2 IP_vtep2
2 MAC1 is learnt, encap w/ IP unicast
MAC Hdr IP Hdr UDP Hdr VXLAN Hdr MAC1 MAC2 ARP MAC Hdr IP Hdr UDP Hdr VXLAN Hdr MAC1 MAC2 ARP
DA:IP_vtep1 VXLAN ID: Resp DA:IP_vtep1 VXLAN ID: Resp
SA:IP_vtep2 100 SA:IP_vtep2 100
L2/L3 network
infra
VM2 sends ARP reply to respond VM1’s ARP request
VTEP – Virtual Tunneling End Point
17
VXLAN Protocol – Gateway
Gateway MAC1 MAC2 Data VM2
MAC1 MAC2
MAC1 MAC2 Data
ESX1 ESX 2
VTEP1 VTEP2
MAC Hdr IP Hdr UDP Hdr VXLAN Hdr MAC1 MAC2 Data MAC Hdr IP Hdr UDP Hdr VXLAN Hdr MAC1 MAC2 Data
DA:IP_vtep1 VXLAN ID: DA:IP_vtep1 VXLAN ID:
SA:IP_vtep2 100 SA:IP_vtep2 100
L2/L3 network
infra
18
VXLAN Protocol – Gateway
Gateway VM2
MAC1 MAC2
MAC1 MAC2 Data
ESX1 ESX 2
VTEP1 VTEP2
IP Hdr SW MAC1 Data
DA: Yahoo L2/L3 network
MAC
SA: GW IP
infra
19
VXLAN Logical View – 2 VMs in same VXLAN
VM VM
VXLAN
[Link] [Link] [Link]/24
[Link]
Gateway
[Link]
External Network
[Link]/24
Internet
VM to VM communication
VM to Internet communication
20
VXLAN Physical View – 2 VMs in same VXLAN
[Link]
Gateway
VM [Link] VM
[Link]
[Link]
VTEP VTEP
VDS VDS
ESX Host ESX Host
VLAN 10 L3 Switch
VLAN 20
Router
Internet
VM to VM communication
VTEP – Virtual Tunneling End Point
VM to Internet communication
21
VXLAN Logical View – 2 VMs in different VXLAN
VM VM
[Link] [Link]
VXLAN Blue VXLAN Purple
[Link]/24 [Link]/24
[Link] Gateway [Link]
[Link]
External Network
[Link]/24
Internet
VM to VM communication between 2 VXLANs
22
VXLAN Ecosystem: Performance, Visibility & Interoperability
VMware
Edge
Visbility
VXLAN Overlay
L2
Network Gateway
Service
Physical IP Network
Server
offloads
23
Summary
VXLAN disassociates workloads from Physical Network, allowing
mobility across clusters or clouds in future
VXLAN enables deploying networks and services in minutes
VXLAN is built on a rich partner ecosystem
24
The New Standard in Data Center Switching
The Role of the Physical Switch in a
VXLAN Environment
Use Case: Hosting
[Link]
[Link]
[Link]
[Link]
[Link]
[Link]
[Link] [Link] [Link]
Physical Hosting Virtual Hosting
Use Case: Bridging Virtual to Physical
VNI MIT VNI CalBears
VNI Auburn
Any port, any workload, any service, any where...
Automated Learning of Multicast/VNI State
vSphere 5
New VNI - CalBears
Multicast Group - [Link]
VNI ID - 650782
Interface Ethernet 24
VXLAN VTEP VNI CalBears
Interface Loopback0
VXLAN VTEP Gateway VNI Calbears
IP Address [Link]/24
<--Network
VM- Oski
VNI - CalBears
Incast SLA Assurance
Poor Service
Solution: VNI Segregation
1. Switch detects congestion (LANZ)
2. Switch samples traffic
3. Switch determines VNI RED is bad
4. Switch installs class-map mapping
RED to a separate queue
How do we
troubleshoot a
tunneled,
encapsulated,
multicast
environment???
VM Tracer for VXLAN
Full physical to
virtual visibility
Network audit to
ensure reachability
Automated provisioning
Workflow without
finger pointing
Other awesome
capabilities
Visibility - so where is my VM now?
spine0
leaf1 leaf2
esx10 esx11
VNI ‘Test’: [Link]
Aubie WarEagle vshield vm-tiger
Visibility - so where is my VM now?
spine0: show vmtracer interface vxlan Auburn
VTEP: ESX1 Role: vSwitch Switch/Port: [Link]/eth15
Name VNIC Status State IP Address
Aubie Network Interface 1 Up/Up vMotion [Link]
WarEagle Network Interface 2 Up/Up VM-FT-A [Link]
BooBama Network Interface 1 Up/Down -- [Link]
VTEP: ar24 Role: Router
spine0
Switch/Port: [Link]/loopback0
NAT/PAT leaf1 Status #ARPs IP Address leaf2
No Up/Up 45 [Link]
VTEP: ar22 Role: Port-VTEP Switch/Port: [Link]/eth2
FQDN IP MAC VLAN Status
[Link] [Link] 00-00-45-ab-12-fe 5 Up/Up
128.218.10.x 128.218.11.x
esx1 esx11
VNI ‘Test’: [Link]
Aubie WarEagle vshield vm-tiger
Visibility - so where is my VM now?
spine0: show vmtracer interface vxlan Auburn
VTEP: ESX1 Role: vSwitch Switch/Port: [Link]/eth15
Name VNIC Status State IP Address
Aubie Network Interface 1 Up/Up vMotion [Link]
WarEagle Network Interface 2 Up/Up VM-FT-A [Link]
BooBama Network Interface 1 Up/Down -- [Link]
VTEP: ar24 Role: Router
spine0
Switch/Port: [Link]/loopback0
NAT/PAT leaf1 Status #ARPs IP Address leaf2
No Up/Up 45 [Link]
VTEP: ar22 Role: Port-VTEP Switch/Port: [Link]/eth2
FQDN IP MAC VLAN Status
[Link] [Link] 00-00-45-ab-12-fe 5 Up/Up
128.218.10.x 128.218.11.x
esx1 esx11
VNI ‘Test’: [Link]
Aubie WarEagle vshield vm-tiger
Summary
Arista simply provides the best network for
VMware
Native vSphere integration into the physical
network to automate, and operationalize the
infrastructure
Simplified systems architecture to reduce
complexity and improve TCO
Optimized power and efficiency to maximize
compute density and enable profitable public, or
cost-effective private cloud deployments
Co-authors and leading innovators of VXLAN to
enable stateful L3 vMotion
Thank You!!
Q&A
arungoel@[Link]
kduda@[Link]
37
Other Networking & Security Sessions to attend
INF-NET1927: VMware vCloud® Networking: an Extensible and
Open Platform
INF-NET1932: On Demand Virtual Networking with vShield Edge
INF-NET2161: Vmware Networking 2012: Enabling the Software
Defined Network
INF-NET2166: Leaders Series: “How I Build My SDN-based Cloud”
INF-NET2965: Leaders Series: "I've Improved IT delivery with
vCloud Networking"
38
FILL OUT
A SURVEY
EVERY COMPLETE SURVEY
IS ENTERED INTO
DRAWING FOR A
$25 VMWARE COMPANY
STORE GIFT CERTIFICATE
INF-NET2162
VXLAN Deep Dive
Arun Goel, VMware, Inc.
Kenneth Duda, Arista Networks
#vmworld (hashtag here)
