VXLAN - Bringing Hypervisor &

Nexus Together

Wayne Davis – Technical Solutions Architect

BRKDCN-2200
Agenda

• Ready, Set, Tunnel - VXLAN Refresher

• Design Details - Under the Hood
• Avoid Resume Generating Events
• Best Practices
• Case Study Deployment Scenario's
• Roadmap – Whats Next?
• Wrap It Up

BRKDCN-2200
Session Rat hole’s
• Concentrate on deployment VXLAN – F & L
• How to Jumpstart ESX
• Introduction to Nexus 1000v
• All components of the design(s) choices
• Configuration Installation “Gotha's”
• Deep Dive into ACI
• Security “Line by Line cfg”
• Troubleshooting Deep dive Design

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Which Encapsulation?
VXLAN NVGRE

LISP MPLS

FabricPath
BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Questions
• Is it a standards based protocol used for traffic flows?
• Would you consider using BGP as a Control Plane in your Data Center ?
• Barriers to Adoption – Configuration Complexity ? Automation help ?
• Importance of being Standards-Based ? Proof of Interoperability
• Reliability and Scale out design *Important*?
• Active/Active Data Center design, is it possible?

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
 Introduction – What is VXLAN ?
MAC-in-IP Encapsulation Tunnel
Ethernet Frames
Ethernet Frames Endpoints
NETWORK

A IP Addr A
IP Addr
1.1.1.1 2.2.2.2
Switch Switch
B
Encap IP Network Dcap
B

C IP/UDP Packets C
PLANE

CRC
DATA

Outer Outer Outer Outer Outer Outer VXLAN ID Inner Inner Optional Original
CRC
MAC MAC 802.1Q IP DA IP SA UDP (24 bits) MAC MAC Inner Ethernet
DA SA DA SA 802.1Q Payload

VXLAN Encapsulation Original Ethernet Frame

16 M Segments

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
VXLAN: Flood-&-Learn vs EVPN Control Plane
Flood-&-Learn EVPN Control Plane

Overlay Services L2+L3 L2+L3

Underlay Network IP network with ECMP IP network with ECMP

Encapsulation MAC in UDP MAC in UDP

Peer Discovery Data-driven flood-&-learn MP-BGP

Peer Authentication Not available MP-BGP

Host Route Learning Local hosts: Data-driven flood-&-learn Local Host: Data-driven
Remote hosts: Data-driven flood-&-learn Remote host: MP-BGP
Host Route Distribution No route distribution. MP-BGP

L2/L3 Unicast Forwarding Unicast encap Unicast encap

BUM Traffic forwarding Multicast replication Multicast replication

Unicast/Ingress replication Unicast/Ingress replication

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
VXLAN - VTEP
VXLAN terminates its tunnels on VTEPs (VXLAN Tunnel End Point).
Each VTEP has two interfaces, one is to provide bridging function for local hosts, the
other has an IP identification in the core network for VXLAN encapsulation / de-
encapsulate.

Transport IP Network

VTEP VTEP
IP Interface IP Interface

Local LAN Segment Local LAN Segment

End System End System End System End System

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
 VXLAN - BGP-EVPN
IBGP Route Reflector*
R/R R/R
(on spine or different box)

BGP Peers VXLAN Overlay

on VTEPs
VTEP VTEP VTEP VTEP VTEP

Use Multi-Protocol BGP with EVPN Address family for :

Host Reachability Information

Tunnel Endpoints Location
• Mac Address + IP Address

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
VXLAN EVPN - Solution Advantages
Early ARP
Suppresses flooding for Unknown Unicast
Termination
ARP

Security Authenticate Tunnel Endpoints

Distributed Anycast Seamless and Optimal vm-mobility

Gateway Forwarding in the overlay

Ingress Replication Unicast Alternative to Multicast underlay

Active/Active Active/Active and Resilient Multipathing

Multipathing using vPC on Nexus

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Design Details – Under the Hood
Under the Hood – “Choice”
•
N1K - • N9K - EVPN • Application Centric
Segmentation Infrastructure

• VxLAN 1.0 / 2.0 • Requires 9k • Supports Any

switches Hypervisor
• Multiple OS support • Stateful Firewall
• Can be upgraded Support
• VSG VM and (NxOS to ACI) • Single Pane of Glass
Custom Attributes
Mgmt.
• VxLAN GW
• Appliance based • Container design
(anycast)
option Model
• Jump Data Centers • Security per vNIC
• Multi-technology with L2 domains
Design
• Broadcast
suppression
BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Under The Hood – Physical Cisco Nexus 9396PX

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
Cisco Nexus 9396PX

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

BCN 1

STS
BCN 1
ACT 2
STS

ACT 2

Cisco Nexus 9396PX

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

40 Gig
BCN 1

STS

ACT 2

Cisco Nexus 9396PX

Cisco Nexus 9396PX
Cisco Nexus 9396PX Cisco Nexus 9396PX

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

BCN 1
BCN 1
STS
BCN 1
BCN 1 STS
ACT 2
STS ACT
STS 2
ACT 2 ACT 2

ACI Leaf ACI Leaf ACI Border

ACI Border Leaf
Leaf

FW
10 Gig

PWR

PWR

PWR

PWR

PWR

PWR

PWR

PWR

PWR

PWR

PWR

PWR

PWR

PWR

PWR

PWR

PWR

PWR

PWR

PWR

PWR

PWR

PWR

PWR

PWR
SYS

SYS
SYS

SYS

SYS

SYS

SYS

SYS

SYS

SYS

SYS

SYS

SYS

SYS

SYS

SYS

SYS

SYS

SYS

SYS

SYS

SYS

SYS

SYS

SYS
CONSOLE
Cisco
!
ASA 5545-X

VPC
BOOT ALARM
ACTIVE VPN Adaptive
Security
PS1 HD1
Appliance
PS0 HD0 0

UCS
C240 M3

Cisco
ASA 5545-X

BOOT ALARM
ACTIVE VPN Adaptive

Servers
Security
PS1 HD1
Appliance
PS0 HD0 0

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

1/10 GIGABIT ETHERNET 1/2/4/8G FIBRE CHANNEL 1/10 GIGABIT ETHERNET 1/2/4/8G FIBRE CHANNEL

ID ID
N55-M16UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 N55-M16UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
STAT STAT

ADC

ADC 1.1 1.1

G
POWER 1 ~ POWER 2 ~
UCS

UCS B200 M3 UCS B200 M3

UCS 5108

!
SLOT SLOT
1 2
! Reset M3 ! Reset
Console UCS B200 Console

UCS B200 M3

1.1 1.1

Reset
Console

POWER 1 ~
!
POWER 2 ~ SLOT SLOT

ADC
G 3 4

FW
! Console Reset

UCS B200 M3 UCS B200 M3

SLOT SLOT
5 6
! Console Reset ! Console Reset

UCS B200 M3 UCS B200 M3

SLOT SLOT
7 8
! Console Reset
! Console Reset

OK FAIL OK FAIL OK FAIL OK FAIL

N2K N2K
Cisco Nexus 2148T Cisco Nexus 2148T
1GE Fabric Extender 1GE Fabric Extender UCS B200 M3 UCS B200 M3

UCS 5108
STAT
STAT
!
SLOT SLOT
1 2
! Reset M3 ! Reset
UCS B200
ID
Console Console
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1 2 3 4
ID 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1 2 3 4
UCS B200 M3

Reset
Console
!

SLOT SLOT

SEC
3 4
! Console Reset

UCS B200 M3 UCS B200 M3

vPC
SLOT SLOT
5 6
! Console Reset ! Console Reset

UCS B200 M3 UCS B200 M3

SLOT SLOT
7 8
! Console Reset
! Console Reset

OK FAIL OK FAIL OK FAIL OK FAIL

HP
BladeSystem
c7000
Enclosure

Bay
1 Bay
8

UCS B200 M3 UCS B200 M3

VM
UCS 5108

VM
!

VM
SLOT SLOT
1 2
! Reset M3 ! Reset
Console UCS B200 Console

UCS B200 M3

Bay
9 Bay
16
Reset

VM
Console
!

SLOT SLOT
3 4
DS
Module
! Console Reset

UCS B200 M3 UCS B200 M3

PS
1 PS
6

SLOT SLOT
5 6
! Console Reset ! Console Reset

VM
UCS B200 M3 UCS B200 M3

Chassis SLOT
7

! Console Reset
! Console Reset
SLOT
8

Servers OK FAIL OK FAIL OK FAIL OK FAIL

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
 VLAN 111 Nexus 7000

VLAN 222 150.150.150.0/30 40G

10G
VLAN 10 A V B

vlan 1011 .1 D .2
vlan 2022
C
Nexus 9396 Nexus 9396 .1 .1
RR
10G

10.9.9.0/30 99.99.99.0/30 ACI-9336 Spine

VXLAN Overlay
.2 10G
Nexus 9396 Nexus 9396
.2
ACI-9396 Leaf

VEM

172.16.111.111 172.16.222.222
VSM 10.96.126.17
10.96.126.80 10.111.111.50 10.222.222.16
HYPERVISOR
BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Under the Hood - Topo
Tunnel

N7k-1 N7k-2

LAN Extension

VXLAN L2 L3 – FW, VXLAN L2 VXLAN L2

Gateway SLB Gateway Gateway
N9k N9k N9k N9k

VM

VM VM OS VM-B
VM-A
OS OS VM-C

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Under The Hood - Nexus 7000 - Transport
Both N7k’s Transit Networks Nexus 7000 - A

feature tacacs+ interface Vlan900 • interface Ethernet1/1

cfs eth distribute description ACI-VLAN900
no shutdown
feature pim
bandwidth 80000000
• no switchport
feature eigrp no ip redirects
feature udld ip address 99.99.99.1/30 • ip address 10.9.9.1/8
feature interface-vlan no ipv6 redirects
feature hsrp no ip passive-interface eigrp 813 • ip router eigrp 813
feature lacp ip pim sparse-mode
feature dhcp • no shutdown
feature vpc interface Vlan901
description Transit_vlan_901_between_sydney23-
feature sflow
sydney24
no shutdown
ip address 150.150.150.2/30
ip router eigrp 813
no ip passive-interface eigrp 813
ip pim sparse-mode

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Design Details - Nexus 9000
Under the Hood – Nexus 9000
Tunnel

N7k-1 N7k-2

LAN Extension

VXLAN L2 L3 – FW, VXLAN L2 VXLAN L2

Gateway SLB Gateway Gateway

N9k N9k N9k N9k

VM

VM VM OS VM-B
VM-A
OS OS VM-C

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Under the Hood - Each Device Begins
Enable VXLAN and MP-BGP EVPN Control Plane
Enable VXLAN
feature nv overlay
Enable VLAN-based VXLAN (the currently
feature vn-segment-vlan-based
only mode)
feature bgp
Enable BGP
nv overlay evpn

Enable EVPN control plane for VXLAN

Other features may need to be enabled Enable OSPF if it’s chosen to be the
underlay IGP routing protocol
feature ospf
feature pim Enable IP PIM multicast routing in the
underlay network
feature interface-vlan
Enable VLAN SVI interfaces if the VTEP
needs to be IP gateway and route for the
VXLAN VLAN IP subnet.

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Under the Hood - Tenant Creation
VXLAN – virtual routing / forwarding
Create a VXLAN Tenant VRF

vrf context vxlan-n1k-vm-a

Specify the Layer-3 VNI for VXLAN routing
vni 22200 within the tenant VRF
rd auto
address-family ipv4 unicast
route-target import 22200:22200
Define VRF RD (route distinguisher)
route-target export 22200:22200
route-target both auto evpn

Define VRF Route Target and import/export

policies in address-family ipv4 unicast
vrf context vxlan-n1k-vm-b
vni 22210
rd auto
address-family ipv4 unicast Example to create a 2nd tenant VRF
route-target import 22210:22210 following the above steps
route-target export 22210:22210
route-target both auto evpn

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Under the Hood - Layer-3 (VNI) Routing – VM(A)
Configure Layer-3 VNI per EVPN Tenant VRF Routing Instant
vlan 2220
name vrf-L3-vm-a Create the VLAN for the Layer-3 VNI.
vn-segment 22200 One Layer-3 VNI per tenant VRF routing
instance
interface Vlan2220
description vrf-L3-vm-a-routing
no shutdown
vrf member vxlan-n1k-vm-a
Create the SVI interface for the Layer-3 VNI
vrf context vxlan-n1k-vm-a Put this SVI interface into the tenant VRF
vni 22200 context
rd auto
address-family ipv4 unicast
route-target import 22200:22200
route-target export 22200:22200 Associate the Layer-3 VNI with the tenant
route-target both auto evpn VRF routing instance.

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Under the Hood - Layer-3 (VNI) Routing – VM(B)
Configure Layer-3 VNI per EVPN Tenant VRF Routing Instant
vlan 1110
name vrf-L3-vm-b
vn-segment 22210
Define Layer-3 VNI for a 2nd tenant
interface Vlan1110 following the same steps in the previous
description vrf-L3-vm-b-routing slide
no shutdown
vrf member vxlan-n1k-vm-b

vrf context vxlan-n1k-vm-b

vni 22210
rd auto
address-family ipv4 unicast
route-target import 22210:22210
route-target export 22210:22210
route-target both auto evpn

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Under the Hood - Layer-2 VXLAN Network Identifier
Map VLANs to VXLAN VNIs and Configure their MP-BGP EVPN Parameters

vlan 222
vn-segment 20000 Map VLAN to VXLAN VNI
vlan 111
vn-segment 21000

evpn
vni 20000 l2
rd auto
Under EVPN configuration, define RD
route-target import auto and RT import/export policies for each
route-target export auto Layer-2 VNIs
vni 21000 l2
rd auto
route-target import auto
route-target export auto

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Under the Hood - Interface SVI – Layer 2
Create SVI interface for Layer-2 VNIs for VXLAN routing

Create SVI interface for a Layer-2 VNI.

interface Vlan111 Associate it with the tenant VRF.
no shutdown
vrf member vxlan-n1k-vm-a All VTEPs for this VLAN/VNI should have the
ip address 10.111.111.1/8 same SVI interface IP address as the
fabric forwarding mode anycast-gateway distributed IP gateway.

Enable distributed anycast gateway for this

interface Vlan222 VLAN/VNI
no shutdown
vrf member vxlan-n1k-vm-b
ip address 10.222.222.1/8
fabric forwarding mode anycast-gateway

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Under the Hood - Distributed Gateway – Anycast
Configure distributed gateway virtual MAC
address
One virtual MAC per VTEP
All VTEPs should have the same virtual MAC
address

fabric forwarding anycast-gateway-mac 0000.1111.2222

interface Vlan111
no shutdown
vrf member vxlan-n1k-vm-a Configure virtual IP address
All VTEPs for this VLAN should have the same
ip address 10.111.111.1/8
virtual IP address
fabric forwarding mode anycast-gateway

Enable distributed gateway for this VLAN

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Under the Hood - Network Virtualization Endpoint
Configure VXLAN tunnel interface nve1
interface nve1
no shutdown Specify loopback0 as the source interface
source-interface loopback0
Define BGP as the mechanism for host
host-reachability protocol bgp reachability advertisement
member vni 20000
suppress-arp Associate tenant VNIs to the tunnel
interface nve1
mcast-group 239.1.1.1 Define the mcast group on a per-VNI basis
member vni 21000 Enable arp suppression on a per-VNI basis
suppress-arp
mcast-group 239.1.1.2 Add Layer-3 VNIs, one per tenant VRF
member vni 22200 associate-vrf
member vni 22210 associate-vrf

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Under the Hood - VXLAN Tunnel Interface
Configuration – Cont’d
Configure VXLAN tunnel interface nve1

interface loopback 0
ip address 10.111.222.1/32 The loopback interface to source VXLAN
ip ospf network point-to-point tunnels
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Under the Hood - BGP – “Yes” in the LAN
router bgp 65535
router-id 10.111.222.1
log-neighbor-changes Address-family ipv4 unicast for prefix-
address-family ipv4 unicast based routing
address-family l2vpn evpn
neighbor 10.1.2.1 remote-as 65535 Address-family l2vpn evpn for evpn host
update-source loopback0 routes
address-family ipv4 unicast
address-family l2vpn evpn Define MP-BGP neighbors.
send-community extended Under each neighbor define address-family
neighbor 10.1.2.2 remote-as 65535 ipv4 unicast and l2vpn evpn
update-source loopback0
address-family ipv4 unicast Send extended community in l2vpn evpn
address-family l2vpn evpn address-family to distribute EVPN route
send-community extended attributes

vrf vxlan-n1k-vm-a
address-family ipv4 unicast Under address-family ipv4 unicast of each
advertise l2vpn evpn tenant VRF instance, enable advertising
vrf vxlan-n1k-vm-b EVPN routes
address-family ipv4 unicast
advertise l2vpn evpn

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Under the Hood - Route Reflector
router bgp 65535
router-id 10.1.2.1
log-neighbor-changes Address-family ipv4 unicast for prefix-
address-family ipv4 unicast based routing
address-family l2vpn evpn
retain route-target all Address-family l2vpn evpn for EVPN vxlan
template peer vtep-peer host routes
remote-as 65535 Retain route-targets attributes
update-source loopback0
address-family ipv4 unicast iBGP RR client peer template
send-community both
route-reflector-client
address-family l2vpn evpn Send both standard and extended
send-community both community in address-family ipv4 unicast
route-reflector-client
neighbor 10.111.222.1
Send both standard and extended
inherit peer vtep-peer
community in address-family l2vpn evpn
neighbor 10.1.2.12
inherit peer vtep-peer

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Design Details - Nexus 1000
Cisco Nexus 1000V Architecture
Virtual Appliance
VSM-1 (active) NX-OS
Network
Admin
Control Plane
VSM-2 (standby)

Supervisor-1 (Active)
Supervisor-2 (StandBy)
Back Plane

Linecard-1
Linecard-2
… NX-OS
Data Plane
Linecard-N

Modular Switch VEM-1 VEM-2 VEM-N

VSM: Virtual Supervisor Module Hypervisor Hypervisor Hypervisor

VEM: Virtual Ethernet Module Server
Admin

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Under the Hood - Nexus 1000v
Tunnel
N7k-1 N7k-2

111.111.111.1
10.222.222.1

LAN Extension

VXLAN L2 L3 – FW, VXLAN L2 VXLAN L2

Gateway SLB Gateway Gateway
10.222.222.50

111.111.111.x

VM
VM
VM OS VM-B
VM-A OS
OS
VM-C 10.222.222.49

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Under the Hood - Nexus 1000v transport
n1kv-wayne# sh run port-profile type ethernet UPLINK
switchport mode trunk
version 5.2(1)SV3(1.5a) switchport trunk allowed vlan 1-2,100-300
hostname n1kv-wayne channel-group auto mode on mac-pinning
no shutdown
system vlan 1-2
state enabled
vrf context management vmware port-group
ip route 0.0.0.0/0 10.96.126.254 port-profile type vethernet L3-Control
vlan 1-2,100-300 switchport mode access
switchport access vlan 1
no shutdown
port-channel load-balance ethernet source-mac capability l3control
port-profile default max-ports 32 system vlan 1
port-profile type ethernet Unused_Or_Quarantine_Uplink state enabled
shutdown vmware port-group
description Port-group created for Nexus 1000V internal usage. Do not use.
state enabled
vmware port-group interface Vethernet1
inherit port-profile L3-Control
port-profile type vethernet Unused_Or_Quarantine_Veth
description VMware VMkernel, vmk2
shutdown
vmware dvport 100 dvswitch uuid "75 3e 37 50 a5 6b ef f6-85 60 6a 7a 7f b6
description Port-group created for Nexus 1000V internal usage. Do not use. 3d"
state enabled vmware vm mac 0050.5671.47DA
vmware port-group

interface Vethernet3
port-profile type vethernet EVPN-VXLAN inherit port-profile vm-222
switchport mode access description Windows-7-222, Network Adapter 1
switchport access vlan 111 vmware dvport 256 dvswitch uuid "75 3e 37 50 a5 6b ef f6-85 60 6a 7a 7f b6
3d"
vmware vm mac 0050.56B7.0108
BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Under the Hood - VXLAN Forwarding Basics - VSM

Forwarding mechanisms similar to Layer 2

bridge: Flood & Learn
VEM learns VM’s Source (MAC, Host
VXLAN IP) tuple

Broadcast, Multicast, and Unknown Unicast VM VM VM VM

Traffic
VM broadcast & unknown unicast traffic are
sent as multicast

Unicast Traffic
Unicast packets are encapsulated and sent
directly (not via multicast) to destination VEM 1 VEM 2
host VXLAN IP (Destination VEM)

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Under the Hood - VM Host - VXLAN Topo

Guest Machine(s)
configured for setup

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Under the Hood -
L3 – N9k Enhanced VXLAN – VSM
B+U – no “M”
VLAN CLI Model VSM config
feature segmentation
vlan 222 segment mode unicast-only
name – n1k
port-profile type vethernet vxlan-n1k
capability l3control
interface vlan 222 vmware port-group
ip address 10.222.222.1 switchport mode access
ip router eigrp 22
switchport access vlan 222
capability vxlan
interface Ethernet3/2 no shutdown
switchport system vlan 1
switchport mode trunk
rate-mode dedicated force state enabled
channel-group 222 mode active
no shutdown
• VMkernel interface acts as VTEP
• VSM Control Mode should be L3
Normal SVI’s
• Bridge domain is configured as Unicast or
Unicast Mac Distribution

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Under the Hood - VMKernel

port-profile type vethernet vxlan-n1k

capability l3control
vmware port-group
switchport mode access
switchport access vlan 222
capability vxlan
no shutdown
system vlan 1
state enabled

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Under the Hood – VSM Bridge Domain

port-profile type vethernet bd-22222

port-profile type vethernet vmk-l3-vxlan-vtep switchport access bridge-domain BD-vxl
switchport mode access no shutdown
switchport access vlan 222 state enabled
capability vxlan vmware port-group
no shutdown
capability l3control
state enabled
vmware port-group

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Under the Hood - Port Profile Attachment

N1K - DVS

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Under the Hood – VTEP status
vsm-vxlan# show bridge-domain bd-22222
Bridge-domain bd-22222 (2 ports in all)
Segment ID: 22222 (Manual/Active)
Mode: Unicast-only (override)
MAC Distribution: Disable (override)
Group IP: NULL
State: UP Mac learning: Enabled
Veth4, Veth18

vsm-vxlan# show bridge-domain bd-22222 vteps

Bridge-domain: bd-22222
VTEP Table Version: 21
Port Module VTEP-IP Address VTEP-Flags
---------------------------------------------------------------------------
Veth1 3 10.111.111.49 (D) <---Designated VTEP (vmk)
Veth2 4 10.111.111.50 (D)

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Avoid Resume Generating Event(s)
Best Practice(s)
Success What Should We Do?
What’s the desired outcome?

Deployment
Hit the
EASY BUTTON
Virtual Switch Update
Manager
LCM
• Backups
• High Availability Options
Life Cycle Management – VSM / VEM • Software Repository

P&S
What Should We Do? • HW Limits
Performance & Scalability
• Optimization • SW Limits
• Decision Trees

Enterprise Architecture Framework – Network, Security, Server, Virtualization

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
 BP
Performance & Scalability
• ESX – 5.2(1)SV3(1.2) • Hyper-V – 5.2(1)SM3(1.1)
• 256 VEMs, 12K vEth count • VxLAN 1.0 & HVN
• VxLAN 2.0 (BGP Control Plane)
• Hyper-V – 5.2(1)SM1(5.2a)
• VxLAN UDP Port Configurable
• SCVMM 2012 SP1 & R2
• N1K Virtual Switch Update Manager
• Windows Server 2012 & R2
• Distributed NetFlow
• VSG VM and Custom Attributes
• IGMP Multicast Offload (1k Groups)
• Universal Licensing
• BPDU Guard & Storm Control
• Cisco TrustSec, IPv6 Enhancements • KVM – 5.2(1)SK3(2.1)
• IceHouse
• ESX – 4.2(1)SV2(2.2)
• RHEL-OSP – OpenStack Platform Inst
• Dynamic Fabric Automation Leaf
• VxLAN GW
• VDP – VSI Discovery Protocol
• pVLAN
• Universal Licensing
• UUFB blocking
BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
 BP
Life Cycle Management - VSM Control Modes
• L3 Mode
• This is not routing
• L3 is the recommended & default
• Easier to troubleshoot
• Cross Firewalls & L3 boundaries
• Requires an IP address be assigned to the VEM (vmk)
• Uses UDP 4785 for both source and destination
• Sourced from mgmt0 by default

• L2 mode (Legacy)
• Requires L2 connectivity through control0 interface to all VEM modules
• Deprecated but supported on ESX
• Not supported with Hyper-V or KVM

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
 BP
Life Cycle Management - VSM vMotion

• Manual vMotion/Live Migration is supported

• VMware DRS is NOT recommended for Primary & Secondary VSMs
• Aggressive DRS could lead to excessive VSM-VEM heartbeat packet drops
• Best practice to keep Primary and Secondary VSM outside DRS control
• Use anti-affinity rules where possible
• FT is not supported

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
 BP
Life Cycle Management - VSM Backups
• A running-config is not enough to restore due to PSS
• VSM on ESXi / HyperV
• Clone to a template
• Restore from an older template + running-config
• Both VSMs must be powered down

• VSM on Nexus 1110

• Export a VSM to a file
• Import the saved VSM to restore

• VSM on ESXi Snapshots

• Not officially supported
• I/O latency cost associated with expanding the differential file

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
 BP
Life Cycle Management - VSM Interfaces
• Control • Management
• VSM-VSM HA Heartbeats • SSH console access
• VSM-VEM Heartbeats • SNMP, HTTP, XML
• VSM-VSM Synchronization • vCenter Communication
• BGP Control Plane • HA Heartbeat Backup

• Packet
• CDP, IGMP*, SNMP • Interface Order is always the same!
• Layer3 Mode VSM-P eth0: control
• Collapsed ctrl0 & pkt into mgmt0
eth1: mgmt0
• VSM-VEM communication on mgmt0
eth2: packet
• Dedicated Control:
svs mode L3 interface [control | mgmt0]

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
 BP
Life Cycle Management - VEM Deployment
• L3 control requires a VMKernel NIC on N1K DVS
• We need an L3 interface to forward control traffic
• 200/100/10ms latency between VSM & VEM

• Recommend using the ESXi management VMKernel NIC

• Migrate management vmk behind VEM
• Doesn’t require static routes on ESXi hosts

• Put additional vmks on different subnets (vMotion / Storage)

• UCS “Dynamic vNICs” in Service-Profiles
• VEM and VM-FEX are mutually exclusive

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
VEM Deployment – VMKs on same subnet
BP

• VMware uses a single TCP/IP stack

for all VMK interfaces VMK1 VMK0
192.168.10.100 192.168.10.200
• Don’t use multiple VMKs on the
same subnet on different virtual VEM vSwitch

switches VMware ESX

• No way to pin traffic to an uplink

interface.
• One interface gets picked for all
traffic on that subnet
• VMware KB article 2010877
• Only one default gateway per host

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
 VEM - Port-Profiles Secret Sauce BP
port-profile type vethernet vmk-l3
port-profile type ethernet uplink
capability l3control
vmware port-group
vmware port-group
switchport mode trunk
VMK1 switchport mode access
switchport trunk allowed vlan 10,119
switchport access vlan 119
channel-group auto mode on mac-pinning
capability vxlan
no shutdown
no shutdown
system vlan 119
system vlan 119
state enabled
state enabled

port-profile type vethernet vm-vlan10

VM2 vmware port-group vmnic0 Eth3/1
switchport mode access
switchport access vlan 10
PO1
VM1 no shutdown
state enabled
vmnic1 Eth3/2

vEthernet PP (default) Ethernet PP

-Virtual Interfaces (vEthernet x) -Physical Interfaces (Ethernet x/y)
-Typically Access Ports or Bridge Domains -Typically Trunk Ports
-Configuration: VLAN, ACLs, VxLAN, QoS -Configuration: Port-Channel, ACLs, QoS
BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
 BP

VSUM – Virtual Switch Update Manager

• Install, Migrate, Upgrade, Monitor Nexus 1000V and ACI AVS
• Standalone VM
• Nexus 1000V Binaries are Self-Contained
• Integrated in vSphere Web Client through Plugin
• VMware only today
• Single instance manages all N1k on a vCenter No
Charge
• Manages existing N1k DVS

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
 BP

VSUM – Plugin Icon

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
 BP
VSUM – Installing Nexus 1000V VSM
1 2

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
 BP

Upgrades - Deployment
• First always read and follow the upgrade guides
• Order matters: VSM then VEM
• Take a backup of the VSMs
• On ESXi use the clone to template option (Powered Down)
• On Nexus 1110s / Cloud Services Platform use the export command
• Backup the running-config

• Generate a tech-support before the upgrade

• If something goes wrong STOP and call TAC
• Use a maintenance window
• VEM upgrades require ESXi hosts to be in Maintenance Mode
• Use N1k Upgrade Utility Matrix to Plan a combined N1k+vSphere Upgrade
BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Design Details – Application
Centric
Infrastructure

BRKDCN-2200
ACI Relationship Map

9 6 2 7 4 5

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
End-Points end EPG membership
 Device connected to network directly or indirectly

Server
 Has address (identity), location, attributes (version,
patch level)
 Can be physical or virtual or container
Virtual Machines & Containers • Examples:
• End Point Group (EPG) membership defined by:
• Ingress physical port (leaf or FEX)
• Ingress logical port (VM port group)
• VLAN ID
• VXLAN (VNID)
Storage • IP address
• IP Prefix/Subnet
• VM-based attributes
• NVGRE (VSID) (future)
• Layer 4 ports (future)

Client

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
ACI – Segmentation

Micro-segmentation

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
 LEAF 222 LEAF 222
1/7 1/17
VLAN 222 VLAN 111

111.111.111.11
111.111.111.10
BaseCL-
BaseCL-VXLAN
• Background VXLAN
BD1
• 111.111.111.10 and 111.111.111.11 in CL-VXLAN
• Create CL-VXLAN(useg) Can talk to each other as they are in same EPG

• Put 111.111.111.10/32 in EPG VM-1(useg)

• Both VM – talk to GW + Each other ! Configure CL-VXLAN(useg)

bypasses IP classification

LEAF 222 LEAF 222

1/7 1/17
VLAN 222 VLAN 111

111.111.111.10 111.111.111.11

CL-
VXLAN(useg) CL-VXLAN
BD1

Still can talk to each other

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Case Study Deployment
Scenario's

BRKDCN-2200
 #1
Case Study Deployment: Basic Tunneling
• VSM is a Virtual Machine
• Control plane for the Nexus 1000V switch
• VEM packet forwarding not impacted by reloads
• VSM HA pair distributed across multiple host

• Responsible for:
• Programming and Managing Virtual Ethernet Modules
(VEM)
• Communicating with Management Applications
(vCenter, SCVMM, Horizon Dashboard, etc.) Hypervisor

VEM
VSM
VM VM VM

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Case Study - Deployment #2

SCE SCE
SAP MSFT RAC RHEL SAP MSFT RAC RHEL
SPINE SPINE
SERVER ACCESS SERVER ACCESS

N2k N2k N2k N2k

N2k
N2k
N2k
N2k
N2k
N2k
N2k
N2k LEAF LEAF
3

UNIFIED COMPUTE SYSTEM UNIFIED COMPUTE SYSTEM

FI FI

FI FI
NGFW NGFW

SVC Block

EXT INT I-NET I-NET INT EXT

DFW TPA

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
Case Study - ACI LEVEL #3
Data Farm

OSPF

Data Farm

VM VM VM

DMZ
BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Roadmap
Now is the future
• Migrate Customers from Nexus 1010/1010-X/1110-S/1110-X
• Dedicated Cisco Cloud Services Platform appliance ( CSP 2100 )
• Preparation for Nexus 1000 release 3 – BGP control plane interoperability with
Nexus 9000
• Whitepaper to follow – design guidance on VM scale and extended attribute
parity
• Look @ ACI – you just might “love it”

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
Complete Your Online Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 Amazon gift card.
• Complete your session surveys
through the Cisco Live mobile
app or from the Session Catalog
on CiscoLive.com/us.

Don’t forget: Cisco Live sessions will be available

for viewing on-demand after the event at
CiscoLive.com/Online

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Thank you
R&S Related Cisco Education Offerings
Course Description Cisco Certification
CCIE R&S Advanced Workshops (CIERS-1 & Expert level trainings including: instructor led workshops, self CCIE® Routing & Switching
CIERS-2) plus assessments, practice labs and CCIE Lab Builder to prepare candidates
Self Assessments, Workbooks & Labs for the CCIE R&S practical exam.

• Implementing Cisco IP Routing v2.0 Professional level instructor led trainings to prepare candidates for the CCNP® Routing & Switching
• Implementing Cisco IP Switched CCNP R&S exams (ROUTE, SWITCH and TSHOOT). Also available in
Networks V2.0 self study eLearning formats with Cisco Learning Labs.
• Troubleshooting and Maintaining
Cisco IP Networks v2.0

Interconnecting Cisco Networking Devices: Configure, implement and troubleshoot local and wide-area IPv4 and IPv6 CCNA® Routing & Switching
Part 2 (or combined) networks. Also available in self study eLearning format with Cisco Learning
Lab.

Interconnecting Cisco Networking Devices: Installation, configuration, and basic support of a branch network. Also CCENT® Routing & Switching
Part 1 available in self study eLearning format with Cisco Learning Lab.

For more details, please visit: http://learningnetwork.cisco.com

Questions? Visit the Learning@Cisco Booth or contact [email protected]

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Design Cisco Education Offerings
Course Description Cisco Certification
Designing Cisco Network Service Architectures Provides learner with the ability to perform conceptual, intermediate, and CCDP® (Design Professional)
(ARCH) Version 3.0 detailed design of a network infrastructure that supports desired capacity,
performance, availability required for converged Enterprise network (Available Now)
services and applications.

Designing for Cisco Internetwork Solutions Instructor led training focused on fundamental design methodologies used CCDA® (Design Associate)
(DESGN) Version 3.0 to determine requirements for network performance, security, voice, and
wireless solutions. Prepares candidates for the CCDA certification exam. (Available Now)

For more details, please visit: http://learningnetwork.cisco.com

Questions? Visit the Learning@Cisco Booth or contact [email protected]

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
Data Center / Virtualization Cisco Education Offerings
Course Description Cisco Certification
Introducing Cisco Data Center Networking (DCICN); Learn basic data center technologies and skills to build a CCNA® Data Center
Introducing Cisco Data Center Technologies (DCICT) data center infrastructure.

Implementing Cisco Data Center Unified Fabric (DCUFI); Obtain professional level skills to design, configure, CCNP® Data Center
Implementing Cisco Data Center Unified Computing (DCUCI) implement, troubleshoot data center network infrastructure.
Designing Cisco Data Center Unified Computing (DCUDC)
Designing Cisco Data Center Unified Fabric (DCUFD)
Troubleshooting Cisco Data Center Unified Computing
(DCUCT)
Troubleshooting Cisco Data Center Unified Fabric (DCUFT)

Product Training Portfolio: DCNMM, DCAC9K, DCINX9K, Gain hands-on skills using Cisco solutions to configure,
DCMDS, DCUCS, DCNX1K, DCNX5K, DCNX7K deploy, manage and troubleshoot unified computing, policy-
driven and virtualized data center network infrastructure.

Designing the FlexPod® Solution (FPDESIGN); Learn how to design, implement and administer FlexPod Cisco and NetApp Certified
Implementing and Administering the FlexPod ® Solution solutions FlexPod® Specialist
(FPIMPADM)

For more details, please visit: http://learningnetwork.cisco.com

Questions? Visit the Learning@Cisco Booth or contact [email protected]

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
Network Programmability Cisco Education Offerings
Course Description Cisco Certification
Integrating Business Applications with Network Learn networking concepts, and how to deploy and troubleshoot Cisco Business Application
Programmability (NIPBA); programmable network architectures with these self-paced courses. Engineer Specialist Certification
Integrating Business Applications with Network
Programmability for Cisco ACI (NPIBAACI)

Developing with Cisco Network Programmability Learn how to build applications for network environments and effectively Cisco Network Programmability
(NPDEV); bridge the gap between IT professionals and software developers. Developer Specialist Certification
Developing with Cisco Network Programmability
for Cisco ACI (NPDEVACI)

Designing with Cisco Network Programmability Learn how to expand your skill set from traditional IT infrastructure to Cisco Network Programmability
(NPDES); application integration through programmability. Design Specialist Certification
Designing with Cisco Network Programmability
for Cisco ACI (NPDESACI)

Implementing Cisco Network Programmability Learn how to implement and troubleshoot open IT infrastructure Cisco Network Programmability
(NPENG); technologies. Engineer Specialist Certification
Implementing Cisco Network Programmability
for Cisco ACI (NPENGACI)

For more details, please visit: http://learningnetwork.cisco.com

Questions? Visit the Learning@Cisco Booth or contact [email protected]

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
Cloud Cisco Education Offerings
Course Description Cisco Certification
Understanding Cloud Fundamentals Learn how to perform foundational tasks related to Cloud computing, and the essentials
(CLDFND) of Cloud infrastructure
CCNA Cloud
Introducing Cloud Administration Learn the essentials of Cloud administration and operations, including how to provision,
(CLDADM) manage, monitor, report and remediate.

Implementing and Troubleshooting the Learn how to implement and troubleshoot Cisco Cloud infrastructure: compute,
Cisco Cloud Infrastructure (CLDINF) network, storage.

Learn how to design private and hybrid Clouds including infrastructure, automation,
Designing the Cisco Cloud (CLDDES)*
security and virtual network services
CCNP Cloud
Automating the Cisco Enterprise Cloud Learn how to automate Cloud deployments – provisioning IaaS (private, private with
(CLDAUT)* network automation and hybrid) and applications, life cycle management
Building the Cisco Cloud with Application Learn how to build Cloud infrastructures based on Cisco Application Centric
Centric Infrastructure (CLDACI)* Infrastructure, including design, implementation and automation

Learn how to manage physical and virtual infrastructure using orchestration and
UCS Director Foundation (UCSDF)
automation functions of UCS Director.

* Available Q2CY2016

For more details, please visit: http://learningnetwork.cisco.com

Questions? Visit the Learning@Cisco Booth or contact [email protected]

BRKDCN-2200 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 75