#CLUS

Catalyst 9000
Switching
Architecture

TECARC-2900

#CLUS
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session

How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space

Webex Teams will be moderated cs.co/ciscolivebot#TECARC-2900

by the speaker until June 16, 2019.

#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
 Campus of the Future
New Trends Drive New
Requirements for the Network
Campus of the Future
New Demands Driving New Requirements

#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
 Enterprises are expanding to the Cloud
This expansion is
driving fundamental
change across every
IT infrastructure domain

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
1920x1080
Full HD

3840x2160
4K

8K Video #CLUS
7680x4320

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Augmented Reality Virtual Reality Mixed Reality

Latency Sensitive Bandwidth Hungry Compute Intensive

 50
Billion
Connected Devices
by 2020
 Emerging
New Protocols

60%
IOT devices
might NOT be WiFi
Wifi6
What does this really
mean for Campus of
the Future?

#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
 <100 ms
Latency

Up to 200 Mbps
Bandwidth
New Clients Connectivity Requirements for
Campus of the Future

Design your Networks for Low Latency & Higher Speeds

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
 Segmentation
IoT BMS
Users
Devices Devices
…

Secure On Fingerprints End to End

Encryption
Boarding & Identity Policy

Security and Segmentation Requirements

for Campus of the Future Security

Design your Networks for End to End Security & Segmentation

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
 Data Analytics in Real Time

Forensic Troubleshooting

Telemetry Data

Data Requirements Open Interfaces & Models

For Campus of the Future

Data is Essential to Operate & Manage the Networks

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
 Always ON
99.99999999%
Availability Requirements
For Campus of the Future Availability
Always Available is the New Requirement
#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
 Full Stack Cisco DNA Center
Applications

APIs

Domain controllers

Powered by Cisco IOS XE

Physical and virtual

Infrastructure
Catalyst 9K
Application-Specific
Integrated Circuit (ASIC)

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Catalyst 9K Family

Built for Campus of the Future

#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
The goal of this seminar is to give you an
in-depth view of Catalyst 9000 Family of
products, and how it enables a
“New Era of Networking”

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Your Speakers Today!

Kenny Lei Minhaj Uddin Sai Zeya Muhammad Imam

Technical Marketing Engineer Technical Marketing Engineer Technical Marketing Engineer Manager Technical Marketing

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
 Agenda
Sections Duration Time Speaker

Opening & Introduction 30 Mins 9:00 – 9:30 Muhammad

Stackable Access Platforms 30 Mins 9:30 – 10:00 Minhaj

Modular Acc/Agg Platforms 30 Mins 10:00 – 10:30 Sai

Modern Campus Platforms Break -10:30 – 10:45

Fixed Core/Agg Platforms 30 Mins 10:45 – 11:15 Sai

Modular Core Platforms 30 Mins 11:15– 11:45 Kenny

UADP ASIC 45 Mins 11:45 – 12:30 Kenny

Architecture building blocks
IOS-XE Software 30 Mins 12:30 – 1:00 Muhammad

Lunch – 1:00 – 2:00

High Availability 90 Mins 2:00 – 3:30 Minhaj

Break – 3:30 - 3:45

Foundational Features
Quality of Service 30 Mins 3:45 – 4:15 Muhammad

Security 15 Mins 4:15-4:30 Kenny

Advance Managing and Programmability 15 mins 4:30 – 4:45 Kenny

Analytic Features Application Hosting 45 Mins 4:45 – 5:30 Sai

Closing & Wrap up 30 Mins 5:30 to 6:00 Muhammad

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Let’s get started

#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Catalyst 9000
Family
Catalyst 9000 (9K) – A Growing Family
Catalyst 9400
Catalyst 9400
Catalyst 9400Catalyst 9400
Catalyst 9400
Catalyst 9400 Catalyst 9600
Catalyst 9400

Catalyst 9400

Catalyst 9500 Catalyst 9500

Catalyst 9300 Catalyst 9500

Catalyst 9300 Catalyst 9500
Catalyst 9300

Catalyst 9200 Catalyst 9300

The CAT 9K Family

IOS-XE 16
Common Software Architecture

UADP 2.0
Common Hardware Architecture

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
The Latest Addition - Catalyst 9600

Modular Campus Core – Modernizing the Catalyst 6K

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
And there is one more
Catalyst 9K

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
The First Catalyst Wireless LAN Controller

Catalyst 9100

BRKEWN-2670 - Introduction to Cisco Catalyst 9800 Wireless Controller

Catalyst 9800 Powered by IOS XE

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Building Blocks
Catalyst 9K – Common Building Blocks

2.0/3.0

Building Blocks to Face the challenges of Campus of the Future

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
UADP - Next Generation of ASIC Innovation
Investment Protection
Flexible Pipeline

Universal Deployments
Adaptable Tables

Enhanced Scale/Buffering
Multicore resource share

Up to 2X to 4X
Up to 384K Flex Shared Up to 1.6T
Counters Lookup Bandwidth Forwarding + TCAM

Up to 20B Transistors Up to 36MB Up to 64K x2

Embedded Packet Buffer Netflow Records
16nm Technology with latest ASIC Microprocessors

Flexible & Programmable ASIC – Adapts to the New Technologies

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Open IOS XE – A Modern Operating System
IOS XE 16.x

IOSd Hosted Apps IOS Sub Systems

Resiliency & HA
IOS Sub
IOSd Blob
IOS Sub
Systems LXC* LXC*
IOS Sub
Systems
Systems
LXC* Guestshell

Common Infrastructure / HA
IOS XE Database
Programmability & Open Models

Management Interface

IOS-XE
Module Drivers
DB
Container Support
Kernel 3rd Party App Hosting

Open, Model Driven & Secure Operating System

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Catalyst 9K Family - x86 CPU

x86 CPU*

x86 based 3rd Party Apps

*Catalyst 9200 has Embedded CPU in the UADP ASIC

x86 CPU enables hosting containers and 3rd party apps

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Catalyst 9K Family – External Storage Options
M2 SATA USB 3.0

Internal - Plugs into Sup External - Hot Swappable

External - Hot Swappable

Up to 1 TB Up to 120 GB

For Local Logging – 3rd Party App Hosting - Containers

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
User Centric
Design of Catalyst
9K
Catalyst 9K Family – Blue Beacon

Blue Beacon
on Every System &
Components

Identification of Devices has never been Easier

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Catalyst 9K Family with Built-in RFID

Sample RFID Tag Data

SN = 'FOC2109Q023’
RFID
PID on Every Device
= 'C9500-24Q'
VID = 'V00'
TAN and FRUable
= '68-100900-02'
TAN Rev = '10'
Components
CLEI = 'UNDEFINED' of Catalyst
Index
Encode
= '900'
9400
= 'SGTIN-198‘
Filter = '0‘
Partition = '5'
Company = '0746320'
Built-in Passive RFID

Inventory Management (Tracking) has never been Easier

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
 Catalyst 9400 – Cool Fan Trays

Fan Trays Serviceable

from Front and Back

Barometer Temprature
Sensors

Variable Speed Fans

Flexibility in Cabling & Maintenance

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Catalyst 9K Family – Industrial Design & Ergonomics

Circle Pattern Silver/Nickel Based. Cisco Medium Gray Grab area in Rounded Frame
Hex Packing Smooth finish Smooth finish molded plastic
2.5 mm.

Best Safety Certifications

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
The Catalyst 9K Family

Catalyst 9200 Catalyst 9300 Catalyst 9400 Catalyst 9500 Catalyst 9600
Fixed Access Switches Modular Access & Distribution Switches Fixed Core & Distribution Switches Modular Core & Distribution

Built on Cisco’s Innovative Hardware & Open IOS-XE

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Catalyst 9300
Catalyst 9300 – Leading Fixed Access Switch

1.8-GHz quad-core x86 CPU

Built-in RFID
USB 2.0 flash drive 8 GB of DDR4 DRAM 1x UADP 2.0 ASIC
(passive) 16 GB flash

Unmatched POE
USB Console Flexible Fixed or Modular
Mini-USB type B Multigigabit Capable Resiliency – Perpetual/Fast Uplinks
High power - 60W UPOE

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Catalyst 9300– Back View
External Storage
USB 3.0 Removable storage Stack Cables Redundant Fans Redundant Power
(120GB SSD)

Optional Power Supplies

Stackwise-480/320 Stackpower*
(AC+DC)

* Supported only on Cisco Catalyst 9300 Series modular uplink models

#CLUS(C9300 SKUs). © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco Catalyst 9300 Series Switches
1G fiber models – expanding to FTTD and 1G fiber aggregation applications

24 ports 1G fiber C9300-24S

48 ports 1G fiber C9300-48S

Stackable with all C9300 Models

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Cisco Catalyst 9300 Series Switches
C9300L Fixed Uplink models

24 Port Data/PoE Model C9300L-24

48 Port Data/PoE Model C9300L-48

Stackable with all C9300L Models

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
 Cisco Catalyst 9300 Series Switches
New generation of fixed access
Modular uplinks Fixed uplinks
(C9300 SKUs) (C9300L SKUs)

Copper – 24/48 Ports Copper – 24/48 Ports

Data-Only

POE+ - 30W

UPOE – 60W
Data Or POE+
Fiber SFP

C9300L Stack kit

Data-Only

* Modular uplinks only

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Multigigabit
Ethernet
 The Problem - Gigabit Bottleneck
Wifi >1G

Cat 5e Cables

Limited to 1G!

Existing Gigabit Gigabit Ethernet has been Market needs an innovative

infrastructure is insufficient around since 1999 and has technology to support
to handle .11ac growth now become the bottleneck >1Gbps over existing cables
beyond 1Gbps

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
 The Solution – Cisco MultiGigabit
WiFi >
1G
Cat 5e Cables

2.5-5G!
MultiGigabit MultiGigabit
Switch Capable AP

Cisco MultiGigabit

Is a game-changing innovation Enables 2.5 and 5 Gbps up to Supports all PoE standards
allowing enterprise networks to 100m on legacy cables up to 60W
evolve beyond 1G

Delivers up to 5X Speeds in Enterprise without replacing Cabling Infrastructure

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
 Multigigabit – IEEE Standard

Cisco Blog: Network World:

http://blogs.cisco.com/enterprise/nbase-t-alliance-achieving-a-new - http://w ww.networkworld.com/article/3124948/lan-w an/ieee-sets-
new -ethernet-standard-that-brings-5x-the-speed-w ithout-disruptive-

802.3bz
industry-standard
cable-changes.html

Multigigabit Speeds are now IEEE Standard!

Ethernet Alliance
EEE Post: http://w ww.ethernetalliance.org/w p-
http://standards.ieee.org/findstds/standard/802.3bz-2016.html content/uploads/2016/09/EA_IEEE802bz_FINAL_26Sep16.pdf

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
 What Speeds Are Supported on MultiGigabit
Ports?
10 M
MultiGigabit Phys Are Different than 1Gigabit Phys

MultiGigabit Ports Are Capable of the Following Speeds 100 M

100M / 1Gig / 2.5Gig / 5Gig / 10Gig
No 10M on MultiGigabit Ports
1000 M
2.5Gig and 5Gig Are now standard

The Non-MultiGigabit Ports Are the Same as Previous 2.5 G

Line Cards / Products – Support 10M/100M/1Gig Speeds

MultiGigabit Phys Are Same on Across our MultiGigabit 5G

Switch Family
Half Duplex on Multigigabit ports is not supported
10 G

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
Catalyst 9300 Multigigabit Family

24 x 100/1/2.5/5/10G Ports

36 x 2.5 G 12 x Multigigabit
Ports Ports

48 x 5 G Ports

Highest 2.5G & mGig Density in the Industry

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
 Cisco Catalyst 9300 Series Switches
Uplink options

Cisco® Catalyst® 9300 Series Cisco Catalyst 9300 Series

modular uplink models fixed uplink models

4x 1G fixed uplinks

4x Multigigabit 4x 1 Gbps 2x 1/10/25 Gbps

SFP 8x 10 Gbps 2x 40 Gbps
copper SFP/SFP+ SFP/SFP+ QSFP
C9300-NM-4M C9300-NM-4G C9300-NM-8X C9300-NM-2Y C9300-NM-2Q 4x 10G fixed uplinks

Modular Uplink options on all C9300 SKUs Fixed uplink option on C9300L SKUs

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
Catalyst 9300 – Power Supplies & Stacking
Power Supplies Stacking

350WAC 715WAC 1100WAC 750WDC

C9300

Platinum Rated
C9300L

0.5, 1 and 3 meter Options

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
 C9300-48 Block Diagram
Stackwise 480
X86 1.8-GHz
Packet buffer (8 MB) Packet buffer (8 MB)
quad-core CPU

Forwarding controller Forwarding controller FPGA

DRAM – 8 GB
Reassembly Rewrite Reassembly Rewrite Flash
crypto crypto 16 GB

Ingress FIFO Egress FIFO Ingress FIFO Egress FIFO

USB 2.0

Core 1 Core 0
USB 3.0
Network interface
Mgmt Console
10Gx4/40Gx1
1G x8 10Gx4/40Gx1
1G x8

PHY PHY PHY PHY PHY PHY 40G PHY 40G PHY
0 1 2 3 4 5 0 1

TX 0-7 TX 0-7

Cage 1 Cage 2 Cage 1 Cage 2

1-12 13-24 25-35 36-48 1-4 1-4

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
 C9300 Multigigabit-24 Block Diagram
X86 1.8-GHz
quad-core CPU
Stackwise 480 Stackwise 480
FPGA
ASIC 0 Packet buffer (16 MB) ASIC 1 Packet buffer (16 MB)
DRAM – 8 GB
Flash
Forwarding controller Forwarding controller Forwarding controller Forwarding controller 16 GB

Reassembly Rewrite Reassembly Rewrite Reassembly Rewrite Reassembly Rewrite USB 2.0
crypto crypto crypto crypto
Ingress Egress Ingress Egress Ingress Egress Ingress Egress USB 3.0
FIFO FIFO FIFO FIFO FIFO FIFO FIFO FIFO
Core 1 Core 0 Core 1 Core 0 Mgmt Console

Network interface Network interface

10Gx4/40Gx1
10G x4 10Gx4/40Gx1
10G x4

Mgig Mgig Mgig Mgig Mgig Mgig

40G PHY 40G PHY
PHY PHY PHY PHY PHY PHY
0 1
0 1 2 3 4 5
TX 0-3 TX 0-3

Cage 1 Cage 2 Cage 1 Cage 2

1-6 7-12 13-18 19-24 1-4 1-4

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
 C9300L-48 Block Diagram
StackWise 320
X86 2.2-GHz
Packet buffer (8 MB) Packet buffer (8 MB)
quad-core CPU

Forwarding controller Forwarding controller FPGA

DRAM – 8 GB
Reassembly Rewrite Reassembly Rewrite Flash
crypto crypto 16 GB

Ingress FIFO Egress FIFO Ingress FIFO Egress FIFO

USB 2.0

Core 1 Core 0
USB 3.0
Network interface
Mgmt Console
10Gx4/40Gx1
1G x8 10Gx4/40Gx1
1G x8

PHY PHY PHY PHY PHY PHY 40G PHY 40G PHY
0 1 2 3 4 5 0 1

TX 0-7 TX 0-7

Cage 1 Cage 2 Cage 1 Cage 2

1-12 13-24 25-35 36-48 1-4 1-4

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
 Cisco Catalyst 9300 Multigigabit-48UXM
• Block diagram X86 1.8-GHz
Stackwise 480 quad-core CPU

FPGA
ASIC 0 Packet buffer (16 MB) ASIC 1 Packet buffer (16 MB)
DRAM – 8 GB
Flash
Forwarding controller Forwarding controller Forwarding controller Forwarding controller 16 GB

Reassembly Rewrite Reassembly Rewrite Reassembly Rewrite Reassembly Rewrite USB 2.0
crypto crypto crypto crypto
Ingress Egress Ingress Egress Ingress Egress Ingress Egress USB 3.0
FIFO FIFO FIFO FIFO FIFO FIFO FIFO FIFO
Core 1 Core 0 Core 1 Core 0 Mgmt Console

Network interface Network interface

10Gx4/40Gx1
10Gx4 10Gx4/40Gx1
10Gx4 10Gx4
2.5G x4 2.5G x4

MGig 2.5G 2.5G 2.5G 2.5G 2.5G MGig 2.5G 2.5G 2.5G 2.5G MGig
40G PHY 40G PHY
PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY
0 1
1 0 1 2 3 4 2 0 1 2 3 3
TX 0-3 TXI 4-7 TX 8-12 TX 0-3 TXI 4-7 TX 8-12 TX 0-7 TX 0-7

Cage 1 Cage 1 Cage 1 Cage 2 Cage 1 Cage 1 Cage 2

37-40 1-18 41-44 19-36 45-48 1-4 1-4

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
 Cisco Catalyst 9300 Multigigabit-48UN
• Block diagram X86 1.8-GHz
Stackwise 480 quad-core CPU

FPGA
ASIC 0 Packet buffer (16 MB) ASIC 1 Packet buffer (16 MB)
DRAM – 8 GB
Flash
Forwarding controller Forwarding controller Forwarding controller Forwarding controller 16 GB

Reassembly Rewrite Reassembly Rewrite Reassembly Rewrite Reassembly Rewrite USB 2.0
crypto crypto crypto crypto
Ingress Egress Ingress Egress Ingress Egress Ingress Egress USB 3.0
FIFO FIFO FIFO FIFO FIFO FIFO FIFO FIFO
Core 1 Core 0 Core 1 Core 0 Mgmt Console

Network interface Network interface

10Gx4/40Gx1
5G x4 10Gx4/40Gx1
5G x4

Mgig Mgig Mgig Mgig Mgig Mgig Mgig Mgig Mgig Mgig Mgig Mgig 40G 40G
PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY
0 1 2 3 4 5 0 1 2 3 4 5 0 1

TX 0-3 TX 0-3

Cage 1 Cage 2 Cage 1 Cage 2

1-24 25-48 1-4 1-4

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Higher Scale with 9300
Forwarding Resources Feature Resources

• MAC: 32K
Lookup
Table
Lookup
Table
Lookup
Table
Lookup
Table
• Security ACL: 5k
Lookup Lookup Lookup Lookup
Table Table Table Table

• Host Route: 24k • QoS ACL: 5k

•
Lookup Lookup

IGMP Groups: 8k
Table Table
Lookup
Table
Lookup
Table • Service ACL: 4k
Lookup
Table
Lookup
Table
Lookup
Table
Lookup
Table
• PBR
• LPM Route: 8k
Lookup Lookup Lookup Lookup Lookup
• Netflow ACL
Lookup Lookup Lookup
Table Table Table Table Table • SPAN Table
Multicast Route: 8k
Table Table
• • MACsec
• CoPP
• SGT: 8k
Lookup
Table
Lookup
Table
Lookup
Table
Lookup
Table
Lookup
Table •
Lookup
Tunnel
Table
Lookup
Table
Lookup
Table
• LISP

Netflow Entries: 64k per ASIC

Lookup Lookup
Netflow
Lookup Lookup Lookup Lookup Lookup Lookup
Table Table Table Table Table Table Table Table

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Stackwise-480 &
Stack Power
How many can I stack together?
Modular uplink models Fixed uplink models
C9300 SKUs C9300L SKUs

8 switches
8 switches
Stacking supported among C9300 SKUs only Stacking supported among C9300L SKUs only

Mixed stacking is not supported between C9300 and C9300L SKUs

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Cisco Catalyst 9300 Series Switches
The stack ring – StackWise-480 on C9300 SKUs

• 6 rings in total
• 3 rings go east
• 3 rings go west
• Each ring is 40 Gbps
• 240 Gbps unidirectional
Stack interface • Spatial reuse = 480 Gbps
of doppler

Assuming 4x 24-port Cisco® Catalyst® 9300 Series modular uplink models

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Cisco Catalyst 9300 Series Switches
The stack ring – StackWise-320 on C9300L SKUs

• 4 rings in total
• 2 rings go east
• 2 rings go west
• Each ring is 40 Gbps
• 160 Gbps bidirectional
• Spatial reuse = 320 Gbps
ASIC stack interface

Assuming 4x 24-port Cisco® Catalyst® 9300 Series fixed uplink models

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
 Unicast Packet Path
2
4
3
1
Destination Stripping
Packet travels ½ the rings
Taken out of stack by
destination

Creating
Assuming Segments
4 x 24-port
9300 Switches Re-ordering
segments

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
 Unicast Packet Path on the Stack Ring
Multiple Communications – Spatial Reuse
2
4
3
1
Destination Stripping
Packet travels ½ the
rings
Taken out of stack by
Assuming destination
4 x 24-port
9300 Switches

4
2
3
1

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
 Multicast Packet Path on the Stack Ring
4
2
3
1
Source Stripping
Packet travels the full rings
Taken out by source, when
packet reach back

Assuming
4 x 24-port
9300 Switches

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
 StackPower – overview
“Zero-footprint” redundant power system (RPS) deployment
• Provides RPS functionality with zero
RPS footprint
• Pay-as-you-grow architecture – similar
to the data stack
• 1+N redundancy with inline power
• Up to 4 switches in a StackPower ring
• Multiple StackPower possible within one
data stack
• Up to 8 switches in a star topology with
an expandable power system (XPS)
StackPower is not supported on C9300L SKUs

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
Power Redundancy Options
Zero Footprint RPS OR XPS
Day 1

eXpandable Power System (XPS)

StackPower - Zero Footprint RPS

Stack of 4 switches Stack of 8 switches

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
How StackPower Works?
StackPower
715 W
• Pools Power from All PS

• All Switches in StackPower share the

available Power in Pool
DataStac 715 W 1100W
k • Each Switch is given their Minimum
Power Budget
1100
715 W
W
715
W

Total Input Power 2530W

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
 Power Budget Modes

1100 1100
715 W 715 W
W W
715 715
W W

2530W – 30W 1430W – 30W

Power Sharing Mode Redundant Mode

• The Default Mode • User Configurable

• Sum of All PS – 30~60W • Sum of All PS – Largest PS - 30~60W
Global StackPower Reserve = 30W

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
 Power Priority
Load Shedding
• Standalone Mode • Stack Mode

Low Priority
Low Priority

Load Shedding Based on configured

priority Load Shedding Based on configured
priority
1. Low Priority Ports
2. High Priority Ports 1. Low Priority Ports
2. High Priority Ports
3. Switch Priority – Highest Priority
#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Catalyst
9200
Catalyst 9200 Series

In-built Memory
Built in External Storage 4GB/2GB Memory Unmatched Full POE+
RFID (USB 2.0) Embedded CPU 4GB Flash Resiliency – Perpetual/Fast POE+

Flexible Fast and Light ASIC Most Dense downlink offering Modular Or
UADP 2.0 mini 24 x 1Gig, 48 x 1Gig Fixed Uplink offering
4 x 1Gig, 4 x 10Gig

Right Sized Switching for simple Branch Deployments

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
Catalyst 9200 Series switching 1G Model SKUs
Catalyst 9200 Series switching SKUs

9200 (Modular Uplinks and Fans) 9200L (Fixed Uplinks and Fans)

24X1G Ports
Data 24X1G Ports Data
48X1G Ports 48X1G Ports

24X1G Ports 24X1G Ports

POE+ POE+
48X1G Ports 48X1G Ports

4 x 1G and 4 x 10G Uplinks 4 x 1G and 4 x 10G Uplinks

Modular Power Supplies available on all the SKUs

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
Modular uplink options on Catalyst 9200
Series switches

• 4 x 1 Gig • 4 x 10 Gig
• SFP Transceivers • SFP/SFP + Transceivers
• Supported on all modular SKUs • Supported on all modular SKUs*

Modular uplinks supported on Catalyst 9200 Series modular SKUs

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
Resilient power supplies

Silver Rated (80% efficiency) Platinum Rated (90% efficiency)

600WAC 1000WAC
Supported only on 24 Supported only on 48
125WAC Port POE+ SKUs Port POE+ SKUs
Supported only on 1G Data SKUs
Load sharing (1+1) mode supported for PoE+ SKUs

Power Supplies are Field Replaceable Units

Redundant Power Supply should be identical

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
StackWise-160/80 with SSO

Up to 8
member stack

Stacking Kit Ordered

• StackWise-160 supported on all modular Catalyst 9200 Series Separately

switching models
• StackWise-80 supported on all fixed Catalyst 9200 Series
switching models Stack Adapters Stacking Cable

• Same Cisco IOS XE and license required on all members

Stacking cable comes with three options:
50cm, 1m, 3m
*Mix stacking not supported between StackWise-160 and StackWise-80
#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
 The stack ring – StackWise 160/80
ASIC Stack Interface

• 2 rings in total
• 1 ring goes East
• 1 ring goes West
• Each ring is 40/20 Gbps
• 80/40 Gbps bi-direction
• Spatial Reuse= 160/80 Gbps

ASIC Stack Interface

Assuming 4 x 24-port Catalyst 9200 Series modular switches

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
Lookup tables
Forwarding Resources Feature Resources

Modular SKUs Fixed SKUs Modular SKUs Fixed SKUs

MAC 32k 16k Security ACL 1k 1k

• PACL
Host Route 10k 8k • VACL

IGMP Groups 1k 1k • RACL

Indirect Route 4k 3k QoS ACL 1k 1k

Multicast Route 1k 1k
Netflow ACEs 128 128
SGT 2k 2k

Netflow Netflow Entries: 16k per ASIC

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
 Fixed Access Positioning EVPN BGP
GRE

MPLS Stackwise-
1/10G 480
/40G ETA
L2 Small
Scale L2 + L3 Stack-
UADP
2.0 Power Guest
Shell
Stackwise- Higher Scale Hot
1/10G POE+
160/80
X86 CPU
Patching
and Buffer

App- POE+/ Multigig

Cold Hosting UPOE GIR abit
Patching

Catalyst 9200 Catalyst 9300

Branch Office and Small Campus Small to Large Campus

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
Modular Platform
- Catalyst 9400
Catalyst 9400
480G BW
per slot
Redundancy
is now
Table-stakes

5KW PoE
Per slot

4-Slot 7-Slot 10-Slot

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
Chassis

4 – Slot 7 - Slot 10 - Slot

Supervisor 2 (Redundant)
Line Cards 2 5 8
Ports 96x 10/100/1000 240x 10/100/1000 384x 10/100/1000
48 mGig; 56 SFP/SFP+ 120 mGig; 128 SFP/SFP+ 192 mGig; 200 SFP/SFP+
2x QSFP+ 2x QSFP+ 2x QSFP+
Dimension W:17.5”; D:16.25”; H:6RU W:17.5”; D:16.25”; H:10RU W: 17.5”; D:16.25”; H: 13RU
BW per LC Slot 480G 480G 480G
BW between Sup Slots 720G High Density 10G Ports, 100G Uplinks
Power Supply 4 PS (N+1 and N+N) 8 PS (N+1 and N+N) 8 PS (N+1 and N+N)
PoE per slot 4,800W Ready for future higher power PoE devices
Cooling Side to Side (Front-to-Back for PS)

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
Sup-1 - Overview
Line Card Slot BW:
720G LCs/Uplinks 7 Slot: 80G
10 Slot: 80G (> 150Byte)

UADP 2.0 XL ASICs

2.4Ghz Quad Core x86

M.2 SATA SSD CPU
(Optional: Upto 1TB)
16G DRAM and 16G Flash

USB 2.0/3.0

MACSec256 Uplinks:
8x 10G / 2x 40G

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
C9400-SUP-1XL - Overview Line Card Slot BW:
4 Slot: 240G (>150Byte)
720G LCs/Uplinks 7 Slot: 120G (>150Byte)
10 Slot: 80G (> 150Byte)

Templates: Distribution,
UADP 2.0 XL ASICs Core, SD Boarder, NAT

M.2 SATA SSD

(Optional: Upto 1TB) 16G DRAM and 16G Flash

2.4Ghz Quad Core x86

USB 2.0/3.0 CPU

MACSec256 Uplinks:
8x 10G / 2x 40G

Optimized for Distribution/Core deployment

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
C9400-SUP-1XL-Y - Overview Line Card Slot BW:
4 Slot: 240G (>150Byte)
720G LCs/Uplinks 7 Slot: 120G (>150Byte)
10 Slot: 80G (> 150Byte)

Templates: Distribution,
UADP 2.0 XL ASICs Core, SD Boarder, NAT

M.2 SATA SSD

(Optional: Upto 1TB) 16G DRAM and 16G Flash

2.4Ghz Quad Core x86

USB 2.0/3.0 CPU

MACSec256 Uplinks:
2x 25G / 8x 10G / 2x 40G

Optimized for Core deployment

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
 Supervisors
Line Card Slot BW: Template: Uplinks:

4 Slot: 80G
7 Slot: 80G Access
C9400-SUP-1
10 Slot: 80G
1G
10G
40G

C9400-SUP-1XL
Access,
4 Slot: 240G
Core,
7 Slot: 120G
SD Boarder,
10 Slot: 80G
NAT 1G
10G
C9400-SUP-1XL-Y
25G
40G

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
Line Cards - Copper
RJ45 (Data)
48x 10/100/1000
TrustSec and MACSec(256)

C9400-LC-48T

RJ45 (UPoE)
48x 10/100/1000
PoE/PoE+; PoE/PoE+/UPoE
TrustSec and MACSec(256)

C9400-LC-48P C9400-LC-48U

RJ45 (mGig)
24x 10/100/1000 + 24x
100/1G/2.5G/5G/10G
PoE/PoE+/UPoE
TrustSec and MACSec(256)
C9400-LC-48UX

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
Line Cards - Fiber

SFP (1G)
24x or 48x 100/1000
TrustSec and MACSec(256)
C9400-LC-24S C9400-LC-48S

Fiber (1G/10G)
24x 1G/10G
TrustSec and MACsec(256)

C9400-LC-24XS

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
Power Supplies
• Modular Design: 4 PS for 4 slot chassis; 8 PS for 7 and 10 slot chassis
• Shared: Power for both Data and Inline Power
• Platinum PS: 90%+ efficiency
• PS:
• 3200W AC PS With 240V input. (1570W with 120V input. 16A input)
• 2100W AC PS With 240V input. (940W with 120V input. 10.4A input)
• 3200W DC PS With -40V to -72V input.

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
Power
Normal PS failure

Combined
(Default)
Load sharing on all PSs Load sharing on functional PSs

Redundant

Load sharing on active PSs Standby PS becomes active

Standby PS in output disabled System enters alarm state

Failed PS ACTIVE STANDBY

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
 Power Redundancy: N+N and N+1
• Default active is PS1-4 and standby is PS5-
8 (C9404R: Active: PS1-2; Standby PS3-4) SW(config)#power redundancy-mode redundant ?
N+N Redundant N+N (N is active, N is standby)
• Standby power slots are configurable N+1 Redundant N+N (N is active, 1 is standby)
SW(config)#power redundancy-mode redundant N+1 ?
<1-8> standby slot in N+N mode
SWR(config)#

• Default active is PS1-7 and standby is SW(config)#power redundancy-mode redundant ?

PS8 (C9404R: Active: PS1-3; Standby N+N Redundant N+N (N is active, N is standby)
PS4) N+1 Redundant N+N (N is active, 1 is standby)
SW(config)#power redundancy-mode redundant N+1 ?
• Standby power slot is configurable
<1-8> standby slot in N+1 mode
SWR(config)#

ACTIVE STANDBY

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
 Power Priority
• All components in the system are
assigned with power priority level
• Supervisors and Fan Tray has the same
highest priority level
• Lower slot# has the higher power priority
level by default if “power supply autoLC
shutdown” is configured
Highest
Priority Configurable
• Configurable power priority for line card Priority
slots
C94(config)#power supply autoLC priority ?
<1-7> Physical slot number
<cr>
C94(config)

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
 Fan Tray
• Redundant - N+1 fan

• Flexible Service - fan tray can be replaced

from the portside or the back
• Efficient - Variable speed per fan depends
on the load, temperature and altitudes
(=>lower noise).
• Air flow - Side to side air flow

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
Architecture
Centralized Architecture
Supervisor
Forwarding Control Plane
Features Open Container
Embedded Controllers IOS-XE HA communication

Passive BackPlane
Upto 480G BW per slot

Line Card Line Card Line Card

…
STUB STUB STUB
ASIC ASIC ASIC

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
Sup-1/Sup-1XL/Sup-1XL-Y Block Diagram
Switch Backplane

240G 24x 24x 16x 40G SupIO

SLI SLI SLI PCIe Ethernet
8x (I2C)
SLI
UADP2.0 XL UADP2.0 XL UADP2.0 XL
#1 #2 #3 Quad Core CPU
(Control Traffic, Containers
for APPs)
720 Gbps

M.2 SATA
ASIC # 4-Slot 7-Slot 10-Slot SDRAM
(optional)
UADP #1 Slot 1 Slot 2 and 7 Slot1, 9, and 10

UADP #2 Slot 4 Slot 1 and 5 Slot 2, 3 and 4

4x10G / 4x10G /
UADP #3 Uplinks Slot 6 and Slot 7, 8, and 1x 25G / 1x 25G / Console/
USB
Uplinks Uplinks 40G 40G Mgmt

SLI – Switch Link Interfaces

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
Port to ASIC Mapping
switch#show platform software fed active ifm mappings
Interface IF_ID Inst Asic Core Port SubPort Mac Cntx LPN GPN Type Active
GigabitEthernet1/0/1 0x7 2 1 0 0 0 4 4 1 101 NIF Y
GigabitEthernet1/0/2 0x8 2 1 0 1 1 4 4 2 102 NIF Y
GigabitEthernet1/0/3 0xf5 2 1 0 2 2 4 4 3 103 NIF Y
GigabitEthernet1/0/4 0xa 2 1 0 3 3 4 4 4 104 NIF Y
GigabitEthernet1/0/5 0xb 2 1 0 4 4 4 4 5 105 NIF Y
GigabitEthernet1/0/6 0xc 2 1 0 5 5 4 4 6 106 NIF Y
GigabitEthernet1/0/7 0xd 2 1 0 6 6 4 4 7 107 NIF Y
GigabitEthernet1/0/8 0xe 2 1 0 7 7 4 4 8 108 NIF Y
GigabitEthernet1/0/9 0xf 2 1 0 8 0 0 0 9 109 NIF Y
GigabitEthernet1/0/10 0x10 2 1 0 9 1 0 0 10 110 NIF Y
GigabitEthernet1/0/11 0x11 2 1 0 10 2 0 0 11 111 NIF Y
GigabitEthernet1/0/12 0x12 2 1 0 11 3 0 0 12 112 NIF Y
GigabitEthernet1/0/13 0x13 2 1 0 12 4 0 0 13 113 NIF Y
GigabitEthernet1/0/14 0x14 2 1 0 13 5 0 0 14 114 NIF Y
GigabitEthernet1/0/15 0x15 2 1 0 14 6 0 0 15 115 NIF Y
GigabitEthernet1/0/16 0x16 2 1 0 15 7 0 0 16 116 NIF Y
GigabitEthernet1/0/17 0x17 2 1 0 16 0 5 5 17 117 NIF Y
GigabitEthernet1/0/18 0x18 2 1 0 17 1 5 5 18 118 NIF Y
GigabitEthernet1/0/19 0x19 2 1 0 18 2 5 5 19 119 NIF Y
GigabitEthernet1/0/20 0x1a 2 1 0 19 3 5 5 20 120 NIF Y
GigabitEthernet1/0/21 0x1b 2 1 0 20 4 5 5 21 121 NIF Y
GigabitEthernet1/0/22 0x1c 2 1 0 21 5 5 5 22 122 NIF Y
GigabitEthernet1/0/23 0x1d 2 1 0 22 6 5 5 23 123 NIF Y
GigabitEthernet1/0/24 0x1e 2 1 0 23 7 5 5 24 124 NIF Y
GigabitEthernet1/0/25 0x1f 2 1 0 24 0 1 1 25 125 NIF Y
<SNIP>

switch#

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
Sup-1/Sup-1XL Uplink - Single Sup

Default Mode (8x 10GE)

2x 40G
interface FortyGigabitEthernet<slot>/0/[9-10]
enable

Mix Mode (4x 10GE + 1x 40G)

interface FortyGigabitEthernet<slot>/0/10
enable

Mix Mode (4x 10GE + 1x 40G)

interface FortyGigabitEthernet<slot>/0/9
enable

Active Disabled

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 97
Sup-1/Sup-1XL Dual Sups - Uplink Redundancy

Default Mode (8x 10GE)

40G Uplink Mode

interface FortyGigabitEthernet<slot>/0/9
enable

Active Disabled
#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
Sup-1/Sup-1XL Dual Sups - Mix Uplink Mode

Mix Mode (4x 10GE + 1x 40GE)

interface FortyGigabitEthernet<slot>/0/9
enable

Mix Mode (1x 40GE + 4x 10GE)

interface FortyGigabitEthernet<slot>/0/9
enable

Active Disabled
#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 99
C9400-Sup-1XL-Y (25G Uplinks)
Tw <slot#>/0/1 Tw <slot#>/0/5

Ten <slot#>/0/1 Ten <slot#>/0/5

Ten <slot#>/0/2 Ten <slot#>/0/6 Four <slot#>/0/9
Ten <slot#>/0/3 Ten <slot#>/0/7 Four <slot#>/0/10
Ten <slot#>/0/4 Ten <slot#>/0/8
If port 9 (and/or 10) is enabled, port 1-4 (and/or 5-8) are disabled.
=> 40G ports are enabled.

If port 9 (and/or 10) is disabled (which is default)

If Tw <slot#>/0/1(and/or 5) is enabled, Ten <slot#>/0/1 – 4 (and/or 5-8) are disabled.
If Tw <slot#>/0/1(and/or 5) is disabled, Ten <slot#>/0/1 – 4 (and/or 5-8) are enabled. (which is default)

Note:
1. TW ports are not auto-sense with speed for 10G/1G, manual speed configuration is needed. (same for dual-rate SFP28)

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 100
 48x1G RJ45 Line Card (PoE+/UPoE)
Switch Backplane

6x
SLI

IEEE
Management 1588
Stub ASIC Sub-System

10G Power
(USGMII) Sub-system

Octal PHY Octal PHY Octal PHY Octal PHY Octal PHY Octal PHY

RJ45 RJ45 RJ45 RJ45 RJ45 RJ45 PoE

1-8 9-16 17-24 25-32 33-40 41-48 Sub-System
Speed10/100/1000
Line Rate on all Ports; UPoE on all Ports
#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 101
 48x1G Line Card (RJ45 Data or SFP)
Switch Backplane

6x
SLI

IEEE
Management 1588
Stub ASIC Sub-System

10G
(USGMII)

Octal PHY Octal PHY Octal PHY Octal PHY Octal PHY Octal PHY

RJ45 RJ45 RJ45 RJ45 RJ45 RJ45

1-8 9-16 17-24 25-32 33-40 41-48
Speed10/100/1000
Line Rate on all Ports
#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 102
24x 1/10G SFP/SFP+ Line Card
Switch Backplane

6x 6x 6x 6x
SLI SLI SLI SLI
IEEE
Stub ASIC Stub ASIC Stub ASIC Stub ASIC Management 1588
Sub-System

6x 10G 6x 10G 6x 10G 6x 10G

(USXGMII)

PHY PHY PHY PHY

SFP/SFP+ SFP/SFP+ SFP/SFP+ SFP/SFP+

1-6 7-12 13-18 19-24

Sup-1: 80G with C9410R, C9407R and C9404R.

Sup-1XL: 80G with C9410R; 120G with C9407R; 240G with C9404R.

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 103
 C9400-LC-24XS Port-Group
With XL Supervisors

7 Slot Chassis: 12 Port-Group 2 Ports in a port-group

1 2 3 4 5 6 7 8 9 10 11 12

10 Slot Chassis: 8 Port-Group 3 Ports in a port-group

1 2 3 4 5 6 7 8

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 104
C9400-LC-24XS Port-Group – 7 Slot Chassis
R4-C94-2041#show platform hardware iomd 5/0 portgroups
Port Interface Status Interface Group
Max
Group Bandwith Bandwidth
• Bandwidth shared within port-group
1 TenGigabitEthernet5/0/1 up 10G
• 12 port-group in the 7 slot chassis 1 TenGigabitEthernet5/0/2 down 10G 10G

2 TenGigabitEthernet5/0/3 up 10G
• Modes: dynamic, performance and 2 TenGigabitEthernet5/0/4 down 10G 10G
static
3 TenGigabitEthernet5/0/5 up 10G
3 TenGigabitEthernet5/0/6 down 10G 10G
• For 10G line rate performance:
• Configure: “hw-module 4 TenGigabitEthernet5/0/7 up 10G
subslot <slot#/0> mode 4 TenGigabitEthernet5/0/8 down 10G 10G
<SNIP>
performance” 11 TenGigabitEthernet5/0/21 up 10G
• 8 Port @ line-rate, other ports 11 TenGigabitEthernet5/0/22 down 10G 10G
are disabled
12 TenGigabitEthernet5/0/23 up 10G
12 TenGigabitEthernet5/0/24 down 10G 10G

R4-C94-2041#show

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 105
C9400-LC-24XS Port-Group – 10 Slot Chassis
mac1#show platform hardware iomd 10/0 portgroups
Port Interface Status Interface Group Max
Group Bandwith Bandwidth

• Bandwidth shared within port-group 1 TenGigabitEthernet10/0/1 admindown 10G

1 TenGigabitEthernet10/0/2 admindown 10G
1 TenGigabitEthernet10/0/3 admindown 10G 10G
• 8 port-group in the 10 slot chassis
2 TenGigabitEthernet10/0/4 admindown 10G
2 TenGigabitEthernet10/0/5 admindown 10G
• Modes: dynamic, performance and 2 TenGigabitEthernet10/0/6 admindown 10G 10G
static
3 TenGigabitEthernet10/0/7 admindown 10G
3 TenGigabitEthernet10/0/8 admindown 10G
• For 10Gline rate performance: 3 TenGigabitEthernet10/0/9 admindown 10G 10G
• Configure: “hw-module <SNIP>
subslot <slot#/0> mode 7 TenGigabitEthernet10/0/19 admindown 10G
7 TenGigabitEthernet10/0/20 admindown 10G
performance” 7 TenGigabitEthernet10/0/21 admindown 10G 10G
• 5 Port @ line-rate, other ports
are disabled 8 TenGigabitEthernet10/0/22 admindown 10G
8 TenGigabitEthernet10/0/23 down 10G
8 TenGigabitEthernet10/0/24 admindown 10G 10G

mac1#

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 106
 mGig RJ45 Line Card
Switch Backplane

6x 6x 6x 6x
SLI SLI SLI SLI
IEEE
Stub ASIC Stub ASIC Stub ASIC Stub ASIC Management 1588
Sub-System

2x 6x 2x 6x 2x 6x 2x 6x

Power
Sub-system
4x Octal PHY
2x Quad mGig PHY 2x Quad mGig PHY 2x Quad mGig PHY
(use 6 out of 8)

RJ45 RJ45 RJ45 RJ45 PoE

1-24 25-32 33-40 40-48 Sub-System
Speed10/100/1000 Speed100/1G/2.5G/5G/10G

Sup-1: 80G with C9410R, C9407R, and C9404R

Sup-1XL/1XL-Y: 80G with C9410R; 120G with C9407R; 240G with C9404R

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 107
 C9400-LC-48UX Port-Group
With XL Supervisors

7 Slot Chassis: 12 Port-Group 4 Ports in a port-group

1 2 3 4 5 6 7 8 9 10 11 12 2 3 1 5 6 4 8 9 7 11 12 10

10 Slot Chassis: 8 Port-Group 6 Ports in a port-group

1 3 5 7 1 3 5 7
1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8
2 4 6 8 2 4 6 8

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 108
C9400-LC-48UX Port-Group – 7 Slot Chassis
R4-C94-2041#show platform hardware iomd 6/0 portgroups
Port Interface Status Interface Group
Max
Group Bandwith Bandwidth
• Bandwidth shared within port-group
1 GigabitEthernet6/0/1 up 1G
1 GigabitEthernet6/0/2 up 1G
• 12 port-group in the 7 slot chassis 1 TenGigabitEthernet6/0/29 up 10G
1 TenGigabitEthernet6/0/30 up 10G 10G

• Modes: dynamic, performance and 2 GigabitEthernet6/0/3 up 1G

static 2 GigabitEthernet6/0/4 up 1G
2 TenGigabitEthernet6/0/25 up 10G
2 TenGigabitEthernet6/0/26 up 10G 10G
• For 10G line rate performance: <SNIP>
• Configure: “hw-module 11 GigabitEthernet6/0/21 down 1G
subslot <slot#/0> mode 11 GigabitEthernet6/0/22 down 1G
11 TenGigabitEthernet6/0/43 up 10G
performance” 11 TenGigabitEthernet6/0/44 up 10G 10G
• 8 Port @ line-rate, other ports
are disabled 12 GigabitEthernet6/0/23 down 1G
12 GigabitEthernet6/0/24 down 1G
12 TenGigabitEthernet6/0/45 up 10G
12 TenGigabitEthernet6/0/46 up 10G 10G

R4-C94-2041#

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 109
C9400-LC-48UX Port-Group – 10 Slot Chassis
mac1#show platform hardware iomd 9/0 portgroups
Port Interface Status Interface Group Max
Group Bandwith Bandwidth

• Bandwidth shared within port-group

1 GigabitEthernet9/0/1 admindown 1G
1 GigabitEthernet9/0/2 admindown 1G
• 8 port-group in the 10 slot chassis 1 GigabitEthernet9/0/3 admindown 1G
1 TenGigabitEthernet9/0/25 admindown 10G
1 TenGigabitEthernet9/0/26 admindown 10G
• Modes: dynamic, performance and 1 TenGigabitEthernet9/0/27 admindown 10G 10G
static
2 GigabitEthernet9/0/4 admindown 1G
2 GigabitEthernet9/0/5 admindown 1G
• For 10Gline rate performance: 2 GigabitEthernet9/0/6 admindown 1G
• Configure: “hw-module 2 TenGigabitEthernet9/0/28 admindown 10G
subslot <slot#/0> mode 2 TenGigabitEthernet9/0/29 admindown 10G
2 TenGigabitEthernet9/0/30 admindown 10G 10G
performance” <SNIP>
• 5 Port @ line-rate, other ports 8 GigabitEthernet9/0/22 admindown 1G
are disabled 8 GigabitEthernet9/0/23 admindown 1G
8 GigabitEthernet9/0/24 admindown 1G
8 TenGigabitEthernet9/0/46 admindown 10G
8 TenGigabitEthernet9/0/47 admindown 10G
8 TenGigabitEthernet9/0/48 admindown 10G 10G

mac1#

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 110
Flex Tables
Forwarding Resources Feature Resources

• MAC: 64K
Lookup Lookup Lookup Lookup • Security ACL: 18K
Lookup Lookup Lookup Lookup
Table
• Host Route: 48K –
Table Table Table Table Table Table Table
• QoS ACL: 18K
112K
Lookup
Table
Lookup
Table
Lookup
Table
Lookup
Table • Service ACL: 18K
Lookup
Table
Lookup
Table
Lookup
Table
Lookup
Table

• IGMP Groups: 16K •

•
PBR/NAT
Netflow ACL
• LPM Route: 64K
Lookup Lookup Lookup Lookup Lookup Lookup Lookup Lookup
Table Table Table Table Table • SPANTable Table Table
• MACsec
• Multicast Route: 16K
Lookup Lookup Lookup Lookup Lookup
• CoPP
Lookup Lookup Lookup

• SGT: 16K
Table Table Table Table Table •
•
Tunnel
LISP
Table Table Table

Netflow Entries: 128K per ASIC

Lookup Lookup
Netflow
Lookup Lookup Lookup Lookup Lookup Lookup
Table Table Table Table Table Table Table Table

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 111
Catalyst 9400 Templates (For Sup-1XL/1XL-Y)
16.6 – 16.8 16.9 and beyond
Access Access Core SDA NAT
LPM 64K 64K 64K 64K 64K
Host 48K 48K 32K 80K 48K
Layer2 Multicast 16K 16K 16K 16K 16K
Layer3 Multicast 16K 16K 32K 16K 32K
MAC Address 64K 64K 16K 16K 16K
SGT 8K 8K 8K 8K 8K
Flexible Netflow 128K/ASIC 128K/ASIC 128K/ASIC 128K/ASIC 128K/ASIC
Security ACL 18K 18K 18K 18K 18K
QoS ACL 18K 18K 18K 18K 3K
PBR/NAT 2K 2K 2K 2K 16K
Tunnel 1K 1K 1K 1K 1K
LISP 1K 1K 1K 1K 1K
MPLS L3VPN VRF 256 256 256 256
MPLS Label 8K 16K 24K 16K
MPLS L3VPN Routes VRF 16K 32K 32K 32K
N/A
MPLS L3VPN Routes Prefix 4K 4K 4K 4K
MVPN MDT Tunnels 256 1K 1K 1K
L2VPN EOMPLS Attachment 256 1K 1K 1K

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 112
#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Catalyst 9500
Cisco Catalyst 9500 Series
New generation of purpose-built fixed core/aggregation

UADP 2.0 UADP 3.0

Cisco Catalyst Cisco Catalyst

9500-16X 9500-24Y4C
Cisco Catalyst Cisco Catalyst
9500-40X 9500-48Y4C
Cisco Catalyst Cisco Catalyst
9500-24Q 9500-32QC
Cisco Catalyst Cisco Catalyst
9500-12Q 9500-32C

Modular Storage for

Modular fans Modular uplinks
power supplies application hosting

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 115
Cisco Catalyst 9500 Series
New generation of purpose-built fixed mid-range core/aggregation

Cisco Catalyst
9500-16X
Cisco Catalyst
9500-40X

Cisco Catalyst
9500-24Q
Cisco Catalyst
9500-12Q

UADP 2.0 XL

Industry’s first 40G enterprise switch

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 116
Cisco Catalyst 9500
High-level overview
2.4-GHz quad-core x86 CPU
Built-in RFID UADP 2.0 XL ASIC Every port 40G,10G*,
USB 2.0 flash drive 16 GB of DDR4 DRAM StackWise Virtual
(passive) 240 Gb Bandwidth and 1G* capable
32 MB / ASIC packet buffer

C9500 24Q
24 x 40G

C9500-12Q
12 x 40G

C9500-40X
40 x 1/10G

C9500-16X
16 x 1/10G

* With QSA Adaptor #CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 117
Cisco Catalyst 9500
Network Modules

C9500-NM-2Q C9500-NM-8X
Cisco Catalyst 9500 Series Network Module Cisco Catalyst 9500 Series Network Module
2-port 40 Gigabit Ethernet with QSFP+ 8-port 1/10 Gigabit Ethernet with SFP/SFP+

• Uplink Modules supported on C9500-40X and C9500-16X

• OIR Supported on all Uplink Modules
• Breakout support on C9500-NM-2Q

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 118
Redundant Power Supplies and Fans

Redundant 1+1 Redundant N+1

120 GB
950W AC / DC power supplies fans
USB 3.0storage

Power supply highlights: Fan Highlights:

• Hot-swappable • Variable-speed high-efficiency fans

• AC/DC/mixed power supplies supported • Detect ambient temperature and adjust fan speeds

• Maximum output 12V/950W at 220V/110V AC input • Individual Fan are OIR capable up to 120 secs

• More than 90% power efficiency at 50% to 100% of load • Front-to-back airflow

• Redundant load sharing (1+1) mode only • Can still operate with individual fan tray failure

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 119
Cisco Catalyst 9500-16X
Block diagram
X86 2.4-GHz
quad-core CPU
Packet buffer (16 MB) Packet buffer (16 MB)
FPGA
DRAM – 16 GB
Forwarding controller Forwarding controller
Flash
16 GB
Reassembly Rewrite Reassembly Rewrite
crypto crypto

Ingress FIFO Egress FIFO Ingress FIFO Egress FIFO

USB 2.0
Core 1 Core 0

Network interface Network interface Mgmt Console

ASIC-0

10Gx8 10Gx8 10Gx8

PHY PHY PHY

SFP Cage SFP/QSFP Cage

SFP 1-8 /
SFP 1-16
QSFP 1-2

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 120
 Cisco Catalyst 9500-40X
Block diagram
X86 2.4-GHz
quad-core CPU
720 Gbps ASIC Interconnect
FPGA
DRAM – 16 GB
Packet buffer (16 MB) Packet buffer (16 MB) Packet buffer (16 MB) Packet buffer (16 MB) Flash
16 GB

Forwarding controller Forwarding controller Forwarding controller Forwarding controller

Reassembly Rewrite Reassembly Rewrite Reassembly Rewrite Reassembly Rewrite

crypto crypto crypto crypto
Ingress
Egress FIFO
Ingress
Egress FIFO
Ingress Ingress USB 2.0
Egress FIFO Egress FIFO
FIFO FIFO FIFO FIFO
ASIC-1
Core 1 Core 0 Core 1 Core 0
Mgmt Console
Network interface Network interface Network interface Network interface
ASIC-1 ASIC-0
10Gx8 10Gx8 10Gx8 10Gx8 10Gx8 10Gx8

PHY PHY PHY PHY PHY PHY

SFP Cage SFP Cage SFP/QSFP Cage

SFP 1-8 /
SFP 1-24 SFP 25-40
QSFP 1-2

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 121
 Cisco Catalyst 9500-12Q
Block diagram
X86 2.4-GHz
quad-core CPU
720 Gbps ASIC Interconnect
FPGA
DRAM – 16 GB
Packet buffer (16 MB) Packet buffer (16 MB) Packet buffer (16 MB) Packet buffer (16 MB) Flash
16 GB

Forwarding controller Forwarding controller Forwarding controller Forwarding controller

Reassembly Rewrite Reassembly Rewrite Reassembly Rewrite Reassembly Rewrite

crypto crypto crypto crypto
Ingress
Egress FIFO
Ingress
Egress FIFO
Ingress Ingress USB 2.0
Egress FIFO Egress FIFO
FIFO FIFO FIFO FIFO
ASIC-1 ASIC-0
Core 1 Core 0 Core 1 Core 0
Mgmt Console
Network interface Network interface Network interface Network interface
ASIC-1 ASIC-0

10Gx8 10Gx8 10Gx8 10Gx8 10Gx8 10Gx8

PHY PHY PHY PHY PHY PHY

QSFP Cage QSFP Cage

QSFP 1-6 QSFP 7-12

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 122
 Cisco Catalyst 9500-24Q
Block diagram
X86 2.4-GHz
quad-core CPU
1.44Tbps ASIC Interconnect
FPGA
DRAM – 16 GB
Flash
Core 0 Core 1 Core 0 Core 1 Core 0 Core 1 Core 0 Core 1
16 GB

ASIC-2 ASIC-1 ASIC-0 USB 2.0

ASIC-3 ASIC-2 ASIC-1 ASIC-0
Mgmt Console

10Gx8 10Gx8 10Gx8 10Gx8 10Gx8 10Gx8 10Gx8 10Gx8 10Gx8 10Gx8 10Gx8 10Gx8

PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY

QSFP Cage QSFP Cage QSFP Cage QSFP Cage

QSFP 1-6 QSFP 7-12 QSFP 13-18 QSFP 19-24

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 123
Cisco Catalyst 9500-24Q/12Q/40X/16X
Port-to-ASIC mapping

Command to verify the port-to-ASIC mapping:

show platform software fed switch active ifm mappings lpn

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 124
Cisco Catalyst 9500 Series
New generation of purpose-built fixed high-end core/aggregation

Cisco Catalyst
9500-24Y4C
Cisco Catalyst
9500-48Y4C

Cisco Catalyst
9500-32QC

Cisco Catalyst
9500-32C

UADP 3.0

Industry’s first 25G/100G enterprise switch

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 125
Cisco Catalyst 9500 100G/25G
High-level overview 2.4-GHz quad-core x86 CPU
16 GB of DDR4 DRAM Next-generation
Built-in RFID 16 GB flash Every port 100G, 40G,
USB 3.0 flash drive UADP 3.0 ASIC StackWise Virtual
(passive) 36 MB /ASIC unified packet buffer 10G* and 1G* capable
1.6Tb Bandwidth

C9500 – 32C
32 x 40/100G

C9500-32QC
16 x 100G (or)
32 x 40G

C9500-24Y4C
24 x 1/10/25G +
4 x 40/100G

C9500-48Y4C
48 x 1/10/25G +
4 x 40/100G

* With QSA adapter

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 126
Redundant Power Supplies and Fans
C9500-32C

Redundant 1+1 1600W AC/DC 5 standalone fans – 240-, 480-, or 960-GB

power supplies N+1 redundancy SATA SSD storage

Power supply highlights: Fan Highlights:

• Hot-swappable • 5 variable-speed high-efficiency fans at rear of chassis
• AC/DC/mixed power supplies supported • Detect ambient temperature and adjust fan speeds
• Maximum output is 1600W at 220V and 1000W at 110V • Fans are hot-swappable
• More than 90% power efficiency at 50% to 100% of load • Front-to-back airflow
• Redundant load sharing (1+1) mode only • Can still operate with one fan unit failure

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 127
Redundant Power Supplies and Fans
C9500-32QC, 24Y4C, 48Y4C

Redundant 1+1
Redundant 1+1 240-, 480-, or 960-GB
650W AC and 930W DC
fan tray SATA SSD storage
power supplies

Power supply highlights: Fan Highlights:

• Hot-swappable • Dual variable-speed high-efficiency fan trays
• AC/DC/mixed power supplies supported • Thermal sensor to detect ambient temperature and
adjust fan speeds
• Maximum output 12V/650W at 220V/110V AC Input
• Fan trays are hot-swappable
• More than 90% power efficiency at 50% to 100% of load
• Front-to-back airflow
• Redundant load sharing (1+1) mode only
• Can still operate with individual fan tray failure

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 128
Cisco Catalyst 9500-32QC
Configuration modes
24P 40G + 4P 100G – default configuration

40G 40G 40G 40G 40G 40G 40G 40G 40G 40G 40G 40G 100G 100G 100G 100G

40G 40G 40G 40G 40G 40G 40G 40G 40G 40G 40G 40G

16P
32P100G
40G

40G
100G 40G
100G 40G
100G 40G
100G 40G
100G 40G
100G 40G
100G 40G
100G 40G
100G 40G
100G 40G
100G 40G
100G 40G
100G 40G
100G 40G
100G 40G
100G

40G 40G 40G 40G 40G 40G 40G 40G 40G 40G 40G 40G 40G 40G 40G 40G

Mix Mode

100G 40G 100G 100G 40G 100G 100G 40G 100G 100G 100G 40G 100G 100G 40G 100G

40G 40G 40G 40G 40G

Note: Other configuration options are supported, including mix and match of speeds

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 129
 Cisco Catalyst 9500-32C
Block diagram
X86 2.4-GHz
400 Gbps 400 Gbps 400 Gbps 400 Gbps quad-core CPU

FPGA
ASIC 0 Packet buffer (36 MB) ASIC 1 Packet buffer (36 MB)
DRAM – 16 GB
Flash
Forwarding controller Forwarding controller Forwarding controller Forwarding controller 16 GB

Reassembly Rewrite Reassembly Rewrite Reassembly Rewrite Reassembly Rewrite USB 2.0
crypto crypto crypto crypto
Ingress Egress Ingress Egress Ingress Egress Ingress Egress USB 3.0
FIFO FIFO FIFO FIFO FIFO FIFO FIFO FIFO
Core 1 Core 0 Core 1 Core 0 Mgmt Console

Network interface Network interface

PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI
0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7

QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28
Cage 1 Cage 2 Cage 3 Cage 4 Cage 5 Cage 6 Cage 7 Cage 8 Cage 9 Cage 10 Cage 11 Cage 12 Cage 13 Cage 14 Cage 15 Cage 16

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 130
 Cisco Catalyst 9500-32QC
Block diagram
X86 2.4-GHz
Packet buffer (36 MB) quad-core CPU

Forwarding controller Forwarding controller FPGA

DRAM – 16 GB
Reassembly Rewrite Reassembly Rewrite Flash
crypto crypto 16 GB

Ingress FIFO Egress FIFO Ingress FIFO Egress FIFO

USB 2.0

Core 1 Core 0
USB 3.0
Network interface
Mgmt Console
NIF 58-61 NIF 32-35 NIF 28-31 NIF 0-3

PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI
0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7

QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28
Cage 1 Cage 2 Cage 3 Cage 4 Cage 5 Cage 6 Cage 7 Cage 8 Cage 9 Cage 10 Cage 11 Cage 12 Cage 13 Cage 14 Cage 15 Cage 16
1/ 3/ 5/ 7/ 9/ 11/ 13/ 15/ 17/ 19/ 21/ 23/ 25/ 27/ 29/ 31/
33 2 34 4 35 6 36 8 37 10 38 12 39 14 40 16 41 18 42 20 43 22 44 24 45 26 46 28 47 30 48 32

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 131
 Cisco Catalyst 9500-48Y4C
Block diagram
X86 2.4-GHz
Packet buffer (36 MB) quad-core CPU

Forwarding controller Forwarding controller FPGA

DRAM – 16 GB
Reassembly Rewrite Reassembly Rewrite Flash
crypto crypto 16 GB

Ingress FIFO Egress FIFO Ingress FIFO Egress FIFO

USB 2.0

Core 1 Core 0
USB 3.0
Network interface
Mgmt Console
10G/25Gx8 10G/25Gx8 20G/25Gx8 20G/25Gx8

PHY PHY PHY PHY PHY PHY PHY PHY

0 1 2 3 4 5 6 7

SFI 0-7 SFI 0-3 SFI 4-7 SFI 0-7 SFI 0-7 SFI 0-3 SFI 4-7 SFI 0-7 SFI 0-7 SFI 0-7

SFP28 Cage 1 SFP28 Cage 2 QSFP28 Cage 1 QSFP28 Cage 2

SFP 1-12 SFP 13-24 SFP 25-35 SFP 36-48 49 50 51 52

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 132
 Cisco Catalyst 9500-24Y4C
Block diagram
X86 2.4-GHz
Packet buffer (36 MB) quad-core CPU

Forwarding controller Forwarding controller FPGA

DRAM – 16 GB
Reassembly Rewrite Reassembly Rewrite Flash
crypto crypto 16 GB

Ingress FIFO Egress FIFO Ingress FIFO Egress FIFO

USB 2.0

Core 1 Core 0
USB 3.0
Network interface
Mgmt Console
10G/25Gx8 20G/25Gx8

PHY PHY PHY PHY PHY

0 1 2 6 7

SFI 0-7 SFI 0-3 SFI 4-7 SFI 0-7 SFI 0-7 SFI 0-7

SFP28 Cage 1 QSFP28 Cage 1 QSFP28 Cage 2

SFP 1-12 SFP 13-24 25 26 27 28

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 133
 Cisco Catalyst 9500-32C/32QC/24Y4C/48Y4C
Port-to-ASIC mapping

400 Gb/s 400 Gb/s 400 Gb/s 400 Gb/s

AS I C 0 AS I C 1
P ack et buffer (36 MB) P ack et buffer (36 MB)

Forwarding controller Forwarding controller Forwarding controller Forwarding controller

R e a s sembly R e as sembly R e as sembly R e as sembly

Rewrite Rewrite Rewrite Rewrite
cr y pto cr y pto cr y pto cr y pto

Ingress Ingress Ingress Ingress

Egress FIFO Egress FIFO Egress FIFO Egress FIFO
FIFO FIFO FIFO FIFO

C o re 1 C o re 0 C o re 1 C o re 0

N e twork interface N e twork interface

PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI SFI
0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7 0-3 4-7

QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28
Cage 1 Cage 2 Cage 3 Cage 4 Cage 5 Cage 6 Cage 7 Cage 8 Cage 9 Cage 10 Cage 11 Cage 12 Cage 13 Cage 14 Cage 15 Cage 16

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32

Command to verify the port-to-ASIC mapping:

show platform software fed active ifm mappings

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 134
Cisco Catalyst 9500 Breakout Options
4x 25G C9500 – 32C
4x 10G

C9500 24Q

C9500-12Q

4x 10G
C9500-40X

16.10.1
C9500-16X
16.9.1

16.8.1a

Not Supported
In Stackwise Virtual System, Breakout cables are not supported as SVL/DAD links.
#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 135
 Cisco Catalyst 9500 Breakout Interface
C9500-32C

Device# configure terminal

Device (config)# hw-module breakout <port-num>

Hu1/0/1 --- > Hu1/0/1/1, Hu1/0/1/2, Hu1/0/1/3, Hu1/0/1/4

C9500-12/24Q/16X/40X

Device# configure terminal

Device (config)# hw-mod breakout module 1 port <port-num> switch <switch_num>

Fo 1/0/1 --- > Te 1/0/1, Te 1/0/2, Te 1/0/3, Te 1/0/4

Fo 1/0/2 --- > Te 1/0/5, Te 1/0/6, Te 1/0/7, Te 1/0/8
..
Fo 1/0/24 -- > Te 1/0/93, Te 1/0/94, Te 1/0/95, Te 1/0/96

Fo 1/1/1 --- > Te 1/1/1, Te 1/1/2, Te 1/1/3, Te 1/1/4

Conversion process do not require system reload to make configuration effective

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 136
Cisco Catalyst 9500 QSA Options

C9500 – 32C

C9500-32QC

C9500-24Y4C

C9500-48Y4C

1G/10G supported

Only 10G supported

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 137
Cisco Catalyst 9500 QSA Options
C9500 24Q

C9500-12Q

C9500-40X

C9500-16X

1G/10G supported

Only 10G supported

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 138
QSA Limitation

1G Optics with QSA 10G Optics with QSA

C9500-24Q C9500-32C C9500-24Q (16.5.1a) C9500-32C (16.8.1) C9600
C9500-12Q C9500-32QC C9500-12Q (16.6.1) C9500-32QC (16.8.1) (16.11.1)
(16.8.1) (16.10.1) C9500-NM-2Q (16.6.1) C9500-48Y4C (16.9.1)
C9300-NM-2Q (16.6.1) C9500-24Y4C (16.9.1)
GLC-SX-MMD, GLC-SX-MMD SFP-10G-SR/SR-S SFP-10G-SR/SR-S SFP-10G-SR/SR-S
GLC-LH-MMD, GLC-LH-SMD SFP-10G-LR/LR-S SFP-10G-LR/LR-S SFP-10G-LR/LR-S
GLC-ZX-SMD GLC-TE SFP-10G-ER/ER-S SFP-10G-ER/ER-S SFP-10G-ER/ER-S
GLC-TE GLC-T SFP-10G-ZR/ZR-S SFP-10G-ZR/ZR-S SFP-10G-ZR/ZR-S
GLC-T SFP-H10GB-CU1M= SFP-H10GB-CU1M=
GLC-BX-D/U SFP-H10GB-CU3M= SFP-H10GB-CU3M=
SFP-GE-S/L/T/Z SFP-H10GB-CU5M= SFP-H10GB-CU5M=
CWDM-SFP10G-xxxx

QSA is supported as data ports for Stackwise Virtual and Standalone deployment.

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 139
Cisco Catalyst 9500 Series
Switch Database Management (SDM) template

Core template User-customizable Distribution template

Maximizes system resources template Maximizes system resources
for unicast and multicast Allows customizable* for MAC and security
routing and security ACL TCAM resources
(Default: C9500 Switches)
(Default: C9500 100G/25G)

SD-Access template NAT template

Maximizes system resources Maximizes the NAT
for security to support fabric configurations on the switch
deployment Cisco® Catalyst®
9500 Series

* Cisco Catalyst 9500 High Performance Switch Security ACL TCAM only

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 140
Cisco Catalyst 9500 40G/10G
SDM templates and scale numbers
Distribution template
Feature Core template SDA template NAT template
(default)
Indirect/LPM Routes (IPv4/IPv6) 64K / 32K 64K / 32K 64K / 32K 64K / 32K
Direct/Host Routes (IPv4/IPv6) 48K / 24K 32K / 16K 80K / 16K 48K / 24K
Multicast routes (IPv4/IPv6) 16K / 8K 32K / 16K 16K / 8K 32K / 16K
MAC address table 64K 16K 16K 16K
Flexible netflow 128K/ASIC 128K/ASIC 128K/ASIC 128K/ASIC
SGT label 8K 8K 8K 8K
Ingress
Security ACL 18K 18K
Egress
Ingress
QOS ACL 18K 3K
Egress
Ingress 1K 1K
Netflow ACL
Egress 2K 2K
Ingress
SPAN 1K 1K
Egress
PBR/NAT 2K 16K
CPP 1K 1K
Tunnel termination and MACSEC 1K 1K
LISP 1K 1K

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 141
Cisco Catalyst 9500 100G/25G
SDM templates and scale numbers
Core template
Feature Distribution template SDA template NAT template
(default)
Routes (IPv4/IPv6) 114K / 114K 212K / 212K 212K / 212K 212K / 212K
Multicast routes (IPv4/IPv6) 16K / 16K 32K / 32K 32K / 32K 32K / 32K
MAC address table 82K 32K 32K 32K
Flexible netflow 98K 64K 64K 64K
SGT label 32K 32K 32K 32K
Ingress 12K 8K 12K
Security ACL
Egress 15K 19K 8K
Ingress 8K 8K 4K
QOS ACL
Egress 8K 8K 4K
Ingress 1K 1K 1K
Netflow ACL
Egress 1K 1K 1K
Ingress 0.5K 0.5K 0.5K
SPAN
Egress 0.5K 0.5K 0.5K
PBR/NAT 3K 2K 15.5K
CPP 1K 1K 1K
Tunnel termination and MACSEC 3K 3K 2K
LISP 1K 2K 1K

Customizable ACL TCAM resources

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 142
 Cisco Catalyst 9500 Series
SDM template – CLI

Command Purpose

Step 1 Show sdm prefer Display the template in use

Step 2 Configure terminal Enter Global Configuration mode

Sdm prefer
Specify the SDM template to be
Step 3 {core|distribution|nat|sda|template
used on the switch
-modification*}

Step 4 End Return to Privilege exec mode

Step 5 Reload Reload the operating system

Step 6 Show sdm prefer Display the template in use

* - Only on Cisco 9500 High Performance SKU’s

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 143
 Cisco Catalyst 9500 Series
SDM Customizable template – CLI

Command to modify ACL TCAM Allocation

C9500-32C-2(config)#sdm prefer template-modification ?
default Default prefered template
fspan Filter Span
nfl NFL ACLs
qos QOS
security-acl Security ACLs

C9500-32C-2(config)# sdm prefer template-modification security-acl input allowed-range

Total_size : 27648 Suggested split percentage for input : 29 33 37 40 48 49 52 60 63 67 71

C9500-32C-2(config)#sdm prefer template-modification security-acl input 25 input-ipv4 75 output-ipv4 75

Allocated Security Acl Input (IPv4:4608, Non-IPv4:3584) entries, Output (IPv4:13824, Non-IPv4:5632) entries input=29.63
input_ipv4=56.25, output_ipv4=71.05

Modifications to preferred template have been stored, but cannot effect until the next reload. Allocations will be an approximation of user
specified percentages. Use 'show sdm prefer' to see proposed values.

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 144
 Cisco Catalyst 9500 Series
SDM Customizable template – CLI Security-ACL
Input=25%
Input V4 – 75%
Allocation
Output v4 – 75%

Command to modify ACL TCAM Allocation 8K(Input)

4.5K(v4)
3.5K(non-v4)

C9500-32C(config)#sdm prefer template-modification security-acl input 25 input-ipv4 75 output-ipv4 75 27K

13.5K(v4)
Allocated Security Acl Input (IPv4:4608, Non-IPv4:3584) entries, Output (IPv4:13824, Non-IPv4:5632) entries 19K(Output)

Command
input=29.63 input_ipv4=56.25, to verify the current template allocation :
output_ipv4=71.05
5.5K(non-v4)
show sdm prefer
Modifications to preferred template have been stored, but cannot effect until the next reload. Allocations will Input=50%
Security-ACL
be Security-ACL
Allocation
Input V4 – 75%
DEFAULT
Allocation Output v4 – 75%
an approximation of user specified percentages. Use 'show sdm prefer' to see proposed values.
6.5K(v4)
9.5K(v4)
12K(Input)
13K(Input)
5.5K(non-v4)
3.5K(non-v4)
27K
27K 6.5K(v4)
14K(Output) 10.5K(v4)
15K(Output)
3.5K(non-v4)
8.7K(Non-v4)
Security-ACL Input=75%
Allocation Input V4 – 75%
Output v4 – 75%

13.5K(v4)
19K(Input)
27K
5.5K(v4)

4.5K(v4)
8K(Output)
3.5K(non-v4)

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 145
Cisco Catalyst 9500 Series comparison
Cisco® Catalyst 9500 Cisco Catalyst 9500
Capabilities (per ASIC)
Series (UADP 2.0) 100G/25G (UADP 3.0)
Switching and forwarding capacity 240 Gbps/360 Mpps 1.6 Tbps/1 Bpps

ASIC interconnect bandwidth 2x 360 Gbps 2x 400 Gbps

Buffer capability 16 MB per Core (32MB /ASIC) 36 MB across Cores

Switch Database Management (SDM) template Fixed templates Customizable templates

NetFlow capabilities Dedicated NetFlow table Shared NetFlow table

v4 FIB scale Total 228,000* Total 412,000*

v4 and v6 scale v6 reduced by half v4 and v6 same scale

Storage 120GB USB 3.0 Up to 1TB M2 SATA SSD

StackWise Virtual Supported Supported

Spanning-Tree Instances – RPVST+ 128 1K

Virtual Routing & Forwarding(VRF) – v4 256 1K

* Maximum ASIC Capability

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 146
Catalyst 9600
Cisco Catalyst 9600 Series Switches
Powered
by UADP
3.0

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 148
Cisco Catalyst 9600 Series
Chassis

Dual-serviceable 4 line card slots

fan tray

2 supervisor slots
Built-in RFID (dedicated)

6.4 Tbps per slot

Blue Beacons from each
(system/fan tray, supervisor slot
sup, line cards)

Dimensions
Modular power (HxWxD inches)
supplies 13.95 x 17.4 x 16.1
(8RU)

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 149
Cisco Catalyst 9600 Series
C9606R chassis port density

Maximum
Density with
Port speed chassis
supervisor 1
density
100G 48 128
40G 96 128
25G 192 192
10G 192 192
1G* 192 192

Line Rate non-blocking

*Roadmap

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 150
Cisco Catalyst 9600 Series
Supervisor 1

9.6 Tbps 2.4 Tbps per slot

8 core X86 CPU

3x UADP 3.0 ASIC
@2.0 Ghz
UADP
UADP UADP

CPU
M.2 SATA SSD
16G DDR4 memory
(optional: up to 1 TB)

Mgmt ports: copper

Built-in RFID
and fiber

2x USB3
Blue Beacon
1x mini-B USB console

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 151
Cisco Catalyst 9600 Series
Line cards

C9600-LC-24C - 100G/40G (fiber)

• 24 ports
• QSFP28/QSFP+
• Supports 100G and 40G

C9600-LC-48YL - 25G/10G/1G* (fiber)

• 48 ports
• SFP28/SFP+/SFP
• Supports 25G, 10G, and 1G

*Roadmap
The Y in the product ID (PID) indicates the hardware capability

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 152
Cisco Catalyst 9600 Series
100G/40G Line card - C9600-LC-24C

• All 24 ports are capable of 100G (QSFP28)/40G (QSFP+)

• Hardware-ready with QSA (for 1G/10G)
• With Supervisor Engine 1
• 100G: Every 2 ports in a port-group. The odd number of ports can be 100G and the next even number port is
disabled. (Maximum of 12x 100G, line rate with 187 byte or higher)
• 40G - 24x 40G (line rate with 148 byte or higher)

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 153
C9600-LC-24C – Port Numbering with
Supervisor Engine 1
• 40G numbering from 1 to 24
• 100G number from 25 to 48

1 3 5 7 9 11 13 15 17 19 21 23
40G port
numbering 2 4 6 8 10 12 14 16 18 20 22 24

25 27 29 31 33 35 37 39 41 43 45 47
100G port
numbering 26 28 30 32 34 36 38 40 42 44 46 48

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 154
C9600-LC-24C with supervisor engine 1
• This line card appears in 40G mode by default
• Future supervisors can support 100G speed on all ports at the same time

40G 40G 40G 40G 40G 40G 40G 40G 40G 40G 40G 40G
Default mode
(all ports 40G) 40G 40G 40G 40G 40G 40G 40G 40G 40G 40G 40G 40G

100G 40G
100G
configuration 40G

Fo<slot#>/0/1 • Enable Hu1/0/25 as 100G Fo<slot#>/0/23

interface HundredGigE1/0/25 enable
Hu <slot#>/0/25 • Disabled Fo1/0/1 and 1/0/2 Hu <slot#>/0/47

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 155
Cisco Catalyst 9600 Series
25G/10G/1G Line card - C9600-LC-48YL

• All 48 ports support 25G/10G/1G

• Hardware capable of 10/100M
• Line rate with 25G/10G/1G (at 187 bytes for 25G; any packet size with 10G/1G)
• Any port, any supported speed
• Port reference is always ”TwentyFive<slot#>/0/<port#>” and port speed is auto-detected based on the
inserted transceiver

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 156
Cisco Catalyst 9600 Series
Fan tray

• N+1 (8+1) fan redundancy

• Flexible service - fan tray can be
replaced from the portside or
the back
• Efficient - variable speed per fan
depends on the load, temperature,
and altitudes (=>lower noise)
• Airflow – side-to-side airflow

Fan tray hot-swappable needs to be done within 120 seconds

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 157
Cisco Catalyst 9600 Series
Power supplies

• Chassis has 4 slots for power supply

• Individual on/off switch for each power supply
• Supports a mix of AC (@220V) and DC
power supplies

• Supports both 110V and 220V input

• 2 KW output with 220V (1050W with 110V)
AC • Platinum rate power supply
• Redundant mode: Combined and N+1

• Supports input range of -40V to -72V

• 2 KW output
DC • Platinum rate power supply
• Redundant mode: Combined and N+1

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 158
Architecture
Architecture
Centralized architecture
• Centralized architecture =>
UADP3.0 Supervisor x86 Uninterrupted
supervisor switchover
Forwarding Open Control plane
Features Cisco® containers • Centralized architecture
embedded controllers IOS-XE HA communication
(Forwarding, queuing,
and security are done
on the supervisor) =>
Unlock new capability
Passive backplane Up to 6.4 T BW per slot with a supervisor upgrade
• Transparent line cards =>
Compatible with new sup
Line card Line card Line card • Passive backplane =>
High MTBF
PHY PHY PHY
• X86 CPU + storage =>
App hosting

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 160
 Supervisor engine 1 – Block diagram
Switch backplane

1.6T 64x 64x 64x

SupIO
NIF NIF NIF PCIe Ethernet FPGA
(I2C)

UADP 3.0 UADP 3.0 UADP 3.0

#1 OBFL
#2 #3 8-core CPU (1 Gb)

1.6 Tbps

ASIC interconnect M.2 SATA SDRAM

(optional)

USB console/
2x USB3 Console/Mgmt SFP+

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 161
Supervisor engine 1 – ASICs to LC mapping

• ASIC #1: First third

of the ports
ASIC #1 ASIC #2 ASIC #3 • 48-port module: 1-16
• 24-port module: 1-8

• ASIC #2: Middle third

of the ports
• 48-port module: 17-32
• 24-port module: 9-16
ASIC #1 ASIC #2 ASIC #3

• ASIC #3: Last third

of the ports
• 48-port module: 33-48
• 24-port module: 17-24

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 162
Cisco Catalyst 9600 – Supervisor 1
Port-to-ASIC mapping

Command to verify the port-to-ASIC mapping:

show platform software fed active ifm mappings

C9600-Bottom#show platform software fed active ifm mappings

Interface IF_ID Inst Asic Core Port SubPort Mac Cntx LPN GPN Type Active
FortyGigabitEthernet1/0/1 0x7 0 0 0 0 0 0 0 1 101 NIF N
FortyGigabitEthernet1/0/2 0x8 0 0 0 8 0 2 1 2 102 NIF N
FortyGigabitEthernet1/0/3 0x9 0 0 0 16 0 16 0 3 103 NIF N
FortyGigabitEthernet1/0/4 0xa 0 0 0 24 0 18 1 4 104 NIF N
FortyGigabitEthernet1/0/5 0xb 1 0 1 8 0 14 1 5 105 NIF Y
FortyGigabitEthernet1/0/6 0xc 1 0 1 0 0 12 0 6 106 NIF Y
FortyGigabitEthernet1/0/7 0xd 1 0 1 24 0 30 1 7 107 NIF Y
FortyGigabitEthernet1/0/8 0xe 1 0 1 16 0 28 0 8 108 NIF Y
FortyGigabitEthernet1/0/9 0xf 2 1 0 0 0 0 0 9 109 NIF Y
<SNIP>
FortyGigabitEthernet1/0/16 0x16 3 1 1 16 0 28 0 16 116 NIF Y
FortyGigabitEthernet1/0/17 0x17 4 2 0 0 0 0 0 17 117 NIF Y
<SNIP>
FortyGigabitEthernet1/0/24 0x1e 5 2 1 16 0 28 0 24 124 NIF N
HundredGigE1/0/25 0x1f 0 0 0 0 0 0 0 25 125 NIF Y
<SNIP>

C9600-Bottom#$

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 163
100G/40G line card block diagram
Switch backplane

4x NIF 4x NIF 4x NIF 4x NIF 4x NIF 4x NIF PCIE PCIE

Sup-A Sup-B Sup-A Sup-B Sup-A Sup-B Sup-A Sup-B

PHY-1 PHY-2 PHY-12 FPGA

OBFL
(1 Gb)
2x QSFP28 2x QSFP28 2x QSFP28
1-2 3-4 23-24

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 164
25G/10G/1G line card block diagram
Switch backplane

4x NIF 4x NIF 4x NIF 4x NIF 4x NIF 4x NIF PCIE PCIE

Sup-A Sup-B Sup-A Sup-B Sup-A Sup-B Sup-A Sup-B

PHY-1 PHY-2 PHY-12 FPGA

OBFL
(1 Gb)
4x SFP28 4x SFP28 4x SFP28
1-4 5-8 45-48

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 165
Cisco Catalyst 9600 Series – Supervisor engine 1
Switch Database Management (SDM) template

User-customizable
Core template template Distribution template
Maximizes system resources Allows customizable Balances system resources
for Layer 3 unicast and multicast ACL TCAM resources between Layer 3 routes and
routes (default) Layer 2 MAC and Netflow

SD-Access template NAT template

Maximizes system resources
Maximizes the NAT
for policy to support fabric
configurations on the switch
deployment Cisco® Catalyst®
9600 Series

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 166
Cisco Catalyst 9600 Series
SDM templates and scale numbers
Core template
Feature Distribution template SDA template NAT template
(default)
Routes (IPv4/IPv6) 114K/114K 212K/212K 212K/212K 212K/212K
Multicast routes (IPv4/IPv6) 16K/16K 32K/32K 32K/32K 32K/32K
MAC address table 82K 32K 32K 32K
Flexible NetFlow 98K/ASIC 64K/ASIC 64K/ASIC 64K/ASIC
SGT label 32K 32K 32K 32K
Ingress 12K 8K 12K
Security ACL
Egress 15K 19K 8K
Ingress 8K 8K 4K
QOS ACL
Egress 8K 8K 4K
Ingress 1K 1K 1K
NetFlow ACL
Egress 1K 1K 1K
Ingress 0.5K 0.5K 0.5K
SPAN
Egress 0.5K 0.5K 0.5K
PBR/NAT 3K 2K 15.5K
CPP 1K 1K 1K
Tunnel termination and MACsec 3K 3K 2K
LISP 1K 2K 1K
Customizable ACL TCAM resources
#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 167
Cisco Catalyst 9600 Series
SDM template – Customizable TCAM section

C9600-Bottom#sho sdm prefer

Showing SDM Template Info

This is the Core template.

Security Ingress IPv4 Access Control Entries* : 6656 (current) - 6656 (proposed)
Security Ingress Non-IPv4 Access Control Entries* : 5632 (current) - 5632 (proposed)
Security Egress IPv4 Access Control Entries* : 6656 (current) - 6656 (proposed)
Security Egress Non-IPv4 Access Control Entries* : 8704 (current) - 8704 (proposed)

QoS Ingress IPv4 Access Control Entries* : 4608 (current) - 4608 (proposed)
QoS Ingress Non-IPv4 Access Control Entries* : 3584 (current) - 3584 (proposed)
QoS Egress IPv4 Access Control Entries* : 4608 (current) - 4608 (proposed)
QoS Egress Non-IPv4 Access Control Entries* : 3584 (current) - 3584 (proposed)

Netflow Input Access Control Entries* : 1024 (current) –1024 (proposed)

Netflow Output Access Control Entries* : 1024 (current) – 1024 (proposed)

Flow SPAN Input Access Control Entries* : 512 (current) – 512 (proposed)
Flow SPAN Output Access Control Entries* : 512 (current) – 512 (proposed)

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 168
Cisco Catalyst 9600 Series
SDM customizable template – CLI

Security-ACL
Customizable range: allocation
Default
10% - 90%
7K (v4)
• Between input and 12K (input)
output 5K (non-v4)
27K
• Between IPv4 and 7K (v4)
15K (output)
non-IPv4 8K (non-v4)

Example 1 Example 2 Example 3

Input =10% Input = 50% Input = 90%

Security-ACL Security-ACL Security-ACL
Input V4 – 75% Input V4 – 75% Input V4 – 75%
allocation allocation allocation
Output v4 – 75%
Output v4 – 75% Output v4 – 75%
2K (v4) 9.5K (v4) 18K (v4)
3K (input) 13K (input) 24K (input)
1K (non-v4) 3.5K (non-v4) 6K (v4)
27K 27K 27K
24K (output) 18K (v4) 10.5K (v4) 2K (v4)
14K (output)
3K (output)
6K (non-v4) 3.5K (non-v4) 1K (non-v4)

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 169
Cisco Catalyst 9600 Series – Supervisor Engine 1
SDM customizable template – CLI

Command to modify ACL TCAM allocation

C9600(config)#sdm prefer template-modification?
default Default prefered template
fspan Filter Span
nfl NFL ACLs
qos QOS
security-acl Security ACLs Allowed ranges

C9600 (config)# sdm prefer template-modification security-acl input allowed-range

Total_size: 27648 Suggested split percentage for input: 11 18 22 25 33 37 40 48 49 52 60 63 67 75 78 82 89

C9600(config)#sdm prefer template-modification security-acl input 15 input-ipv4 15 output-ipv4 85

Allocated Security Acl Input (IPv4:1024, Non-IPv4:4096) entries, Output (IPv4:18432, Non-IPv4:4096) entries
input=18.52 input_ipv4=20.00, output_ipv4=81.82
Modifications to the preferred template have been stored, but cannot take effect until the next reload. Allocations will be an
approximation of user-specified percentages. Use 'show sdm prefer' to see proposed values.

C9600(config)#

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 170
Catalyst 9600
Design
Consideration
 Catalyst 9600 for Multidomain Campus Core
Lowest TCO

L3/Collapsed Core L3 Core + MPLS PE Fabric Border

Fabric

• Reduced Complexity, • Segmentation, Scale, • Segmentation &

Resiliency & Scale LAN/WAN Consistency Automation/Programmability
• IPv4/IPv6, Unicast & Multicast, • MPLS VPNs (L2 & L3), • Turnkey Solution: SD-Access
QoS & ACL Scale MPLS over GRE • DIY : BGP EVPN VxLAN

One Platform. Any Place. Any Speed (1G to 100G)

#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Example of a Traditional Three-Tiers Campus
Design
Core
• Catalyst® 9606R Switch with 4x C9600-LC-24C for connection
to the distribution layer, WAN, and between core devices Core
ECMP or
port-channel * 100G • Oversubscription ratio of 4:1 from distribution to core

Distribution … • Catalyst 9606R with 1x C9606-LC-24C as the uplink

connection to the core and 3x C9606-LC-48YL as the
downlink connection to the access layer Distribution
25G • Oversubscription ratio of 20:1 from access to distribution
ECMP or
port-channel *
10G

• Catalyst 9410 with 1G copper and/or mGig modules with

Access … Sup1-XL-Y Access
• 8x Catalyst 9300 StackWise-480 with 10G/25G uplinks

2.5G
* StackWise® Virtual is on the roadmap 1G

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 173
Access Layer - POD
Cisco® Catalyst® 9400
Distribution • A Catalyst 9410 switch provides a total of 384 ports of 1G
• Catalyst 9410 can also provide 192x1G + 192x mGig ports
(up to 10G)

Catalyst 9300 StackWise®-480

ECMP or • Stack of 8 can provide a total of 384 ports of 1G or 2.5G (mGig)
port-channel* 2x 2x
10G 25G • Stack of 8 can also provide 384 ports of 1G and mGig combination

Aggregated downlink BW:

Access
• 384G with 384x 1G
• 960G with 384x 2.5G
384x 384x
1G 2.5G
Uplinks BW needed for 20:1 oversubscription from access
to distribution
POD
• 2x 10G for 384x 1G
• 2x 25G for 384x 2.5G
* StackWise® Virtual is on the roadmap

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 174
Distribution Layer - Block
Cisco® Catalyst® 9606R Switch Downlinks:
• 3x C9600-LC-48YL per Catalyst 9606R
4x 10x 4x 10x
100G 100G
• A total of 144 x 10G/25G ports per chassis
100G 100G
• Aggregate downlink BW per Catalyst 9606 Switch
1. With 10G uplinks: 144x 10G = 1.44T
Distribution 2. With 25G uplinks: 144x 25G = 3.6T

144x 144x 144x 144x Uplinks

10G 25G 10G 25G
• 1x C9606-LC-24C per Catalyst 9606R
• To maintain 4:1 oversubscription between distribution and core
layers
2x 2x 2x 2x 1. With 10G uplinks: BW = 1.44T/4 = 360G => 4x 100G ports
10G 25G 10G 25G 2. With 25G uplinks: BW = 3.6T/4 = 900G => 10x 100G ports

Access POD POD (The remaining 100G/40G ports can be used for ECMP or
1 … 144 StackWise Virtual when it is available.)

384x 384x 384x 384x

1G 2.5G 1G 2.5G
Each distribution block can aggregate 144 access PODs.
That’s 144 x 384 = 55,296 of 1G, or 2.5G ports
* StackWise® Virtual is on the roadmap

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 175
Core Layer with 1G in the Access Layer
Cisco® Catalyst® 9606R Switch
12x 12x • 4x C9606-LC-24C
100G 100G • 75% of ports (36x 100G) to distribution
• 25% of ports (12x 100G) for connections between the two
Core cores and the WAN
• Two of the core devices will provide 72x 100G for the
36x 36x distribution layer
10G 10G

• 1G aggregation
8x 8x • With 8x 100G per distribution block,
10G 10G two Catalyst 9606R Switches with the above configuration
can aggregate 72/8, or 9 distribution blocks
Block Block
Distribution
1 … 9

288x 288x
10G 10G
The total number of 1G ports:
9x 144 x 384 = 497,664 of 1G ports

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 176
Core Layer with 2.5G in the Access Layer
Cisco® Catalyst® 9606R Switch
12x 12x • 4x C9606-LC-24C
100G 100G • 75% of ports (36x 100G) to distribution
• 25% of ports (12x 100G) for connections between the two
Core cores and the WAN
• Two of the core devices will provide 72x 100G for the
36x 36x distribution layer
100G 100G

• 2.5G aggregation
20x 20x • With 20x 100G per distribution block, two of Catalyst
100G 100G 9606R Switches with the above configuration can
aggregate 72/20, or 3 distribution blocks
Block Block
Distribution
1 … 3

288x 288x
10G 10G
The total number of 2.5G ports:
3x 144 x 384 = 165,888 of 2.5G ports

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 177
Core Layer with 2.5G in the Access Layer
With 4x Catalyst 9606 in the core

12x 12x 12x 12x 12x 12x

100G 100G 100G 100G 100G 100G

Core

36x 36x 36x 36x 36x 36x

100G 100G 100G 100G 100G 100G

10x 10x 5x
5x 5x 5x
100G 100G 100G 20x
20x 100G 100G 100G
100G
100G

Block Block Block Block

Distribution
1 … 3 1 … 6

288x 288x 288x 288x

10G 10G 10G 10G

The total number of 2.5G ports: The total number of 2.5G ports:
3x 144 x 384 = 165,888 of 2.5G ports 6x 144 x 384 = 331,776 of 2.5G ports

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 178
Summary
Two Cisco Catalyst 9606R Switches in the core can provide:
1. 497K of 1G ports, or
Core
2. 165K of 2.5G ports

ECMP or Oversubscription = 4:1

port-channel *
Uplinks (40/100G module): 4x 100G 10x 100G
Downlinks (10/25G modules): 144x 10G 144x 25G
Distribution

Oversubscription: 20:1

ECMP or
Uplinks (Supervisor or uplink module): 2x 10G 2x 25G
port-channel *
Downlinks (1G/mGIG module): 384x 1G 384x 2.5G

Access …

* StackWise® Virtual is on the roadmap

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 179
UADP ASIC
ASICs are a Pillar of Cisco Innovation…
#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Traditional Networking ASICs - Fixed Pipelines Can lookup these
Fields

Parses &
Understands Fixed Fixed
Parser
number of Bytes
MAC IPv4 ACL QoS
Ether
net
IP Payload Look Look Look Look
up up up up
VXLAN Ether
net
IP UDP
VXLA
N
Ether
net
IP Payload

GRE Ethern
IP GRE
Ethern
IP Payload
et et

MPLS Ethern Fast Memory Lookup Tables

Label IP Payload
et

Not Supported in
Traditional Hardware QoS
Look
ACL
Look
…
Look
…
Look
ASIC up up up up

Fixed Pipeline

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 182
New ASICs for New Technology ?

Marketing
Architecture RTL Design Synthesis Floor Planning Fabrication
Requirements

2 – 4 Years

Building a new ASIC takes a lot of time & money

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 183
How about CPUs ?

CPUs are highly CPUs are not as

Programmable fast

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 184
Traditional Networking ASICs vs CPUs

Performance Performance

Flexibility Flexibility

Traditional General
Networking Purpose
ASIC CPU

Purpose Built – High Performance General Purpose – Highly Flexible

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 185
Cisco Innovation – UADP ASIC

Performance

In 2013 Cisco Introduced Flexibility

UADP
(Unified Access Data Plane) Programmability

UADP brings Flexibility without compromise on Performance

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 186
UADP 1.x

UADP 1.0 UADP 1.1

1.6 Billion 3.2 Billion
Transistors Transistors
36 nm 36 nm

Catalyst Catalyst Catalyst

3850 3650 3650
Catalyst 3850 Catalyst 3650 Catalyst SFP Fiber Multigigabit Mini Multigigabi
Copper Catalyst t
3850 SFP+

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 187
UADP 2.0

7.46B Transistors
28nm Technology
Catalyst 9K
Family
#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 188
UADP Evolution – 1.0 vs 2.0

Cores Ports MHz 500/625/

Dual 1/10/25/ 750 MHz
Core 40G Up to
Single 1/10G Ports 375 MHz 240Gbps
Core Ports 56 Gbps

UADP 1.0 UADP 2.0 UADP 1.0 UADP 2.0 UADP 1.0 UADP 2.0

TCAM Buffers 32MB Xconnect Up to

54K
TCAM Buffers 720G
12K 6MB 240G ASIC
connectivity
ASIC
TCAM Buffers
connectivity

UADP 1.0 UADP 2.0 UADP 1.0 UADP 2.0 UADP 1.0 UADP 2.0

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 189
UADP 2.0 Family

UADP 2.0 mini UADP 2.0 UADP 2.0 XL

Total Bandwidth Up to 100G Up to 160G Up to 240G

Frequency 500 MHz 500 MHz 625, 750 MHz

Table Sizes Mini Standard XL Tables

TCAM Entries 5K 20K 54K

Buffers 6MB 16MB 32 MB

Stack Bandwidth 160G 240G 720G

Stack Ring 1 1 2

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 190
UADP 2.0 Mini
Architectural simplicity with powerful innovations
Investment Protection
Flexible Pipeline

Enhanced Scale/Buffering
Embedded
CPU

1/2.5/5/10/40G
100GE 6MB
Supports Different
Bandwidth Packet Buffer
Speeds

16K Flexible SDM Templates

Netflow Records Programmable Modules Flexibility

160/80G Up to 2X to 4X
Stacking Capacity forwarding + TCAM Catalyst 9200

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 191
 UADP 3.0
Customizable ASIC 36-MB
templates unified buffer

Double-width tables 3x more FIB scale

1G, 10G, 20G,

~20B transistors Up to 1.6 TB
25G, 40G, and 100G
bandwidth
16-nm technology speeds

Catalyst 9500 High Performance and Catalyst 9600

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 192
UADP evolution
UADP 2.0 vs. 3.0 per-ASIC capabilities

Throughput Ports Forwarding

Up to 240 1, 10, 25, 40G 1, 10, 25,

500, 625,
Gbps Up750,
to 1.6875
Tbps 375 Mpps 1000 Mpps
Ports 40, 100G
750
500, MHz
625, MHz,875
750, 1 GHz
MHz, Ports
750
Up to MHz
240 Gbps Up to1 1.6
GHzTbps

UADP 2.0 UADP 3.0 UADP 2.0 UADP 3.0 UADP 2.0 UADP 3.0

Buffers Backplane
720G 1600G
32 MB 36 MB (36x 15G)
Shared buffers Unified buffers (32x 28G)
Stack
ASIC interconnect
interconnect
16 MB 16 MB 36 MB

UADP 2.0 UADP 3.0 UADP 2.0 UADP 3.0

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 193
UADP ASIC 2.0 and 3.0 comparison
Capabilities (per ASIC) UADP 2.0 XL UADP 3.0

Switching and forwarding capacity 240 Gbps/360 Mpps 1.6 Tbps/1 Bpps

Stack bandwidth Upto 2x 360 Gbps Upto 2x 800 Gbps

Buffer capability 2x 16 MB 36 MB shared buffer

Switch Database Management (SDM) template Fixed templates Customizable templates

NetFlow capabilities Dedicated NetFlow table Shared NetFlow table

v4 FIB scale Total 228,000* Total 412,000*

v4 and v6 scale v6 reduced by half v4 and v6 same scale

* Maximum ASIC Capability

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 194
UADP ASIC Core

ASIC Family Core

UADP 2.0 mini Single

UADP 2.0/2.0XL Dual

UADP 3.0 Dual

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 195
UADP Core Architecture
Inter-ASIC interface

SQS AQM
PBC – Packet Buffers Complex
Q Q Q
IQS EQS

Ingress Forwarding Look up Egress Forwarding

Controller Tables Controller
(IFC) (EFC)

Rewrite engine

Ingress FIFO Egress FIFO

MACsec Encryption Recirculation MACsec

engine engine

Network Interfaces - Front Panel Ports

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 196
Some of the Key Capabilities of UADP ASIC

Flex Parser
& Recirculation Adaptable Tables
Micro Engines
Programmable Capability
Pipelines

No Compromise on Performance

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 197
UADP – Programmable Pipeline
Inter-ASIC interface

SQS AQM
PBC – Packet Buffers Complex
Q Q Q
IQS EQS

Ingress Forwarding Look up Egress Forwarding

Controller Tables Controller
(IFC) (EFC)

Rewrite engine

Ingress FIFO Egress FIFO

MACsec Encryption Recirculation MACsec

engine engine

Network Interfaces - Front Panel Ports

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 198
Programmable Pipelines – Closer Look
Final decision on Flex parser
packet’s future 256 B

IGR Flex parser

Look Look Look Look
up up up up
Tabl Tabl Tabl Tabl
e e e e
Stage Look Look Look Look Stage
#17
Stage
up
Tabl
Flexible
up
Tabl
up
Tabl
up
Tabl
#1
Stage
17 ingress #..
e
Look lookup tables
e
Look
e
Look
e
Look #2 8 egress
Stage up up up up Stage
Programmable #.. Tabl Tabl Tabl Tabl #.. Programmable
e e e e
stages Stage Look
(shared
Look Look Look Stage stages
#2 up up up up #..
Stage
#1
Tabl
e across cores)
Tabl
e
Tabl
e
Tabl
e
Stage
#8
Look Look Look Look
up up up up
Tabl Tabl Tabl Tabl
e e e e
Flex parser EGR

Flex parser At each stage, 2 Final decision on

256 B simultaneous lookups packet’s future

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 199
Proven Investment Protection
with UADP 1.0

2013 2015 2017

UADP 2.0 extends it to the Catalyst 9K Family

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 200
Some of the Key Capabilities of UADP ASIC

Flex Parser
& Recirculation Adaptable Tables
Micro Engines
Programmable Capability
Pipelines

No Compromise on Performance

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 201
UADP – Recirculation Engine
Inter-ASIC interface

SQS AQM
PBC – Packet Buffers Complex
Q Q Q
IQS EQS

Ingress Forwarding Look up Egress Forwarding

Controller Tables Controller
(IFC) (EFC)

Rewrite engine

Ingress FIFO Egress FIFO

MACsec Encryption Recirculation MACsec

engine engine

Network Interfaces - Front Panel Ports

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 202
UADP – Recirculation First Pass
Inter-ASIC interface

SQS AQM
PBC – Packet Buffers Complex
Q Q Q
IQS 2. Forwarding 3. Skip Egress EQS
Lookup = packet to Lookup
be decapsulated
Ingress Forwarding Look up Egress Forwarding
1. Apply Ingress
Controller Tables Controller 4. Decapsulate and
Policies of Physical (IFC) (EFC) forward to
Port, Vlan/L3 Recirculation
Interface
Rewrite engine

Ingress FIFO Egress FIFO

MACsec Encryption Recirculation MACsec

engine engine

Network Interfaces - Front Panel Ports

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 203
UADP – Recirculation Second Pass
Inter-ASIC interface

SQS AQM
PBC – Packet Buffers Complex
Q Q Q
IQS 2. Forwarding 3. Egress Policies of EQS
lookup to determine Physical
Tx port/VLAN Port/VLAN/L3
Ingress Forwarding Look up Egress ForwardingInterface
1. Apply Ingress
Controller Tables Controller 4. Final Rewrite
Client Policy (IFC) (EFC) information

Rewrite engine

Ingress FIFO Egress FIFO

MACsec Encryption Recirculation MACsec

engine engine

Network Interfaces - Front Panel Ports

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 204
Some of the Key Capabilities of UADP ASIC

Flex Parser
& Recirculation Adaptable Tables
Micro Engines
Programmable Capability
Pipelines

No Compromise on Performance

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 205
UADP – Micro Engines
Inter-ASIC interface

SQS AQM
PBC – Packet Buffers Complex
Q FSE PLC HSH ILE Q Q
IQS Ingress Pipeline
ASE ELE
Egress Pipeline
NFL EQS

Ingress Forwarding Look up Egress Forwarding

Controller Tables Controller
(IFC) (EFC)

Rewrite engine

Ingress FIFO Egress FIFO

MACsec Encryption Recirculation MACsec

engine engine

Network Interfaces - Front Panel Ports

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 206
Integrated & Micro Engines can help with …

Encryption Netflow Hash Table Search

Fragmentation Policy TCAM TCAM Search

ReAssembly ReWrite LPM Search

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 207
Some of the Key Capabilities of UADP ASIC

Flex Parser
& Recirculation Adaptable Tables
Micro Engines
Programmable Capability
Pipelines

No Compromise on Performance

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 208
UADP – Lookup Tables
Inter-ASIC interface

SQS AQM
PBC – Packet Buffers Complex
Q Q Q
IQS EQS

Ingress Forwarding Look up Egress Forwarding

Controller Tables Controller
(IFC) (EFC)

Rewrite engine

Ingress FIFO Egress FIFO

MACsec Encryption Recirculation MACsec

engine engine

Network Interfaces - Front Panel Ports

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 209
Flex Tables
SRAM TCAM
Lookup Lookup Lookup Lookup Lookup Lookup Lookup Lookup
Table Table Table Table Table Table Table Table

Lookup
Table
Lookup
Table
Lookup
Table
Lookup
Table
Lookup
Table
Lookup
Table
Lookup
Table
Lookup
Table
Pool of
Lookup
Lookup Lookup Lookup Lookup Lookup Lookup Lookup Lookup
Table Table Table Table Table Table Table Table Tables
Lookup Lookup Lookup Lookup Lookup Lookup Lookup Lookup
Table Table Table Table Table Table Table Table

Flexible Tables consists of different types of TCAM &

SRAM

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 210
Flex Tables - Example
SRAM TCAM QoS Entries

Lookup Lookup Lookup Lookup Lookup Lookup Lookup Lookup

Table Table Table Table Table Table Table Table

Lookup Lookup Lookup Lookup Lookup Lookup Lookup Lookup

Table Table Table Table Table Table Table Table

Lookup Lookup Lookup Lookup Lookup Lookup Lookup Lookup NAT Entries
Table Table Table Table Table Table Table Table

Lookup Lookup Lookup Lookup Lookup Lookup Lookup Lookup

Table Table Table Table Table Table Table Table

Tunnels

Security ACL

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 211
Flex Tables - Example
SRAM TCAM QoS Entries

Lookup Lookup Lookup Lookup Lookup Lookup Lookup Lookup

Table Table Table Table Table Table Table Table

Lookup Lookup Lookup Lookup Lookup Lookup Lookup Lookup

Table Table Table Table Table Table Table Table

Lookup Lookup Lookup Lookup Lookup Lookup Lookup Lookup NAT Entries
Table Table Table Table Table Table Table Table

Lookup Lookup Lookup Lookup Lookup Lookup Lookup Lookup

Table Table Table Table Table Table Table Table

Tunnels

Security ACL

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 212
Flex Tables - Example
QoS Entries

SRAM TCAM
Lookup Lookup Lookup Lookup Lookup Lookup Lookup Lookup
Table Table Table Table Table Table Table Table

Lookup Lookup Lookup Lookup Lookup Lookup Lookup Lookup

Table Table Table Table Table Table Table Table

Lookup Lookup Lookup Lookup Lookup Lookup Lookup Lookup NAT Entries
Table Table Table Table Table Table Table Table

Lookup Lookup Lookup Lookup Lookup Lookup Lookup Lookup

Table Table Table Table Table Table Table Table

Tunnels

Security ACL

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 213
Catalyst 9K
Sample Profiles Cross Domain
Mix of L2/L3 Network Address
Capabilities Translation Policy

Scale Profile Aggregation Profile NAT Profile Border Profile

IPv4 / IPv6 LPM 64K / 32K 64K / 32K 64K / 32K
IPv4 / IPv6 Host 48K / 24K 48K/ 24K 80K/ 40K

Multicast Route 16K (IPv4) 32K (IPv4) 16K (IPv4)

8K (IPv6) 16K (IPv6) 8K (IPv6)

IGMP/MLD Snooping 16K 16K 16K

MAC Address 64K 16K 16K
SGT Label 8K 8K 8K
Netflow 128K/ASIC 128K/ASIC 128K/ASIC
Security ACL 18K 18K 4K
QoS ACL 18K 3K 16K
Service ACL- PBR / NAT 2K 16K 16K

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 214
Packet walks
UADP block diagram
Inter-ASIC interface

SQS AQM
Packet Buffer Complex
Q Q Q
IQS EQS
Ingress Forwarding Egress Forwarding
Controller Controller

Packet rewrite
Recirculation

Ingress Egress
FIFO FIFO
Encryption
MACsec

Network Interface

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 216
Abbreviations used

IFC Ingress Forwarding Controller PBC Packet Buffer Complex

IQS Ingress Queues EFC egress Forwarding Controller

EQS Egress Queueing and scheduling AQM Active Queue Management

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 217
 Unicast: Within the ASIC
Inter-ASIC interface
4
1. Received, 5. EQS schedule PBC
processed by Packet Buffer SQS AQM to send a copy to
MACSec if needed EFC and a copy to
and into FIFO 3 5 Q Q ReWrite (includes
EQS descriptor)

2. A copy to buffer, Ingress Forwarding Egress Forwarding

a copy to IFC Controller Controller 6. EFC sends results to
ReWrite
6
3. Goes through IFC, 7. Rewrite the packet
result descriptor and send though the
send to PBC Packet rewrite egress FIFO

2 Recirculation 7
4. Descriptor has local 8. Packet is encrypted
destination, PBC by MACSec if
Ingress Egress
sends the info needed
FIFO FIFO
to EQS
1 MACSec
Encryption 8

Network Interface

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 218
 Unicast: Across ASICs/Stack Members
Inter-ASIC interface
1. Received, 4 5 6
SQS AQM 6. PBC received the
processed by Q frame and sends
MACSEec if needed Packet Buffer the info to EQS
IQS Q Q
and into FIFO
3 7 EQS
2. A copy to buffer, 7. EQS schedule PBC
a copy to IFC Ingress Forwarding Egress Forwarding to send a copy to
EFC and a copy to
Controller Controller ReWrite (includes
3. Goes through IFC,
descriptor)
result descriptor
8
send to PBC
8. EFC sends results to
4. Descriptor has Packet rewrite ReWrite
remote destination,
PBC sends the info 2 Recirculation 9. Rewrite the packet
to IQS sent it to egress
Ingress Egress FIFO
9
5. IQS schedule PBC FIFO FIFO
10. Packet is encrypted
to send the packet
1 MACsec
Encryption by MACSec if
with descriptor to 10
needed
Inter-ASIC Interface
Network Interface

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 219
 Multicast: Egress local
Inter-ASIC interface
7 4
1. Received, SQS AQM 5. AQM within EQS
processed by Packet Buffer generate the list of
MACSec if needed Q Q egress port based
and into FIFO 5 on descriptor,
3 EQS
schedule for each
egress port.
2. A copy to buffer, Ingress Forwarding Egress Forwarding
a copy to IFC Controller Controller
6. For each egress
port, frame goes
3. Goes through IFC, though the EFC,
result descriptor ReWrite, Egress
send to PBC Packet rewrite FIFO, and
encrypted by
2 6 MACSec if needed
Recirculation
4. Descriptor has local
destination, PBC Ingress Egress 7. Once the
sends the info replication for the
FIFO FIFO
to EQS last port is done,
1 MACsec
Encryption PBC remove the
packet from the
Network Interface buffer

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 220
 Multicast: Egress remote Replication done on
egress => Efficient use
of BW
1. Received, Inter-ASIC interface
processed by 4 5 6 9 6. PBC received the
MACSec if needed SQS AQM frame and sends the
and into FIFO Q
Packet Buffer info to EQS
IQS Q Q
2. A copy to buffer, 7
3 EQS 7. AQM within EQS
a copy to IFC
generate the list of
Ingress Forwarding Egress Forwarding egress port based
3. Goes through IFC, on descriptor,
result descriptor Controller Controller schedule for each
send to PBC egress port.
8
4. Descriptor has
8. For each egress
remote destination,
Packet rewrite port, frame goes
PBC sends the info
though the EFC,
to IQS
ReWrite, Egress
2 Recirculation FIFO, and encrypted
5. IQS schedule PBC to
by MACSec if
send the packet with Ingress Egress needed
descriptor to Inter- FIFO FIFO
ASIC Interface
9. Once the replication
1 MACsec
Encryption for the last port is
Descriptor contains both
local and remote done, PBC remove
destinations Network Interface the packet from the
buffer

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 221
Open IOS XE Software
Historical View of Cisco IOS
IOS XE
IOS 9.x IOS 15.x 16.1
M&T Release
LAN Switching,
Remote Access, Cat3850
WAN Switching
ASR1000, ISR
Cisco
founded

1984 1986 1993 2007 2009 2010 2015

Operating IOS XE IOS XE

System BinOS NOVA
6.0 3.x 3.x
Cisco’s ASR1000 Cat4K
Shipped First
AGS Router

Timeline unevenly distributed

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 223
 Monolithic
Growing

All in One

#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Challenges with Classis IOS
Monolithic
One Big Process
Process not independent of one another

Unified control/data plane

Non Modular
No clean separation between path calc.
and forwarding

Common Memory
Config/oper data centralized, not easily
shared

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 225
Open IOS-XE
IOS IOS XE 3.7.x(SE) Open IOS XE 16.5.1

IOS IOSd Hosted Apps IOSd Hosted Apps

IOS Sub
WCM IOS Sub
Systems Containers* Containers*
Features IOSd Blob IOS Sub
Systems
Features Components Components Systems
Wireshark Containers* Wireshark

Config & Operational

Common States
CommonInfrastructure
Common Infrastructure//HA
HA Infrastructure / HA
Management
ManagementInterface
Management Interface Interface
IOS-XE
ModuleDrivers
Module Drivers Module Drivers
DB

Kernel
Kernel Kernel

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 226
Open IOS XE – Key Architectural Enhancements
IOS XE Denali 16.5.1

IOSd Hosted Apps

IOS Sub Systems
IOS Sub
IOS Sub
Systems Containers* Containers*
IOSd Blob IOS Sub
Systems
Systems
Containers* Wireshark

Common Infrastructure / HA
IOS XE Database
(Crimson Database)

Management Interface

IOS-XE
Module Drivers
DB
Kernel
LXCs Support

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 227
Open IOS XE – IOS Sub Systems

IOS Sub Systems BGP

IOSd Blob IOS Sub Systems
OSPF
IOS Sub Systems
Failure of one of
MPLS, etc
the Sub Systems
Keeps Rest of
the System intact
IOS-XE DB

IOS Sub Systems Enhances IOS Resiliency

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 228
Open IOS XE – DB
Link STP OSPF
Logs
State State State

Link MST
Logs
State State

IOS-XE DB
BGP Tunnel
State State

The DB contains the

Operational and Configurational
States

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 229
Open IOS XE – DB

Higher Application UP Time

IOSd Sub
Systems IOS-XE DB

Config & Operational

States
Quicker Recovery

Decoupling Code & Data

protects the Operational & Better Convergence
Configurational States

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 230
Open IOS XE – DB

Link STP OSPF

Logs
State State State

Link MST
Logs
State State

Data Models

BGP Tunnel
State State

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 231
App Hosting – Dockers Based

APP1 APP4

Model Driven APIs

XML JSON GPB
NETCONF RESTCONF gRPC
Yang Models

Crimson Interface

Open Apps
Crimson DB

App Hosting Environment Control Plane

Linux Kernel
Data Plane

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 232
Open IOS XE – Containers
IOS XE Denali 16.5.1

IOSd Hosted Apps

Containers Containers

Containers Wireshark

Common Infrastructure / HA

Management Interface
IOS-XE
Module Drivers
DB
Kernel

Decoupled Execution Space

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 233
Benefits for our Customers
One Release Train Single Binary across Catalyst 9K

RAFA
(Run Any Feature Anywhere) MPLS, GRE, NAT, etc.

Patch Updates Patching Available NOW!

Comprehensive
NETCONF, RESTCONF, YANG Models
Programmability

Trustworthy & Secure

Platform
ASLR 64 bit, Mandatory Access Control

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 234
IOS XE – Same Software on all 9K Platforms

Simple Certification &

Qualification

Easier Image Management

Same Binary Image

Easier Image Management
On all C9K

cat9k_iosxe.16.05.01a.SPA.bin
Single Binary for the entire Catalyst 9K Family
#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 235
 IOS XE
16.x

Catalyst 9200 Catalyst 9300 Catalyst 9400 Catalyst 9500 Catalyst 9600

IOS XE Lite IOS XE

Binary Image Same Binary Image

Catalyst 9000 runs the same Operating System

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 236
#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
High Availability on
C9K
Goals
• Efficiently utilize available bandwidth
• Dynamically respond to all types of disruptions
• Leverage most effective design techniques that meet the design
requirements

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 239
Where Can Outages Occur?
Remote
• Unplanned Outage
• Planned Outage

Core L3 Protocols
Impact
Application Layer
Link or Device Failure

Distribution L2 Protocols
Impact
Protocol Layer

L3 Link
L2 Link
Access
Physical Layer

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 240
Options to Mitigate the outages
Remote

• Add more links

• Add more devices
Core
• Leverage FHRP like HSRP and VRRP
• Change the timers
• Tune the application performance
• Etc…
Distribution

L3 Link
Convergence Time?
L2 Link
Failover Detection?
Access

What is the best way ?

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 241
Cisco IOS High Availability Strategy:
Based on Customer Needs
Provide continuous access to
applications, data, and content
anywhere, anytime
System Level • Robust Hardware
Resiliency • Modular and Flexible Software

Network • IOS XE Features for faster

Level convergence and recovery
Resiliency

Embedded • IOS XE Intelligence for proactive

faults/events and configuration
Management tracking

Address every potential cause of downtime

with functionality, design, or best practice
#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved.
24 Cisco Public 242
Enterprise Campus Network Designs

Multi-Tier Layer2/3 Topology Multi-Tier Layer3 Topology

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 243
Industry‘s Broadest Portfolio of
End-to-End High Availability Technologies
Requirements Technologies
 In-Service Software Upgrade (ISSU)
 IP NSF/SSO
 MPLS NSF/SSO—LDP, VPNs
 IOS Software Modularity
 Fast Software Upgrade
System-Level  Fast Reload
Resiliency  Control Plane Policing

 NSF/GR Awareness (BGP, OSPF, IS-IS, EIGRP, LDP,)

 Routing Convergence
Optimization
Incremental SPF optimization
Network- Level IP Event Dampening
Resiliency  Multicast Sub-second Convergence
 Fast Convergence (OSPF, IS-IS)
 Bi-Directional Forwarding Detection (BFD)
 Embedded Event Manager
Embedded  Embedded Resource Manager (ERM)
Management

#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
High Availability
Architecture in
Campus
 High Availability Architecture in Campus – SSO

Stateful Switchover (SSO) Stackwise Virtual

Catalyst 9500 Catalyst 9500
SSO Aware Applications SSO
Active Standby
Forwarding Information Base
IEEE 802.1x
PAgP / LACP
…and more

SSO Compliant Applications

Routing Protocols, Netflow, etc

Active
SSO
Standby
Stackwise-480
Active SUP Active
Standby SUP SSO
Standby

Catalyst 9400 Catalyst 9300

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 246
 SSO by itself Does Not
Provide Redundancy for the
Routing Protocols

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 247
 Routing Protocol Redundancy With NSF
Active Supervisor/Switch Standby Supervisor/Switch
EIGRP RIB OSPF RIB ARP Table EIGRP RIB OSPF RIB ARP Table

Prefix Next Hop Prefix Next Hop IP MAC Prefix Next Hop Prefix Next Hop IP MAC
192.168.0 192.168.0.1
10.0.0.0 10.1.1.1 10.1.1.1 aabbcc:ddee32 - - - - - -
192.168.55..0 192.168.55.1
10.1.0.0 10.1.1.1 10.1.1.2 adbb32:d34e43 - - - - - -

10.20.0.0 10.1.1.1 192.168.32.0 192.168.32.1 10.20.1.1 aa25cc:ddeee8 - - - - - -

SSO
FIB Table FIB Table
Redundancy
Prefix Next HOP Prefix Next HOP
Facility
10.1.1.1 aabbcc:ddee32 10.1.1.1 aabbcc:ddee32

10.1.1.2 adbb32:d34e43 10.1.1.2 adbb32:d34e43

192.168.0.0 aa25cc:ddeee8 Checkpoint 192.168.0.0 aa25cc:ddeee8

Facility

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 248
 Routing Protocol Redundancy With NSF
Active Supervisor/Switch Standby Supervisor/Switch
EIGR RIB OSPF RIB ARP Table EIGRP RIB OSPF RIB ARP Table
P
Prefix Next Hop Prefix Next Hop Prefix Next Hop IP MAC
Prefix Next Hop IP MAC
192.168.0 192.168.0.1
- - - - - -
10.0.0.0 10.1.1.1 10.1.1.1 aabbcc:ddee32
192.168.55..0 192.168.55.1
- - - - - -
10.1.0.0 10.1.1.1 10.1.1.2 adbb32:d34e43
192.168.32.0 192.168.32.1 - - - - - -
10.20.0.0 10.1.1.1 10.20.1.1 aa25cc:ddeee8

FIB Table SSO FIB Table

Prefix Next HOP Redundancy Prefix Next HOP

10.1.1.1 aabbcc:ddee32 Facility 10.1.1.1 aabbcc:ddee32

10.1.1.2 adbb32:d34e43 10.1.1.2 adbb32:d34e43

192.168.0.0 aa25cc:ddeee8 Checkpoint 192.168.0.0 aa25cc:ddeee8

Facility

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 249
Routing Protocol Redundancy With NSF
Standby Supervisor/Switch
EIGRP RIB OSPF RIB ARP Table

Prefix Next Hop Prefix Next Hop IP MAC

192.168.0 192.168.0.1
1-0.0.0.0 -10.1.1.1 -10.1.1.1 a-abbcc:ddee32
192.168.55..0 192.168.55.1
-10.1.0.0 -10.1.1.1 -10.1.1.2 -adbb32:d34e43

-10.20.0.0 1-0.1.1.1 192.168.32.0 192.168.32.1 -10.20.1.1 -aa25cc:ddeee8

FIB Table

Prefix Next HOP

10.1.1.1 aabbcc:ddee32

10.1.1.2 adbb32:d34e43

192.168.0.0 aa25cc:ddeee8

GR/NSF Signaling per

protocol

Synchronization per protocol

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 250
High Availability Architecture in Campus –
SSO/NSF
Stackwise Virtual
Catalyst 9500-24Q Catalyst 9500-24Q
SSO
Active Standby NSF Aware

Non Stop Forwarding(NSF) or

Graceful Restart

OSPF,BGP,LDP, etc
Routing Protocols

Active
NSF Capable
SSO
Standby
Stackwise-480
Active SUP Active
Standby SUP SSO
Standby

Catalyst 9400 Catalyst 9300

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 251
High Availability in
Fixed Access -
Catalyst 9300
High Availability in Campus – Fixed Access
Stackwise Virtual
Catalyst 9500-24Q Catalyst 9500-24Q
SSO
Active Standby NSF Aware

Routing Protocols
Fixed Access

Active
NSF Capable
SSO
Standby
Stackwise-480
Active SUP Active
Standby SUP SSO
Standby
Catalyst 9300

Catalyst 9400

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 253
Enhanced Fast
Software Upgrade
 Achieving High Availability on Catalyst 9300
Enhanced Fast Software Upgrade
• eFSU provides a mechanism to upgrade and
Control-Plane
downgrade the software image by segregating the
RIB
Control plane and Data Plane update Prefix Next Hop

10.0.0.0 10.1.1.1
• It updates the control plane by leveraging the NSF/GR
10.1.0.0 10.1.1.1
Architecture with Flush and Re-Learn mechanism to 10.20.0.0 10.1.1.1
reduce the impact on the data plane

Data Plane

FIB Table

Prefix Next HOP

10.1.1.1 aabbcc:ddee32

10.1.1.2 adbb32:d34e43

192.168.0.0 aa25cc:ddeee8

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 255
 Fast Software Upgrade
• Regular Upgrade Vs Enhanced Fast Software Upgrade Process

#Install add file image activate commit Enhanced Fast Software Upgrade

#Install add file image activate reloadfast commit

< 30 seconds of
traffic impact

Traffic is impacted throughout the upgrade cycle

* Limited Controlled Availability in 16.10.1

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 256
 Enhanced Fast Software Upgrade
• Supported and Unsupported Designs without Stackwise-480

Layer 2/3 Access Layer Designs– eFSU Supported

STP
L2 Only L2 Only L3 connections with
x x Vlan1-10 Routing Protocols

Access Access OSPF

Layer Layer ISIS

Unsupported Designs

L2 Extensions with L2 Only Design with

Access Layer Device LACP and PagP
Port-channels

Access MEC MEC

Access Future
Layer Layer

#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Enhanced Fast Software Upgrade
CLI Commands

• FSU is supported only in install mode

• One step command which activates the fast software upgrade and
commits it

9300# install add file flash:cat9k_iosxe.BLD_V1610 activate reloadfast commit

• Fast Reload without Software upgrade

9300# Reload Fast

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 258
Enhanced Fast Software Upgrade
Restrictions
• Enhanced FSU is not supported on a Stackwise-480
• Enhanced FSU is only supported and tested on Catalyst 9300-48U model
for ( 16.10.1* )
• Enhanced FSU is not supported on the switch configured with
LACP/PAGP Port-channels

* All 9300 models will be supported in 16.11.1 Release

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 259
High Availability in
Modular Chassis -
Catalyst
9400/9600
High Availability in Campus – Modular Access
Stackwise Virtual
Catalyst 9500-24Q Catalyst 9500-24Q
SSO
Active Standby NSF Aware

Routing Protocols
Modular Access

Active
NSF Capable
SSO
Standby
Stackwise-480
Active SUP Active
Standby SUP SSO
Standby

Catalyst 9400 Catalyst 9300

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 261
In-Service
Software
Upgrade(ISSU)
with Dual
Supervisors
 Supervisor Redundancy
Eliminate single points of failure
for hardware and software components

Control/data plane resiliency Control Plane

 Separation of control and forwarding plane
 Seamless restoration of Route Processor control and data plane
failures
<200 ms

MANAGEMENT PLANE
Active Sup
SSO
Link resiliency and Load Balancing Standby Sup
 Reduced impact of Line Card hardware and software failures

Line Card
Planned outages
 Seamless software and hardware upgrades
FORWARDING/DATA PLANE

Catalyst 9400/9600
#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 263
 Dual Supervisor ISSU
ISSU Overview
Control Plane

• ISSU provides a mechanism <200 ms

to perform software upgrades and downgrades

MANAGEMENT PLANE
Active Sup
without taking the switch out of service SSO
Standby Sup
• Leverages the capabilities of NSF and SSO to allow
the switch to forward traffic during Supervisor IOS
upgrade (or downgrade) Line Card

• Key technology is the ISSU Infrastructure

• Allows SSO between different versions
FORWARDING/DATA PLANE

Catalyst 9400/9600

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 264
ISSU Process
Dual Supervisors
Start ISSU • ISSU Process leverages SSO/NSF
Architecture

• Uplinks on both active and standby SUP

Uplinks are forwarding traffic

Active Supervisor
SSO
Standby Supervisor

Line Card

• Convergence is less than 200 msec

Catalyst 9400

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 265
C9K ISSU
Dual Supervisor ISSU

3 Step Process
• Install add file <tftp/ftp/flash/disk:*.bin>
Granular Control on
the upgrade process
• Install activate ISSU
with ability to rollback
• Install commit

1 Step Process
• Install add file <tftp/ftp/flash/disk:*.bin>activate ISSU commit Single Command
to perform
complete ISSU

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 266
 C9K ISSU Workflow
1. ISSU Started, Image is
expanded on Active and Standby

V1 S1 Active
If S2 fails to become standby
it will revert back to step 1

V1 S2 Standby Abort Timer

Starts

2. Standby Reloads
with the new V2
Image

5. ISSU V2 S1 Standby
V1 S1 Active
Expired Abort timer will
Complete revert to Step 2 and then
V2 S2 Active Step 1 V1 V2 S2 Standby

Abort Timer
Expired

Abort Timer
Stopped
V1 V2 S1 Standby
3. Auto-Switchover causes S2 to
4. ‘Commit’ Keyword become new active and S1
stops the abort timer
V2 S2 Active
reloads with the new V2 image

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 267
High Availability in
Distribution/Core-
Catalyst
9400/9500/9600
High Availability in Campus – Distribution/Core

Stackwise Virtual
Catalyst 9500-24Q Catalyst 9500-24Q
SSO
Active Standby NSF Aware

Routing Protocols
Modular Access

Active
NSF Capable
SSO
Standby
Stackwise-480
Active SUP Active
Standby SUP SSO
Standby

Catalyst 9400 Catalyst 9300

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 269
Stackwise Virtual
 Stackwise Virtual
Topology Comparisons

Traditional SV - Physical SV- Logical

HSRP or
VRRP

LACP SVL
or PAGP

STP or LACP or MEC

MST PAGP

Access Switch Access Switch Access Switch Access Switch Access Access Switch
Stack Stack
Switch Stack

Benefits of Stackwise Virtual

Simplify Operations by Eliminating STP, FHRP and Multiple Touch-Points

Double Bandwidth & Reduce Latency with Active-Active Multi-chassis EtherChannel (MEC)

Minimizes Convergence with Sub-second Stateful and Graceful Recovery (SSO/NSF)

271
271
#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
 Traditional L2 / L3 Campus Multi-
Layer
Switches
Non-Stack
L2
Switches
Campus Core Network Design

94 Total Devices of Image

& Configuration Management
168 Port-Channels
168 Access Trunks
4032 User Ports

Design Considerations:
STP Loop Prevention
CAM & ARP Tuning
FHRP Tuning / Priority
Routing Protocol Tuning
PIM Tuning / DR priority
Building 1 Building 2 Building 3 Building 4
1000 Ports 1000 Ports 1000 Ports 1000 Ports 94 Separate Configurations
of Hostname, VLAN DB, IP/GW, SNMP,
NTP, TACACS, VTY, etc.

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 272
 Stackwise Virtual Core with Access Stacking
VSS

Stackwise Stacked
Virtual L2
Switches Switches
Campus Core Network Design

25 Total Devices of Image

& Configuration Management
24 Port-Channels
24 Access Trunks
4032 User Ports

Design Considerations:
STP Loop Prevention
CAM & ARP Tuning
FHRP Tuning / Priority
Routing Protocol Tuning
PIM Tuning / DR priority

Building 1 Building 2 Building 3 Building 4

1000 Ports 1000 Ports 1000 Ports 1000 Ports 25 Separate Configurations
of Hostname, VLAN DB, IP/GW,
SNMP, NTP, TACACS, VTY, etc.

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 273
Stackwise Virtual
Architecture
 Stackwise Virtual Architecture
Control Plane

• Unified Control Plane Active

• Manage, Configure and

troubleshoot two switches Standby

as a single switch

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 275
 Stackwise Virtual Architecture
Data Plane

Catalyst 9500-24Q Catalyst 9500-24

• Active/Active Data Plane

Active Stackwise-Virtual Link Standby

• Both the switches are capable of

forwarding the traffic locally Port-Channel

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 276
 Stackwise Virtual Components

• Stackwise Virtual Link Dual-Active Detection Link

• Dedicated Stacking Link facilitating Catalyst 9500-24Q Catalyst 9500-24
communication between the
switches
• Dual Active Detection Link Active Stackwise-Virtual Link Standby

• Dedicated Connection to check and

avoid dual-active scenario
Port-Channel

• Multi-Chassis Ether-channel
• Port-Channel Spanning across
Stackwise virtual switches
• L2 and L3 Port-channels

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 277
 Stackwise Virtual Link
Dual-Active Detection Link
• Inter-Chassis System Link Catalyst 9500-24Q Catalyst 9500-24Q
o No network protocol operations
o Invisible in network topology
o Transparent to network level troubleshooting

• SVL Control Link SVH L2 L3 Payload CRC

Dist-1# show stackwise-virtual link

o Carries all system internal control traffic
Stackwise Virtual Link(SVL) Information: Stackwise-Virtual Link
o Single member-link and dynamic election during -----------------------------------------------------
bootup Flags:
--------
o Shared interface for network/data traffic Link Status
---------------

• Payload Overhead
U-Up D-Down
Protocol Status
---------------------
o Every single packet encapsulated with 64B of S-Suspended P-Pending E-Error T-Timeout R-Ready
StackWise Virtual Header (SVH) -----------------------------------------------------------------------
Switch SVL Ports Link-Status Protocol-Status
o Non-bridgeable and Non-routeable. ------ --- ----- ----------- ----------
-----
o SVL must be directly connected between two stack- 1 1 FortyGigabitEthernet1/1/1 U R
member switch systems FortyGigabitEthernet1/1/2 U R
2 1 FortyGigabitEthernet2/1/1 U R
FortyGigabitEthernet2/1/2 U R

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 278
 SVL – Capacity Planning

SW-1 SVL SW-2

Catalyst 9k Catalyst 9k

• Plan SVL bandwidth capacity to reduce congestion point, handle failures and specific configurations

• Three major points to consider for planning :

o Total Uplink BW Per Stack-Member. Ability to handle data re-route during uplink failures without network congestion
o Handling egress data to single-homed devices (Non-recommended design)
o Remote network services such as SPAN

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 279
 StackWise Virtual – Multi-Chassis EtherChannel
• Multi-Chassis EtherChannel (MEC) in StackWise Virtual
enables cross stack-member link bundling into single
logical L2/L3 Interface SW-1 SVL SW-2

• StackWise Virtual supports 128 maximum MEC –

Catalyst 9k Catalyst 9k
Port-Channel ID 1-127 available for L2/L3 network configurations
Port-Channel ID 128 is internally reserved for SVL purpose

• MECs can be deployed in three modes –

Cisco PAgP, LACP and Static (ON)

• Combining StackWise Virtual and Layer 2 or Layer 3 MEC

builds simplified, scalable and highly resilient campus
network

• MEC is an primary network design component to enable –

Simplified STP loop-free network topology
Consistent L3 control-plane and network design as traditional
Standalone mode system
Deterministic sub-second network recovery

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 280
Stackwise Virtual
Inter Chassis SSO/NSF The original Standby Switch now takes over as
2 the new Virtual Switch Active

Virtual Switch initiates Graceful Restart (NSF)

SW-1 SW-1 Non Stop Forwarding of packets continues

using hardware entries synched to Switch 2

NSF Aware neighbors exchange protocol

updates with the new Virtual Switch Active

Switch 2
Switch Is Down Virtual Switch Active

Virtual Switch incurs a failure of the

1 (SSO) Active Switch 1

The Standby Switch detects failure by

loss of all SVL ports, or no replies to
SSO keep-alive packets Switch 1

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 281
 OSPF
StackWise Virtual – Non-Stop Forwarding (NSF)
D6-9500sv-1(config)#router ospf <ID>
Core
D6-9500sv-1(config-router)#nsf cisco | ietf

EIGRP
D6-9500sv-1(config)#router eigrp <AS ID>
D6-9500sv-1(config-router)#nsf
SW-1 SVL SW-2
BGP
D6-9500sv-1(config)#router bgp <AS ID> Catalyst 9k Catalyst 9k
D6-9500sv-1(config-router)#bgp graceful-restart

MPLS LDP
D6-9500sv-1(config)#mpls ldp graceful-restart

• StackWise Virtual is NSF-Capable and NSF-Helper system.

• NSF capabilities for all Layer 2 protocols and several Layer 3 Unicast and Multicast routing protocols. Including VRF and
MPLS.

• NSF is mandatory configuration for graceful recovery during switch over conditions. Default on for Multicast protocols, manual
configuration required for each Unicast and MPLS LDP protocol.

• Implement IETF based OSPF NSF capability with “nsf ietf” CLI if OSPF neighbor is based on Cisco NXOS.

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 282
 High Availability
Dual-Active Detection
In a SVL Domain, one switch is elected
as Active and the other as Standby

All Neighbors view SVL as a

single Entity, single MAC, single IP

Since the SVL is always configured

as a Port Channel, the chance of the SVL

entire SVL going down is remote…

However… IT IS POSSIBLE! 

Recommend to deploy the SVL with 2 or more links, distributed across ASIC’s for highest redundancy

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 283
 High Availability
Dual-Active Detection

If the entire SVL bundle fails, the SVL Domain will enter
into a “Dual Active” scenario
Both switches transition to SSO Active state, and share
the same network configuration
• IP addresses, MAC address, Router IDs, etc.
This can cause communication problems in the network!

3 Step Process
SVL
Dual-Active Detection - using any detection method
1 enabled in the system.

Previous SVL Active shuts down ALL interfaces, and enters

2 “Recovery Mode”… preventing further network disruption

Dual-Active Recovery - when the SVL recovers, the switch

3 in Recovery Mode will reload to boot into a preferred standby
state

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 284
High Availability
Dual-Active Protocols

Fast Hello Enhanced PAGP

Hello Hello
Switch 1 Switch 2 Switch 1 Switch 2
Active Standby Active Standby

 Direct L2 Point-to-Point Connection  Requires ePAGP capable

neighbor:

Sub-Second Convergence
 Sub-Second Convergence
 Typically ~50-100ms
 Typically ~200-250ms

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 285
Stackwise Virtual
Network Designs
 StackWise Virtual – Access Network Design
• Single-home network design is non-recommended
SW-1 SVL SW-2

• Cannot leverage any distributed StackWise Virtual

architecture benefits. Catalyst 9k Catalyst 9k

• Non-congruent L2 or L3 network design with –

Centralized network control-plane processing over SVL
Asymmetric forwarding plane. Ingress data may traverse
L2 L3
over SVL interface and oversubscribe the ports Access Access

• Single-point of failure in various faults – Link/SFP/Stack-

Member failure, SSO switchover, ISSU etc.

• Cannot be trusted switch for dual active detection

purpose using ePAgP.

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 287
 Unicast Forwarding Path

Multicast Forwarding Path

StackWise Virtual – Core Network Design

MEC Design
VSS Enabled Core – ECMP Design Standalone
Standalone Core
Core – ECMP Design
– EtherChannel Design

VSL

VSS-Core Standalone-Core

SVL SVL
SV-Dist SV-Dist

•• Improved System
ECMP network Performance
doubles – Singleload
control-plane MEC hatredundant
and reduces 50% control-plane load in Core
topologies
•• Simple
UnicastTopology – Abstracts
routing protocol hardware
installs ECMP. layer with routing
Multicast single neighbor and single
installs single bestInterface
Outgoing forwarding
Listpath
(OIL)
•• StackWise-Virtual
Improved egress forwarding
Network Performance decisionunicast
– Consistent is across all ECMPdesign.
forwarding paths Increase in multicast switching capacity in core
• Protocol and scale-dependent network recovery
• Improved App Performance – Increased unicast and multicast load sharing input variables
• Resilient – Protocol and scale-independent network recovery

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 288
Stackwise Virtual
ISSU
 StackWise Virtual – Software Upgrade
Auto Software Upgrade Cisco Prime Infra SWIM Upgrade In-Service Software Upgrade (ISSU)
Cisco Prime Infra

IOS version – Y IOS version – X

SVL SVL

Without Manual Upgrade

FTP | SFTP | SCP

IOS version – Y IOS version – Y

X
SVL

With
Auto Upgrade

• StackWise Virtual members must have common • Cisco IOS software upgrade from centralized • Cat 9500 series systems deployed in
IOS software version to pair in SSO redundancy Cisco Prime Infrastructure Software Image StackWise Virtual mode will support ISSU
state Management (SWIM)
• StackWise Virtual ISSU support is currently
• Stack member with version mis-match with • Supports internal or external file distribution targeted for 16.7.0
ACTIVE switch will fail to RPR mode. server with – FTP, SFTP and SCP protocols
• Plan for network downtime during software
• Enable “software auto-upgrade enable” • Upgrade single or multiple StackWise Virtual upgrade on both StackWise Virtual systems
command to automate upgrade process. domains based on automated schedule or
on-demand.
• System must boot in Install mode (Default and
Recommended). Auto Upgrade not supported in
Bundle mode.

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 290
 Stackwise Virtual ISSU
ISSU Overview

Dual-Active Detection Link

Catalyst 9500-24Q Catalyst 9500-24Q
• ISSU provides a mechanism 16.9.3
16.9.2
to perform software upgrades and
downgrades without taking the
Stackwise-Virtual Link
switch out of service
• Leverages the capabilities of NSF
and SSO to allow the switch to
forward traffic during Supervisor IOS
upgrade (or downgrade)
• Key technology is the ISSU
Infrastructure
• Allows SSO between different versions

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 291
C9K ISSU
Stackwise Virtual ISSU and Dual Supervisor ISSU

3 Step Process
• Install add file <tftp/ftp/flash/disk:*.bin>
Granular Control on
the upgrade process
• Install activate ISSU
with ability to rollback
• Install commit

1 Step Process
• Install add file <tftp/ftp/flash/disk:*.bin>activate ISSU commit Single Command
to perform
complete ISSU

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 292
Stackwise Virtual ISSU
ISSU Process

Install ISSU
Dual-Active Detection Link
Catalyst 9500-24Q Catalyst 9500-24Q
Auto-Switchover 1st Sub-second
2nd Sub-second 16.9.3
16.9.2 16.9.3
16.9.2 traffic
traffic convergence
convergence
Stackwise-Virtual Link

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 293
 C9K ISSU Workflow
1. ISSU Started, Image is expanded
on Active and Standby

V1 S1 Active
If S2 fails to become standby
it will revert back to step 1

V1 S2 Standby Abort Timer

Starts

2. Standby Reloads with

the new V2 Image

5. ISSU V2 S1 Standby
V1 S1 Active
Expired Abort timer will
Complete revert to Step 2 and then
V2 S2 Active Step 1 V1 V2 S2 Standby

Abort Timer
Expired

Abort Timer
Stopped
V1 V2 S1 Standby
3. Auto-Switchover causes S2 to
4. ‘Commit’ Keyword become new active and S1
stops the abort timer
V2 S2 Active
reloads with the new V2 image

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 294
Graceful Insertion and
Removal (GIR)
 High Availability Architecture in Campus – GIR
Core

Catalyst 9500-24Q Catalyst 9500-24Q

Routed Access

Routing Protocols

Active
SSO
Standby
Stackwise-480
Active SUP Active
Standby SUP SSO
Standby

Catalyst 9400 Catalyst 9300

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 296
Graceful Insertion and Removal on Catalyst 9000
Isolation of Switch from network Gracefully

Stop Maintenance

Distribution Layer
Start Maintenance

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 297
Graceful Insertion and Removal

Upgrades with no or Minimal Traffic Loss

Simple
Comprehensive Node Isolation Framework Customizable
Non-Traffic
Impacting
Easy Execution with a single command

Highly Customizable workflow

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 298
 L2 and L3 Topology with GIR Isolation
9300#start maintenance
Template default will be applied.
Do you want to continue?[confirm]
*Mar 25 17:43:20.162: %MMODE-6-
MMODE_CLIENT_TRANSITION_START: Maintenance Isolate
start for router isis 1
*Mar 25 17:43:50.213: %MMODE-6-
MMODE_CLIENT_TRANSITION_COMPLETE: Maintenance Isolate
complete for router isis 1
*Mar 25 17:43:50.213: MMODE-6-
MMODE_CLIENT_TRANSITION%_START: Maintenance Isolate
start for shutdown l2
Set-overload-
*Mar 25 17:44:20.214: %MMODE-6-
MMODE_CLIENT_TRANSITION_COMPLETE: Maintenance Isolate Set-overload-
bit ISIS
complete for shutdown l2 bit
Set-overload-bit
*Mar 25 17:44:20.214: %MMODE-6-MMODE_ISOLATED: System
is in Maintenance

Order for Maintenance:

BGP -> IGPs in parallel (ISIS) -> L2

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 299
 L2 and L3 Topology with GIR Isolation
9300#stop maintenance
*Mar 25 19:15:40.235: %MMODE-6-
MMODE_CLIENT_TRANSITION_START: Maintenance
Insert start for shutdown l2
*Mar 25 19:16:10.237: %MMODE-6-
MMODE_CLIENT_TRANSITION_COMPLETE: Maintenance
Insert complete for shutdown l2
*Mar 25 19:16:10.237: %MMODE-6-
MMODE_CLIENT_TRANSITION_START: Maintenance
Insert start for router isis 1
*Mar 25 19:16:40.288: %MMODE-6- No set-overload-
MMODE_CLIENT_TRANSITION_COMPLETE: Maintenance
Insert complete for router isis 1
No set-overload-
bit ISIS
*Mar 25 19:16:40.612: %MMODE-6-MMODE_INSERTED: No set-overload-bitbit
System is in Normal Mode

Order for Maintenance:

L2  IGPs in parallel (ISIS) -> BGP

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 300
 Graceful Insertion and Removal
Default and Customizable Templates
• Default Template 9300L#show system mode maintenance template default
System Mode: Normal
• System Generated Profile based on default maintenance-template details:
the switch configuration
router isis 1
shutdown l2
9300L#show system mode maintenance template test
• Customized Template System Mode: Normal
Maintenance Template test details:
• User Configured Profile based on shutdown l2
specific configuration or use case

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 301
 Graceful Insertion and Removal
• Snapshots
Switch#show system snapshots compare before_maintenance
• Automatic Snapshots after_maintenance
• Snapshots are automatically ================================================================================
Feature Tag .before_maintenance .after_maintenance
generated when entering and ================================================================================
exiting maintenance mode [interface]

--------------------------------------------------------------------------------
[Name:Vlan1]
• Captures operational data packetsinput
[Name:GigabitEthernet1/0/3]
181587 **181589**

from the running system like packetsinput 101531 **101550**

Vlan’s, Routes etc. broadcasts

packetsoutput
80893
211568
**80910**
**211594**
[Name:GigabitEthernet1/0/8]
output 00:00:00, **00:00:04,**
packetsinput 6915 **6918**
packetsoutput 57677 **57706**

• User Configured Snapshots [Name:GigabitEthernet1/0/17]

packetsinput 101528 **101550**
broadcasts 80891 **80910**
packetsoutput 211570 **211600**
• Snapshots can be collected
manually for comparing and
troubleshooting

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 302
 Graceful Insertion
• Maintenance Profile Optionsand Removal

• On-Reload 9300(config)#system mode maintenance

• If the switch is reloaded in 9300(config-maintenance)#?
maintenance mode, the
maintenance mode submode configuration commands: default
switch will come back in Set a command to its defaults
maintenance mode
exit Exit from maintenance configuration mode
failsafe Client ack timeout
no Negate a command or set its defaults
• Failsafe
on-reload On reload maintenance mode configuration
• Timeout for Client
template use maintenance-template
Acknowledgement
timeout maintenance duration

• Duration
• The Switch will come out of
maintenance after the
configured duration

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 303
Configuration Profiles
• Maintenance-mode profile is applied when entering GIR mode,
• Normal-mode profile is applied when GIR mode is exited.

Automatic Profiles Custom Profiles

• Generated by default • User created profile for maintenance-
mode and normal-mode using
• GIR is applied to all protocols running “templates”
on the system
• Flexible selection of protocols for
• GIR state machine uses Registry isolation
mechanism to interface with client
protocols • Use: maintenance windows and
isolation during troubleshooting using
• Use: Maintenance Windows preconfigured templates

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 304
Open IOS-XE
Patchability
Ready for Software Patching

SMU is an emergency point fix positioned for expedited delivery to a customer in case of a network down or
revenue affecting scenario.

Cold Patching: Install of a SMU will require a system Hot Patching: Install of a SMU does not require a
reload in the first release. It is traffic impacting. reload.

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 306
Why SMUs are needed?
Software Upgrades are Challenging

Cost

• Expensive Upgrades - Business Loss

• Each device upgrade causes Network outage

Time SMU
• Reduced IT staff slows software roll out Point Fixes
Reduces Validation –
• Physical presence required Scope & Time

Scope

• New Code requires bug analysis, certification

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 307
 SMU Lifecycle – CLI SMU SMU

SMU
Repository
Switch# install add …
Switch# install remove …
show install active
SMU Committed Copy to Device
show install committed
Memory: Process: Memory: Process:
show install inactive

Switch# install commit … Catalyst 9k

show install packages
Switch# install activate …

SMU Removed
Memory: Process: SMU Applied
Memory: Process:

Switch# install deactivate … Switch# install commit …

SMU Committed
Memory: Process:
#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 308
Software Update Creation – Work Flow
Commit
Service NO
Fixed to next
DDTS Impacting
? SW
?
TAC/HTTS release
YES
NO
Investigat
Request Patch
e

DE Engineer Create SW update

SMU Council
approval
...
Create
DT Engineer SW Update

Unit /Dev Test

Documentation
Build Engineer Integration Test

Post on CCO

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 309
 SMU Management Options
Problem: SMU Life Cycle Mgmt. at Scale is a challenge with (1) Device types (2) SW versions

There are three potential solutions

Programmable APIs
CLI Controller (Cisco DNA-C) (3rd Party tools -
Chef/Puppet/Ansible)

 Small Scale Deployments  Mass Scale Deployments  Mass Scale Deployments

 Per Device Access  SMU Analysis

 Standard Programmatic Interfaces
 Full Control  SMU Life Cycle Mgmt • Open Standards APIs
• Consistent across multiple
 SMU Alerts and Notification
platforms
 SMU Orchestration across Geo’s
 Script Support (Shell, Perl,
Python)

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 310
Enterprise Campus Network Designs

Multi-Tier Layer2/3 Topology Multi-Tier Layer3 Topology

Stackwise
Virtual

ISSU
Stackwise
Virtual
GIR

Stackwise-480 Stackwise-480
Dual Sup/ Dual Sup/ Stackpower
Stackpower
Power Power
Redundancy FSU Redundancy FSU

ISSU ISSU

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 311
High Availability on Catalyst 9000
Catalyst 9300 Catalyst 9400 Catalyst 9500/9600
Graceful Insertion & Removal(GIR)
Supported Protocols: ISIS, OSPF,BGP, HSRP,VRRP

Software Maintenance Upgrade

• Cold Patching
• Hot Patching

• Stackwise-480 • Stackwise Virtual • Stackwise Virtual

• Stack Power • ISSU(Stackwise • ISSU with Stackwise
• Fast Software Virtual) Virtual
Upgrade • ISSU ( Dual 16.9.2

• Enhanced Fast Supervisor)

Software Upgrade

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 312
#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Quality of Service
(QoS)
Why QoS ?

Bandwidth Savvy
Guaranteeing voice
Video Quality Business
quality
Applications

protect network
de-prioritizing non-
infrastructure to protecting the
business
deal with abnormal control planes
applications
events

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 315
Determining Business Relevance
How Important is an Application to Your Business?

Relevant Default Irrelevant

• These applications directly • These applications may/may not • These applications do not
support business objectives support business objectives support business objectives and
(e.g. HTTP/HTTPS/SSL) are typically consumer-oriented
• Applications should be
classified, marked and • Applications of this type should • Applications of this type should
treated marked according to be treated with a Default be treated with a “less-than Best
industry best-practice Forwarding service Effort” service
recommendations
RFC 4594 RFC 2474 RFC 3662

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 316
 Catalyst 9000 Campus QoS Design
Application DSCP 2P6Q3T

Network Control (CS7) EF PQ Level 1 (10%)

Internetwork Control CS6 CS5
CS4 PQ Level 2 (20%)
VoIP EF
CS7 & CS6 Q6
Broadcast Video CS5
CS3 & CS2 (BWR 10%)
Multimedia Conferencing AF4
Relevant Q5
AF4
Realtime Interactive CS4 (BWR 10% + WTD)
Multimedia Streaming AF3 AF3 Q4
(BWR 10% + DSCP-Based WTD)
Signaling CS3
Q3
Transactional Data AF2 AF2
(BWR 10% + DSCP-Based WTD)
Network Management CS2
AF1 Q2
Bulk Data AF1 (BWR 5% + DSCP-Based WTD)
CS1
Irrelevant Scavenger CS1

Default Best Effort DF DF Q1 (BWR 25%)

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 317
QoS is a set of tools

#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Catalyst 9000 – QoS Tools

Conditional
Policing Marking
Trust
(By Classification
Default)
Unconditional
Marking

Network Ingress Tools/Actions

Interface
WTD
Q0
Conditional WTD
Policing Q1
Marking WTD
Q2
WTD
Scheduler Q3 2P6Q3T
1P7Q3T
Q4 WTD
0P8Q3T
Unconditional Q5 WTD

Marking WTD
WRED
Q6
Q7 WTD
Egress Tools/Actions

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 319
Catalyst 9000 Family – Consistent QoS
Highlights

1P7Q3T or 8 Queues 8 Queues

MQC
2P6Q3T per Port per Port

Trust HQoS No DBL Buffer

By Default 2-Level WRED 8MB – 36MB
Depending on the ASIC

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 320
 Trust & Conditional Trust

Trust Model Untrust Model

• Trust Specific Devices – Cannot

Trusted trust multiple devices at the same
time

interface GigabitEthernet 1/0/1

• Catalyst 9000 family trust all ports by Default trust device cisco-phone [or]
(DSCP based) trust device cts [or]
trust device ip-camera [or]
trust device media-player
DSCP/Prec/CoS on the incoming packets are
retained interface GigabitEthernet 1/0/1
trust device cisco-phone
service-policy input CISCO-IPPHONE

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 321
Traffic Classification
• A class-map can be defined as a logical OR “match-any”

• And now “match-all” is supported on Catalyst 9000

OR AND
Switch(config-cmap)# class-map match-any VOICE Switch(config-cmap)# class-map match-all VOICE
Switch(config-cmap)# match ? Switch(config-cmap)# match ?
access-group Access group access-group Access group
class-map Class map class-map Class map
cos IEEE 802.1Q… cos IEEE 802.1Q…
dscp Match DSCP … dscp Match DSCP …
ip IP specific values ip IP specific values
non-client-nrt Match non-client NRT non-client-nrt Match non-client NRT
precedence Match Precedence… precedence Match Precedence…
protocol Protocol protocol Protocol
qos-group Qos-group qos-group Qos-group
vlan VLANs to match vlan VLANs to match

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 322
 Marking
[class-maps omitted for brevity]
policy-map MARKING-POLICY
class VOIP
• Three types of marking policies: set dscp ef
• Conditional Policer based marking class MULTIMEDIA-CONFERENCING
• Unconditional Explicit marking set dscp af41
class SIGNALING
• Table Map based marking set dscp cs3
class TRANSACTIONAL-DATA
• Marking with ‘set’ set dscp af21
class BULK-DATA
• Marking with Table Maps set dscp af11
class SCAVENGER
• Table-maps can be applied only on class- set dscp cs1
default class default
Catalyst3650(config-pmap-c)#set ? set dscp dscp table COS2DSCP
cos Set IEEE 802.1Q/ISL class…
dscp Set DSCP in IP(v4) and IPv6…
ip Set IP specific values table-map COS2DSCP
precedence Set precedence in IP(v4) and IPv6…
qos-group Set QoS Group map from 5 to 46
default copy

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 323
 Policing

1 Rate 2 Color 2 Rate 3 Color

CIR PIR CIR

CBS PBS CBS

No No B>Tc No
B<Tc B>Tp

Packet of Yes Packet of Yes Yes

Conform Exceed Size B Violate Exceed Conform
Size B

Action Action Action Action Action

police cir 100000000 bc 3125000 conform- police cir percent 10 pir percent 50
action set-dscp-transmit af41 exceed-action conform-action transmit exceed-action set-
drop dscp-transmit af11 violate-action drop

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 324
 TCAM Resources
QoS TCAM Resources Cat9300 Cat9400/cat950
0
IPv4 Entries 5000 (256 bits) Entries 18000 (256 bits)
Entries
IPv6 Entries Half the IPv4 (512 bits) Half the IPv4 (512

TCAM Entries (shared between cores)

bits)
Class-maps (Ingress) 255 255
Class-maps (egress) 255 255
Table-maps (ingress) 16 16
Table-maps (egress) 16 16
Aggregate Policers Per ASIC: 4K (2K per Per ASIC: 4K (2K per
Core) (1R2C), 2K (1K Core) (1R2C), 2K (1K
per core) (2R3C) per core) (2R3C)
(not shared across (not shared across
cores) cores)
Wired Queues/port 8 queues (2 can be 8 queues (2 can be
priority) priority)

256 Bits Entries Each

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 325
 Buffer Size Comparison per Platform
UADP2.0XL UADP3.0
UADP2.0 Mini

3.4 MB Egress
10 MB Egress 27 MB Egress
0.6 MB
FIFO
0.5 MB 0.75 MB – 1
MB Stack 1.5 MB 5 MB
– 1 MB
Ingress FIFO FIFO
1.5MB – 3.5 2.6 MB
per ASIC 6 MB
0.4MB- 1.4
MB Stack Stack
1.5 MB Ingress
Ingress
UADP2.0

5 MB Egress

per Core16 MB per Core 36 MB

0.75 MB per ASIC 16 + 16 MB per ASIC 36 MB
FIFO
0.5 MB 1 MB – 1.75
– 1 MB MB Stack
Ingress
per Core 8 MB
per ASIC 8+8 MB

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 326
Auto QoS
• Generate templates of Class-maps and Policies based on best practices
per connected device type
• Different template Versions (Latest: 4.0)

auto qos voip {cisco-phone | cisco-softphone | trust}

auto qos video {cts | ip-camera | media-player}
auto qos classify [police]
auto qos trust {cos | dscp}

Reference:
www.cisco.com/en/US/docs/solutions/Enterprise/Video/autoqosmediacampus.pdf

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 327
IQS Scheduling to Stack Interface

Stack Interface

Q PBC – Packet Buffers Complex SQS AQM

IQS Q Q
EQS

DSCP DSCP
46 0

IQS will schedule

High Priority packets
first over stack

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 328
Dynamic Threshold Scalability (DTS)
Switch • Shared buffer is good for burst
absorption.

Unused
Dynamic Shared Pool (DTS based) • Dedicated buffer is good for
predicated performance for each

Unused
port.

Unused
• Buffer management is flexible:
Dedicated plus shared.

Configurable dedicated

Unused
•
threshold per port/queue
Unused

Unused
Unused
• Configurable global maximum
shared threshold
• Automatically adjusted depends
on the available shared pool
Port 1 Port 2 Port N
#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 329
DTS – Dynamic Fair Buffer Sharing
Maximum buffer per • SoftMin – Minimum Shared buffer given to the port
queue (Configurable)
• SoftMax – Maximum Shared buffer the port can
SoftMax consume from shared Pool

• Port Soft Start – Moment when the Softmax start

Buffer Threshold

to reduce with step base

Maximum Burst • Port Soft End – Moment when SoftMin and

absorption SoftMax are equal

SoftMin

Dedicated for fairness

Shared Buffer Utilization
75% 100%
Port Soft Start Port Soft End
#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 330
 Default Buffer Allocation – Cat 9300
Soft Max
1200 Units

Soft Max
600 Units
Soft Max Soft Min
2400 Units 300 Units Soft Max
Hard Max 14400 Units
Soft Max 600 Units
400 Units Soft Min
150 Units Q0 Q1 Hard Max 0
Hard Max
Soft Max
100 Units Soft Min
9600 Units
Q0 Q1 Hard Max 0 3600 Units
Hard Max
2400 Units
Q0 Q1 Hard Max 0

1 Buffer Unit == 256 bytes

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 331
Default Buffer Allocation – Cat 9400
Soft Max Soft Max
Soft Max 3600 Units Soft Max 1056 Units
700 Units 700 Units
Soft Min Soft Min
Hard Max 225 Units Hard Max 264 Units
176 Units 176 Units
Q0 Q1 Hard Max 0 Q0 Q1 Hard Max 0

Soft Max
Soft Max 10800 Units
700 Units
Soft Min
Hard Max 337 Units
176 Units
Q0 Q1 Hard Max 0

1 Buffer Unit == 256 bytes

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 332
 Default Buffer Allocation – Cat 9500
Soft Max
7200 Units

Soft Max
3600 Units
Soft Max Soft Min
1800 Units Soft Max
4800 Units 28800 Units
Hard Max
Soft Max 1200 Units
800 Units
Hard Max
Soft Min Q0 Q1 Hard Max 0
800 Units Soft Min
200 Units Soft Max
19200 Units 7200 Units
Q0 Q1 Hard Max 0
Hard Max
4800 Units
Q0 Q1 Hard Max 0

1 Buffer Unit == 256 bytes

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 333
 Default Buffer Allocation per port Speed
Platform Port 100 / 1 / 2.5 / 5 Gbps 10 Gbps 40 Gbps
Speed (if applicable)

Queue HardMax SoftMax HardMax SoftMax HardMax SoftMax

Cat 9300 Q0 100 400 600 2400 2400 9600

Cat 9400 Q0 176 700 176 700 176 700
Cat 9500 Q0 200 800 1200 4800 4800 19200

SoftMin SoftMax SoftMin SoftMax SoftMin SoftMax

Cat 9300 Q1 150 600 300 1200 3600 14400

Cat 9400 Q1 225 3600 264 1056 337 10800
Cat 9500 Q1 800 3600 1800 7200 7200 28800

Notes: Softmax for Q0 is 4 times Hardmax.

Softmax for Q1 is 4 times “Soft Min”
Q1 doesn’t have Hardmax.

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 334
Security
Catalyst 9000 Security Features

• Mission-critical application
Visibility visibility (NBAR)*
Consistently delivered
• Full NetFlow telemetry
throughout the
Cisco® Catalyst® 9000 family
• Highest level of macro and
micro segmentation with
Segmentation SD-Access
• Multidomain policy integration

• MACsec 256-bit link encryption

Threat
• Encrypted Traffic Analytics*
defense
• Trustworthy solutions

* Roadmap on Cisco Catalyst 9500 High Performance and Catalyst 9600 Series

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 336
Security-
Trustworthy
Solutions
 Cisco Catalyst 9000 Platform Trustworthy Solutions
Design/ Plan/ Service/End
Source Make Quality Delivery
Develop Order of Life (EOL)

PnP SUDI Physical security practices + security technology innovations + logical security processes Secure boot
support Boot sequence
Two-way trust check

Integrity
Image signing
Authentic OS
verification
Malware protection

Hardware Runtime
authenticity defenses
Genuine hardware 64-bit ASLR

Cisco® trustworthy systems use industry best practices to help ensure full development lifecycle integrity and end-to-end
security

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 338
Cisco Trust Anchor Module (TAm)

• HW Based Entropy
Integrity Applications
• HW Authenticity Check
TAM Services Libraries • Secure PnP
• Integrity Verification

Crypto Functions
• Anti-Tamper Chip Design
Tamper-Proof Storage • Built-In Crypto Functions
Boot • Secure Storage
SUDI
Measurements

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 339
Secure Unique Device Identification (SUDI)

• Tamperproof ID for the device

• Binds the hardware identity to a key
pair in a cryptographically secure
X.509 certificate PID during
manufacturing
• Connections with the device can be
authenticated by the SUDI credential
• IEEE 802.1AR Compliant

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 340
Boot Sequence

Bootloader Operating System

Fetch Hardware Start-up Operating

Instruction set System

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 341
Cisco Secure Boot
Anchors Secure Boot in Hardware to Create a Chain of Trust
Cisco Secure Boot
Boot Code Integrity Anchored in Hardware

Step 1 Step 2 Step 3 Step 4  Only authentic signed

Cisco software boots
up on a Cisco platform

 The boot process stops

Hardware CPU CPU CPU if any step fails to
Anchor
Microloader Bootloader OS authenticate
Microloader

Microloader Bootloader OS launched  IOS “show software

checks checks OS authenticity” command
bootloader illustrates the results

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 342
 Secure Boot Verification during boot up
Microloader doesn’t display verification, if verification fails then the box doesn’t boot at all.

Initializing Hardware ...

System integrity status: 00000610 1 2

Rom image verified correctly
ROMMON Secure boot verification
System Bootstrap, Version 15.4(3r)S, RELEASE SOFTWARE (fc1)
Copyright (c) 1994-2014 by cisco Systems, Inc.
<snip>

<snip>
##########################################################
Boot image size = 425853700 (0x19620304) bytes

Package header rev 1 structure detected

Calculating SHA-1 hash...done
validate_package: SHA-1 hash:
IOS Secure boot verification calculated 334207fa:464503d3:2e7abd5f:160919d0:b425523b
expected 334207fa:464503d3:2e7abd5f:160919d0:b425523b

RSA Signed RELEASE Image Signature Verification Successful.

Package Load Test Latency : 6511 msec
Image validated
3 4
<snip>

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 343
 Secure Boot Verification after bootup
Switch#show software authenticity running ROMMON
<snip> (other packages not displayed) ------
Image type : Production
PACKAGE cat3k--universalk9.16.03.05..SPA.pkg Signer Information
------------------------------------------------------------------ Common Name : CiscoSystems
Image type : Production Organization Unit : IOS-XE

4
Signer Information
Common Name : CiscoSystems 2 Organization Name
Certificate Serial Number
:
:
CiscoSystems
53A3B3D2
Organization Unit : IOS-XE Hash Algorithm : SHA512
Organization Name : CiscoSystems Signature Algorithm : 2048-bit RSA
Certificate Serial Number : 54F33A2E Key Version : A
Hash Algorithm : SHA512
Signature Algorithm : 2048-bit RSA Verifier Information
Key Version : A Verifier Name : ROMMON
Verifier Version : System Bootstrap, Version 15.4(3r
Verifier Information
Verifier Name : mono Microloader
Verifier Version : 16.03.05 -----------
Image type : Release
SYSTEM IMAGE Signer Information
------------ Common Name : CiscoSystems
Image type
Signer Information
: Production
1 Organization Name
Certificate Serial Number
:
:
CiscoSystems
f01632135f43ae4bc1c4ca63a289b727
Common Name : CiscoSystems Hash Algorithm : HMAC-SHA256
Organization Unit : IOS-XE Verifier Information

3 Organization Name
Certificate Serial Number
:
:
CiscoSystems
54F33B36
Verifier Name
Verifier Version
: Hardware Anchor
: F01023R12.1817bb4af2014-05-23
Hash Algorithm : SHA512
Signature Algorithm : 2048-bit RSA
Key Version : A

Verifier Information
Verifier Name : ROMMON
Verifier Version : System Bootstrap, Version 15.4(3r

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 344
After Secure Boot, IOS Software Verifies that
Hardware is Authentic

STEP 5 STEP 6

STEP 1 STEP 2 STEP 3 STEP 4

Trust Anchor Trust Anchor
module module

Hardware CPU CPU CPU CPU CPU

Anchor
Microloader Bootloader OS OS OS

Microloader

Microloader Bootloader OS Launched Hardware Trust Anchor

Checks Checks OS Authenticity module stores
Bootloader Check values

Software authenticity Hardware authenticity

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 345
HW Authenticity Check
TAm

• Trust Anchor Module (TAm)

securely stores HW Identity
(SUDI)
• After the operating system is up
and running…
• IOS-XE automatically verifies
that the HW is genuine
5) Host Verification:
• Host nonce
• SUDI Cert Chain
• Signature Verification
• Product Info

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 346
Cisco Runtime Defenses
Address Space
Layout Randomization Object-Size Checking
(ASLR)

X-Space Safe-C Libraries

Hardware, Operating System, Compiler, and Development Best Practices

To protect against Buffer-Overflow and Return-Oriented Programming Attacks

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 347
Trustworthy Features on Cat 9000 Family
Catalyst 9000 Family
Features
(Open IOS-XE)

Image Signing Yes

Secure Boot Yes

Anti-Counterfeit Check Yes

Trust Anchor Module Yes

PnP SUDI Support Yes

Run Time Defenses Yes

X.509v3 SSH Authentication Yes

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 348
Automation and
Programmability
Automation and
Programmability
Why Programmability?

Reduction of human error

Easier Troubleshooting Time and money cost savings

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 351
IOS XE 16.X enables…

Cisco Solutions 3rd Party Integration DIY Solutions

Cisco DNAC Standards Based Interoperability Custom Development
Model Driven Model Driven
Programmability Telemetry

Zero Touch Guest Shell

Policy Automation Analytics Provisioning (On Box Python)
SDN Controllers CI/CD Tools NMS Systems App Hosting

Intent-based Intent-based Intent-based

Network Infrastructure Network Infrastructure Network Infrastructure

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 352
 IOS XE: Automating Network Device Lifecycle
Goal:
 Get devices into an operational state
Provisioning Automation Tools:
Goal:
 PXE, ZTP, PnP
 Continuously upgrade
network, incrementally  Python Scripting
and safely
Tools: Install
 Patching
 Config/Replace
Goal:
 Apply configuration to the
Upgrade Configure device

Goal: Tools:
 Add dynamic services,  Data Models
optimize behavior and  Programmable Interfaces
trouble shooting
Optimize
 Python Scripting
Tools:
 Operating Data Models
 Telemetry

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 353
Day 0 Provisioning Automation
Pre-boot Execution Zero Touch Cisco Network Plug
Environment (PXE) Client Provisioning and Play
Boot Server ZTP Server

PXE Booting ZTP Agent PnP Agent

ZTP script Switch PnP Agent Switch
Switch

Boot Image Network Device Device

Source
Programmable Open Open Open (via DNAC)
Interfaces

Ideal for heterogeneous / Optimized for Cisco networks

multi-vendor network environments (highly secure and scalable)

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 354
Configuration Management Today

CLI CLI

Physical and Virtual Network Infrastructure

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 355
 CLI YANG Models

Human Oriented Interface Machine Oriented Interface

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 356
What is YANG
• Data modeling language

• Yet Another Next Generation

• Text-based, readable
• Hierarchical, modular, and extensible
• Can represent operations or capabilities (RPC’s)
• Describes: configuration – operational state/parameters – actions – notifications
• Separate operation and configuration modules (trees)
• Vendor neutral [IETF, OpenConfig] and vendor/device specific models
• Each node may contain different combination of models and versions
• SNMP MIB’s can be translated to YANG modules
• IETF https://datatracker.ietf.org/wg/netmod/documents/
#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 357
Structured vs Unstructured Data
Un-structured Structured

John Smith 42 14155551212 Name: John Smith

Age: 42
Phone: +1-415-555-1212
What is this?

• His age? Keys Values

• The year he graduated college?
• Employee number?

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 358
Open Programmable APIs

Simplified View

NETCONF
RESTCONF
gRPC

NETCONF RESTCONF gNMI gRPC Protocols

CLI
Parser
Data Model YANG Models
Device Features
Device
Interface BGP QoS ACL … Features

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 359
Data Models

NETCONF
RESTCONF
gRPC

NETCONF RESTconf gNMI gRPC

Data Models
Data Model Open Native Open Native
Models Models Models Models

Configuration Operational
Device Features

Interface BGP QoS ACL …

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 360
Configuration vs. Operational data

Config-data Operational-data

• Tells the device what to do • How the device is operating

• Can write configuration data • Output form show command (other
than show run)
• Can read configuration data
• This is read-only
Examples: Examples:
switch> show run interface Loopback0 switch> show interface Loopback0
switch(config)# interface Loopback0

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 361
Data Models: Open vs Native

Open Native
Models Models

Industry definition Cisco definition

Compliant with standard Unique to a Cisco operating
(IETF, ITU, OpenConfig, etc) system

Example: ietf-diffserv-policy.yang Example: Cisco-IOS-XR-ipv4-bgp-cfg.yang

(IETF Diffserv data model) (IOS-XR BGP data model)

Open Models are a subset of the Native Models

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 362
Data Models

NETCONF
RESTCONF
gRPC NETCONF RESTCONF gNMI gPRC Protocols

NETCONF RESTconf gNMI gRPC XML JSON JSON kvGPB Encoding

Data Model SSH HTTP/HTTPs HTTP/HTTPs HTTP/2 Transport

Device Features

Interface BGP QoS ACL …

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 363
Streaming Telemetry

Export enriched, consistent and concise data with context from

network devices for a better user and operator experience

Periodic or Structured Data Scalable Reduced CPU

On-Change Load

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 364
Publication and Subscriptions
Dial-in vs Dial-out
Mac, Linux, etc

Dial-in Dial-out
Dynamic Configured

NETCONF RESTCONF gNMI gRPC

YANG Data Models

Open Native
Intent-based
Network Infrastructure
Configuration and Operation

Device Features
SNMP
Interface BGP QoS ACL …

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 365
 Model Driven
Telemetry

Telemetry Publication Options

Time, or event based

Q. What is the minimum publication interval?

A. On IOS XE 16.10: 1 second

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 366
Recap

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 367
Application
Hosting
Application Hosting in the Enterprise

Cloud Gateways with

IT Operations & Security Agents & Serverless Edge Customer Specific
Monitoring Tools Functions Compute Applications

Consolidate Physical Reduce App Latency

Enhance Visibility & Derive New Insights
Infrastructure &
Security Enforcement and Respond
Optimize App Traffic

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 369
Application Hosting in Catalyst 9K Platforms
IoT & Enterprise

Cisco Application Framework

Container Container Container Container

IOS XE Third- Third- Custom Custom

party party app app
app app

Cisco IOS XE Kernel

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 370
 Catalyst 9000 switch storage and compute
Catalyst 9500
Resource type Catalyst 9200 Catalyst 9300 Catalyst 9400 Catalyst 9500* Catalyst 9600*
High Perf*

Networking Front panel ports (1G) No Yes Yes* No No No

Memory No 2GB up to 8GB up to 8GB up to 8GB up to 8GB

CPU No 1 core (25%) 1 core (25%) 1 core (25%) 1 core (25%) 2 core (25%)
Resources
120GB 240-960GB 120GB 240-960GB 240-960GB
Storage No
(USB3.0/SSD) (SATA) (USB3.0/SSD) (SATA) (SATA)

Catalyst 9500
Catalyst 9300/9500 Catalyst 9400
high-performance For local storage and app
USB 3.0 M2 SATA M2 SATA hosting production
120GB 240/480/960GB 240/480/960GB • 3rd party USB drives in front
panel are not supported
• Applications can be hosted via
CLI too
Back panel Plug into removable SUP Back panel

* Roadmap
#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 371
IOS XE performance and security protection

• Memory and CPU usage for Apps

are bounded using Control groups
(cgroups).
• Process and files access for Apps
are isolated and restricted (using
user namespace)
• Disk usage is isolated using
separate storage.

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 372
Docker Container

}
App A App B
• Virtual Machine Bins/Libs Bins/Libs
Includes application, binaries & GBs
Guest OS Guest OS
libraries, an entire guest OS. Hypervisor
Host OS
Server

• Docker Container App A App B

}
format for Linux containers that makes the Bins/Libs Bins/Libs
MBs
process of creating and maintaining Docker Engine
Host OS
containers easier. Server

C9K supports native Docker container starting from IOS XE 16.12 release.

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 373
Docker Workflow
1 Dockerfile 2 Build Docker Image
docker build -t <app> .
or

docker pull <app>

4 Deploy App
docker save myapp > myapp.tar

3 App Descriptor
(Optional)

DockerTM

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 374
 Layer 2 Connectivity to
App Hosting Environment
Eth0
DockerTM

Trunk
10, 20

AppGigEthernet Port

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 375
App Lifecycle Management – State Transitions

app-hosting install appid myapp package usbflash1:myapp.tar

app-hosting activate appid myapp

app-hosting start appid myapp

install activate start

uninstall deactivate stop

app-hosting stop appid myapp

app-hosting deactivate appid myapp

app-hosting uninstall appid myapp

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 376
Application Management

Cisco DNA
Cisco
Center
DNA-C

REST
CLI
REST

Cisco Application Framework

Docker Docker Docker Docker Docker

3rd 3rd 3rd

Party Custom Custom
Party Party
App App
App App App

Host OS (IOS XE Kernel)

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 377
App Hosting Recap

Catalyst 9000

x86 CPU Linux-based OS Memory/Storage

install activate start

DockerTM
uninstall deactivate stop

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 378
 Cisco Catalyst 9000
switching application ecosystem

tshark

ISC DHCP
Server iPerf

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 380
 Sandbo
x

Eco
System
Exchang
e

https://developer.cisco.com/
#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 381
Closing & Wrap
up…
#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Catalyst 9K has fundamentally
changed the Networks

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 384
Multigigabit

UPoE
100G
40G
25G
10G Fabric Enabled Wireless
5G
1G 2.5G Embedded Wireless LAN Controller
10M 100M

With Catalyst 9K Or Networks are Ready for 11ax and New Speeds

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 385
 Security Secure Infrastructure Secure Transport

Encrypted Traffic Analytics Trustworthy Systems MACSEC

Traffic Hardware Man-in-

Analytics Authenticity the-Middle

Malware Compliance Two Way Run-time Wire-tapping Impersonation

Detection Trust Defense

With Catalyst 9K Our Networks are more Secure than ever

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 386
 Platform Resilience
Sub Second
Convergence

1+1 Redundancy ISSU

Active and Standby eFSU
SSO/NSF Hot Patching

Highly Available Software Upgrade

Networks Instant Fix

Stackwise Virtual
GIR
HSRP/VRRP

With Catalyst 9K Our Networks are Highly Available most Resilient

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 387
Full PoE+/UPoE AVB & PTP IoT Readiness

2 Event Fast
Classification POE

Perpetual
UPOE

802.3bt Type 3 (60W) Digitized Audio & Video PoE Innovations

LED Building Management Digital Court Rooms Media IoT IP Based

Lights Systems Rooms Devices End Points

With Catalyst 9K Our Networks are Ready for IoT

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 388
Application Hosting Model Based API & Streaming
& Containers Programmability Telemetry

Flexibility and Control Network Automation Monitoring the Network

Network Analytics Monitoring Build Your Own Consistent Models Real Time Monitoring On-change &
Tools Tools e.g. Kibana Periodic Telemetry
Candidate Datastore

With Catalyst 9K Our Networks are Programmable

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 389
Software Defined Access DNA Assurance

With Catalyst 9K Our Networks are Software Defined

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 390
 The Catalyst 9000 Family of Switches

Catalyst 9200 Catalyst 9300 Catalyst 9400 Catalyst 9500 Catalyst 9600
Fixed Access Switches Modular Access & Distribution Switches Fixed Core & Distribution Switches Modular Core & Distribution

Enabling Campus of the Future

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 391
 Core
Catalyst 9500/9600

L2 / L3 (IPv4, IPv6) Fixed & Modular Form Factors

Multicast
ACL Scale
Netflow Distribution
Catalyst 9400/9500
First Hop Security
Fixed & Modular Form Factors
MACSEC
Etc.

Client
Access Catalyst 9200
Catalyst 9300/9400 for
Small & Medium Size
Fixed & Modular Form Factors

End to End Catalyst 9000 for Traditional Campus Networks

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 392
 Multi Domain
Integration
Other Sites

Catalyst 9500/9600
Border + Control
Fixed & Modular Form Factors

Any Fabric
SD-Access: Border, CP Catalyst 9400/9500
BGP/EVPN, VXLAN, EdgeForm Factors
Fixed & Modular
Group based Policy Intermediate Nodes

Segmentation
Unified Policy Edge

Edge
Security
Catalyst 9300/9400
Fixed & Modular Form Factors
IoT Network Employee Network

End to End Catalyst 9000 for Fabric Based Networks

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 393
 Site 1 MPLS Site 2

CE PE CE PE

MPLS L2 / L3 VPN
Catalyst 9300/9400/9500/9600 mVPN Catalyst 9300/9400/9500/9600
QoS/HQoS
NAT

Comprehensive MPLS Features for Core + MPLS PE Deployments

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 394
Recognitions &
References
Industry Recognitions…

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 396
Catalyst 9K Book

Cisco Catalyst 9000

A New Era of Networking

eBook Available on Cisco.com

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 397
Visit World of Solutions…

Catalyst 9300/9500 Catalyst 9400 Catalyst 9K Demos

Campus of the Future

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 398
Complete your
online session • Please complete your session survey
evaluation after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live water bottle.
• All surveys can be taken in the Cisco Live
Mobile App or by logging in to the Session
Catalog on ciscolive.cisco.com/us.
Cisco Live sessions will be available for viewing
on demand after the event at ciscolive.cisco.com.

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 399
Continue your education

Demos in the
Walk-in labs
Cisco campus

Meet the engineer

Related sessions
1:1 meetings

#CLUS TECARC-2900 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 400
Thank you

#CLUS
#CLUS