#CiscoLive

Troubleshooting
5G Architectures

Ravi Narahari, Sr. Technical Leader, Darth Vader’s Mentor

BRKSPG-2040

#CiscoLive
Cisco Webex App

Questions?
Use Cisco Webex App to chat
with the speaker after the session

How
1 Find this session in the Cisco Live Mobile App

2 Click “Join the Discussion”

3 Install the Webex App or go directly to the Webex space Enter your personal notes here

4 Enter messages/questions in the Webex space

Webex spaces will be moderated

by the speaker until June 9, 2023. https://ciscolive.ciscoevents.com/ciscolivebot/#BRKSPG-2040

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
 • 5G Technology Summary
• Cloud
• Investigating where is 5G
Network Broken
• IP Transport
Agenda • Debugging case studies
• Wrap Up

BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
5G Technology
Summary
5G is Packet switched. More IP Troubles ?
1980s 1990s 2000s 2010s 2020s

1G 2G 3G 4G 5G

Radio and Voice/Data Evolution Service and Network Revolution

Voice TXT Video? Internet ?

TDM Voice TDM Voice All Packet All Packet
Limited Data Capabilities Packet Data Capable Virtualised Infra. Containerised Infra.
Network Appliances Network Appliances

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Architecture Evolution
4G Architecture EPC
eNodeB

CPRI
backhaul
RU DU Backhaul aggregation IP Edge IP Core
Central Datacenter

Decompose, virtualise & centralise Decompose, virtualise & distribute

vCore vCore API

5G Architecture Edge Datacenters User plane User plane
vCore Service
Control plane Architecture
MANO
SR SDN
SR SR SR SR Analytics/
Telemetry
Fronthaul backhaul Automate

vRAN IP Edge IP Core

Central Datacenter
Legacy 3G, LTE, LTE-A Managed
5G <6GHz (Micro, Pico) Video
MEC
5G mmW (Pico, PTMP)
WiFi next-G AP SDN Network Fabric based on Segment Routing
CIoT/LPWA Orchestration
Data Center Fabric based on NFV Automation
E2E Network Slicing

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Not all greenfield – Evolution path
• Operators have existing transport networks and services
• 5G services support is/will be added together with existing services
• The architecture needs to support brownfield scenarios
• Converge of multi-services on common network infrastructure is a key
requirement to reduce infrastructure costs and simplify operations
• There is a need to support legacy interfaces (CPRI) and the industry
recognizes such requirement:
• IEEE specified Radio over Ethernet technology (RoE)
• ORAN Alliance WG-7 developed the specifications for Fronthaul
Gateway Interworking Function
• Cloud and non cloud architecture

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
 Reference 5G Converged Architecture During Trouble shooting
Cisco Cross Domain Automation
CSR
FH NX-OS + NDFC NX-OS + NDFC NX-OS + NDFC Public Cloud
eMBB ACI Remote ACI Fabric ACI Fabric
Leaf

CSR Far Edge Edge Core DC

Legacy 2G/3G MH

DU
FWA / Premium
xHaul
Services
vCSR MH Pre-Agg Aggregation Core
Enterpris DU vRouter
e VPNs
Cloud Services
CSR
IOS-XR Routing Portfolio
Video BH
Surveillance NCS5xx/NCS57xx/NCS55xx/ASR9K/8000
DU
CU

End-to-End IP with Segment Routing

Issues may be in any of this eco system

Optimized for Multi-Service Flexible and Precise network-based Seamless integration Consistent Open, automated,
Environments programmable timing and across Transport, DC QoS and Service centralized
(2G, 3G, 4G, 5G, FWA, Enterprise, architecture synchronization and Cloud Performance management at scale
Greenfield, Brownfield)
#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Cloud
 Open RAN – VNF, Containers – what’s Broken ?

eNB/ Proprietary H/W, • Open interfaces with vendor

gNB closed CPRI & mgmt.
diversity
• Disaggregated software, cloud-
based models

Decompose
Functions
RU DU CU
Disaggregate
Virtual

H/W + S/W
RU: Radio Unit
COTS COTS
DU: Distributed Unit
CU: Centralized Unit

Open Interfaces and Management COTS: Commercial Off-The-Shelf

(support challenges)

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
 Start at RAN and then 5G Core Interfaces if
Problem not Obvious
N13
AUSF UDM

N8 N10
N12
N11 N7 N5
AMF SMF PCF AF

N14 N15
N1 N2 N4
Control Plane
Data Plane

F2 F1 N6
RU
C/U
N3 Data NW
UE1 C/U DU CU
Site1 UPF
E1 N9 Internet
Xn
C/U (C/U) N3

F2 F1
RU
UE2 C/U C/U CU
DU
Site2
E1
C/U
Start from left to right, RU to DU etc..

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Kubernetes Overview
Traditional -> Virtual Machine -> Container as time goes by...

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
 Cisco Cloud Native Architecture for Mobile Core
Service Based APIs

Life Cycle Mgmt

User Plane Control Plane App Services Common Services
Services Protocol Load Active Inventory
Balancing
UPF SMF AMF
Deploy
API Gateway
TCP Opt PCF NRF Telemetry
Common Validate
agents
Mobility Security NSSF UDSF Scale
Logging Service
Platform
CLAMP

Istio

Infrastructure Bare Metal OS OpenStack NFVI VMWare

as a Service
Hardware

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Cloud Native Application Architecture / Micro
Services
• Dynamic Discovery of
Protocol Load Balancer Services application containers
Metal LB
HTTP Load
Rest PFCP IKEv2/ GTP-C
SCTP • Protocol proxy and/or
Endpoint Endpoint IPsec (LI) GTP-U
Balancer termination, e.g., Diameter,
GTP, HTTP2

SMF Access Application Services HSMF Access

• Core application/business
Policy Svc
Node Event Policy Svc logic
Manager Exposure • Varying degrees of
Charging Svc Charging Svc
Microservice granularity
UPF Manager UPF Manager
• Native scale out and upgrade
• Stateless services

Policy DB Chrg DB State Management Services NF DB • State separation layer

ID Session
• Supports different data storage
PDU PFCP Event
Database Lookup Contexts DB Session DB Expo DB
Timer Svc technologies – in memory
caches to full fledge DBs
• Scale out technologies
Disk Memory/TMPFS/5G DB
#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
 CEE pod list (Cisco Specific)
• ”Common Execution Environment”
combines all the applications and
services that are used by all
network functions (such as NAME
logging, alerting, collecting of READY NODE
statistics etc.) alert-logger-74446c8fd6-9krdt 1/1
• Output on right is derived from oam-2
command alertmanager-0 1/1
“kubectl get pods -n cee-global –o wide” oam-2
• Pods marked with “*” in the node alertmanager-1 1/1
column run on all nodes oam-3
(shortened here for visibility) alertmanager-2 1/1
oam-1
#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Verify Status – NSSF, AMF
root@198:~/conf# kubectl get pods -n nssf -o wide

NAME READY STATUS

RESTARTS AGE IP NODE
activemq-0 1/1
Running 0 3d 10.42.0.73 worker2

root@198:~/conf# kubectl get pods -n amf -o wide

NAME READY STATUS
RESTARTS AGE IP NODE
amf-amf-documentation-documentation-5fbb577548-srddh 1/1 Running 0
2d 10.43.128.120 worker6
amf-amf-pats-executor-6bf4d7bc4d-gncfb 1/1 Running 0
2d 10.33.0.59 worker1
amf-amf-pats-repo-ff988f847-7nd2b 1/1 Running 0

Ensure to look for status Pending

master1:~$ kubectl get pod -A -o wide | grep -i pending

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Session Establishment between SMF & UPF
[local]POD7-UP# show sx peers <truncated output>

Wednesday April 24 16:30:18 EDT 2019

+---Node Type: –€ - CPLANE (–) - UPLANE

|
|+--Peer Mode: (–) - Active (–) - Standby
|
||+-Association (–) - Idle (–) - Initiated
||| State: (–) - Associated (–) - Releasing
||| (–) - Released
Recovery | Current Max Peer
vvvvv v Group Name Node ID Peer ID Timestamp
v Sessions Sessions State
----- ---- -------------------- ------------------------------ ---------- -------------
------ ---- --------- --------- ---------
CAAND 5 ingress 198.18.134.13 33554434 2019-04-
24:15:57:10 1 0 0 ACTIVE

Total Peers: 1

[local]POD7-UP#

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Interface Specific Error codes
• smf-rest-ep logs for smf-ims : | Interface n11 | Count Error Code Transaction Log | 175 404
N11N1N2MessageTransferReq(19) Received N1N2 response error - status code: 404 | 54 403
N11N1N2MessageTransferReq(19) Received N1N2 response error - status code: 403 | 5 404
N11EbiAssignmentReq(22) Received Assign Ebi Response error - status code: 404

• Interface n10 | Count Error Code Transaction Log | 1843 404 N10UnsubscribeForNotificationReq(222) Received
Unsubscribe-to-Notification response error - status code: 404 | 895 404 N10SubscriptionFetchReq(36)
Received sm subscription response error - status code: 404 | 288 404 N10DeregistrationRequest(45) Received
deregistration response error - status code: 404 | 6 500 N10DeregistrationRequest(45) Received deregistration
response error - status code: 500 | 6 500 N10RegistrationRequest(33) Received registration response error -
status code: 500

• Interface n40 | Count Error Code Transaction Log | 289 404 N40ChargingDataReq(63) Received Charging Data
Response error - response status code: 404 | 221 410 N40ChargingDataReq(63) Received Charging Data
Response error - response status code: 410

• Interface n7 | Count Error Code Transaction Log | 5 404 N7SmPolicyDeleteReq(54) Received sm policy control
delete error - response status code: 404 |

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
SMF – IMS Logs
• 4445 Error Message : Session not found

• 4318 [ERROR] [smf-service.smf-app.messageprocessor] Timer EventId = [0], Timer Type = [UnknownTimer]

• 1702 Error Message : 403 : Key not found in indexing

• 1702 [ERROR] [MasterBlueprint.go:426] [infra.transaction.core] Error deleting session [DatastoreDelete] 403 :

Key not found in indexing

• 1665 [ERROR] [SessionTimer.go:157] [smf-service.smf-app.gen] RemoveTimer: Unable to Delete timer type [11]
but has valid handle

• 1289 [ERROR] [smf-service.smf-app.udm] UDM Subscription Fetch Failure | 1289 [ERROR] [smf-service.smf-
app.sgw] UDM Subscription Fetch Failure

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Kubernetes - Check all the namespace created
• root@198:~# kubectl get ns
NAME STATUS AGE
amf Active 3d
base Active 203d
cnee Active 3d
default Active 206d
helm Active 105d
kube-public Active 206d
kube-system Active 206d
nrf Active 3d
nssf Active 3d
pcf Active 3d
smf Active 3d

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Where is 5G
Network Broken ?
Approach to Troubleshooting – Blame Game
• First thing to know is what is broken
• Logically look for issues as a stack – Mobile or IP
• HW and SW defects
• Line of sight
• Radio
• Mobile core
• IP Transport

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Approach to Troubleshooting – Ask the questions
• UE – 5G SoC ? – Handset / Device 5G capability
• NSA , SA ? – 4G CP and 5G UP
• Inline CUPS
• Apps – PAS, PCF
• Through put issues
• Data leak
• NR side / Xhual / 5G Core

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
MEC in 5G – Content, Latency, Thruput ?

5G nodes are CNFs!

• Some MEC nodes will be cloud-native as well
• Edge DC platform will have to accommodate both VNFs and CNFs

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
 5G Core Issues – Think of APIs, Kubernetes ? —
end points, tls not matching, version mismatch
Manages Session states Central database that Policy enforcement and
and authenticates and contains user-related and flow-based charging
tracks a user across the subscription related
network information

Mobility Management Subscriber Information Policy and Charging

AMF UDM/UDR/AUSF PCF

Gateway
Gateway Control
Plane
SMF
IP Packets to/from UEs
User Plane Function
UPF

The gateway anchors the data

session, manages QoS, provides
Radio Access deep packet inspection and IP
Network (RAN) address allocation for the UE.
The gateway is split into a control
User Equipment plane and user plane function. UPF User Plane Function
(UE) Enabling the User plane to be
deployed remotely from the control
SMF Session Management Function
plane AMF Access & Mobility Function
AUSF Authentication Server Function
UDM User Data Management
UDR User Data Repository
PCF Policy Control Function

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Infrastructure
• Resources – Apps- VM based vs Kubernetes allocation --
• Kubernetes – automation – match with App infrastructure – synergy
• Look for issues – Topic
• Accessibility – VM vs Kubernetes – 100s of logs

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
5G Private Networks – troubleshoot from left to
right Mgmt & Mgmt &
Orch Orch

AMF SMF UPF

5G CNFs
Full Private Local or SP
Spectrum
PCF UDM xxF
Public,
Private or
gNode
Deployment B Edge
Router
SP Cloud

Enterprise LAN & FW

Mgmt &
Orch
PCF UDM
Hybrid Cloud Local or SP EDGE PoD UPF
AMF SMF
yyF
Public,
Private Deployment Spectrum
gNode
Edge
Private or
SP Cloud
B
Router
Enterprise LAN & FW

“Enterprise”
Slice
Macros Slice SP
Spectrum UDM SMF UPF PCF Enterprise
Traffic
Deployment gNode
B
NSSF

AMF SMF UPF PCF

Public “Consumer”
Slice

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
 5G SA and 5G NSA Components to look for
issues – NSA is for eMBB, SA is for uLL,IOT

Non Standalone Standalone (SA) 5G

Standalone LTE (NSA) LTE & 5G NR NR under 5GC
Under EPC under EPC
4G 4G 5G
Core Core Core
Multiple Paths

eNodeB eNodeB gNodeB gNodeB

GSMA Option 1 GSMA Option 3 Device must support 4G and 5G GSMA Option 2
radio in NSA mode. 4G used to
“attach” and bring up bearer.
REQUIRES both 4G and 5G
spectrum/radios
#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
QoS Framework

• Flow based QoS, instead of bearer based

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
 Service Classification and Protection
Timing
CU UPF
RU DU
Mgnt / OOB
Slice G
OAM
CSR Access Pre-Agg Aggregation Core
U-Plane / Slice R
Converged traffic
RAN Slice Transport Slice Transport Service level Slice
C-Plane VLAN or IP VLAN + IP + Port VLAN + IP + Port + Service QCI /
DSCP

Service Classification and Quality of Service Service convergence and Routing

o CU plane traffic requirements preserved ? o Allowing traffic convergence ?
o Ensuring Timing service always received best o Do we have Resiliency and Redundancy
latency and jitter ? o Fast Convergence with TI-LFA tested and working ?
o Protecting against traffic bursts and spikes ? o Traffic steering for flexible workload deployment
o Guarantee quality of service to all other traffics ? validated ?

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
5G Network Slicing – Recall QoS ID
End to End Cross Domain Orchestration and Service Assurance

Low Latency Slice

Disjointness Slice

Best Effort Slice

Central Office Metro and Core Network Domain Datacenter Domain

Unified underlay and overlay networks with Segment Routing

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
IP Transport
Segment
Routing
Network issues as a Programed Fabric App. Driven
N/W and
Network Services Orchestrator (NSO) Usecases

REST/API Automation
Applications NC/
NC/ Yang
Yang REST Data
/API Collection
& Analytics
Path Compute Element
Telemetry Device and
Telemetry
Services
PCEP BGP-LS BGP-LS Onboarding
PCEP
EVPN
Based
Access Edge Core Services

Controller
Based
Forwarding
IGP with SR Extensions (TI-LFA) IGP with SR Extensions (TI-LFA) IGP with SR Extensions (TI-LFA)
Segment
Services: BGP - L3VPN/L2VPN(EVPN)
Routing
Transport: Segment-Routing with TI-LFA Transport

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
 Segment Routing – Any source routing issues ?
Evolve MPLS with Segment Routing
Mission – Route the luggage to Berlin
via Mexico and Madrid

Segment Routing
London
Toronto 1. A unique and global luggage tag
Seattle Berlin is attached to the luggage with
the list of stops to the final
New-York Madrid destination

Mexico
MEX 2. At each stop, the luggage is simply
routed to the next hop listed on the
MAD luggage tag
BER

Path can be controlled

MAD BER RESULT:
Simple and scalable
BER

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Segment Routing – Technology Overview
Prefix SID
Adjacency SID
(Node SID)
Locally Significant
Globally Significant
Unidirectional

16001 16003 16005

24010 24035

24012 24034 24010

16002 16004 16006

24024 24045

Segment = Instructions such as

"go to node N using the shortest path"

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Segment ID Construct

• Configured under IGP Routing Protocol

• Requires: Enabling SR & Configuring Prefix-SID

• Configure “Absolute Value” or “Index”
• Optional: Configure SR-Global-Block (SRGB).
• Default 16000 – 23999 (higher in newer versions)
• SRGB & Index advertised using IGP

SRGB Index SID

16000 4 16004

• Result: No LDP Needed for label distribution

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Segment Routing – Programming The Path
Go To ”RED”

RED = SID-List:
24012
16001 16003 16005
24024
16003 24010 24035
16005
16006 24012 24034 24010

16002 16004 16006

24012; INT=”x”
24024 24045

Reached ”RED”

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
 Crosswork Network Controller
Lifeline to troubleshoot
Crosswork Network Controller

Crosswork Common UI & API

Use Cases
C Service Provisioning for L3VPN and L2VPN
Crosswork Optimization Active Topology &
Engine Inventory Service-Oriented Transport Provisioning (SR-TE)
Cisco NSO +
Real Time Network Service Core Function
Optimization Inventory Packs
Topology & Inventory
Model-based
Crosswork Data Service & Bandwidth Optimization
SR-PCE Gateway Device
Path Computation Provisioning Real time network optimization
and Programming Data Collection to meet SLA

Extended for Network Slicing

Multivendor
Multi-domain
Physical and Virtual

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
 Segment Routing Underlay and EVPN Overlay
NG Subscriber
Data Management
E2E Orchestration

NG-6a/p
Service
Capability
Exposure
Residential
NG RAN NG Lawful
Intercept
Access NG Policy
NRs NG-Gx Control
Network Slice
Selection
NG C-Plane
Function Function
Charging/
Sign
al
AgthD
Str e
n
G
P
12 J W M
3
4B T E

LTE-AP
H 5X
K 6
Q
7C 8N
U 9
F
*#
I L
R 0YO

NG-CH
V
S Z

NG-X
Billing
Things Wi-Fi
Slice
NG U-Plane VAS VAS
Hot Spot Selection
NB-IoT Function(s) NG-Gi SDN
Logical Network
Other Slicing Functions IP Services

HetNet RAN Virtualization CPU

& cRAN
Controller/Orchestration

Services: BGP - L3VPN/L2VPN(EVPN)

Transport: Segment-Routing

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
MPLS Control and Forwarding Operation with Segment
Routing
Services
MP-BGP
No changes to
IPv4 IPv6
IPv4 IPv6 VPWS VPLS control or
PE1 PE2 VPN VPN
forwarding plane

Packet
Transport LDP RSVP Static BGP IS-IS OSPF IGP or BGP label
distribution for
PE1 IGP PE2
IPv4 and IPv6.
MPLS Forwarding
Forwarding plane
remains the same

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
IGP-SR/TI-LFA/SR-LDP Configuration issues

router isis 1 router ospf 1

address-family ipv4 unicast router-id 1.1.1.1
metric-style wide segment-routing mpls
segment-routing mpls segment-routing forwarding mpls
! IGP-SR area 0
interface Loopback0 interface Loopback0
address-family ipv4 unicast passive enable
prefix-sid index 1 prefix-sid index 1

router isis 1 router ospf 1

interface GigabitEthernet0/0/0/2 fast-reroute per-prefix
address-family ipv4 unicast TI-LFA fast-reroute per-prefix ti-lfa
fast-reroute per-prefix
fast-reroute per-prefix ti-lfa

router isis 1 router ospf 1

address-family ipv4 unicast SR/LDP Co-existence segment-routing mpls
segment-routing mpls sr-prefer segment-routing sr-prefer
LDP->SR migration

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
 What is EVPN
RFC 7432
• EVPN family introduces next generation solutions for
Ethernet services EVPN
• BGP control-plane for Ethernet Segment and MAC
distribution learning over MPLS or VXLAN data-plane
• Same principles and operational experience as in IP P2P Multipoint
VPNs
• No use of Pseudo wires
EVPN-VPWS
• Uses MP2P tunnels for unicast
• Multi-destination frame delivery via ingress replication RFC 7623
(via MP2P tunnels) or LSM
draft-ietf-bess-evpn-vpws RFC 7432
• Multi-vendor solutions
EVPN PBB-EVPN

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
Ethernet VPN
Highlights Data-plane address
Control-plane address
advertisement / learning
learning from Access over Core
• Next generation solution for Ethernet
multipoint (E-LAN) services
VID 100
• PEs run Multi-Protocol BGP to SMAC: M1
DMAC: F.F.F
PE1 PE3
advertise & learn Customer MAC
addresses (C-MACs) over Core
CE1 CE3
• Same operational principles of L3VPN
C- C-
• Learning on PE Access Circuits via MAC:M1 PE2 PE4 MAC:M3

data-plane transparent learning

• No pseudowire full-mesh required
• Unicast: use MP2P tunnels BGP MAC adv. Route
• Multicast: use ingress replication over MP2P EVPN NLRI
MAC M1 via PE1
tunnels or use LSM
• Standardized at IETF – RFC 7432
#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
 Control-plane address

BGP Ethernet VPN

Data-plane address advertisement / learning
learning from Access over Core

Verification VID 100

SMAC: M1

RP/0/RSP0/CPU0:router# show evpn evi vpn-id 2001 neighbor

DMAC: F.F.F PE1 PE3

Neighbor IP vpn-id CE1 CE3

----------- -------- C- C-
MAC:M1 PE2 PE4 MAC:M3
20.20.20.20 2001

30.30.30.30 2001
BGP MAC adv. Route
Verify the BGP L2VPN EVPN summary. EVPN NLRI
MAC M1 via PE1
RP/0/RSP0/CPU0:router# show bgp l2vpn evpn summary

...

Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd

20.20.20.20 0 200 216739 229871 200781341 0 0 3d00h 348032

30.30.30.30 0 200 6462962 4208831 200781341 10 0 2d22h 35750

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
EVPN - Components L2 and L3 in the same instance!
EVPN Instance (EVI) BGP Route
Ethernet Segment BGP Routes
Attributes

SHD Route Types Extended Communities

CE1

MAC
BD VRF ESI1 [1] Ethernet Auto-Discovery (AD) Route ESI MPLS Label
PE1 [2] MAC Advertisement Route ES-Import
MHD [3] Inclusive Multicast Route MAC Mobility
MAC
VRF

BD CE2
ESI2 PE2 [4] Ethernet Segment Route Default Gateway
PE
(5) IP Prefix Advertisement Route Router’s MAC

• EVI spans all PEs • Represents a ‘site’ • EVPN and PBB-EVPN • New BGP extended
participating in an EVPN connected to one or more define a single new BGP communities defined
• MAC-VRF: A VRF table for PEs NLRI used to carry all • Expand information
MACs on a PE • Uniquely identified by a EVPN routes carried in BGP routes,
• Encompass one or more 10-byte global Ethernet • NLRI has a new SAFI (70) including:
bridge-domains, Segment Identifier (ESI) • Routes serve control MAC address moves
depending on service • Could be a single device plane purposes, C-MAC flush notification
interface type or an entire network including: Redundancy mode
Port-based Single-Homed Device (SHD) MAC / IP address reachability MAC / IP bindings of a GW
VLAN-based (shown above) Multi-Homed Device (MHD) MAC mass withdrawal Split-horizon label encoding
VLAN-bundling Single-Homed Network (SHN) Split-Horizon label adv.
VLAN aware bundling (NEW) Multi-Homed Network (MHN) Aliasing
Multicast endpoint discovery
Redundancy group discovery
Designated forwarder election
#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
 MPLS Data Plane Operations – push label
Segment 16004 Prefix-SID 16004
NP-flag=0, E-flag=0
1 2 3 4

Push Swap Pop - FIB entry remote prefix P

with prefix-SID N
operation : push N
16004 16004
egress interface : oif spt(N)
Payload Payload Payload Payload

RP/0/0/CPU0:Node1#show cef 1.1.1.4/32

1.1.1.4/32, version 277, internal 0x4004001 0x0 (ptr 0xacce39a4) [1], 0x0 (0xaccde760), 0x450 (0xacd8b8)
local adjacency 10.0.0.2
Prefix Len 32, traffic index 0, precedence n/a, priority 1
via 99.1.2.2, GigabitEthernet0/0/0/0, 5 dependencies, weight 0, class 0 [flags 0x0]
path-idx 0 NHID 0x0 [0xacbb3bf0 0x0]
next hop 99.1.2.2
local adjacency
local label 16004 labels imposed {16004}

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
 Verifying IP Forwarding (global table)
RP/0/0/CPU0:xrvr-1#show cef 1.1.1.2/32
1.1.1.2/32, version 652, internal 0x4004001 0x0 (ptr 0xacca7ba4) [2], 0x0 Global prefix
(0xacca27a8), 0x450 (0xacfd1c10)
Updated Jan 28 09:00:55.333
local adjacency 99.1.3.3
Prefix Len 32, traffic index 0, precedence n/a, priority 1
via 99.1.3.3, GigabitEthernet0/0/0/0, 8 dependencies, weight 0, class 0 2 paths (ECMP)
[flags 0x0]
path-idx 0 NHID 0x0 [0xacbbaeac 0x0]
next hop 99.1.3.3
local adjacency
local label 16002 labels imposed {16002} Push label stack:
via 99.1.5.5, GigabitEthernet0/0/0/1, 8 dependencies, weight 0, class 0 {Prefix-SID}
[flags 0x0]
path-idx 1 NHID 0x0 [0xacbbac54 0x0]
next hop 99.1.5.5 Label stack:
local adjacency {Prefix-SID}
local label 16002 labels imposed {16002}

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
 Verifying IP Forwarding (VRF RED)
RP/0/0/CPU0:xrvr-1#show cef vrf RED 10.0.0.0/30
10.0.0.0/30, version 27, internal 0x14004001 0x0 (ptr 0xacca79a4) [1], 0x0 VPN prefix (VRF RED)
(0x0), 0x410 (0xacd3372c)
Updated Jan 27 10:05:33.906
Prefix Len 30, traffic index 0, precedence n/a, priority 3
via 1.1.1.2, 5 dependencies, recursive [flags 0x6000]
path-idx 0 NHID 0x0 [0xacfc0e24 0x0]
next hop VRF - 'default', table - 0xe0000000
next hop 1.1.1.2 via 16002/0/21 BGP nexthop, egress PE
next hop 99.1.3.3/32 Gi0/0/0/0 labels imposed {16002 90001}
next hop 99.1.6.6/32 Gi0/0/0/1 labels imposed {16002 90001}

2 paths (ECMP)
Push label stack:
{Prefix-SID to egress PE, BGP VPN label}

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
 Verifying MPLS Forwarding
RP/0/0/CPU0:xrvr-3#show mpls forwarding
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
16001 16001 SR Pfx (idx 1) Gi0/0/0/0 99.2.3.2 0 prefix-SIDs of
16002 Exp-Null-v4 SR Pfx (idx 2) Gi0/0/0/0 99.2.3.2 0 remote nodes
16004 Pop SR Pfx (idx 4) Gi0/0/0/1 99.3.4.4 0
16005 16005 SR Pfx (idx 5) Gi0/0/0/1 99.3.4.4 0
16010 16010 SR Pfx (idx 10) Gi0/0/0/0 99.2.3.2 0
16010 SR Pfx (idx 10) Gi0/0/0/1 99.3.4.4 0
24032 Pop SR Adj (idx 1) Gi0/0/0/0 99.2.3.2 0
24034 Pop SR Adj (idx 3) Gi0/0/0/1 99.3.4.4 0 adjacency-SIDs

3 2 1
Gi0/0/0/0

Gi0/0/0/1
4 5 10
Node n advertises prefix-SID 16000+n

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
 Verifying MPLS Forwarding
RP/0/0/CPU0:xrvr-3#show mpls forwarding
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------ Remote prefix-SID
16001 16001 SR Pfx (idx 1) Gi0/0/0/0 99.2.3.2 0 Neighbor prefix-SID
16002 Exp-Null-v4 SR Pfx (idx 2) Gi0/0/0/0 99.2.3.2 0 Explicit-Null
16004 Pop SR Pfx (idx 4) Gi0/0/0/1 99.3.4.4 0
16005 16005 SR Pfx (idx 5) Gi0/0/0/1 99.3.4.4 0 Neighbor prefix-SID
16010 16010 SR Pfx (idx 10) Gi0/0/0/0 99.2.3.2 0 PHP on
16010 SR Pfx (idx 10) Gi0/0/0/1 99.3.4.4 0 Remote prefix-SIDs
24032 Pop SR Adj (idx 1) Gi0/0/0/0 99.2.3.2 0
ECMP
24034 Pop SR Adj (idx 3) Gi0/0/0/1 99.3.4.4 0

3 2 1
Gi0/0/0/0

Gi0/0/0/1
4 5 10
Node n advertises prefix-SID 16000+n

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
 SID index 1
1.1.1.2 1.1.1.1

Verifying IS-IS Database

RP/0/0/CPU0:xrvr-1#show isis database verbose xrvr-1 1.1.1.4 1.1.1.6
DIS
IS-IS 1 (Level-2) Link State Database
LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OL
xrvr-1.00-00 * 0x00000073 0x4eba 1161 0/0/0
Area Address: 49.0001
NLPID: 0xcc
NLPID: 0x8e
MT: Standard (IPv4 Unicast) Two topologies: IPv4 and IPv6
MT: IPv6 Unicast 0/0/0
Hostname: xrvr-1
IP Address: 1.1.1.1
IPv6 Address: 2001::1:1:1:1
Router Cap: 1.1.1.1, D:0, S:0 Single SRGB for both IPv4 and IPv6
Segment Routing: I:1 V:1, SRGB Base: 16000 Range: 8000 I:1 IPv4; V:1 IPv6
Label range 16,000 – 23,999
<...>

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
 SID index 1
1.1.1.2 1.1.1.1 1.1.1.4

Verifying OSPF Database

RP/0/0/CPU0:xrvr-1#show ospf database self-originate originate 1.1.1.5 1.1.1.3
DR

OSPF Router with ID (1.1.1.1) (Process ID 1)

Router Link States (Area 0)

Link ID ADV Router Age Seq# Checksum Link count

1.1.1.1 1.1.1.1 13 0x8000000b 0x00b7ba 4

Type-10 Opaque Link Area Link States (Area 0)

Area-scoped (Type10) Opaque LSAs
Link ID ADV Router Age Seq# Checksum Opaque ID
4.0.0.0 1.1.1.1 1085 0x80000003 0x000a66 0 Router Information Opaque LSA
7.0.0.1 1.1.1.1 1085 0x80000003 0x00e1b2 1
8.0.0.4 1.1.1.1 823 0x80000003 0x0042c0 4
8.0.0.7 1.1.1.1 13 0x80000009 0x0030f4 7
Opaque LSAs for Segment Routing

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Issues with ACI Leaf delivering clock to O-DUs
and O-RUs
Far Edge DC#1 Far Edge DC#2
T-TSC T-TSC T-TSC T-TSC T-TSC T-TSC

O-DU O-DU O-DU vDU vDU vDU

Midahaul/ S S
S S S S
Backhaul

T-BC
T-BC M M T-BC T-BC M M T-BC

N9K- N9K-
N9K-
C93180YC-FX3
S C93180YC-FX3
N9K-
C93180YC-FX3
S M C93180YC-FX3

PLC PLC PLC PLC

Splitter Splitter Splitter Splitter

PLC splitter between cell site and Far Edge DC

PLC splitter between cell site and Far Edge DC

M M M S
GNSS T-GM GNSS T-GM GNSS T-GM T-BC
M M M M

O-RU O-RU O-RU O-RU

T-TSC T-TSC T-TSC T-TSC

Cell Site#1 Cell Site#2 Cell Site#3 Cell Site#4

54
Debugging Case
studies
RCAs
• MPLS L3 connectivity to some cell sites when a core network link broke
down- Service was restored after issuing clear mpls ldp neighbor*.
• NCS-5508 -config SR policy on NSO print error – Policy Config issue
• Fallback from SR-Policy to Native SR after RPFO – Defect
• Micro Services - abnormal Micro-service memory usage - The
Workaround to restart the rest-ep was already applied, and the container-
memory-usage was cleared.
• Throughput issue in 5G - Suspecting issue is due to QOS Data & Control
plane Buffer exhaust - Qos buffer enhancement with different buffer
options
• ACI | Stale Route found in ACI | Route do not exist in ACI but is seen
receiving from ACI from external site - defect - Issue got resolved after
upgrading Border Leaf to version
#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
RCAs
• 5G AMF - Three pods in pending state - Issue recovered after
restart of kubectl, no issues on kubernetes settings
• 5G SA // AMF De-registration call flow - registration-request fails –
Stop sending Http2 go away
• UDM peer is not getting selected properly as per the static
configuration - restart of the rest-ep endpoint that display the
problem
• G SA : Call Drop - SMF Not Initiating Dedicated Bearer During
EPSFB - Defect
• Bad 5G site UE speed is around 80Mbps and good site is around
800Mbps – IP Transport issue
#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Summary
 • Troubleshooting 5G involves
• NR
• Mobile Core
• PDU
Cloud
Key Takeaways •

• IP Routing
• Apps
• Cisco support

BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
 Cisco Private 5G Learning Map
Start
June 4 | 2:00 pm June 5 | 11:00 am June 7 | 4:00 pm
TECSPG-2432 BRKENS-2950 BRKSPG-3004
New Adventures in Wireless: The Journey Is your Enterprise Network Ready for P5G Monolithic or Polylithic packet cores?
of WiFi6 and Private 5G Networks for the The case for specialized use-case-
Enterprise based mobile packet cores
June 5, | 8:00 am June 5 | 11:30 am
June 8 | 09:30 am
BRKSEC-2085 PSOSPG-1002 BRKSPG-2044
Architecting Enterprise Security in a Wi-Fi Leading Your Digital Transformation with
5G Use Cases Flight Line of the
plus Private 5G World Cisco Private 5G Network Offer
Future and Smart Warehouse
June 5 | 8:30 am June 6 | 3:00 pm June 8 | 01:00 pm
BRKSPG-2042 BRKEWN-2030 IBOSPG-2007
Architecting Private 5G for resiliency, WiFi6 and Private 5G for the Getting Started with Private 5G
security, and enterprise network Enterprise – a ‘Better Together’
convergence Journey
June 5 | 10:30 am June 7 | 2:30 pm June 8 | 1:00 pm
BRKSPM-1006 PSOGEN-1033 BRKGEN-2001
The 5G System as a Spectrum Unlock business outcomes from Cisco P5G - A Robust and Secure
Management Solution connectivity with a Private 5G solution Architecture
Finish

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
 Cisco 5G Learning Map
Start
June 4 | 9:00 am June 5 | 2:30 pm June 6 | 2:30 pm
TECIOT-2584 BRKSPG-1002 IBOSPM-2030
Designing IoT Wireless Networks Don’t Just Connect, Grow your IoT 5G Transport Design Considerations
Business with Cisco IoT Cellular Combining Onsite and Cloud-Based
Connectivity Management Deployments
June 5, | 8:30 am June 5 | 3:00 pm June 6 | 4:00 pm
BRKNWT-2203 BRKIOT-1126 BRKSPM-2027
Automation-first Approach to Network Connecting Moving Assets with Cisco IoT Holistic Security in 5G Deployments
Infrastructure Modernization for 5G & Solutions
Beyond
June 7 | 10:30 am
June 5 | 1:00 pm June 6 | 10:30 am
BRKSPG-2063 BRKSPG-2315 BRKSPG-2133
Evolution of the Transport Network
Design, Deploy and Manage Transport Cloud-Ready Converged SDN Transport
Architecture in the Context of 5G and
Slicing using SDN Controller and Assurance
Open RAN
June 5 | 1:00 pm June 6 | 1:00 pm June 8 | 8:30 am
BRKARC-2094 BRKSPG-2401 BRKSPG-3050
Hiking the Band Canyon with 5G: New Use Cisco Secure Edge Protection – Protecting the Synchronizing 5G Mobile Networks
Cases, New Business Outcomes 5G Edge against DDoS Attacks

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Cisco 5G Learning Map
June 8 | 9:30 am
IBOSPG-2006
DISH Wireless, World’s first 5G
Network with a Hybrid Cloud

June 8 | 10:30 am
BRKNWT-2301
DevNetOps Automation Approach to
Network Infrastructure Modernization for 5G
and Beyond
June 8, | 11:00 am
BRKSPG-2040
Troubleshooting 5G Architectures

June 8 | 1:00 pm
BRKMER-2001
Postcards from the 5G Edge: Meraki
Cellular Gateways
Finish

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Fill out your session surveys!

Attendees who fill out a minimum of four session

surveys and the overall event survey will get
Cisco Live-branded socks (while supplies last)!

Attendees will also earn 100 points in the

Cisco Live Game for every survey completed.

These points help you get on the leaderboard and increase your chances of winning daily and grand prizes

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
 • Visit the Cisco Showcase
for related demos

• Book your one-on-one

Meet the Engineer meeting

• Attend the interactive education

with DevNet, Capture the Flag,
Continue and Walk-in Labs

your education • Visit the On-Demand Library

for more sessions at
www.CiscoLive.com/on-demand

BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Thank you

#CiscoLive
Gamify your Cisco Live experience!
Get points for attending this session!

How:
1 Open the Cisco Events App.

2 Click on 'Cisco Live Challenge’ in the side menu.

3 Click on View Your Badges at the top.

4 Click the + at the bottom of the screen and scan the QR code:

#CiscoLive BRKSPG-2040 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
#CiscoLive