Deploying CMG On AWS-V1
Deploying CMG On AWS-V1
1 © 2021 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
CMG on AWS
Agenda/ Objectives
2 © 2021 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
Project Requirement and
High-Level Architecture
3 © 2021 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
Project Description
• DISH intends to provide voice/video, data and SMS service to its customers by
launching DISH 5G Core network on AWS cloud for different US Markets.
• Ensuring a consistent level of customer experience for DISH subscribers is important.
• Scope of the requirement document is to cover the design to deploy NOKIA 5G SA
Core in AWS:
• Solution Architecture and High-level Network design for 5G Core on AWS
• Detailed Use cases relevant for DISH 5G Launch
• Detailed external interfaces relevant for interworking with DISH Network
• IP interconnect, Network Topology & high-level redundancy aspects relevant for interworking for access &
Peering network for DISH 5G Launch.
• Cloud design and placement
• Dimensioning
• Geo-Redundancy
4 © 2021 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
UnTrusted WiFi
Solution Architecture
RDC NDC
WiFi
Internet IPsec (Swu) To 5G
Roaming
sftp GSMA
SWm
ePDG IMEI DB
SCP SEPP http/spml
http OSS/BSS
http/xml
UDM/
DNS NRF DRA AAA 5G-EIR AUSF X3
Ud/ldap
N8/N12 Lawful Interception Gateway(LIG), Law
T-Mobile AMF N17
X1
Enforcement Agency(LEA)
LDAP ldap X2 (SS8)
DEA S2b N15
UDR Prov GW
LTE RAN N22
S1 MME NSSF
S2b N21 10G Cheyenne
D
x NetAct NSP
S11 N11
S11 N20
10G
S6a SMSF UHHd
N28 Nudr/N36
S1U SGW N1/N2
N40
CHF PCF+BSF
S8-C
DEA
Ud/ldap
SMF/
S2b
SGW-C/ S6b
DISH 5G RAN
(RU, Cell Site, LDC, P-EDC) S8U PGW-C SGd
SMS/MMS Aggregato rs &
DNS N4 S5 Intercarrier Hubs(BICS)
SWx
HTTP
UPF/
MM7
5GRAN N3 SGW-U/ MM1 MMSC MM4/SMPP
F1 S8-U
PGW-U SMPP SMPP
MAP
Wireless Emergency Alert
SMSC HSS/HLR http
(InterOp)
S5 MAP MAP
N4 Rx
MAP C/D Cx
SMPP E911 calls
RTP(Mb)
IMS SMPP IMS ENUM ESINet/ Legacy PSAP
2G/3G Roaming HUB (BICS) DNS Location Mgmt Function(LMF)
Impact Ici(sip) (Interado)
I-SBC
PSTN B-EDC SMPP IP-SM-GW ISC DNS
Gm Other Peering Networks & N11
UPF/ I/S/E- Mx I-BCF ICi (Peerless)
GSM C SGW- Izi
A N3 U/ REST CSCF, TCF
MSC/ U
PGW-U
Smith
VMAS
Ma Ix
NRF S8-U
Internet
S8-U
AMF
V-NSSF Legend
DRA Not shown, all Diameter messages will be routed through DRA.
Non Nokia ISV DISH Sites on
V-PCF V-SMF NOKIA
SCP Not shown, all SBI messages will be routed through SCP Vendor in DISH AWS
5GR
AN V-UPF
Control Plane
DISH Sites
User Plane
5 © 2021 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
Geographical Placement
5G Build out Overview, IP Transport. Hub and Spoke Design
US-West2 Region US-East2 Region US-East1 Region
B-
B- B-
B- B-
B- B-
B- B-
B- B-
B- B-
B- B-
B- B-
B-
B- B- B- B- B- B- B- B- B-
EDC11
B-EDC11
EDC11 EDC11
B-EDC21
EDC11 EDC11
B-EDC31
EDC11 EDC11
B-EDC41
EDC11 EDC11
B-EDC51
EDC11 EDC11
B-EDC61
EDC11 EDC11
B-EDC71
EDC11 EDC11
B-EDC81
EDC11 EDC11
B-EDC91
EDC11
EDC11 EDC11 EDC11 EDC11 EDC11 EDC11 EDC11 EDC11 EDC11
The v-Routers provide an Overlay Network, only integration point from Networking is the ENI/ENA interfaces on the NF where the V Router
is the next Hop/ Gateway to NOKIA NFs. Underlay and VPC networking constructs are all handled by DISH/AWS and not visible to NOKIA .
6 © 2021 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
Geographical Placement
5G Build out Overview, Application Connectivity. Full Mesh
US-West2 US-East2 US-East1
Region Region Region
NDC3 NDC2 NDC1
West2A West2B East2A East2B East1A East1B
B-
B- B-
B- B-
B- B-
B- B-
B- B-
B- B-
B- B-
B- B-
B-
B- B- B- B- B- B- B- B- B-
EDC11
EDC11
B-EDC11 EDC11
EDC11
B-EDC21 EDC11
EDC11
B-EDC31 EDC11
EDC11
B-EDC41 EDC11
EDC11
B-EDC51 EDC11
EDC11
B-EDC61 EDC11
EDC11
B-EDC71 EDC11
EDC11
B-EDC81 EDC11
EDC11
B-EDC91
EDC11 EDC11 EDC11 EDC11 EDC11 EDC11 EDC11 EDC11 EDC11
The v-Routers provide an Overlay Network, only integration point from Networking is the ENI/ENA interfaces on the NF where the V Router
is the next Hop/ Gateway to NOKIA NFs. Underlay and VPC networking constructs are all handled by DISH/AWS and not visible to NOKIA .
7 © 2021 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
NF Placement and Cloud Design
8 © 2021 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
Nokia NF Placement across NDC/RDC/EDC
VNF
NDC-CNF-
Normal
NDC (West2)
IS
NDC-CNF-
AZ2 Performance
VNF VMC : Mgmt Cluster VNF VMC Cluster – NDC VNF AZ1 RDC/B-EDC CNF
Performance
ENUM+UtAp SDL AAA SDL-Assist
CBAM
CNF EKS Generic Cluster (IPVLAN / CPU Sharing) - AWS RDC(West2-AZ2) B-EDC5
I/S/ECSCF+
RTT-TTY
IPSM-GW UDM / AUSF MMTAS-SubP RDC(West2-AZ1) B-EDC4
NG_CSCF NG_IPSM NG_UDM_AUSF NG_IMS2
B-EDC3
STIR-Shaken SCC-AS HSS / HLR DevMgmt
B-EDC2
NG_STI NG_SCCAS NG_HLRHSS
CNF EKS Performance Cluster (SRIOV / CPU Isolation / Kernel Optimiz.) NG_AMF NG_EPDG
UPF / PGW-U
data
A-SBC I-SBC A-SBC I-SBC MRF A/I-SBC MRF
SMF / UPF / PGW-U
Media Media Signaling Signaling ZTS
PGW-C voice
NG_A_Media NG_I_Media NG_SBC_Signaling NG_Transcoding NG_SBC_ZTS NG_MRF NG_UPF
NG_SMF NG_UPF
9 © 2021 Nokia
AWS - Dimensioning
Host requirements and
Networking
10 © 2021 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
Summary of the NF capacity Planning
Y1 (600k subs)
2x SMF ( Performance cluster NFs ) per RDC AZ m5n.24xlarge 8 12 156 328 256
1x ePDG ( RDC Performance cluster NFs ) per RDC AZ m5n.24xlarge 4 6 124 192 128
2x UPF-Voice ( Performance Cluster NFs ) per RDC AZ m5n.24xlarge 8 12 172 456 256
1x UPF –Data ( Performance Cluster NFs ) per B-EDC (LZ) m5.24xlarge 4 6 156 292 128
11 © 2021 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
DPDK on AWS
Deploying CMG CNF using AWS DPDK
Setup required to prepare the worker nodes to enable DPDK ENA PMD to be used:
• Pre-steps:
• Update kernel and install required packages
• Download DPDK package and Amazon drivers from GitHub repo.
• Install DPDK and the Amazon drivers
• Setup Hugepage memory, CPU isolation and reboot.
• Bind the DPDK interfaces to the VFIO-PCI driver
AWS has automated deployment steps as part of their infrastructure bring-up/CICD pipeline.
12 © 2021 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
SMF CNF Networking, Performance Cluster
13 © 2021 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
AWS SMF EC2/Pods/ETH Port Mapping & Resource Allocation
RDC
SMF [m5n.24xlarge: 96/384/100Gpbs]
EC2 Worker 1 Worker 2 Worker 3 Worker 4
VRF
Eth/Pod OAM-A LB17 OAM-B LB18 MG01 MG02
VPC/CNI 00 x x x x x x
OAM 01 x x
DSF 02 x x x x
03 x x
5GC OAM
04 x x
05 x x
5GC SIG
06 x x
07 x x
5GC LI
08 x x
09
5GC EPC
10
11
5GC N3
12
13
5GC N6
VCPU Allocation 14
Node Pods assignment 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
worker1 smf-llb1,smf-oam-a smf-llb1(12) smf-oam-a(12)
worker2 smf-llb2, smf-oam-b smf-llb2(12) smf-oam-b(12)
worker3 smf-lmg1 smf-lmg1(12)
worker4 smf-lmg0 smf-lmg0(12)
Memory Allocation
Node Pods assignment 02 04 06 08 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48
worker1 smf-llb1,smf-oam-a smf-llb1 (24) smf-oam-a (24)
worker2 smf-llb2, smf-oam-b smf-llb2 (24) smf-oam-b (24)
worker3 smf-lmg1 smf-lmg1 (32)
worker4 smf-lmg0 smf-lmg0 (32)
14 © 2020 Nokia
Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
UPF-Voice CNF Networking, Performance Cluster
15 © 2021 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
AWS UPF-v EC2/Pods/ETH Port Mapping & Resource Allocation
RDC
UPF-v [m5n.24xlarge: 96/384/100Gpbs]
EC2 Worker 1 Worker 2 Worker 3 Worker 4
VRF
Eth/Pod OAM-A LB17 OAM-B LB18 MG01 MG02
VPC/CNI 00 x x x x x x
OAM 01 x x
DSF 02 x x x x
03 x x
5GC OAM
04 x x
05 x x
5GC SIG
06 x x
07 x x
5GC LI
08 x x
09 x x
5GC EPC
10 x x
11 x x
5GC N3
12 x x
13 x x
5GC N6
14 x x
VCPU Allocation
Node Pods assignment 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
worker1 upf-llb1,upf-oam-a upf-llb1(12) upf-oam-a(12)
worker2 upf-llb2,upf-oam-b upf-llb2(12) upf-oam-b(12)
worker3 upf-lmg1 upf-lmg1(16)
worker4 upf-lmg2 upf-lmg2(16)
Memory Allocation
Node Pods assignment 02 04 06 08 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64
worker1 upf-llb1,upf-oam-a upf-llb1 (24) upf-oam-a (24)
worker2 upf-llb2,upf-oam-b upf-llb2 (24) upf-oam-b (24)
worker3 upf-lmg1 upf-lmg1 (64)
worker4 upf-lmg2 upf-lmg2 (64)
16 © 2020 Nokia
Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
UPF-Data CNF Networking, Performance Cluster
17 © 2021 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
AWS UPF-d EC2/Pods/ETH Port Mapping & Resource Allocation
BEDC
UPF-d [m5.24xlarge: 96/384/25Gpbs]
Memory Allocation
Node Pods assignment 02 04 06 08 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80 82 84 86 88 90 92 94 96
worker1 upf-llb1,upf-oam-a upf-llb1 (24) upf-oam-a (24)
worker2 upf-llb2,upf-oam-b upf-llb2 (24) upf-oam-b (24)
worker3 upf-lmg1 upf-lmg1 (96)
worker4 upf-lmg2 upf-lmg1 (96)
18 © 2020 Nokia
Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
ePDG CNF Networking, Performance Cluster
19 © 2021 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
AWS ePDG EC2/Pods/ETH Port Mapping & Resource Allocation
RDC
ePDG [m5n.24xlarge: 96/384/100Gpbs]
EC2
VRF Worker 1 Worker 2 Worker 3 Worker 4
Eth/Pod OAM-A LB17 OAM-B LB18 MG01 MG02
VPC/CNI 00 x x x x x x
OAM 01 x x
DSF 02 x x x x
03 x x
5GC OAM
04 x x
05 x x
5GC SIG
06 x x
07 x x
5GC ACC
08 x x
09 x x
5GC EPC
10 x x
11
12
13
VCPU Allocation 14
Node Pods assignment 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
worker1 smf-llb1,smf-oam-a smf-llb1(16) smf-oam-a(16)
worker2 smf-llb2, smf-oam-b smf-llb1(16) smf-oam-a(16)
worker3 smf-lmg1 smf-lmg1(30)
worker4 smf-lmg0 smf-lmg1(30)
Memory Allocation
Node Pods assignment 02 04 06 08 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64
worker1 smf-llb1,smf-oam-a smf-llb1 (16) smf-llb1 (16)
worker2 smf-llb2, smf-oam-b smf-llb1 (16) smf-llb1 (16)
worker3 smf-lmg1 smf-lmg1 (64)
20 © 2020 worker4
Nokia smf-lmg0 smf-lmg1 (64)
Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
CMG NF Geo-Redundancy
21 © 2020 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
NF Placement in NDC/RDC
22 © 2021 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
5G SA Core (IMS)
Packet Core: AMF (2) All Active (AMF set); 2xSMF (1+1) Primary/Secondary; 2xUPFv (1+1) Primary/Secondary 3rd pty
vendor in
DISH
NDC1 UDM+AUSF
UDM+AUSF NDC2 UDM+AUSF NDC3 DNS UDM+AUSF
DNS DNS UDM+AUSF UDM+AUSF Nokia
AAA AAA AAA SDM
W2AZ1: RDC1 W2AZ2: RDC2 E2AZ1: RDC4 E2AZ2: RDC5 E1AZ2: RDC7 E1AZ4: RDC8
SMF+PGW-C SMF+PGW-C SMF+PGW-C SMF+PGW-C SMF+PGW-C SMF+PGW-C
(2) (2) (2) (2) (2) (2)
RAN
RAN RAN RAN
RAN RAN
23 © 2021 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
5G SA Core (IMS)
• AMF (2) All Active. AMF Set (common data) replicated within Regional RDC group
• SMF/UPFv (1+1)
• RDC1 SMF01/UPFv01 Primary/RDC2 SMF02/UPFv02 Secondary, RDC2 SMF01/UPFv01 Primary / RDC1 SMF-02/UPFv02 Secondary
• Maintain PDU session on failover
FUTURE
W2AZ1: RDC1 W2AZ2: RDC2 N4 RDC3
SMF+PGW-C-01 SMF+PGW-C-02
(Pri) SMF+PGW-C
(Sec)
N4
SMF+PGW-C-02 SMF+PGW-C-01
(Sec) (Pri) SMF+PGW-C
UPF+PGW-U-01 UPF+PGW-U-02
(Pri) UPF+PGW-U
(Sec)
N11
N11
UPF+PGW-U-02
(Sec) UPF+PGW-U-01 UPF+PGW-U
(Pri)
N11
AMF-01 AMF-02
(Act) AMF
N3 (Act) N3
N2 N2 N2
RAN RAN
RAN
24 © 2021 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
5G SA Core ePDG
Packet Core: AMF (2) All Active; ePDG All Active 3rd pty
vendor in
DISH
NDC1 UDM+AUSF
UDM+AUSF NDC2 UDM+AUSF NDC3 DNS UDM+AUSF
DNS DNS UDM+AUSF UDM+AUSF Nokia
AAA AAA AAA SDM
RDC1 ePDG-01 ePDG-01 RDC2 RDC4 ePDG-01 ePDG-01 RDC5 RDC7 ePDG-01 ePDG-01 RDC8
RAN
RAN RAN RAN
RAN RAN
25 © 2021 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
5G SA Core ePDG
• AMF (2) All Active. AMF Set (common data) replicated within Regional RDC group
• ePDG
• All Active
FUTURE
W2AZ1: RDC1 W2AZ2: RDC2 RDC3
SMF+PGW-C-02 SMF+PGW-C-01
SMF+PGW-C
(Sec) (Act)
N2 N2 N2
RAN RAN
RAN
26 © 2021 Nokia
Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
5G SA Core (eMBB)
Packet Core: AMF (2) All Active; 2xSMF (1+1) Primary/Secondary; n x UPF-d All Active 3rd pty
vendor in
DISH
NDC1 UDM+AUSF
UDM+AUSF NDC2 UDM+AUSF NDC3 UDM+AUSF
DNS UDM+AUSF UDM+AUSF Nokia
AAA DNS DNS AAA
AAA SDM
W2AZ1: RDC1 W2AZ2: RDC2 E2AZ1: RDC4 E2AZ2: RDC5 E1AZ2: RDC7 E1AZ4: RDC8
SMF+PGW-C SMF+PGW-C SMF+PGW-C SMF+PGW-C SMF+PGW-C SMF+PGW-C
(2) (2) (2) (2) (2) (2)
27 © 2021 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
5G SA Core (eMBB)
• AMF (2) All Active. AMF Set (common data) replicated within Regional RDC group
• SMF (1+1) RDC1 SMF01 Primary/RDC2 SMF02 Secondary, RDC2 SMF01 Primary/RDC1 Secondary . Maintain PDU session on failover
• UPF-d All Active. UPF selection based on TAC.
Stateless Failover; On a failure of UPF in a region, calls are re-established by user on available UPFs in that region
FUTURE
W2AZ1: RDC1 W2AZ2: RDC2 N4 RDC13
N4
SMF+PGW-C-01 SMF+PGW-C-02
(Pri) (Sec) SMF+PGW-C
SMF+PGW-C-02 SMF+PGW-C-01
(Sec) (Pri) SMF+PGW-C
B-EDC : B-EDC : B-EDC :
UPF+PGW-U-111 UPF+PGW-U-121 UPF+PGW-U
N11
UPF+PGW-U-112 UPF+PGW-U-122 UPF+PGW-U N11
N11
UPF+PGW-U-113 UPF+PGW-U-123 UPF+PGW-U
N2 N2 N2
28 © 2021 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
DISH Production: RDC to B-EDC Mapping
US-West2 Region US-East2 Region US-East1 Region
NDC3 NDC2 NDC1
<50ms <15ms
West2A West2B East2A East2B East1A East1B
DEN BEDC PHX BEDC XXX BEDC ORD BEDC IAH BEDC XXX BEDC ATL BEDC NYC BEDC XXX BEDC
LAS BEDC SEA BEDC XXX BEDC CMH BEDC MCI BEDC XXX BEDC BOS BEDC MIA BEDC XXX BEDC
POR BEDC SJO BEDC XXX BEDC MSP BEDC DFW BEDC XXX BEDC PHL BEDC IAD BEDC XXX BEDC
LAX BEDC XXX BEDC XXX BEDC XXX BEDC XXX BEDC XXX BEDC XXX BEDC XXX BEDC XXX BEDC
29 © 2021 Nokia Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements
Nokia Proprietary and Confidential Information – Use Pursuant to Outstanding Non-Disclosure Agreements