Practical 5G KPI Measurement Results on a

Non-Standalone Architecture
Gábor Soós, Dániel Ficzere, Pál Varga Zsolt Szalay
Dept. of Telecommunications and Media Informatics Dept. of Automotive Technologies
Budapest University of Technology and Economics Budapest University of Technology and Economics
Budapest, Hungary Budapest, Hungary
{soos, ficzere, pvarga}@tmit.bme.hu [email protected]

Abstract—While mobile network operators have started networks as well, significant improvements on tools will be
announcing their 5G-capable network implementations, needed for detailed validation and verification on meeting
practical details of their capabilities are not widely supported 5G targets.
through measurements. This paper describes a practically
implemented 5G architecture – based on the 5G Option 3x The upcoming promises of 5G include 1 Gbps data
reference model – and initial results on its KPI (Key transmission, 1 ms latency, 1 million devices/km2 and 10
Performance Indicator) measurements. times less energy consumption (when compared to 4G UE),
Beside being matched against the 5G network expectations, however, the combined effect of these attributes to the
the results are compared to our actual hands-on 4G network
network is not fully known yet. There are dozens of new –
measurements. The analysis of the 5G testbed result shows that
– as expected – 5G outperforms 4G in all basic KPIs, including but theoretical – use-cases for 5G, however, these are not
up- and downlink throughput as well as latency. Furthermore, tested on live network, so it can not be known which will
the practical measurement results on this non-standalone 5G fulfill the expectations. When Facebook became popular, 3G
architecture show that – when traffic preferences are networks could not serve increased multimedia traffic
configured well – latency and jitter is not significantly affected
demands. Something similar could happen for 5G, as we do
by the load of the cell or the core network. This is a
long-awaited proof on 5G KPI promises coming true. not fully know, how the new use-cases will affect the
Keywords—Cellular Mobile Networks, 5G, Practical network. Some services could be overloaded for significant
Measurements, Latency, Jitter, Throughput, PER periods of time. One approach to prepare for such overloads
and transient effects is to examine the existing, ”pre-5G”
I. I NTRODUCTION use-cases and create test environments on 5G networks.

T HE first instances of 5th generation mobile networks

meaning that anybody who has subscription can connect and Both the signalling and data planes of mobile networks
transmit data – but the generic processes for QoS (Quality of need to be analyzed all along their lifecycle – especially
Service) guarantees are not implemented yet. Operators before going live. Ultimately, it would be the users’ Quality
usually run ”friendly customer tests” with no fees for this of Experience (QoE) would define the requirements;
service, however, it cannot be considered as a partial public although best practice shows that initially the basic QoS
test process, as user devices and equipment are expensive metrics must be taken care of. A mixture of these can then
and hardly available at this stage [1]. Furthermore, there is a be correlated with QoE [5].
chance that current 5G-capable user devices will not be fully
A. Basic QoS measurements
operational in the future on the live network. At present,
neither the networking services, nor the User Equipment Being very practical, it is usual that service providers
(UE) have been tuned [2] for the three main use-cases, examine only the basic attributes in case of data
namely enhanced Mobile Broad Band (eMBB), networks [6]. (Such real-life tests do not necessarily include
Ultra-Reliable Low Latency Communications (URLLC) and e.g., packet inter-arrival time or its jitter.) These can be
Massive Internet of Things (MIoT) [3]. In many cases short-period, functionality-based measurements, or long-term
service providers merely publish that their ”5G network is measurements that are often combined with stability tests.
operational”, but do not detail which parts of the 5G Since the transmission system is considered as Device or
standards have already been built, and which parts are to be Network Under Test (DoNUT), there are more parameters
built after further consideration [4]. There are numerous worth to be examined separately. The main, QoS-related
methods and tools that are available for examining LTE KPIs for our measurements are throughput, latency, packet
network and service performance. While currently used inter-arrival jitter and packet loss – which are described in
network and service validation systems can be useful on 5G Section IV and their results are presented in Section V.
B. Higher layer measurements
There are further, more subtle measurements that can be
carried out at higher OSI layers, which can reflect further
aspects of user experience. Some examples are the following.
1) The iPerf tool: A popular application for data
transmission speed measurements is iPerf – and its updated
version, iPerf3 [7]. It can be used for both TCP and UDP
transmission, even to create dynamic traffic load for a
system. Furthermore, iPerf output can be compared easily
even for significantly diversified systems.
2) ICMP-related measurements: Sending ICMP packets is
a common method to examine network latency. Such
applications can be found on almost any x86-based system,
but the output and features are very poor for detailed
analysis. Furthermore, latency can increase for many
network applications in case the traffic load increases; for
such cases ICMP is not able to provide proper insights. A Fig. 1: The architecture of our 5G testbed – as Option 3X
small improvement can be achieved with special settings, but
it is still not a fully adequate solution. TABLE I: Expected parameters of the examined technologies
3) Application-specific measurements: During the Cat 3 [14] Cat 12 [15] 5G – Option 3X [11]
transmission of huge files, latency and transmission speed
3GPP Release Release 8 Release 11 Release 15
can be analyzed at the same time. While this method may
Downlink Peak Rate 100 Mbps 600 Mbps 1500 Mbps
provide information on retransmitted packets as well, their
Uplink Peak Rate 50 Mbps 100 Mbps 100 Mbps
application-specific nature is a major drawback.
Latency (typical) 40ms 20ms 5ms
4) Tools for operator measurements: Before a service is
Bandwidth 1.4-–20 MHz 1.4–80 MHz 100 MHz and 20 MHz
allowed to go live, it is worth examining how the network
Used technology 4G only 4G 4G and 5G
performs within complex traffic scenarios. For this purpose,
targeted measurement tools can be used such as IXIA
solutions [8] or Cisco TRex [9]. Since most features we
It is worth to stress that the client uses 4G radio for uplink
require are freely available with TRex, we utilized it for our
and 5G NR is only used for downlink data transmission.
analysis – but any other system can be used to repeat the
As an initial validation for the testbed, we analyzed
measurements with properly calculated parameters.
fundamental technical promises of 5G, such as peak data
III. O UR 5G T ESTBED C ONFIGURATION rate in up- and downlink, and latency. Table I shows a
The fundamental architecture of 5G networks is defined by comparison on expectations against older 3GPP standards –
the 3GPP Release 15 standard set [10]. Many of its network where 5G is supposed to outperform all others.
elements currently operate in live, previous-generation
B. 5G User Equipment
mobile networks. The Core network is responsible for AAA
functions, service mobility management, besides, it ensures The first 5G capable device [12] category has been
connectivity with the IP network. Another key part of the 5G published in 3GPP Release 15. To fulfill the standard’s
architecture is gNB, which is responsible for the radio requirements, a 3,6 GHz frequency band, 256-QAM
attachment between UEs (the end-device) and the Core modulation and even 100 MHz bandwidth are also
network’s interfaces in case of 5G media. For the basic needed [13]. It is worth to mention that at this stage
understanding, Figure 4.2.3-1 on [10] depicts the generic 5G asymmetric network configuration is recommended. For
architecture, where the main elements are also defined. downlink 5G network’s 100 MHz bandwidth and 256-QAM
modulation are used – but for uplink and signalling
A. The 5G Testbed transmission 4G network is preferred.
The testbed we used is a standardized architecture shown In the 3GPP Release 8 [14] a device type have been
by Figure 1, and known as ”migration path to Non-Standalone published, being capable to co-operate with 4G data
Next Generation Radio with LTE assisted mode connected to networks [16]. Cat 3 radio devices were standardized mostly
Evolved Packet Core (EPC)”, or ”Option 3X” [11]. In here for low budget modems and mobile phones – but they did
the Core network is pre-5G, and the access includes 5G gNB. not spread widely. LTE category 12 have been published in
In this setup, the UE connects primarily to a 4G anchor 3GPP Release 11 [15], and is currently representing the top
point, and the core elements authenticate it on the 4G network. category of the 4G capable devices. This was the first
After the attachment, the 5G NR (New Radio) base station standard using 256-QAM modulation, therefore 600 Mbps
builds a data connection providing a high bandwidth downlink. downlink speed can be achieved.
 IV. M EASUREMENT S ETUP TABLE II: Aggregated (median) results of measurements
Our measurement took place within laboratory 4G- Cat 3 4G- Cat-12 5G - 1 UE 5G - 2 UE
circumstances, while only our devices operated, transmitted Downlink Peak Rate 75 Mbps 420 Mbps 885 Mbps 1465 Mbps
data and generated background traffic. Radio input and Uplink Peak Rate 22 Mbps 87 Mbps 92 Mbps 91 Mbps
output power could be configured arbitrarily, so it can be Latency (one-way) 13 ms 12 ms 3.71 ms 4.96 ms
considered constant for the measurement process. Packet Error Rate 0.3% 0.2% 0.39% 0.67%
Based on DoNUT models, different transmission directions
can be tested – as shown by Figure 1.
values. To measure latency, bandwidth and PER, primarily
A. Testbed elements iperf, iperf3 [7] and Cisco TRex [9] were used. The results
1) 4G, 5G UE and SIM: Our UE supported 256-QAM were further validated by Wireshark. We measured UDP and
using 2 transmitter and 4 receiver antennas in the testbed. TCP stream transmission, and with the help of TRex, IP
SIM cards had the same authorities as the simple 4G SIM packets had been sent with high frequency. It became clear
cards and 5G data bearer also had the same authorities that one device can not exploit the whole bandwidth alone,
during the building process as on 4G. In order to have although by using multiple devices we managed to reach the
reference measurements on 4G, we used a Cat 3 and a Cat DoNUT transmission limit. The critical link located between
12 device as well. the UE and radio equipment, which was the first to reach its
2) RAN: The 4G frequency band was on 1800 MHz with capacity limit.
20 MHz bandwidth in Frequency Division Duplex (FDD)
V. R ESULTS
mode. The 5G bearer was on 3,6 GHz, as we mentioned,
only in downlink transmission could attach to the user with The following findings are the results of a series of 5G non-
100 MHz bandwidth, using 32 Tx, 32 Rx antennas and 256 standalone measurements. For reference comparison we have
QAM modulation. carried out similar measurements on 4G. Our measurement
3) Mobile Core: The network was built by the 3.X results are summarized by Table II, which shows that for 5G,
reference model with the standardized S1AP and S1U [17] even 1 UE downlink peak rate heavily outperforms the 4G
interfaces towards to 4G eNB and 5G gNB. However, the values. However, 5G uplink transmission is an exception as it
inner architecture was built fully virtualized by the 5G has similar performance as 4G. As expected, latency is also
standard. So the MME, SGW and PGW functions’ resources significantly lower at 5G than at 4G.
can be varied based on the load of the network traffic. The A. Latency results
infrastructure base was an off-the-shelf x86, with CentOS
7(1810), with 64 CPUs and 128 GB RAM. There were 16 We examined how the DoNUT affect 5G latency deviation
CPUs actively connected to the Core network. with 100 Byte and 1000 Byte packets. During the tests, the
time between sending packets was between 10 ms and 1000
B. Expectations on measurement parameters ms, in 4 different scenarios. After these tests, we measured
1) Throughput: Higher data transmission rate is achieved latency with continuous background traffic, with its volume
with increasing bandwidth, different coding and better being approximately 50% of the network’s capacity: 700
modulation. In downlink direction even 100 MHz bandwidth Mbps. One-way latency results are shown by Figure 2, when
can be used; the theoretical throughput limit is 1.5 Gbps. the background traffic took 50% of the link capacity, and
Naturally, transmission rate can be tuned with the number of with no backround traffic.
receiver and transceiver antennas as well.
2) Latency: A key promise of 5G is that latency
experienced with 4G will be reduced drastically to around 1
ms, furthermore, latency can be kept low even if the traffic
load is high in the network.
3) Packet Error Rate (PER): The data connection layer of
the radio channel should correct the transmission errors if
packet loss has occurred. We measured PER at a higher
layer, so we expected to detect PER only when lower layers
were not able to restore packets. Our results confirmed this.
C. Measurement methodology
In our setup, the 5G network was treated as a Device or
Network Under Test [18]. One part of the server connected
to UE with cable, while the other one wirelessly to EPC SGi Fig. 2: Latency of 5G links, measured with different
interface (Figure 1). From the sent and received packets the transmission intervals [0.02 - 1 ms] – with background traffic
server could easily calculate latency, throughput and PER (left) and with NO background traffic (right)
Fig. 3: Jitter results (left): 100 Byte packets, 1000 ms sending Fig. 4: Jitter results (left): 100 Byte packets, 20 ms sending
period, (right): 100 Byte packets, 100 ms sending period period, (right): 100 Byte packets, 10 ms sending period

A very important finding is that even the serious background After the basic measurements, detailed analysis of one-way
traffic does not significantly affect the targeted latency values. latency and packet-intrarrival times variance was carried out
regarding 5G networks’ main attributes.
B. Jitter results with background traffic As expected, 5G data transmission latency is significantly
When analyzing the variance of packet inter-arrival times lower than it is for 4G – although sub-millisecond results
during transmission, the examined packets were 100 Byte and require further tuning. As a further result, latency and jitter
1000 Byte and also in 4 different frequency scenarios between values with traffic load are not increasing significantly on the
a packet per 1 and 0.1 seconds. Since we were controlling examined, practical 5G network testbed.
the ratio of packet sending in multiple dimensions, we can
R EFERENCES
calculate not only the variance of the packet arrival time, but
also the variance difference between the expected and real [1] Digitaltrends, Mark Jansen, “5G smartphones are on the way,”
https://www.digitaltrends.com/mobile/5g-capable-phones/, 2019.
arrival. The Equation 1 we used is very similar to calculating [2] 3GPP, “3GPP system standards heading into the 5G era,”
variance, however we replaced the mean ”μ” to the packet https://www.3gpp.org/news-events/1614-sa 5g, 2019.
sending interval as 1 sec to 0.1 sec. [3] Q. Zhang and F. H. Fitzek, “Mission critical IoT communication in 5G,”
in Future Access Enablers of Ubiquitous and Intelligent Infrastructures.
Pn Springer, 2015, pp. 35–41.
(xi − µ)2
σ 2 = i=1 (1) [4] 3GPP, “The path to 5G: as much evolution as revolution,”
n https://www.3gpp.org/news-events/1774-5g wiseharbour, 2016.
[5] P. Varga, G. Kún, G. Sey, I. Moldován, and P. Gelencsér, “Correlating
Figure 3 shows the received packet interarrival times with user perception and measurable network properties: Experimenting with
0.1 and 1 s packet sending periods (in-between packets), and QoE,” in International Workshop on IP Operations and Management.
Figure 4 shows that at 10 and 20 ms packet sending periods [6] X. Lu, W. Cao, X. Huang, F. Huang, L. he, W. Yang, S. Wang, X. Zhang,
and C. Hongsong, “A Real Implementation of DPI in 3G Network,” 12
the jitter is fairly minimal. The further (average) results are 2010, pp. 1–5.
shown by Table III. [7] iPerf, iPerf - The ultimate speed test tool for TCP, UDP and SCTP.
[8] Ixia, “Ixia validate the performance of your networking solutions,”
TABLE III: Packet inter-arrival time variance (jitter) σ 2 [msec] https://www.ixiacom.com/solutions/network-test-solutions, 2019.
[9] Cisco.com, “TRex - Realistic traffic generator ,” https://trex-
Packet size 100 Byte 1000 Byte tgn.cisco.com/, 2019.
Sending ingerval 1 sec 0.1 sec 1 sec 0.1 sec [10] 3GPP, “Summary of Rel-15 Work Items, TR 21.915 version 0.5.0
No background 0.0160618 0.232781 0.0222247 0.371 Release 15,” 3GPP, Version 15, 2018-12.
50% background 0.01724 0.2157 0.0194 0.3426 [11] GTI, Ericsson, ZTE, Nokia, “5G Network Architecture White Paper,”
https://www.3gpp.org/keywords-acronyms/1612-ue-category, 2018-02.
[12] 3GPP, “User Equipment (UE) radio access capabilities, 3GPP TS 38.306
VI. C ONCLUSION version 15.3.0 Release 15,” 3GPP, Version 15, 2018-10.
[13] ——, “Physical channels and modulation, TS 38.306 version 15.3.0
This paper aims to provide some insights of 5G and 4G Release 15,” 3GPP, Version 15, 2018-10.
network capabilities. Adter a brief introduction of the non- [14] ——, “Technical Specifications and Technical Reports for a UTRAN-
based 3GPP system, TS 21.101,” 3GPP, Version 8, 2008-12.
standalone 5G architecture, Option 3X – the paper presents [15] ——, “Technical Specifications and Technical Reports for a UTRAN-
the created testbed containing both 4G and 5G base stations, based 3GPP system, TS 21.101,” 3GPP, Version 11, 2013-03.
an EPC, and various user devices. [16] 3rd Generation Partnership Project, “LTE UE-Category,”
https://www.3gpp.org/keywords-acronyms/1612-ue-category, 2016.
As the main contribution for understanding and validating [17] 3GPP, “Evolved Universal Terrestrial Radio Access Network (E-
practical 5G capabilities, the paper presented multiple, UTRAN); S1 Application Protocol (S1AP),” 3GPP, 36.413 Rel. 8, 2016.
end-device-originated measurements. These fundamental [18] P. Orosz, P. Varga, G. Soós, and C. Hegedus, “QoS Guarantees for
Industrial IoT Applications over LTE - a Feasibility Study,” in IEEE
parameters that were compared against the widely advertised International Conference on Industrial Cyber-Physical Systems (ICPS),
expectations were throughput, latency, variance of packet 2019.
inter-arrival times (jitter) and PER.