applied

sciences
Article
The Effects of High-Performance Cloud System for Network
Function Virtualization
Wu-Chun Chung * and Yun-He Wang

Information and Computer Engineering, Chung Yuan Christian University, Taoyuan 320314, Taiwan
* Correspondence: wcchung@[Link]

Abstract: Since ETSI introduced the architectural framework of network function virtualization
(NFV), telecom operators have paid more attention to the synergy of NFV and cloud computing.
With the integration of the NFV cloud platform, telecom operators decouple network functions
from the dedicated hardware and run virtualized network functions (VNFs) on the cloud. However,
virtualization degrades the performance of VNF, resulting in violating the performance requirements
of the telecom industry. Most of the existing works were not conducted in a cloud computing
environment, and fewer studies focused on the usage of enhanced platform awareness (EPA) features.
Furthermore, few works analyze the performance of the service function chain on a practical cloud.
This paper facilitates the OpenStack cloud with different EPA features to investigate the performance
effects of VNFs on the cloud. A comprehensive test framework is proposed to evaluate the verification
of functionality, performance, and application testing. Empirical results show that the cloud system
under test fulfills the requirements of service level agreement in Rally Sanity testcases. The throughput
of OVS-DPDK is up to 8.2 times as high as that of OVS in the performance test. Meanwhile, the
hardware-assisted solution, SR-IOV, achieves the throughput at near the line rate in the end-to-end
scenario. For the application test, the successful call rate for the vIMS service is improved by up to
14% while applying the EPA features on the cloud.

Citation: Chung, W.-C.; Wang, Y.-H. Keywords: enhanced platform awareness; test framework; cloud computing; network function
The Effects of High-Performance virtualization; performance evaluation
Cloud System for Network Function
Virtualization. Appl. Sci. 2022, 12,
10315. [Link]
app122010315 1. Introduction
Academic Editors: Pedro Valero-Lara In 2012, the European Telecommunications Standards Institute (ETSI) launched an
and Antonio J. Pena industry working group to comply with the Network Function Virtualization (NFV) frame-
Received: 17 September 2022
work and standardization [1]. The primary goal of NFV technologies is to decouple network
Accepted: 11 October 2022
functions from the dedicated hardware to be run on commodity servers as software. With
Published: 13 October 2022
the help of NFV, telecom operators exploit commercial off-the-shelf servers and virtualiza-
tion technologies to provide a variety of network functions. Therefore, telecom operators
Publisher’s Note: MDPI stays neutral
reduce capital expenditure while improving flexibility from the benefits of NFV technolo-
with regard to jurisdictional claims in
gies [2]. Cloud computing also provides scalability and availability for NFV services to
published maps and institutional affil-
be more reliable in business operations [3,4]. These advantages are attractive for telecom
iations.
operators towards NFV on the cloud.
OpenStack, an open-source cloud operation system, plays a critical role in network
function virtualization infrastructure (NFVI) and virtualized infrastructure managers
Copyright: © 2022 by the authors.
(VIM) [5] of the ETSI framework. Telecom operators build an on-premises cloud on
Licensee MDPI, Basel, Switzerland. multiple nodes and deploy virtual network functions (VNFs) on top of the NFVI. However,
This article is an open access article the network functions in virtual machines (VMs) require a hypervisor or virtual machine
distributed under the terms and manager (VMM) to mediate available virtual and physical resources. This extra layer brings
conditions of the Creative Commons challenges to the performance degradation of VNFs [6] on a cloud system. The performance
Attribution (CC BY) license (https:// on computation and networking is dramatically reduced without any improvement.
[Link]/licenses/by/ Many works attempt to improve the efficiency of virtual resources on the cloud
4.0/). from different perspectives. The workflow scheduling problem is addressed in [7–9] to

Appl. Sci. 2022, 12, 10315. [Link] [Link]

Appl. Sci. 2022, 12, 10315 2 of 24

improve the workload execution on the cloud system. The authors of [10–14] focus on VM
placement to improve the power consumption for the cloud system. Some works [15–19]
further take the load-balancing issue into consideration for better resource utilization in
VNF placement. Previous works have proposed effective mechanisms and algorithms to
improve the resource management of VNF on the cloud. However, rare works emphasize
performance enhancement techniques from scratch for system virtualization.
Intel Corp. devotes massive efforts to enhanced platform awareness (EPA) [20] for
virtualization technologies. The bottlenecks of computation performance are enhanced by
CPU pinning for allocating dedicated CPU cores to a virtual machine (VM), and hugepages
for improving effective memory access. A practical cloud system with enhanced tech-
nologies for computation has been preliminarily verified in [21]. For the enhancement
of network performance, Open vSwitch with DPDK (OVS-DPDK) and single root I/O
virtualization (SR-IOV) are two effective technologies to overcome the communication bot-
tleneck. However, most of the previous works mainly focus on the perspective of network
devices [22–25] rather than a comprehensive verification of the overall cloud system under
test (SUT).
In addition, a telecom service usually requires chaining of network functions, named
the service function chain (SFC) [26], to serve the NFV application. The length of a service
chain has an impact on network performance. When the path of a service chain contains
only one VNF on the cloud system, the packets are sent from a traffic generator to the target
VNF via a physical network interface controller (NIC). The packets are then transmitted
back to the traffic generator through a physical top-of-rack (TOR) switch. This communica-
tion scenario is called a physical–virtual–physical (PVP) flow. If the SFC is compounded
by two VNFs, the packets are sent from a traffic generator to one VNF and forwarded to
another VNF before transmitting back to the traffic generator. This scenario is a typical
physical–virtual–virtual–physical (PVVP) flow. When the service path becomes longer, the
workload of components in this path is heavier. Thus, the performance is deteriorated, but
less attention has been paid to this in the literature.
Considering the telecom service as the use case, the telecom operators replace the pro-
prietary hardware with commodity servers for hosting VNFs for the SFC. Before deploying
telecom services on the cloud, the operator has to verify and validate if the cloud system
is adequate for the carrier-grade services. According to the ETSI testing specification [27],
both functionality and performance tests are the basis of the verification process on the NFV
cloud. The functional tests aim to check if the system works correctly. After verifying that
the system is operational, evaluating the performance of the system is also necessary. In
many use cases, network performance is a critical concern in deploying and orchestrating
cloud services. However, the performance is degraded due to the virtualization overhead.
Telecom operators need a systematic approach to verify the network performance with
different enhanced technologies of the cloud system.
In this regard, this paper investigates the enhancement of network techniques to
mitigate the deterioration in a practical cloud system. A comprehensive testing framework
is proposed to evaluate the cloud system with different EPA technologies in three areas,
which are functionality, performance, and application. The functionality tests focus on
instantiating VMs, building a tenant network, and the connectivity of VMs. In performance
tests, both PVP and PVVP scenarios are considered to estimate the network throughputs.
As for the application test, this paper not only deploys a virtual IP multimedia subsystem
(vIMS) on the cloud, but also evaluates the performance of the vIMS via a session initiation
protocol (SIP) stress benchmark. The performance metric of successful call rate is measured
for the vIMS test. Empirical results show that the throughput of the PVP test performs near
the line rate when applying the SR-IOV acceleration, while OVS-DPDK approaches 90%
of the line rate when packet size is 1518. Increasing the number of lcores and pmd-cpus
further enhances the performance by about 50% in OVS-DPDK. In addition, the successful
call rates in vIMS validations with different techniques are similar in lower call rates. When
Appl. Sci. 2022, 12, 10315 3 of 24

the call rate exceeds 100, the improvement brought by OVS-DPDK is up to 14%. To sum
up, the contributions of this paper are:
• Proposing a comprehensive testing framework to investigate a practical cloud system
on different areas of performance effects.
• Conducting performance comparisons between the generic and the enhanced cloud
system for NFV.
• Considering both PVP and PVVP packet paths for different SFC scenarios.
• Deploying the vIMS application on the enhanced cloud system to show effective performance.
The rest of this paper is organized as follows. Section 2 presents the background and
discusses related works to support our research. The comprehensive testing framework
and the test methodologies are proposed in Section 3. Section 4 introduces the experiments
and results. Finally, Section 5 concludes this paper with summarizing remarks.

2. Background and Related Works

This section first introduces the background of enhancement technologies and then dis-
cusses the most relevant works to improve the performance of virtualization on the cloud.

2.1. Background
Regarding the virtualization of network functions, VM and container are two common
technologies for running applications on a shared resource. The VM shares the underlying
hardware via the VMM. Each VM is running with a guest operating system and dependen-
cies of the application. On the other hand, the container simply packages the user program
and corresponding dependencies to run the application. Multiple containers on the same
host share the same application runtime environment of an operating system. Therefore,
the container consumes less warmup time and fewer system resources [28]. However,
running a container relies on sharing the host operating system, which may lead the system
to a security issue. Most of the cloud service providers tend to provide the container service
based on a VM for better isolation and easy management in the data center. Tackling the
performance degradation of a VM is a pressing matter. Accordingly, this paper takes the
VNF as an example to verify the proposed framework.
Regarding the enhanced technologies for the computation, EPA features involve
typical mechanisms in the operating system such as CPU isolation, CPU pinning, and
hugepages. The resource allocation scheme with a nonuniform memory access (NUMA)
awarded topology is also considered as the performance impact in cloud computing. As
for network improvement, many technologies are proposed for providing high-speed
networking in the literature. This paper takes Open vSwitch (OVS), OVS-DPDK, and
SR-IOV as examples for accelerating the packet processing in the cloud system. The basic
concept of each enhanced technology adopted in this paper is introduced as follows.
• CPU Isolation
When the system administrator sets the isolated CPU mask, the system cannot assign
regular jobs to the isolated CPU cores. In other words, each isolated CPU core remains idle
if the system does not dispatch any specific process to the isolated CPU core. Therefore,
the system administrator must set the isolated CPU properly; otherwise, the setting wastes
computing resources for the system.
• CPU Pinning
CPU pinning enables the system to assign a job to specific CPU cores. In general,
CPU isolation and CPU pinning are complementary. CPU isolation reserves a set of CPU
resources, while CPU pinning allows the specific process to acquire the dedicated resources.
If the system administrator enables these two settings, the VM process does not have to
compete for CPU resources with others [29]. Accordingly, eliminating the CPU resource
competition keeps the VM process away from the context switch overhead.
 resources, while CPU pinning allows the specific process to acquire the dedic
sources. If the system administrator enables these two settings, the VM process d
have to compete for CPU resources with others [29]. Accordingly, eliminating t
Appl. Sci. 2022, 12, 10315 resource competition keeps the VM process away from the context switch4 of 24 overhe

• Nonuniform Memory Access

• NUMA isMemory
Nonuniform a memory architecture in a multi-CPU computer system. Figure 1
Access
a host consisting
NUMA of two
is a memory NUMAinnodes.
architecture Eachcomputer
a multi-CPU NUMA system.
node has its 1CPU
Figure cores
depicts a and m
resources.
host Memory
consisting accessnodes.
of two NUMA is faster
EachifNUMA
CPU cores
node hasaccess thecores
its CPU memory on the same
and memory
node. However, if CPU cores have to access the memory from another NUMA n
resources. Memory access is faster if CPU cores access the memory on the same NUMA
crossing
node. Quick
However, Pathcores
if CPU Interconnect (QPI)
have to access the access
memoryresults in increasing
from another NUMA node, access
thelatency
crossing Quick Path Interconnect (QPI) access results in increasing access latency.

Figure
Figure 1. 1. NUMA
NUMA architecture.
architecture.

• Hugepage
• Hugepage
The page table in the memory stores the translation between logical and physical
The The
addresses. page tablelocates
system in thethememory
memory stores
address the translation
according between
to the page number logical
and and
addresses. The system locates the memory address according to the page num
corresponding frame number in the page table. Accessing a particular memory address
requires several mappings to find the accurate translation. A translation lookaside buffer
corresponding frame number in the page table. Accessing a particular memory
(TLB) is responsible for speeding up the effective memory access via caching the recent
requires
address several mappings
translation. The more thetocache
findhits,
thethe
accurate translation.
less access Afor
time there is translation
the memory lookasid
(TLB) is responsible
addressing. for speeding
Hugepage technology enables up thepage
a large effective memory
size to provide moreaccess
memoryviaspace
caching th
address translation. The more the cache hits, the less access time there issofor the
for data access [30]. Therefore, the number of TLB misses and page faults is decreased,
addressing.
as Hugepage
to accelerate the technology
effective memory [Link] a large page size to provide more
•spaceOVS for data access [30]. Therefore, the number of TLB misses and page faul
creased, past,
In the so asVMs
to accelerate
did not sharethe effective
a virtio memory
ring with access.
the user space process in a host [31], i.e.,
vswitch. The packet processing had to be handled in the kernel space. Figure 2 illustrates
• OVS
the network architecture of OVS. ovs-vswitchd is used to set up the OVS data path. The
In the past,
configuration VMs didisnot
of ovs-vswitchd share
stored a virtio
in ovsdb. ring with
Therefore, the user connects
ovs-vswitchd space process
to the in a h
i.e., vswitch. The packet processing had to be handled in the kernel space. Figure
ovsdb-server to retrieve the configuration. When a cloud system adopts the OVS network,
the OVS kernel module looks up the flow table when a packet is received. If the flow
trates the network architecture of OVS. ovs-vswitchd is used to set up the OVS da
matches a network rule, the packet is processed according to the actions associated with the
The When
flow. configuration
the packet of ovs-vswitchd
does not match any is stored
rule in ovsdb.
in the flow Therefore,
table, the OVS kernelovs-vswitchd
module c
to the ovsdb-server to retrieve the configuration. When a cloud system
queues the packet to the user space. This procedure introduces context switches and results adopts t
network,
in the OVS
slowing down kernel
the packet module looks up the flow table when a packet is receive
processing.
flow matches a network rule, the packet is processed according to the actions as
with the flow. When the packet does not match any rule in the flow table, the OV
module queues the packet to the user space. This procedure introduces context s
and results in slowing down the packet processing.
 [Link].
Appl. Sci. 2022,12,12,10315
10315 5 of 24
Appl.2022,
Sci. 2022, 12, 10315 5 of 24

Figure 2. OVS architecture.

Figure 2. 2.
OVS-DPDK
•Figure OVSOVS architecture.
architecture.

• The data plane development kit (DPDK) aims to accelerate the packet processing.
OVS-DPDK
The OVS-DPDK
• network architecture of OVS-DPDK is shown in Figure 3. With the help of vhost-user,
The data plane development kit (DPDK) aims to accelerate the packet processing. The
VMs share
networkThe the memory
data plane
architecture space with
development
of OVS-DPDK userkit
is shown space processes
in(DPDK)
Figure aimsin
3. With the
theto host. Sharing
accelerate
help the memory
of vhost-user, be-
packet pro
VMs
tween
share two
The network
the processes
memory spaceiswith
architectureimplemented
of OVS-DPDK
user by a file
space processes descriptor.
is in
shown The file
in Sharing
the host. Figure 3. descriptor
With between
memory can be
the help ac-
of vho
cessed
VMs
two by the
share is
processes vhost
the process
memory space
implemented (i.e., virtual
by a filewith switch).
user space
descriptor. The NIC
The file processes driver,
descriptor canin be
thesuch as UIO
host. Sharing
accessed or
by the VFIO
mem
[32],
vhostisprocess
responsible for
(i.e., virtual OVS-DPDK
switch). The NIC to handle the
driver, such user
as UIOspace
or VFIO
tween two processes is implemented by a file descriptor. The file descriptor ca interrupt. Therefore,
[32], is responsible OVS-
DPDK can process
for OVS-DPDK the packets
to handle the userinspace
userinterrupt.
space and pass them
Therefore, through the
OVS-DPDK can kernel.
process The
the kernel
cessedinbyuserthespace
vhost process (i.e., virtual switch). The NICbypass
driver, such as UIO
bypass eliminates the overhead of system calls and context switches. OVS-DPDK also pro-
packets and pass them through the kernel. The kernel eliminates
[32],
the is responsible
overhead for and
of system calls OVS-DPDK to handle
context switches. OVS-DPDKthe user
also space
providesinterrupt.
a poll mode Therefor
vides a poll mode driver (PMD) [33] to poll the network queue. The dedicated CPU cores
DPDK can
driver
assigned
(PMD) process
[33]
to PMD
to the packets
poll the
are helpful
network in userthe
queue.
to alleviate space
The and pass
dedicated
interrupt
CPU them through the kernel. Th
overhead.
cores assigned to PMD
bypass eliminates the overhead of system calls and context switches. OVS-DPDK a
are helpful to alleviate the interrupt overhead.

vides a poll mode driver (PMD) [33] to poll the network queue. The dedicated CP
assigned to PMD are helpful to alleviate the interrupt overhead.

Figure 3. Network
Figure 3. Networkarchitecture
architecture of OVS-DPDK.
of OVS-DPDK.

• SR-IOV
With the help of the input–output memory management unit (IOMMU), the VM can
access Peripheral Component Interconnect (PCI) devices directly, called PCI passthrough.
Figure 3. Network architecture of OVS-DPDK.
Appl. Sci.
Appl.2022, 12, 10315
Sci. 2022, 12, 10315 6 of 24

• SR-IOV
via the virtual address. When a VM accesses the network device, the interrupt rem
With the help of the input–output memory management unit (IOMMU), the VM can
is triggered to send an interrupt to the VM instead of the host. With DMA and in
access Peripheral Component Interconnect (PCI) devices directly, called PCI passthrough.
remapping,
Direct memorythe network
access (DMA) interface
remappingis[34] assigned
allows the directly
VM accessto atoVM so that
physical the VMM
memory
affect
via the performance.
the virtual address. When a VM accesses the network device, the interrupt remapping
In thistoregard,
is triggered accessing
send an interrupt to the
the VMNIC directly
instead tohost.
of the a VM canDMA
With significantly
and interrupt improve
work performance. However, allocating a dedicated physical NIC for each VM is
remapping, the network interface is assigned directly to a VM so that the VMM will not
affect the performance.
pensive.
In thisThe SR-IOV
regard, is proposed
accessing the NIC to tackletothis
directly a VM problem. As shown
can significantly in Figure
improve the 4, a N
SR-IOV support canHowever,
network performance. provideallocating
physicalafunction
dedicated (PF) andNIC
physical virtual function
for each VM is too (VF) [35].
has all the functions of the NIC and is responsible for managing the VFs. The VF i
expensive. The SR-IOV is proposed to tackle this problem. As shown in Figure 4, a NIC
plified
with PCIesupport
SR-IOV functioncanand allows
provide the VM
physical to access
function thevirtual
(PF) and physical NIC(VF)
function directly.
[35]. Acco
The PF has all the functions of the NIC and is responsible for managing the VFs. The
the network performance of a VM can be improved in the cloud system. Howev
VF is a simplified PCIe function and allows the VM to access the physical NIC directly.
IOV is a hardware-assisted
Accordingly, technique,
the network performance andcan
of a VM it is
benot easy toinlive-migrate
improved the cloud system. a VM with
IOV
However,VF. SR-IOV
Procuring the specific NIC
is a hardware-assisted is also necessary
technique, while
and it is not easy adopting the
to live-migrate a VM SR-IOV,
increase the cost of capital expenditure and management efforts for the cloud.
with an SR-IOV VF. Procuring the specific NIC is also necessary while adopting the SR-IOV,
so as to increase the cost of capital expenditure and management efforts for the cloud.

Figure 4. Network architecture of SR-IOV.

Figure 4. Network architecture of SR-IOV.
2.2. Related Works
2.2. Related
AlthoughWorks
NFV realizes flexible deployment, virtualization technology introduces the
performance overhead. Rojas-Cessa et al. [36] estimated the performance of virtualized
Although NFV realizes flexible deployment, virtualization technology introdu
and nonvirtualized routers to show the degradation caused by virtualization. Their work
performance overhead.
revealed the performance Rojas-Cessa
impact et [Link][36]
of the number coresestimated
and differentthe performance
packet sizes. The of virt
and nonvirtualized
experimental routers
results also to show
demonstrate the degradation
the benefit causedOn
of using multicores. bythe
virtualization.
other hand, The
revealed the performance impact of the number of cores and different packet siz
the performance degradation of traditional Linux I/O for high-speed networking has been
discussed in previous works [37–39]. The main reasons are summarized as the long paths,
experimental results also demonstrate the benefit of using multicores. On the othe
the frequent interrupts, and the repeated data replications. Therefore, researchers try to
the performance degradation
reduce the duplication of traditional
of packets between Linux
the physical NICI/O
andforthe
high-speed networking h
user space process.
discussed in previous
Previous works works [37–39].
focus on reducing or avoidingThe main reasons
hardware are summarized
interruptions as the lon
after the physical
the
NIC frequent interrupts,
receives packets. and attempts
This paper the repeated data replications.
to investigate the enhanced Therefore,
technologiesresearcher
on
reduce the duplication of packets between the physical NIC and the user space p
packet acceleration for a cloud system.
Previous works focus on reducing or avoiding hardware interruptions after the p
NIC receives packets. This paper attempts to investigate the enhanced technolo
packet acceleration for a cloud system.
In the past, the virtual switch ran as a user space process and could not sh
Appl. Sci. 2022, 12, 10315 7 of 24

In the past, the virtual switch ran as a user space process and could not share the
memory with VMs. Owing to this limitation, the virtual switch had to receive packets
from the kernel space. However, moving the packets between kernel space and user space
introduces too many context switches to deteriorate the performance of VMs. To relieve the
bottleneck, Virtual Open System Corp. designed the vhost-user for VM to share its memory
with a virtual switch. Paolino et al. [31] then proposed a virtual switch, named Snabbswitch,
and evaluated the performance of the virtual switch, such as OVS, OVS-DPDK, VFIO, and
SR-IOV. In that work, the authors considered the scenarios of one VM and two VMs, in
which each VM is enhanced with a 1G hugepage. However, only the process of Snabbswitch
is allocated with the dedicated CPU cores. In addition, the experiments in their work were
not conducted in a cloud system.
Pitaev et al. [40] compared the performance of three different I/O architectures: OVS-
DPDK, [Link] VPP, and SR-IOV. The experimental environment contains a host and a packet
generator. The target VM is deployed directly on the KVM hypervisor rather than a cloud
system. The results show that SR-IOV is better than OVS-DPDK or [Link] VPP in IPv4
forwarding and the SFC scenario. In addition, the limitation of OVS-DPDK or [Link] VPP is
that enough cores must be bound for PMD to maintain high throughputs. This constraint
also restricts the maximum number of available VMs. The container technique is becoming
more popular. G. Ara et al. [41] evaluated the network performance of various vswitches
such as Linux-bridge, OVS-DPDK, VPP, and SR-IOV in containers. The results demonstrate
that SR-IOV outperforms OVS-DPDK in terms of throughput.
R. Bonfiglia et al. [23] also tested the effects of OVS-DPDK and OVS on VMs and
containers. The throughputs of a VM and a container are similar when both are applied
to the OVS. In addition, the OVS-DPDK performs higher throughputs when compared to
the OVS. Kourtis, M. A. et al. [22] went further to compare the network performance of a
VNF with/without OVS-DPDK and SR-IOV. The authors used DPI as the VNF for testing.
The results demonstrate that the throughput of a VM with OVS-DPDK or SR-IOV is higher
than that of a VM without the OVS-DPDK or SR-IOV technique. Nevertheless, the previous
works were not conducted on a cloud system. This paper tends to conduct experiments on
a practical cloud system and verify the empirical results, which are rarely given attention.
The authors in [6,42] indicated that the VM in cloud computing has an issue of
performance degradation. F. Callegati [43] et al. compared the performance of the Linux
bridge and OVS in OpenStack. The authors found that the performance of the Linux
bridge is worse than that of OVS. Tsai, M. H. et al. [21] discussed the effect of enabling EPA
on a computing-intensive cloud system. They exploited CPU pinning and hugepages to
improve the computing capability of VMs in NFVI. The results show that the computing
performance of VMs could be enhanced by up to 7%. However, the performance of network
enhancement techniques was not further discussed in the previous work.
To sum up, this paper synthesizes the relevant works on experimental conditions as
shown in Table 1. The conditions include whether the enhanced techniques are applied
or not, and whether the experiments are conducted in a cloud system. This table helps
us find out where the discussion has been inadequate over the years. Based on the table,
hardware accelerator architectures such as SR-IOV are rarely introduced for the verification
on a cloud system. Even if the SR-IOV is adopted, the proposed experimental environment
is not designed for ETSI NFV framework. Furthermore, EPA technologies enhance the
performance of VNFs in either computing or networking. Previous work rarely considers
the corresponding technologies at the same time. Therefore, this paper considers the
OpenStack cloud as the system under test and attempts to assess the effect of EPA features
for the NFV on a practical cloud system.
Appl. Sci. 2022, 12, 10315 8 of 24

Table 1. Summary of related works.

CPU OVS-
Works Hugepage OVS SR-IOV In Cloud
Pinning DPDK
Tsai et al. [21] v v v
Kourtis et al. [22] v
Gallenmuller et al. [24] v
Appl. Sci. 2022, 12, 10315 8 of 24
Kawashima et al. [25] v v v
Huang et al. [28] v v
Paolino et al. [31] v v v v
Shanmugalingam et al. [38] v v
Shanmugalingam et al. [38] v v
Pitaev et al. [40] v v
Pitaev et al. [40] v v
Ara et al. [41] v v
Ara et al. [41] v v
Callegati et al. [43] v v
Callegati et al. [43] v v
Ours v v v v v v
Ours v v v v v v

3. Methodology
3. Methodology
This paper designs a comprehensive testing framework for the NFV cloud. The goal
is toThis paper
verify the designs a comprehensive
performance testing framework
effects of a practical for the
cloud system NFV
with cloud. enhanced
different The goal
istechnologies.
to verify theThis
performance effects of a practical cloud system with different enhanced
section first introduces the proposed framework and then discusses the
technologies.
key factors ofThis section first
the network introduces the proposed framework and then discusses the
performance.
key factors of the network performance.
3.1. Overview of Testing Framework
3.1. Overview of Testing Framework
Figure 5 illustrates an implementation of the proposed test framework in this paper.
Figure 5 illustrates an implementation of the proposed test framework in this paper.
This framework is aligned with the ETSI NFV framework and mainly contains NFVI, VIM,
This framework is aligned with the ETSI NFV framework and mainly contains NFVI, VIM,
VNFM,NFVO,
VNFM, NFVO,and andtest
test host.
host. NFVI
NFVI is composed
is composed of physical
of physical servers
servers incloud
in the the cloud system.
system. The
The virtualization layer pools and extracts the physical resources into
virtualization layer pools and extracts the physical resources into virtualized resources. The virtualized re-
sources. The virtualized resources are allocated and managed by
virtualized resources are allocated and managed by VIM. In this paper, OpenStack plays VIM. In this paper,
OpenStack
the playsThe
role of VIM. thetesting
role oftools
VIM. The the
build testing
testingtools build the testing
environment environment
via OpenStack via
services.
OpenStack
Most testingservices.
tools areMost testingintools
installed are installed
the test host, which in the test host, which
is independent is independent
of the nodes of a
of the nodes of a cloud system. Therefore, the testing is triggered
cloud system. Therefore, the testing is triggered on the host and connected to on the host and con-
OpenStack
nected to OpenStack services via the external network and a TOR
services via the external network and a TOR switch. Only if the tools are authorizedswitch. Only if the tools
are authorized with a credential file are the test processes then activated
with a credential file are the test processes then activated and verified on the OpenStack and verified on
the OpenStack system. According to the ETSI TST specification, the
system. According to the ETSI TST specification, the proposed framework considers a proposed framework
considerscloud
complete a complete
system cloud
undersystem
test. under test. The verifications
The verifications of the cloud of the
SUTcloud SUT empha-
emphasize three
size three
areas: areas: functionality,
functionality, performance, performance, and application
and application testing. testing.

OSS / BSS Test Host

NFV
Orchestrator
(NFVO)
Applica�on
VNF VNF VNF
VNF
VNF
VNF
VNF
VNF VNF
VNF VNF
Cloudify
●●●

Functest NFVBench SIP-stress

VNF
Management
NFV Infrastructure (NFVI) (VNFM) Performance
Virtual Virtual
Virtual Storage
Compute Network
Virtualiza�on
Virtualiza�on Layer Infrastructure
Func�onal
Management
Hardware Resources (VIM)

[Link]
Figure Proposedtesting
testingframework
frameworkfor
forcomprehensive
comprehensiveverifications
verificationson
onthe
theNFV
NFVcloud.
cloud.

3.1.1. Functionality Verification

In the functionality test, the verification focuses on testing the basic operations of a cloud
system. The main goal is to verify the core service operations in a cloud system via a set of test
tools. For example, the system under test in this paper is an OpenStack cloud. The Functest
Appl. Sci. 2022, 12, 10315 9 of 24

3.1.1. Functionality Verification

In the functionality test, the verification focuses on testing the basic operations of a
cloud system. The main goal is to verify the core service operations in a cloud system via a
set of test tools. For example, the system under test in this paper is an OpenStack cloud. The
Functest [44] provides a set of test cases to verify core services for operating the OpenStack
cloud. If the services of OpenStack are active properly, the testing process further evaluates
the operating performance of each service, e.g., the turnaround time of each test case. This
performance evaluation checks if OpenStack services finish the specific operations in time.
If the operations break the time limits, the relevant services work incorrectly. This paper
applies Rally Sanity to check if the services fulfill the predefined SLA.
For the test cases in Functest, this paper adopts Healthcheck and Rally_Sanity to
test the primary functionalities of an OpenStack cloud system. In Healthcheck test suite,
Connection_check is used to check the connectivity of OpenStack Endpoints. Vping_ssh
tests whether the VM can be accessed via SSH from the external network. Vping_userdata
is used to verify if the VMs can communicate with each other through the internal network.
Rally_Sanity evaluates the operating performance of OpenStack services. The primary
services include Nova, Neutron, Cinder, Keystone, and so on. Nova is used to provide
computing instances. Neutron is responsible for the networking as a service. Cinder
provides the block storage service. Keystone is used to provide the authentication service.
These test cases of Rally_Sanity measure the time taken by each scenario to make sure that
all tests fulfill the SLA.

3.1.2. Performance Verification

In the performance test of a SUT, the verification focuses on testing the network
performance of VNFs. The main goal is to benchmark the target VNF and evaluate the
performance metric. This paper adopts NFVBench [45] to evaluate the throughput of VNFs
on the SUT. In NFVBench, TRex is applied as a traffic generator and a TestPMD as a target
VNF to forward the received packets. When launching the performance test, TRex sends
packets from the test host to the target VNF. Before reaching the VNF, the packets are
transmitted throughout the TOR switch and the OpenStack cloud system. VNF then uses
L2 Forward to send the packets back to the traffic generator.
In practice, a telecom application usually deploys multiple VNFs and connects these
VNFs to the SFC. When the number of VNFs in the SFC increases, the packet path of this
SFC is longer. The length of a packet path has an impact on the network performance. This
paper considers different traffic flows for performance test scenarios: PVP and PVVP tests.
The experiment of the PVP scenario is used to estimate the performance of end-to-end
traffic. As shown in Figure 6, the traffic generator installed on the test host sends out the
packets to the target VM deployed on a cloud system. When the VM receives packets, the
VM sends the packets back to the traffic generator through the TOR switch.
The PVVP scenario is conducted for the multiple VNFs of a simple SFC scenario as
shown in Figure 7. The traffic generator sends out the packets to the first VM. When the first
VM receives packets, the first VM forwards the packets to the other VM via the east–west
traffic in OpenStack. The VM then sends the packets back to the traffic generator through
the TOR switch. Lengthening the packet path increases network communications in the
data plane. If a component in the packet path cannot handle the packets, the throughput
worsens. Therefore, this experiment involves both evaluations of the north–south traffic
and the east–west traffic in the SUT.

3.1.3. Application Verification

Regarding the application test, the verification focuses on deploying the NFV applica-
tion on the cloud system. The main goal is to validate the feasibility and effectiveness of
SUT. This paper applies the Clearwater vIMS to act as the telecom service. The architecture
of Clearwater vIMS is shown in Figure 8.
 VNFs to the SFC. When the number of VNFs in the SFC increases, the packet path of this
SFC is longer. The length of a packet path has an impact on the network performance.
This paper considers different traffic flows for performance test scenarios: PVP and PVVP
tests. The experiment of the PVP scenario is used to estimate the performance of end-to-
end traffic. As shown in Figure 6, the traffic generator installed on the test host sends out
Appl. Sci. 2022, 12, 10315 10 of 24
the packets to the target VM deployed on a cloud system. When the VM receives packets,
the VM sends the packets back to the traffic generator through the TOR switch.

Appl. Sci. 2022, 12, 10315 10 of 24

Appl. Sci. 2022, 12, 10315 throughput worsens. Therefore, this experiment involves both evaluations of the north– 10 of 24
south traffic and the east–west traffic in the SUT.

throughput worsens. Therefore, this experiment involves both evaluations of the north–
south
Figuretraffic
Figure 6. andpath
6. Packet
Packet the in
path east–west
in PVP traffic in the SUT.
PVP scenario.
scenario.

The PVVP scenario is conducted for the multiple VNFs of a simple SFC scenario as
shown in Figure 7. The traffic generator sends out the packets to the first VM. When the
first VM receives packets, the first VM forwards the packets to the other VM via the east–
west traffic in OpenStack. The VM then sends the packets back to the traffic generator
through the TOR switch. Lengthening the packet path increases network communications
in the data plane. If a component in the packet path cannot handle the packets, the

Figure 7. Packet path in PVVP scenario.

3.1.3. Application Verification

Regarding the application test, the verification focuses on deploying the NFV appli-
cation on the cloud system. The main goal is to validate the feasibility and effectiveness
of [Link] paper applies the Clearwater
scenario. vIMS to act as the telecom service. The archi-
Figure 7. Packet
Figure Packet path
path in
inPVVP
PVVP scenario.
tecture of Clearwater vIMS is shown in Figure 8.

3.1.3. Application Verification

Regarding the application test, the verification focuses on deploying the NFV appli-
cation on the cloud system. The main goal is to validate the feasibility and effectiveness
of SUT. This paper applies the Clearwater vIMS to act as the telecom service. The archi-
tecture of Clearwater vIMS is shown in Figure 8.

Figure 8. Architecture of Clearwater vIMS.

Figure 8. Architecture of Clearwater vIMS.
The main function of Bono is Proxy Call Session Control Function (P-CSCF). P-CSCF
The main function of Bono is Proxy Call Session Control Function (P-CSCF). P-CSCF
acts as an edge proxy and is responsible for the network address translation and the con-
acts as an edge proxy and is responsible for the network address translation and the
nection between the user equipment and the Clearwater system. Sprout is the SIP route
connection
in [Link] the user equipment
Sprout is responsible and the Clearwater
for user authentication system.
and SIP registration. TheSprout
ma- is the SIP
jor functions
route of Sprout are
in Clearwater. the Interrogating
Sprout is responsibleCallfor
Session
user Control Function and
authentication (I-CSCF) and
SIP registration. The
the Serving
major Call Session
functions of SproutControl
are Function (S-CSCF). S-CSCF
the Interrogating manages
Call Session and controls
Control Functionthe (I-CSCF) and
communications, whileofI-CSCF
Figure 8. Architecture sendsvIMS.
Clearwater SIP requests to the appropriate S-CSCF according
to the information stored in Homer. Application servers provide advanced features. The
features include call forwarding, call recording, etc. Homer is used to store the settings
The main function of Bono is Proxy Call Session Control Function (P-CSCF). P-CSCF
files of the application service. Dime acts as a diameter gateway and is composed of
acts as an edge proxy and is responsible for the network address translation and the con-
Homestead and Ralf. Ralf is used to trigger functions, while Homestead is an HSS cache
Appl. Sci. 2022, 12, 10315 11 of 24

the Serving Call Session Control Function (S-CSCF). S-CSCF manages and controls the
communications, while I-CSCF sends SIP requests to the appropriate S-CSCF according
Appl. Sci. 2022, 12, 10315
to the information stored in Homer. Application servers provide advanced features. The
11 of 24
features include call forwarding, call recording, etc. Homer is used to store the settings files
of the application service. Dime acts as a diameter gateway and is composed of Homestead
and Ralf. Ralf is used to trigger functions, while Homestead is an HSS cache for storing
for storing user information. Vellum is responsible for the state storing and exploits Cas-
user
sandrainformation. Vellum is data.
to store all long-lived responsible for the
Lastly, Ellis state storing
is responsible forand
userexploits
sign-upCassandra
and pass- to store
all long-lived
word [Link]. Lastly, Ellis is responsible for user sign-up and password management.
In
In this
thispaper,
paper,thetheSIP
SIPstress is used
stress to benchmark
is used to benchmark the vIMS application.
the vIMS SIP stress
application. SIP stress
simulates the
simulates thephone
phonecalls
callsamong
among multiple
multiple user equipments
user equipments (UEs) and and
(UEs) records the num-
records the numbers
bers
of of outgoing
outgoing calls,
calls, successful
successful calls,
calls, failedcalls,
failed calls,etc.
etc. The
The successful
successfulcallcallrate
rateis is
measured
measured as the
as the performance
performance metricmetric of vIMS.
of vIMS. A higher
A higher successful
successful callcall
raterate indicates
indicates betterperformance.
better perfor- To
mance. To
deploy thedeploy the Clearwater
Clearwater vIMS on vIMS on the
the cloud cloud system,
system, this paperthis adopts
paper adopts
Cloudify Cloudify
Manager [46]
Manager
to manage [46] to manage
and and orchestrate
orchestrate VNF resources VNF resources in OpenStack.
in OpenStack. Deploying
Deploying vIMSvIMS requires cus-
requires customized configurations according to the SUT environment,
tomized configurations according to the SUT environment, e.g., authentication e.g., authentica-endpoint,
tion endpoint, credential file of the cloud, VM flavor, connection keypair, etc. After setting
credential file of the cloud, VM flavor, connection keypair, etc. After setting the correspond-
the corresponding information of the SUT, Cloudify Manager deploys the vIMS template
ing information of the SUT, Cloudify Manager deploys the vIMS template according to the
according to the configurations. After the deployment is successful, the SIP stress is acti-
configurations.
vated. After the deployment is successful, the SIP stress is activated.

3.2. VerificationProcedures
3.2. Verification Proceduresandand Factors
Factors
The testing
The testingframework
frameworkisismade madeupupofofopen-source
open-source tools
tools andand is easy
is easy to to reproduce. First,
reproduce.
First,testing
the the testing
toolstools
are are containerized
containerized andandcan
canbe be run
run as docker
dockerinstances
instances ononthethe
testtest host.
host. Second,
Second, each each verification
verification hashas to set
to set thethe configurationfile
configuration fileaccording
according to to the SUT
SUTenvi-
environment
ronmentrunning
before before running
the testthe test Finally,
case. case. Finally, the testing
the testing toolstools
areare launchedtotoperform
launched perform the SUT
the SUT
test. test. Incases,
In some some the
cases, the system
SUT SUT system requires
requires some some adjustmentsaccording
adjustments according to to aa specific
specific deployment
deployment of the of the cloud
cloud [Link].
For For example,
example, thethe authenticationendpoint
authentication endpoint varies
varies among
among different
different cloud cloud systems.
systems. TheThe testing
testing shouldmodify
should modify the thecredential
credential filefile
according
accordingto to the
the cloud system. As another example, enabling DPDK requires a dedicated network in-
cloud system. As another example, enabling DPDK requires a dedicated network interface.
terface. The SUT system has to attach network interface to the provider network with
The SUT system has to attach network interface to the provider network with OVS-DPDK.
OVS-DPDK.
Figure 9 depicts an overview of the verification procedure in the proposed test frame-
Figure 9 depicts an overview of the verification procedure in the proposed test frame-
work.
work. Before deployingthe
Before deploying theSUT
SUTsystem,
system,thethefirst
firststep
stepisistotoplan
planandanddetermine
determine which kinds
which
of EPA features should be enabled on an OpenStack cloud for
kinds of EPA features should be enabled on an OpenStack cloud for verification. The sec- verification. The second
ond step is to deploy the OpenStack cloud with corresponding configurations to enable
step is to deploy the OpenStack cloud with corresponding configurations to enable the
the accelerating technologies for the SUT test. The third step is verifying and evaluating
accelerating technologies for the SUT test. The third step is verifying and evaluating the
the cloud
cloud systems
systems withwith differentbenchmarking
different benchmarking tools.
tools. The
Thelastlaststep
stepisiscollecting
collecting and vali-
and validating
dating
the testthe test results.
results. The verification
The verification process
process willwill
be be ended
ended untilall
until allthe
theenhanced
enhanced fea-features are
tures are verified and
verified and accomplished. accomplished.

Determine the Deploy OpenStack Verify and

features with the features evaluate the SUT

No

Yes Test all the Validate the

End
features results

Figure 9.
Figure [Link]
Verificationprocedures.
procedures.

Once
Oncethethedeployment
deployment of of
a customized
a customizedOpenStack is complete,
OpenStack the verification
is complete, and
the verification and
evaluation process tests the cloud system in terms of functionality, performance, and
evaluation process tests the cloud system in terms of functionality, performance, and VNF VNF
application. The functionality tests verify the general operations of a cloud system and the
application. The functionality tests verify the general operations of a cloud system and the
connectivity of VMs. If the basic functions operate correctly, the next step is to evaluate
connectivity of VMs. If the basic functions operate correctly, the next step is to evaluate the
the performance test of VNF networks on the cloud. Afterward, the application-level test
performance test ofthe
is optional to deploy VNF SFCnetworks ontrigger
service and the cloud. Afterward, tools
the corresponding the application-level
for a stress test. test is
optional to deploy the SFC service and trigger the corresponding tools
To evaluate the impacts on the SUT verification, this paper conducts the verificationfor a stress
fac- test. To
tors from different perspectives: packet size, VM size, and call rate.
Appl. Sci. 2022, 12, 10315 12 of 24

evaluate the impacts on the SUT verification, this paper conducts the verification factors
from different perspectives: packet size, VM size, and call rate.
Appl. Sci. 2022, 12, 10315 • Packet Size 12 of 24

Different applications have different requirements for performance characteristics. The

difference in these characteristics requires different packet sizes in various applications [47].
•WhenPacket
theSize
header of each packet is the same, the larger payload results in a larger packet.
Different
The applicationapplications have different
of interactive requirements
video often for performance
requires low characteristics.
latency. Therefore, the large packet
The difference in these characteristics requires different packet sizes in various
size is not suitable for interactive video applications. However, the application applica-
using small
tions [47]. tends
packets When tothesend
header
outofhuge
each amounts
packet is the same, the Iflarger
of packets. payloadswitch
the virtual results cannot
in a afford
larger packet. The
the massive application
packets, some of interactive
packets video often
are dropped andrequires low [Link],
the throughput decreased. In this
the large packet size is not suitable for interactive video applications. However, the appli-
paper, the test framework adopts the NFVBench tool to generate PVP and PVVP traffic to
cation using small packets tends to send out huge amounts of packets. If the virtual switch
investigate the throughput impacts on different packet sizes.
cannot afford the massive packets, some packets are dropped and the throughput is de-
creased.
• VM In Size
this paper, the test framework adopts the NFVBench tool to generate PVP and
PVVP traffic to investigate the throughput impacts on different packet sizes.
In general, packet processing requires CPU resources. For example, a VM running the
•VNF VM Sizetwo cores to forward the packets. The other jobs also consume CPU resources
uses
In general,
on the same VM. packet processing
If the requires
target VNF onlyCPUhasresources.
two vCPU Forcores,
example,
the aCPU
VM resources
running are not
the VNF uses
sufficient fortwo coresprocessing.
packet to forward When
the packets. The other
the number of jobs
vCPU also consume
cores is tooCPU re-the VM is
small,
sources on the same VM. If the target VNF only has two vCPU cores, the CPU resources
not able to handle all the packets. As a result, the dropped packets deteriorate the network
are not sufficient for packet processing. When the number of vCPU cores is too small, the
throughputs. Therefore, this paper compares the throughputs of 2-core VM and 4-core
VM is not able to handle all the packets. As a result, the dropped packets deteriorate the
VM with OVS-DPDK. This evaluation tool in this experiment is also NFVBench. This
network throughputs. Therefore, this paper compares the throughputs of 2-core VM and
experiment
4-core VM with focuses on theThis
OVS-DPDK. importance
evaluationoftool
resource
in thisallocation.
experiment In is this
also paper, CPU pinning
NFVBench.
and CPU isolation techniques are applied to limit the number of dedicated
This experiment focuses on the importance of resource allocation. In this paper, CPU pin- cores.
ning
• and CallCPU
Rate isolation techniques are applied to limit the number of dedicated cores.
• Call Rate
In the application test, this paper applies the vIMS as a VNF application on the cloud.
TheIntest
thenot
application test, thiscloud
only deploys paper vIMS
applieswith
the vIMS as a VNF
different application
enhanced on the but
features, [Link] adopts
The test not only deploys cloud vIMS with different enhanced features, but also adopts
the SIP stress test to evaluate the performance of the application. Figure 10 depicts the
the SIP stress test to evaluate the performance of the application. Figure 10 depicts the
sequence of a phone call in this experiment. First of all, UE1 sends out the invite request to
sequence of a phone call in this experiment. First of all, UE1 sends out the invite request
initiate a call session. The 100 trying prevents the retransmissions of invite requests. When
to initiate a call session. The 100 trying prevents the retransmissions of invite requests.
the UE2
When receives
the UE2 the the
receives invite request,
invite UE2
request, UE2sends
sendsthe the180
180 ringing backand
ringing back andalerts
alertsthethe user. If
user. If the user answers the phone call, UE2 sends a 200 OK. When UE1 receives the 200 OK from
the user answers the phone call, UE2 sends a 200 OK. When UE1 receives the 200
OK
UE2,fromUE1UE2, UE1 replies
replies to UE2towith UE2an with an ACK.
ACK. AfterAfterthesethese
stepssteps are accomplished,
are accomplished, thethe
call session
call session is established.
is established. When the When the conversation
conversation is finished,
is finished, bothboth
UE1UE1andand UE2
UE2 hangup
hang upthe phone
the
callphone call by sending
by sending the BYEthe BYE message.
message. WhenWhen a UEareceives
UE receives a BYE
a BYE message,
message, theUE
the UEreplies the
replies
200 OK theto200
[Link] the BYE. Afterward,
Afterward, the session
the session is terminated.
is terminated. Throughout
Throughout the the ex-
experiment, the
periment,
successful thecall
successful
is measuredcall istomeasured to indicate
indicate how many how many
packets arepackets are processed
processed without a loss. The
without
growthaof loss.
theThe
callgrowth of the call
rate increases rate
the increasesof
workload thethe
workload of the The
application. application. Thecall rate is
successful
successful call rate is declined when the cloud
declined when the cloud vIMS cannot afford the workload. vIMS cannot afford the workload.

User1’s vIMS User2’s

Equipment system Equipment
Invite
Invite
100 Trying
100 Trying
180 Ringing
180 Ringing
200 OK
200 OK
ACK
ACK
BYE
BYE
200 OK
200 OK

Figure
Figure10.10.
SIPSIP
callcall
flow.
flow.
Appl. Sci. 2022, 12, 10315 13 of 24

4. Experiments and Evaluations

In NFV applications, network performance is a critical concern, and enhanced tech-
nologies are proposed to improve packet processing. These technologies bring different
effects of performance improvement to a cloud system. This section firstly verifies the
functional operations of OpenStack cloud and then evaluates the performance of the clouds
with different network enhancement techniques. Lastly, the vIMS application is deployed to
validate effective performance. The cloud system applying the OVS networking is a generic
case to act as the baseline in the experiment. To further improve the performance, the
cloud system enables CPU pinning and hugepages while applying network-accelerating
technologies such as OVS-DPDK or SR-IOV.

4.1. Testbed Environment

Table 2 summarizes the host configurations of the testbed. The test host is equipped
with Intel i7-3770 and 16 G memory. Both control and compute hosts have two Xeon Silver
CPUs and 160 G memory. In addition, all of these hosts have one Intel XXV710 as the
network interface and each host is interconnected via a 10 Gbps TOR switch. The testing
tools are installed in the test host. The OpenStack cloud system is composed of two nodes
in the testbed. The first node acts as the control host for providing the core OpenStack
services, such as Keystone, Nova API, Nova conductor, Neutron, and Glance. The other
node is a compute host for launching VM resources via Nova Hypervisor.

Table 2. Host configurations of the testbed.

Test Host Control Host Compute Host

CPU Intel i7-3770 Xeon(R) Silver 4210 × 2 Xeon(R) Silver 4210 × 2
Cores 4 10 × 2 10 × 2
Memory 4G×4 16 G × 10 16 G × 10
Host OS Ubuntu 18.04 Server Ubuntu 18.04 Server Ubuntu 18.04 Server
Physical NIC XXV710 XXV710 XXV710

Table 3 summarizes the settings in the performance experiments, such as network

enhancement techniques, scenarios, packet sizes, and enhanced configurations. In per-
formance testing, this paper focuses on four experiments. The first test compares the
network performance of the cloud systems with different network acceleration techniques.
The second experiment shows the performance of different traffic flows. The traffic flows
include PVP and PVVP scenarios. The third experiment evaluates the throughputs with
different packet sizes. The fourth experiment shows the impact of different VM sizes.
Because OVS-DPDK requires additional configurations of lcore and pmd-cpu, the last
experiment investigates the performance impact of the number of lcores and pmd-cpus in
OVS-DPDK tests.

Table 3. Experimental settings in performance tests.

Network Technique Generic OVS/OVS-DPDK/SR-IOV

Scenario PVP/PVVP
Packet Size 64/128/256/512/1024/1280/1518
vCPU (DPDK) 2/4
lcore + pmd-cpu (DPDK) 2 + 2/4 + 4

Table 4 summarizes the experiment to estimate the performance of vIMS with different
network techniques. The vCPU of a VM is set to 1 or 2. The call rate varies from 80 to 140.
In the OVS-DPDK scenario, all the components are deployed with the features of CPU
pinning and hugepages. This experiment also considers the impact of lcore and pmd-cpu.
Appl. Sci. 2022, 12, 10315 14 of 24
Appl. Sci. 2022, 12, 10315 14 of 24

In this experiment, the pmd-cpu and lcore are set to 4. Each NUMA node has 2 lcores and
2Inpmd-cpus.
this experiment, the pmd-cpu and lcore are set to 4. Each NUMA node has 2 lcores and
2 pmd-cpus.
Table 4. Experimental settings in application tests.
Table 4. Experimental settings in application tests.
Enhancement Tech Generic OVS/OVS-DPDK
Enhancement Tech Generic OVS/OVS-DPDK
vCPU 1/2
vCPU 1/2
Call Rate 80/100/120/140
Call Rate 80/100/120/140
lcore + pmd-cpu
lcore + pmd-cpu(DPDK)
(DPDK) 4 +4 +
44

4.2. Results
4.2. Results of
of Functional
Functional Verifications
Verifications
Afterthe
After the deployment
deploymentof ofaa cloud
cloud system,
system, thethe first
first experiment
experimentis is to
to verify
verify the
the operation
operation
of the SUT. All test cases should be passed if no error occurs during the functional tests.
of the SUT. All test cases should be passed if no error occurs during the functional tests. The
The results of the functionality test are summarized in Table 5. The test cases of
results of the functionality test are summarized in Table 5. The test cases of Healthcheck
Healthcheck
and Rally_Sanity andareRally_Sanity are shownAinand
shown in Appendices Appendices
B. The result A ofand B. The result
Healthcheck of
verifies
Healthcheck
all verifies
the operations all the operations
of OpenStack of OpenStack
are correct. are correct.
Rally_Sanity checksRally_Sanity checks
if all the tests meetifthe
all
the tests meetofthe
requirements therequirements
SLA. Because of all
thetheSLA. Becausepass
scenarios all the
the scenarios pass the teststhe
tests of Rally_Sanity, of
Rally_Sanity, the OpenStack services can finish their jobs in time.
OpenStack services can finish their jobs in time.

[Link]
Table Resultof
ofthe
thefunctionality
functionalitytest.
test.

Test
TestSuite
Suite Pass/Total
Pass/Total Result ofofTest
Result TestCases
Cases
Healthcheck
Healthcheck 9/99/9 Appendix
AppendixA A
Rally_Sanity
Rally_Sanity 56/56
56/56 Appendix
AppendixBB

4.3. Results
4.3. Results of
of Performance
Performance Verifications
Verifications
Thissection
This sectionpresents
presentsthethe performance
performance evaluation
evaluation and and analysis.
analysis. Firstly,
Firstly, the perfor-
the performance
mance results of different network techniques are evaluated in the experiments.
results of different network techniques are evaluated in the experiments. Afterward, After-
the
ward, the impacts of each verification factor are discussed in terms of packet
impacts of each verification factor are discussed in terms of packet size, VM size, trafficsize, VM
size, and
flow, traffic flow, and
enhanced enhanced
data plane. data plane.

4.3.1. Different
4.3.1. Different Network
Network Techniques
Techniques
The experiment
The experiment uses
uses aa VM
VM with
with two
two cores
cores and
and different
different improvement
improvement technologies.
technologies.
Thetest
The testscenario
scenarioisisPVP.
PVP.
AsAs a baseline,
a baseline, thethe
VMVM in OVS
in the the OVS scenario
scenario does does not apply
not apply en-
enhance-
hancement
ment techniques,
techniques, while
while the the enable
others others enable CPU pinning
CPU pinning or hugepages.
or hugepages. Experimental
Experimental results
results
in in different
different packet
packet sizes are sizes
shownare
inshown
Figuresin11–13.
Figures
The11–13.
reasonThe reason
behind behind
each each ex-
experiment is
periment is
discussed asdiscussed
follows. as follows.

Figure11.
Figure 11. Throughputs
Throughputsof
ofdifferent
differenttechnologies
technologiesin
inPVP
PVPscenario
scenariowhen
whenpacket
packetsize
sizeisis64.
64.
[Link].
Appl. Sci.2022, 12,10315
2022,12, 10315 15
15 of 24
of 24
Appl. Sci. 2022, 12, 10315 15 of 24

Figure12.
Figure 12. Throughputs
Throughputsof
ofdifferent
differenttechnologies
technologiesin
inPVP
PVPscenario
scenariowhen
whenpacket
packetsize
sizeisis512.
512.
Figure 12. Throughputs of different technologies in PVP scenario when packet size is 512.

Figure 13. Throughputs of different technologies in PVP scenario when packet size is 1518.
Figure13.
Figure 13. Throughputs
Throughputsof
ofdifferent
differenttechnologies
technologiesin
inPVP
PVPscenario
scenariowhen
whenpacket
packetsize
sizeisis1518.
1518.
Because OVS receives packets in kernel space, this method still needs a context switch
BecauseOVS
Because OVSreceives
receivespackets
packetsin inkernel
kernelspace,
space,thisthismethod
methodstill stillneeds
needsaa context
context switch
switch
if a packet does not match any rules. The kernel-user space interrupt degrades the network
ifif aa packet
packet does
does not
not match
match any any rules.
rules. The
The kernel-user
kernel-user spacespace interrupt
interrupt degrades
degrades the the network
network
performance. As shown in these three figures, the throughputs with OVS are the lowest.
performance. As
performance. As shown
shown in in these
these three
three figures,
figures, thethe throughputs
throughputs with with OVSOVS areare the
the lowest.
lowest.
When the packet size is 64, the throughput is only 0.16 Gbps. The throughput varies with
When the packet size is 64, the throughput is only 0.16 Gbps. The throughput varies with
When the packet size is 64, the throughput is only 0.16 Gbps. The throughput varies with
the increment in packet size. The maximum throughput is 2.8 Gbps when the packet size
the increment
the incrementininpacket
packetsize. [Link]
maximumthroughput
throughput is is
2.82.8 Gbps
Gbps whenwhen thethe packet
packet size
size is
is 1518. These results show a low performance without any enhancement.
is 1518.
1518. These
These results
results showshow a low
a low performance
performance without
without anyany enhancement.
enhancement.
The difference between OVS and OVS-DPDK is to exploit vhost-net or vhost-user for
The difference
The differencebetweenbetweenOVS OVSand andOVS-DPDK
OVS-DPDKisis to to exploit
exploit vhost-net
vhost-net or or vhost-user
vhost-user for for
NIC emulation. OVS OVS uses vhost-net
vhost-net to enable enable the virtual
virtual switch to to process the the packets in in
NIC emulation.
NIC emulation. OVS uses uses vhost-net to to enable the the virtual switch
switch to process
process the packets
packets in
kernel space. This method introduces the overhead of context switches and deteriorates
kernel space.
kernel space. This
This method
method introduces
introduces the the overhead
overhead of of context
context switches
switches and and deteriorates
deteriorates
the performance.
performance. On the contrary, OVS-DPDK uses vhost-user to enable the guest to com-
the performance. On the contrary, OVS-DPDK uses vhost-user to enable the guest
the On the contrary, OVS-DPDK uses vhost-user to enable to com-
the guest to
municate with the virtual switch. Therefore, OVS-DPDK mitigates the performance deg-
municate with the virtual switch. Therefore, OVS-DPDK mitigates the performance deg-
communicate with the virtual switch. Therefore, OVS-DPDK mitigates the performance
radation. OnOn the other hand, thetheVMVMwith withOVSOVSis not
notimproved
improved with with CPU pinning andand
radation. On thethe
degradation. other
other hand,
hand, the VM with OVS isisnot improved with CPU pinning and
CPU pinning
hugepages. CPU pinning helps the VM possess dedicated cores. These cores avoid the VM
hugepages. CPU
hugepages. CPU pinning
pinninghelps helpsthe the VM
VM possess
possess dedicated
[Link]
Thesecores
coresavoid
avoidthethe VM
VM
sharing
sharing CPU
CPU resources
resources with
with other
other processes
processes in
in the
the host.
host. Therefore,
Therefore, the
the overhead
overhead of
of context
context
sharing CPU resources with other processes in the host. Therefore, the overhead of context
switches is
switches is relieved. Hugepages
Hugepages areused used tospeedspeed upmemorymemory access.
switches is relieved.
relieved. Hugepages are are usedto to speedup up memoryaccess. access.
Considering
Considering the enhancement of the virtual switch, the throughputs
throughputs of using using OVS-
Considering the the enhancement
enhancement of of the
the virtual
virtual switch,
switch, the the throughputs of of using OVS-
OVS-
DPDK are
DPDK are better thanthan that of of OVS. The The throughput of of OVS-DPDK is is 1.4 Gbps
Gbps when the the
DPDK are better better than that that of OVS.
OVS. The throughput
throughput of OVS-DPDKOVS-DPDK is 1.4 1.4 Gbps when when the
packetsize
packet sizeisis64.
64. This
This throughput
throughput of of OVS-DPDK
OVS-DPDK is is 8.2
8.2 times
times as as high
high as as the
the throughput
throughput of of
packet size is 64. This throughput of OVS-DPDK is 8.2 times as high as the throughput of
[Link]
OVS. The maximum
maximum throughput
throughputwith withOVS-DPDK
OVS-DPDKisis9.9 9.9 Gbps
Gbps when
when thethe packet
packet size
size is
is 1518.
1518.
OVS. The maximum throughput with OVS-DPDK is 9.9 Gbps when the packet size is 1518.
This value
This value isis 3.5
3.5 times
times as as high
high as as the
the throughput
throughput with with OVS.
OVS. Therefore,
Therefore, the the throughput
throughput of of
This value is 3.5 times as high as the throughput with OVS. Therefore, the throughput of
OVS-DPDK ranges from 3.5 to 8.2 times as high as the
OVS-DPDK ranges from 3.5 to 8.2 times as high as the OVS. However, the throughputs OVS. However, the throughputs
OVS-DPDK ranges from 3.5 to 8.2 times as high as the OVS. However, the throughputs
withOVS-DPDK
with OVS-DPDKare arelower
lowerthan thanaahardware-based
hardware-basedsolution. solution.
with OVS-DPDK are lower than a hardware-based solution.
Appl. Sci. 2022, 12, 10315 16 of 24

Appl. Sci. 2022, 12, 10315 16 of 24

SR-IOV is the hardware-assisted solution. This technique improves network perfor-
mance significantly and allows a VM to access the physical NIC directly via the virtual
function.
SR-IOV Theis throughput of SR-IOVsolution.
the hardware-assisted approaches This9.5 Gbps when
technique the packet
improves network sizeperfor-
is 64.
When the packet size is higher than 64, the throughputs are
mance significantly and allows a VM to access the physical NIC directly via the virtual 9.9 Gbps. As shown in the
results, the
function. Theperformance
throughput of SR-IOV approaches
of SR-IOV is near the line rate. With
9.5 Gbps whenSR-IOV,
the packetthesize
system
is [Link]
When
dedicated
the packet size virtual functions
is higher than to64,thethe VMs. These virtual
throughputs functions
are 9.9 Gbps. help the
As shown VMresults,
in the access the
the
physical NICof
performance directly.
SR-IOVTherefore,
is near thethe impact
line rate. of VMM
With is relieved.
SR-IOV, the system assigns dedicated
virtual functions to the VMs. These virtual functions help the VM access the physical NIC
4.3.2. Different
directly. Therefore, Packet
the Sizes
impact of VMM is relieved.
As shown in Figures 11 and 13, the throughputs of OVS are 0.16 Gbps and 2.8 Gbps
4.3.2.
whenDifferent
the packet Packet
sizes Sizes
are 64 and 1518. In the OVS network, the throughput in packet size
1518 is 16.4 times as high as
As shown in Figures 11 the
andthroughput in packet of
13, the throughputs size
OVS64. are
In addition,
0.16 Gbpsthe and throughput
2.8 Gbps
of OVS-DPDK in packet size 64 is only 1.4 Gbps. When the packet size is 1518, the through-
when the packet sizes are 64 and 1518. In the OVS network, the throughput in packet size
put of OVS-DPDK is 9.93 Gbps. In the OVS-DPDK network, the throughput in packet size
1518 is 16.4 times as high as the throughput in packet size 64. In addition, the throughput
1518
of is 7 times as
OVS-DPDK inhigh as the
packet throughput
size 64 is onlyin1.4 packet
[Link]
64. Regarding
the packetSR-IOV,
size the through-
is 1518, the
puts are 9.5 of
throughput Gbps, 9.96 Gbps,
OVS-DPDK and Gbps.
is 9.93 9.96 Gbps when
In the the packetnetwork,
OVS-DPDK sizes arethe
64,throughput
512, and 1518. in
packet size 1518 is 7 times as high as the throughput in packet size
Therefore, the impact of packet size is less significant when packet processing capacity is 64. Regarding SR-IOV,
the throughputs are 9.5 Gbps, 9.96 Gbps, and 9.96 Gbps when the packet sizes are 64, 512,
higher.
and 1518.
BecauseTherefore,
the VMthe impactthe
accesses of packet
physicalsize is less
NIC significant
directly, when packetofprocessing
the performance SR-IOV is
capacity is higher.
near the line rate in all cases of different packet sizes. On the contrary, the throughputs of
OVSBecause the VM accesses
and OVS-DPDK the physical
dramatically decreaseNIC directly,
when the performance
the packet of That
size is small. SR-IOV is near
is because
the line rate in all cases of different packet sizes. On the contrary,
the sender generates more packets when the packet size is small. A large workload the throughputs of OVS of
and OVS-DPDK
packet processing dramatically
is not affordabledecrease when switches
for virtual the packet size is the
because small. That is
capacity ofbecause the
each virtual
sender generates more packets when the packet size is small. A large workload of packet
switch for processing packets is limited. The more packets per second the sender gener-
processing is not affordable for virtual switches because the capacity of each virtual switch
ates, the more packets are dropped. Therefore, the throughput of small packet size is lower
for processing packets is limited. The more packets per second the sender generates, the
than that of a large one.
more packets are dropped. Therefore, the throughput of small packet size is lower than
that of a large one.
4.3.3. Different VM Sizes
4.3.3. Considering
Different VMthat the packet processing consumes CPU resources, the next experi-
Sizes
mentConsidering
tests the throughputs of VMs
that the packet with different
processing numbers
consumes of cores. Because
CPU resources, the nextthe through-
experiment
put of the VM with SR-IOV is too high while the throughput of the VM with
tests the throughputs of VMs with different numbers of cores. Because the throughput OVS is too
of
low, this experiment only presents the performance of applying OVS-DPDK, as
the VM with SR-IOV is too high while the throughput of the VM with OVS is too low, this shown in
Figure 14. Results show that the throughput grows in all packet sizes when
experiment only presents the performance of applying OVS-DPDK, as shown in Figure 14. the VM size
increases.
Results show that the throughput grows in all packet sizes when the VM size increases.

Figure14.
Figure 14. Throughputs
Throughputsof
ofVMs
VMswith
withdifferent
differentcores
coresin
inthe
thePVP
PVPscenario.
scenario.

The throughputs
The throughputs ofof aa 2-core
2-core VM
VM are
are 1.4
1.4 Gbps,
Gbps, 5.9
5.9 Gbps,
Gbps, and
and 9.93
9.93 Gbps
Gbps when
when the
the
packet sizes are 64, 512, and 1518, respectively. Regarding a 4-core VM, the throughputs
packet sizes are 64, 512, and 1518, respectively. Regarding a 4-core VM, the throughputs
are 1.42
are 1.42 Gbps,
Gbps, 6.7
6.7 Gbps,
Gbps, and
and 9.96
9.96 Gbps
Gbps when
when the
the packet
packet sizes
sizes are
are 64,
64, 512,
512, and
and 1518,
1518,
respectively. The improvement ranges from 0.2% to 16% in this experiment. The reason
is that if the VM only has two CPU cores, the preemption occurs frequently while the
Appl.
Appl. Sci. 2022, 12,
Sci. 2022, 12, 10315
10315 17
17 of
of 22

respectively.
respectively. The
The improvement
improvement ranges
ranges from
from 0.2%
0.2% to
to 16%
16% in
in this
this experiment.
experiment. The
The reason
reason ii
Appl. Sci. 2022, 12, 10315 that if the VM only has two CPU cores, the preemption occurs frequently while
that if the VM only has two CPU cores, the preemption occurs frequently while the CPU
17 of 24the CPU

core
core isis processing.
processing. Therefore,
Therefore, the
the throughput
throughput isis decreased
decreased ifif the
the CPU
CPU resources
resources areare lim
lim
ited.
ited. Increasing the number of CPU cores relieves the overhead of preemption so that the
Increasing the number of CPU cores relieves the overhead of preemption so that th
throughput
throughput is
CPU
is enhanced.
core is processing. Therefore, the throughput is decreased if the CPU resources are
enhanced.
limited. Increasing the number of CPU cores relieves the overhead of preemption so that
the throughput is enhanced.
4.3.4.
4.3.4. Different
Different Traffic
Traffic Flows
Flows
4.3.4. The
The experiment
experiment
Different deploys
deploys two
Traffic Flows two VMs
VMs withwith different
different improvement
improvement technologies
technologies to to eval
eval
uate the throughput, as shown in Figures 15–17. The SR-IOV
uate the throughput, as shown in Figures 15–17. The SR-IOV brings the best performanc
The experiment deploys two VMs with different improvement brings the
technologies best
to performance
evaluate
among
among
the all
all technologies.
throughput, technologies. The
The throughputs
as shown in Figures throughputs
15–17. The SR-IOVof
of OVS
OVS and
andtheOVS-DPDK
brings OVS-DPDK
best performanceincrease
increase
among when
when the th
packet
packet is
all
is larger.
larger. However,
technologies.
However, the
The throughputs
the packet
packet path
of OVS
path in
and
in this
this experiment
OVS-DPDK
experiment is
increase
is longer
when
longer than
the
than that
packet
that of
is
of the
th
larger.
PVP However,Ifthe packet path in this experiment is longer than thatcomponents
of the PVP scenario.
PVP scenario. If the traffic flow is longer, the workload of the components in this path ii
scenario. the traffic flow is longer, the workload of the
If the traffic flow is longer, the workload of the components in this path is too heavy to
in this path
too
too heavy
heavy to
to decline
decline thethe throughputs.
throughputs. When
When the packet sizes are 64,
64, 512,
512, and
andof1518,
1518, the
decline the throughputs. When the packet sizes arethe packet
64, 512, andsizes
1518, are
the throughputs th
throughputs
throughputs
OVS of
are 0.147 Gbps, OVS
of OVS are 0.147
are 0.147
1 Gbps, and 2.58Gbps,
Gbps, 1 Gbps,
1 Gbps,
Gbps, and 2.58
and 2.58
respectively. Gbps,
Gbps,
The respectively.
respectively.
throughputs The
are 0.8 The throughput
Gbps,throughput
3.5
are
are 0.8
Gbps, 0.8 Gbps,
andGbps, 3.5
3.5 Gbps,
6.2 Gbps Gbps, and
and 6.2
of OVS-DPDK Gbps
6.2 when
Gbps theof
of OVS-DPDK
OVS-DPDK
packet sizes are when
when the packet
the and
64, 512, packet sizes
sizes
1518, are
are 64,
64, 512,
respectively. 512, and
and
1518, respectively.
1518, respectively.
Therefore, Therefore,
in the PVVPTherefore,
scenario, the in the PVVP
in improvement scenario,
the PVVP scenario,
brought by the improvement
theOVS-DPDK
improvement ranges brought
brought by
from 2.4by OVS OVS
DPDK
DPDK
to 5.5 timesranges
ranges from
the OVS. 2.4
2.4 to
to 5.5
fromRegarding times
5.5the
times the OVS.
the the
SR-IOV, OVS. Regarding
Regarding
throughputs arethe
the
aboveSR-IOV,
8 Gbpsthe
SR-IOV, the throughputs
throughputs
in all packet are
ar
above 8 Gbps in all packet sizes. However, the throughputs
above 8 Gbps in all packet sizes. However, the throughputs of SR-IOV are still the best.
sizes. However, the throughputs of SR-IOV are still the best. of SR-IOV are still the best.

Figure
Figure 15.
Figure15. Throughput
[Link]
Throughput of
of different
different
of different technologies
technologies
technologies in
in PVVP
in PVVP PVVP scenario
scenario
scenario when packet
whensize
when packet is [Link]
packet size is
is 64.
64.

Figure
Figure 16.
Figure16. Throughput
[Link]
Throughput of
of different
different
of different technologies
technologies
technologies in
in PVVP
in PVVP PVVP scenario
scenario
scenario when packet
whensize
when packet packet size
size is
is 512. is 512.
512.
Appl. Sci. 2022, 12, 10315 18 of 24
Appl. Sci. 2022, 12, 10315 18 of 24

Figure 17.
Figure [Link]
Throughputof of
different technologies
different in PVVP
technologies scenario
in PVVP when packet
scenario when size is 1518.
packet size is 1518.

Regarding
Regardingpacket
packetloss,
loss,thethe
lossloss
raterate
of OVS or OVS-DPDK
of OVS or OVS-DPDKdecreases when increasing
decreases when increasing
the packet size. Meanwhile, SR-IOV has the lowest drop
the packet size. Meanwhile, SR-IOV has the lowest drop rate amongrate among the other
thenetwork
other network
technologies. The
technologies. Thereason is that
reason SR-IOV
is that mitigates
SR-IOV the interference
mitigates of VMM.
the interference of Therefore, VM
VMM. Therefore, VM
can access the physical NIC directly. On the other hand, the OVS has the highest drop
can access the physical NIC directly. On the other hand, the OVS has the highest drop rate.
rate. That is because OVS introduces a lot of context switches between the kernel-user
That is because OVS introduces a lot of context switches between the kernel-user space.
space. OVS-DPDK mitigates the context switches between the kernel-user space and polls
OVS-DPDK mitigates the context switches between the kernel-user space and polls the
the queues of NICs with dedicated cores. Therefore, the drop rate of OVS-DPDK is lower
queues
than that of [Link] dedicated cores. Therefore, the drop rate of OVS-DPDK is lower than
of NICs
that of OVS.
4.3.5. Enhanced Data Plane
4.3.5. Enhanced Data Plane
Although OVS-DPDK eliminates the context switch in the kernel-user space, the per-
formance is still OVS-DPDK
Although affected by theeliminates
configurations
the of OVS-DPDK.
context switch Applying OVS-DPDK re-
in the kernel-user space, the
quires CPU resources
performance for lcore by
is still affected andthe
pmd-cpu. The lcore of
configurations is used by non-datapath
OVS-DPDK. Applyingthreads
OVS-DPDK
to removeCPU
requires the idle and wrong
resources flowsand
for lcore while the pmd-cpu
pmd-cpu. Theneeds
lcore to
is poll
usedthe byqueues of phys- threads
non-datapath
ical NICs and virtual NICs. If the number of pmd-cpu is too small, the
to remove the idle and wrong flows while the pmd-cpu needs to poll the queues of physical lack of pmd-cpu
will decrease
NICs the throughput.
and virtual NICs. If the Therefore,
numberthisof experiment
pmd-cpu isis too to verify
small,thethe
performance im-
lack of pmd-cpu will
pact of lcore and pmd-cpu to enhance the data plane.
decrease the throughput. Therefore, this experiment is to verify the performance impact of
lcoreAs andshown in Figure 18, the throughput increases when the number of lcore and pmd-
pmd-cpu to enhance the data plane.
cpu grows. When the packet sizes are 64, 512, and 1518, the throughputs of 2 lcores and 2
As shown in Figure 18, the throughput increases when the number of lcore and
pmd-cpus are 1.4 Gbps, 5.9 Gbps, and 9.9 Gbps. Under the same packet sizes, the through-
pmd-cpu grows. When the packet sizes are 64, 512, and 1518, the throughputs of 2 lcores
puts of 4 lcores and 4 pmd-cpus are 1.9 Gbps, 9 Gbps, and 9.9 Gbps. Therefore, the
and 2 pmd-cpus
throughputs of 4 lcores are 1.4 Gbps,
and 5.9 Gbps,
4 pmd-cpus areand 9.9 Gbps.
improved by up Under
to 50%.theFurthermore,
same packet thesizes, the
throughputs of 4 lcores and 4 pmd-cpus are 1.9 Gbps, 9 Gbps,
enhancement of increasing lcore and pmd-cpu is even more obvious than increasing the and 9.9 Gbps. Therefore,
VMthroughputs
the size. The reason of is that each
4 lcores and VM consumes the
4 pmd-cpus arevirtual NIC resource
improved by up toand increases
50%. the
Furthermore, the
workload of pmd-cpu.
enhancement When lcore
of increasing the number of pmd-cpus
and pmd-cpu is small,
is even morethe pmd-cpu
obvious thanmay not
increasing the
process
VM [Link] The
manyreason
packets. isThus, the packets
that each are dropped
VM consumes if pmd-cpus
the virtual NIC are resource
not affordable.
and increases
Appl. Sci. 2022, 12, 10315 19 of 24 may not
the workload of pmd-cpu. When the number of pmd-cpus is small, the pmd-cpu
process so many packets. Thus, the packets are dropped if pmd-cpus are not affordable.

Figure 18.
Figure [Link]
Throughputsof of
OVS-DPDK when
OVS-DPDK lcorelcore
when and pmd-cpu increase.
and pmd-cpu increase.

4.4. Results of Application Verifications

After testing the network performance of a SUT, this experiment deploys vIMS as a
cloud application to show the improvement of a virtual switch. Subsequently, the SIP
stress is installed to benchmark the successful call rate (SCR) of the vIMS with different
Appl. Sci. 2022, 12, 10315 19 of 24

Figure 18. Throughputs of OVS-DPDK when lcore and pmd-cpu increase.

4.4. Results
4.4. Results of
of Application
Application Verifications
Verifications
After testing
After testing the
the network
network performance
performance of of aa SUT,
SUT, this
this experiment
experiment deploys
deploys vIMSvIMS asas aa
cloud application
cloud applicationtotoshow show thethe improvement
improvement of aofvirtual
a virtual switch.
switch. Subsequently,
Subsequently, the SIPthe SIP
stress
stress
is is installed
installed to benchmark
to benchmark the successful
the successful call rate call
(SCR)rate
of (SCR) of the
the vIMS withvIMS with call
different different
rates.
call
A [Link]
higher A higher successful call rate
call rate means means
better better performance.
performance.
In this
In this experiment,
experiment, eacheach component
component of of vIMS
vIMS has has only
only 11 vCPU.
vCPU. As shown in
As shown in Figure
Figure 19,
19,
the successful call rate declines along with the increment in the call rate. When the call rate is
the successful call rate declines along with the increment in the call rate. When the call rate
lower
is lower than 100,
than thethe
100, workload
workload is affordable for for
is affordable both OVSOVS
both andandOVS-DPDK.
OVS-DPDK. Therefore, the suc-
Therefore, the
cessful callcall
successful rates are are
rates higher than
higher 95%.
than In addition,
95%. In addition,the the
successful call call
successful raterate
is significantly im-
is significantly
proved from
improved 72%72%
from to 86% by OVS-DPDK
to 86% by OVS-DPDK when thewhen callthe
ratecall
is 120.
rateThat is because
is 120. That the compo-
is because
nents
the of vIMS with
components ofOVS-DPDK
vIMS withare enhanced by
OVS-DPDK areDPDK,
enhancedCPUby pinning,
DPDK, and hugepages.
CPU pinning,These
and
technologies improve both networking and computing. Therefore, the successful call rate with
hugepages. These technologies improve both networking and computing. Therefore, the
OVS-DPDK performs better than OVS.
successful call rate with OVS-DPDK performs better than OVS.

Figure 19.
Figure 19. Successful
Successful call
call rate
rate of
of vIMS
vIMS with
with OVS
OVS and
and OVS-DPDK.
OVS-DPDK.

When the
When the call
call rate
rate is
is higher
higher thanthan 120,
120, the
the missing
missing call
call rate
rate of
of OVS
OVS ranges
ranges fromfrom 27%
27%
to 48%.
to 48%. The
The reason
reason is
is that
that interactive
interactive videovideo applications
applications tendtend toto use
use smaller
smaller packets
packets [47].
[47].
When many
When many small smallpackets
packetsare aretransmitted
transmitted to to
thethe
application,
application, the the
workload
workload of packet pro-
of packet
cessing is increased.
processing is [Link],
Therefore,the missing call rate
the missing callalso
rateincreases when the
also increases when virtual switch
the virtual
cannot afford the loading. While applying OVS-DPDK, the missing call rate is reduced
switch cannot afford the loading. While applying OVS-DPDK, the missing call rate byis
up to 14%. The next experiment further improves the performance of vIMS with OVS-
reduced by up to 14%. The next experiment further improves the performance of vIMS
DPDK
with in differentinVM
OVS-DPDK sizes. VM
different [Link] illustrate that
Previous experiments the VM
illustrate size
that thebrings
VM sizeim-
pacts on
brings the performance
impacts of VNF. Therefore,
on the performance this experiment
of VNF. Therefore, increases
this experiment the number
increases of CPU
the number
Appl. Sci. 2022, 12, 10315 cores
of CPUfor eachfor
cores component
each component in vIMS. As shown
in vIMS. in Figure
As shown 20, the
in Figure 20,vIMS
the vIMSperformance
performance20is of
im-
is
24
proved if the VM size is increased to 2 CPU cores. The successful call rate increases from
improved if the VM size is increased to 2 CPU cores. The successful call rate increases from
61%
61% to
to 79%
79% whenwhen the
the call
call rate
rate isis 140,
140, which
which means
means that
that the
the enhancement
enhancement is is 18%.
18%.

Figure20.
Figure 20. Successful
Successfulcall
callrate
rateof
ofvIMS
vIMSwith
withdifferent
differentVM
VMsizes
sizesusing
usingOVS-DPDK.
OVS-DPDK.

5. Conclusions
As ETSI proposes the NFV architecture using virtualization technologies, the perfor-
mance degradation of virtualization is a challenge to meet the SLA of the telecom indus-
try. This paper attempts to conduct the performance effects of EPA techniques for the NFV
cloud. The EPA techniques adopted in our testbed include CPU pinning, CPU isolation,
Appl. Sci. 2022, 12, 10315 20 of 24

5. Conclusions
As ETSI proposes the NFV architecture using virtualization technologies, the perfor-
mance degradation of virtualization is a challenge to meet the SLA of the telecom industry.
This paper attempts to conduct the performance effects of EPA techniques for the NFV
cloud. The EPA techniques adopted in our testbed include CPU pinning, CPU isolation,
hugepages, OVS-DPDK, and SR-IOV. This paper further proposes a test framework for
the comprehensive verification and evaluation of the cloud system under test in terms of
functionality, performance, and application. The major advantages of this framework are
threefold. First, this framework is based on open-source projects to eliminate the expendi-
ture on system verification and evaluation. Second, the framework is easy to implement
and expandable. The testing procedures are triggered on the test host with the test contain-
ers for different benchmarking tools. Third, for the performance testing, the verification not
only compares the impacts of different network techniques, but also considers the length of
packet path, different VM sizes, and performance tuning of packet processing, e.g., lcore
and pmd-cpu.
Results show that all the cases in functional tests are passed. The test cases in
Rally_Sanity also fulfill the requirement of SLA. Throughputs of OVS-DPDK are up to 8.2
times as high as that of generic OVS networking. Regarding the performance at the appli-
cation level, the SCR of vIMS is improved by up to 14% with OVS-DPDK. Our experiments
further obverted that the throughput is limited when the packet size is small. This deteri-
oration happens when the virtual switch cannot process a huge number of packets. The
throughput declines when the packets are dropped. Regarding the end-to-end scenarios,
the length of a traffic flow in PVP or PVVP has different impacts on the throughput with
different networking techniques. The results and findings of this work are valuable for fur-
ther reference in workload scheduling, resource placement, or load-balancing mechanisms.
To extend this work, deploying a feature cloud and verifying the SUT automatically are
interesting works. Further research will be to model the specific application performance
based on the proposed testing method and the enhanced technologies, which are also left
for future work.

Author Contributions: Conceptualization, W.-C.C.; Funding acquisition, W.-C.C.; Methodology,

W.-C.C. and Y.-H.W.; Software, Y.-H.W.; Supervision, W.-C.C.; Validation, Y.-H.W.; Writing—review
& editing, W.-C.C. and Y.-H.W. All authors have read and agreed to the published version of the
manuscript.
Funding: This work was supported by the Ministry of Science and Technology of Taiwan, under
Grants MOST108-2218-E-033-008-MY3 and MOST111-2221-E-033-033.
Conflicts of Interest: The authors declare no conflict of interest.

Appendix A

Table A1. Result of Healthcheck.

Test Cases Project Tier Result

connection_check functest healthcheck PASS
tenantnetwork1 functest healthcheck PASS
tenantnetwork2 functest healthcheck PASS
vmready1 functest healthcheck PASS
vmready2 functest healthcheck PASS
singlevm1 functest healthcheck PASS
singlevm2 functest healthcheck PASS
vping_ssh functest healthcheck PASS
vping_userdata functest healthcheck PASS
Appl. Sci. 2022, 12, 10315 21 of 24

Appendix B

Table A2. Result of Rally_Sanity.

Scenario Errors Success (SLA)

[Link] 0 4
Authenticate.validate_cinder 0 4
Authenticate.validate_glance 0 4
Authenticate.validate_heat 0 4
Authenticate.validate_neutron 0 4
Authenticate.validate_nova 0 4
CinderQos.create_and_list_qos 0 4
CinderQos.create_and_set_qos 0 4
CinderVolumes.create_and_delete_snapshot 0 4
CinderVolumes.create_and_delete_volume 0 4
CinderVolumes.create_and_delete_volume-2 0 4
CinderVolumes.create_and_delete_volume-3 0 4
CinderVolumes.create_and_extend_volume 0 4
CinderVolumes.create_from_volume_and_delete_volume 0 4
CinderVolumeTypes.create_and_list_volume_types 0 4
CinderVolumeTypes.create_volume_type_and_encryption_type 0 4
GlanceImages.create_and_delete_image 0 4
GlanceImages.create_and_list_image 0 4
GlanceImages.create_image_and_boot_instances 0 4
GlanceImages.list_images 0 4
HeatStacks.create_check_delete_stack 0 4
HeatStacks.create_suspend_resume_delete_stack 0 4
HeatStacks.create_update_delete_stack 0 4
HeatStacks.list_stacks_and_resources 0 4
KeystoneBasic.add_and_remove_user_role 0 4
KeystoneBasic.create_add_and_list_user_roles 0 4
KeystoneBasic.create_and_delete_role 0 4
KeystoneBasic.create_and_delete_service 0 4
KeystoneBasic.create_and_list_tenants 0 4
KeystoneBasic.create_and_list_users 0 4
KeystoneBasic.create_tenant 0 4
KeystoneBasic.create_tenant_with_users 0 4
KeystoneBasic.create_update_and_delete_tenant 0 4
KeystoneBasic.create_user 0 4
KeystoneBasic.get_entities 0 4
NeutronNetworks.create_and_delete_networks 0 4
NeutronNetworks.create_and_delete_ports 0 4
NeutronNetworks.create_and_delete_routers 0 4
NeutronNetworks.create_and_delete_subnets 0 4
NeutronNetworks.create_and_list_networks 0 4
NeutronNetworks.create_and_list_ports 0 4
NeutronNetworks.create_and_list_routers 0 4
Appl. Sci. 2022, 12, 10315 22 of 24

Table A2. Cont.

Scenario Errors Success (SLA)

NeutronNetworks.create_and_list_subnets 0 4
NeutronNetworks.set_and_clear_router_gateway 0 4
NeutronSecurityGroup.create_and_delete_security_group_rule 0 4
NeutronSecurityGroup.create_and_delete_security_groups 0 4
NovaKeypair.boot_and_delete_server_with_keypair 0 4
NovaServerGroups.create_and_delete_server_group 0 4
NovaServers.boot_server_and_list_interfaces 0 4
NovaServers.boot_server_associate_and_dissociate_floating_ip 0 4
NovaServers.boot_server_from_volume_and_delete 0 4
NovaServers.pause_and_unpause_server 0 4
Quotas.cinder_update 0 4
Quotas.cinder_update_and_delete 0 4
Quotas.neutron_update 0 4
Quotas.nova_update 0 4

References
1. Network Functions Virtualisation (NFV). Available online: [Link]
virtualisation (accessed on 17 September 2022).
2. Hwang, J.; Ramakrishnan, K.K.; Wood, T. NetVM: High performance and flexible networking using virtualization on commodity
platforms. IEEE Trans. Netw. Serv. Manag. 2015, 12, 34–47. [CrossRef]
3. Ferrari, C.; Kovács, B.; Tóth, M.; Horváth, Z.; Reale, A. Edge computing for communication service providers: A review
on the architecture, ownership and governing models. In Proceedings of the 2021 International Conference on Software,
Telecommunications and Computer Networks (SoftCOM), Split, Hvar, Croatia, 23–25 September 2021; pp. 1–6.
4. Google Cloud, Nokia Partner to Accelerate Cloud-Native 5G Readiness for Communication Service Providers. Available
online: [Link]
5g-readiness-for-communication-service-providers/ (accessed on 17 September 2022).
5. ETSI GS NFV; Network Functions Virtualisation (NFV): Architectural Framework. European Telecommunications Standards
Institute: Sophia-Antipolis, France. Available online: [Link]
[Link] (accessed on 10 October 2022).
6. Bourguiba, M.; Haddadou, K.; Korbi, I.E.; Pujolle, G. Improving network I/O virtualization for cloud computing. IEEE Trans.
Parallel Distrib. Syst. 2014, 25, 673–681. [CrossRef]
7. Aziza, H.; Krichen, S. A hybrid genetic algorithm for scientific workflow scheduling in cloud environment. Neural Comput. Appl.
2020, 32, 15263–15278. [CrossRef]
8. Mohammadzadeh, A.; Masdari, M.; Gharehchopogh, F.S.; Jafarian, A. Improved chaotic binary grey wolf optimization algorithm
for workflow scheduling in green cloud computing. Evol. Intell. 2021, 14, 1997–2025. [CrossRef]
9. Mohammadzadeh, A.; Masdari, M.; Gharehchopogh, F.S.; Jafarian, A. A hybrid multi-objective metaheuristic optimization
algorithm for scientific workflow scheduling. Clust. Comput. 2021, 24, 1479–1503. [CrossRef]
10. Tolia, N.; Wang, Z.; Marwah, M.; Bash, C.; Ranganathan, P.; Zhu, X. Delivering energy proportionality with non energy-
proportional systems-optimizing the ensemble. In Proceedings of the 2008 Conference on Power Aware Computing and Systems,
San Diego, CA, USA, 7 December 2008; p. 2.
11. Fu, X.; Zhou, C. Virtual machine selection and placement for dynamic consolidation in cloud computing environment. Front.
Comput. Sci. 2015, 9, 322–330. [CrossRef]
12. Gharehpasha, S.; Masdari, M.; Jafarian, A. Power efficient virtual machine placement in cloud data centers with a discrete and
chaotic hybrid optimization algorithm. Clust. Comput. 2021, 24, 1293–1315. [CrossRef]
13. Gharehpasha, S.; Masdari, M.; Jafarian, A. Virtual machine placement in cloud data centers using a hybrid multi-verse optimiza-
tion algorithm. Artif. Intell. Rev. 2021, 54, 2221–2257. [CrossRef]
14. Gharehpasha, S.; Masdari, M.; Jafarian, A. The placement of virtual machines under optimal conditions in cloud datacenter. Inf.
Technol. Control. 2019, 48, 545–556. [CrossRef]
15. Carpio, F.; Dhahri, S.; Jukan, A. VNF placement with replication for load balancing in NFV networks. In Proceedings of the 2017
IEEE International Conference on Communications (ICC), Paris, France, 21–25 May 2017; pp. 1–6.
16. You, C.; Li, L.M. Efficient load balancing for the VNF deployment with placement constraints. In Proceedings of the ICC
2019—2019 IEEE International Conference on Communications (ICC), Shanghai, China, 20–24 May 2019; pp. 1–6.
Appl. Sci. 2022, 12, 10315 23 of 24

17. Wang, T.; Xu, H.; Liu, F. Multi-resource load balancing for virtual network functions. In Proceedings of the 2017 IEEE 37th
International Conference on Distributed Computing Systems (ICDCS), Atlanta, GA, USA, 5–8 June 2017; pp. 1322–1332.
18. Ghorab, A.H.; Kusedghi, A.; Nourian, M.A.; Akbari, A. Joint VNF load balancing and service auto-scaling in NFV with multimedia
case study. In Proceedings of the 2020 25th International Computer Conference, Computer Society of Iran (CSICC), Tehran, Iran,
1–2 January 2020; pp. 1–7.
19. Zamani, A.; Bakhshi, B.; Sharifian, S. An efficient load balancing approach for service function chain mapping. Comput. Electr.
Eng. 2021, 90, 106890. [CrossRef]
20. OpenStack Enhanced Platform Awareness. Available online: [Link]
[Link] (accessed on 17 September 2022).
21. Tsai, M.H.; Liang, H.T.; Wang, Y.H.; Chung, W.C. Enhanced OpenStack cloud for network function virtualization. In Proceedings
of the 2020 International Computer Symposium (ICS), Tainan, Taiwan, 17–19 December 2020; pp. 185–190.
22. Kourtis, M.A.; Xilouris, G.; Riccobene, V.; McGrath, M.J.; Petralia, G.; Koumaras, H.; Gardikis, G.; Liberal, F. Enhancing VNF
performance by exploiting SR-IOV and DPDK packet processing acceleration. In Proceedings of the 2015 IEEE Conference on
Network Function Virtualization and Software Defined Network (NFV-SDN), San Francisco, CA, USA, 18–21 November 2015;
pp. 74–78.
23. Bonafiglia, R.; Cerrato, I.; Ciaccia, F.; Nemirovsky, M.; Risso, F. Assessing the performance of virtualization technologies for NFV:
A preliminary benchmarking. In Proceedings of the 2015 Fourth European Workshop on Software Defined Networks, Bilbao,
Spain, 30 September–2 October 2015; pp. 67–72.
24. Gallenmuller, S.; Emmerich, P.; Wohlfart, F.; Raumer, D.; Carle, G. Comparison of frameworks for high-performance packet IO.
In Proceedings of the 2015 ACM/IEEE Symposium on Architectures for Networking and Communications Systems (ANCS),
Oakland, CA, USA, 7–8 May 2015; pp. 29–38.
25. Kawashima, R.; Nakayama, H.; Hayashi, T.; Matsuo, H. Evaluation of forwarding efficiency in NFV-nodes toward predictable
service chain performance. IEEE Trans. Netw. Serv. Manag. 2017, 14, 920–933. [CrossRef]
26. Halpern, J.; Pignataro, C. (Eds.) Service Function Chaining (SFC) Architecture. RFC 7665. Available online: [Link]
org/info/rfc7665 (accessed on 17 September 2022).
27. ETSI GS NFV-TST 001. Network Functions Virtualisation (NFV); Pre-Deployment Testing. Report on Validation of NFV Environments
and Services. Available online: [Link]
[Link] (accessed on 10 October 2022).
28. Huang, Y.X.; Chou, J. Evaluations of network performance enhancement on cloud-native network function. In Proceedings of the
30th International Symposium on High-Performance Parallel and Distributed Computing, Virtual Event, 21 June 2021; pp. 3–8.
29. Processor Affinity or CPU Pinning. Available online: [Link]
13/current/[Link] (accessed on 17 September 2022).
30. Wang, X.L.; Luo, T.W.; Hu, J.Y.; Wang, Z.L.; Luo, Y.W. Evaluating the impacts of hugepage on virtual machines. Sci. China Inf. Sci.
2017, 60, 012103. [CrossRef]
31. Paolino, M.; Nikolaev, N.; Fanguede, J.; Raho, D. SnabbSwitch user space virtual switch benchmark and performance optimization
for NFV. In Proceedings of the 2015 IEEE Conference on Network Function Virtualization and Software Defined Network (NFV-
SDN), San Francisco, CA, USA, 18–21 November 2015; pp. 86–92.
32. Linux Drivers. Available online: [Link] (accessed on 17 September 2022).
33. Poll Mode Driver. Available online: [Link] (accessed on
17 September 2022).
34. Yao, J.; Zimmer, V.J.; Zeng, S. A Tour beyond BIOS: Using IOMMU for DMA Protection in UEFI Firmware. Available on-
line: [Link]
[Link] (accessed on 17 September 2022).
35. Usha Devi, G.; Peduru, K.T.; Mallikarjuna Reddy, B. A novel approach to gain high throughput and low latency through SR-IOV.
Int. J. Eng. Technol. 2013, 5, 1245–1251.
36. Rojas-Cessa, R.; Salehin, K.M.; Egoh, K. Evaluation of switching performance of a virtual software router. In Proceedings of the
2012 35th IEEE Sarnoff Symposium, Newark, NJ, USA, 21–22 May 2012; pp. 1–5.
37. Rizzo, L. Netmap: A novel framework for fast packet I/O. In Proceedings of the 21st USENIX Security Symposium, Bellevue,
WA, USA, 8–10 August 2012; pp. 101–112.
38. Shanmugalingam, S.; Ksentini, A.; Bertin, P. DPDK Open vSwitch performance validation with mirroring feature. In Proceedings
of the 2016 23rd International Conference on Telecommunications (ICT), Thessaloniki, Greece, 16–18 May 2016; pp. 1–6.
39. Xu, C.; Wang, H.; Shea, R.; Wang, F.; Liu, J. On multiple virtual NICs in cloud computing: Performance bottleneck and
enhancement. IEEE Syst. J. 2018, 12, 2417–2427. [CrossRef]
40. Pitaev, N.; Falkner, M.; Leivadeas, A.; Lambadaris, I. Characterizing the performance of concurrent virtualized network
functions with OVS-DPDK, [Link] VPP and SR-IOV. In Proceedings of the ACM/SPEC International Conference on Performance
Engineering, Berlin, Germany, 9–13 April 2018; pp. 285–292.
41. Ara, G.; Cucinotta, T.; Abeni, L.; Vitucci, C. Comparative evaluation of kernel bypass mechanisms for high-performance inter-
container communications. In Proceedings of the 10th International Conference on Cloud Computing and Services Science
(CLOSER), Prague, Czech Republic, 7–9 May 2020; pp. 44–55.
Appl. Sci. 2022, 12, 10315 24 of 24

42. Wang, G.; Ng, T.S.E. The impact of virtualization on network performance of amazon ec2 data center. In Proceedings of the 2010
Proceedings IEEE INFOCOM, San Diego, CA, USA, 14–19 March 2010; pp. 1–9.
43. Callegati, F.; Cerroni, W.; Contoli, C. Virtual networking performance in OpenStack platform for network function virtualization.
J. Electr. Comput. Eng. 2016, 2016, 15. [CrossRef]
44. OPNFV Functest. Available online: [Link]
functest-dockers-for-openstack-deployment (accessed on 17 September 2022).
45. NFVbench: A Network Performance Benchmarking Tool for NFVi Full Stacks. Available online: [Link]
nfvbench/en/latest/testing/user/userguide/[Link] (accessed on 17 September 2022).
46. Prerequisites for Installing a Cloudify Manager. Available online: [Link]
prerequisites/ (accessed on 17 September 2022).
47. Sengupta, S.; Yadav, V.K.; Saraf, Y.; Gupta, H.; Ganguly, N.; Chakraborty, S.; De, P. MoViDiff: Enabling Service Differentiation for
Mobile Video Apps. In Proceedings of the 2017 IFIP/IEEE Symposium on Integrated Network and Service Management (IM),
Lisbon, Portugal, 8–12 May 2017; pp. 537–543.