Version of Record: https://www.sciencedirect.

com/science/article/pii/S1383762120300722
1
Manuscript_dc2378ee0f51d06d8d51051bc060eafd

Task Allocation Algorithm and Optimization Model

on Edge Collaboration
Xiaoheng Deng, Jun Li, Enlu Liu
School of Computer Science and Engineering
Central South University
Changsha 410083, China
Email: {dxh, jun li, liuenlu}@csu.edu.cn
Honggang Zhang
Engineering Department
University of Massachusetts Boston
Boston 02125-3393, USA
Email: {honggang.zhang}@umb.edu

Abstract—This paper investigates a mobile edge computing network load pressure [5], and privacy data protection[6].
environment for video analysis tasks where edge nodes pro- In addition, with the rapid development of IoT technology
vide their computation capacities to process the computation and the emergence of various applications that are closely
intensive tasks submitted by end users. First, we introduce
a Cloudlet Assisted Cooperative Task Assignment (CACTA) related to location [7], the data generated by edge device
system that organizes edge nodes that are geographically close that is privacy-sensitive and time-sensitive grows rapidly
to an end user into a cluster to collaboratively work on the [8][9]. Edge computing can take its advantages to meet these
user’s tasks. It is challenging for the system to find an optimal requirements and provide a better user experience, and will
strategy that assigns workload to edge nodes to meet the also play an indispensable role in future computing models
user’s optimization goal. To address the challenge, this paper
proposes multiple algorithms for different situations. Firstly, [10]. In fact, there are abundant mobile terminal resources
considering the situation that historical data cannot be obtained, around the end user for collaborative computing and storage,
a multi-round allocation algorithm based on EMA prediction is and the development of 5G communication technology, which
proposed, and the experimental results prove the efficiency and will also enable the edge collaborative computing to become
necessity of multiple rounds of transmission. To address the a reality [11][12]. Taking video analysis as an example, When
second case of obtaining historical data, this paper introduces
a prediction-based dynamic task assignment algorithm that we need to find lost children or old people, or track suspects,
assigns workload to edge nodes in each time slot based on the video analysis and processing is needed. we can allocate the
prediction of their capacities/costs and an empirical optimal allo- captured videos to a number of nearby end users rather than
cation strategy which is learned from an offline optimal solution cloud center to find the target person as early as possible, thus
from historical data. Experimental results demonstrate that our avoiding the long transmission time required for transmission
proposed algorithm achieves significantly higher performance
than several other algorithms, and especially its performance to the cloud, and the disclosure of privacy.
is very close to that of an offline optimal solution. Finally, Edge computing is a new emerging paradigm that utilizes
we propose an online task assignment algorithm based on Q- computing resources at the network edge to deploy hetero-
learning, which uses the model-free Q-learning algorithm to geneous applications and services. In an edge computing
actively learn the allocation strategy of the system, and the system, applications such as Internet of Things can offload
experimental results verify the superiority and effectiveness of
this algorithm. computation-intensive tasks to nearby computing resources
to meet the increasing demand for high computation power
Index Terms—Edge Computing, Task Assignment, Prediction, and low latency [13][14][15]. The computation and storage
Q-learning, Optimization Algorithm
servers in an edge computing system [16], referred to as edge
nodes, which have the following three major features: firstly,
I. I NTRODUCTION edge nodes are highly heterogeneous [17], they might be edge
Driven by the Internet of Things (IoT), the total amount servers with strong computing capability, or mobile terminals
of data created (and not necessarily stored) by IoT will reach such as tablets and mart phones, ranging from embedded
847 ZB per year by 2021, while in 2016 it was 218 ZB per computing devices to micro data centers. Secondly, edge
year [1]. Edge computing has attracted great attention from nodes are highly mobile as they might belong to different
academics and industry [2][3], and its main idea is to push owners who might be highly mobile. Residence time of
computing resources closer to the location of the mobile user. a mobile device at a particular physical area (like coffee
By leveraging devices located on the edge of the network and shops) is uncertain, but there are also some rules to follow.
away from the cloud, various challenges in cloud computing In addition, edge nodes are highly dynamic [18], unlike
models can be solved, such as real-time requirements [4], cloud computing, edge nodes in an edge computing system

An earlier version of this paper was presented at the 2018 IEEE GLOBECOM Conference and was published in its Proceedings, DIO:10.1109/GLOCOM.2018.8647199
© 2020 published by Elsevier. This manuscript is made available under the Elsevier user license
https://www.elsevier.com/open-access/userlicense/1.0/
 2

might be owned by different entities, which brings additional by the video processing application are then used in
uncertainty to the provisioning of computing resources in an the CACTA system and combined with the empirical
edge system. The highly heterogeneity, mobility and dynamic dataset of the Google Cluster to complete the experiment
nature make edge computing nodes differ from traditional [28]. The experimental results show that the proposed
cloud computing environment where computation resources online prediction algorithm can achieve near-optimal
are centrally managed and controlled [19]. Therefore, edge performance in the environment where the edge node
nodes actual computation capacities and costs may vary over has high computation capability and cost predictability,
time. Given the uncertainty and dynamic nature of edge and its performance is not worse than other algorithms
computing, it is quite challenging to design a system to find in the setting of poor predictability.
an optimal strategy that assigns workload to nearby edge • This paper proposes an online task assignment algorithm
nodes to meet the users optimization goal. In a previous work based on Q learning, which adopts the classical rein-
[20], we proposed a prediction-based assignment optimization forcement learning method to achieve real-time resource
algorithm to find an optimal strategy that assigns workload load balancing and joint optimization. The long-term
to edge nodes in an edge computing environment. This cumulative reward is maximized by setting the reward
manuscript presents an extension of our previous paper. A function to the weighted sum of the completion time and
multi-round allocation algorithm is proposed to supplement the cost of the system. The experimental results show
the Prediction-based Optimal Algorithm (PA-OPT) in the that the algorithm is effective and outperforms other
conference paper, and a task online assignment algorithm existing algorithms.
based on Q learning is proposed. The rest of this paper is organized as follows. Related
To address the challenge, we develop a Cloudlet Assist- work is summarized in Section II. We present in Section III
ed Cooperative Task Assignment (CACTA) edge computing the system architecture and the proposed optimization model.
system, a type of proximity-based Mobile Crowd Service Two prediction-based dynamic algorithms are proposed in
system [21], that organizes edge nodes that are geographi- Section IV. Section V describes the task online assignment
cally close to an end user into a cluster to collaboratively algorithm based on Q learning. The performance evaluations
work on the users tasks. Different from existing work (e.g., of our proposed algorithms are presented and discussed in
[22][23][24][25][26][27]), our system allows an end user to Section VI. Section VII concludes this paper.
specify his/her desired level of quality of service, and then
we propose multiple algorithms for different situations, so II. R ELATED WORK
that the system can adopt a workload assignment algorithm
to satisfy the user’s quality of service requirement. The main 2015-2017 is a period of rapid growth in edge comput-
contributions of the present paper are as follow: ing, which has attracted close attention from industry and
academics at home and abroad. Next, we will usher in
• This paper designs a Cloudlet Assisted Cooperative Task the steady development of edge computing. At present, the
Assignment (CACTA) edge computing system, which main research directions are dynamic scheduling strategies,
adopts a method of dynamically allocating workloads programming models, computational execution frameworks,
to edge nodes through multiple time slots to optimize close integration with vertical industries, and the landing of
task completion time or to jointly minimize the total prototype systems.
task completion time and system cost. Resource management and collaboration: In terms of re-
• Considering the uncertainty and stochastic nature of the source management and collaboration, the existing work is
edge computing system, we formulate the optimization mainly divided into two scenarios. One is to try to allocate
problem by modeling the computation capacities and the computing tasks that are offloaded from edge nodes to mul-
costs of edge nodes as time-varying variables. Then, tiple edge clouds, and the other is to balance the distribution
the offline optimal solution is solved and used as a of edge clouds (ie, Cloudlets) and multiple edge nodes by
benchmark solution for evaluating the performance of considering objective factors (such as energy, overhead) [29].
online algorithm. You et al. considered the priority of the user’s channel
• To address the situation where long-term historical data revenue and locally calculated energy consumption in mobile
is not available to the system, a multi-round alloca- edge computing, and allocates resources to multiple users
tion algorithm based on EMA prediction (PA-EMA) is under the same delay constraint [30]. Wang et al. proposed
designed. This algorithm verifies the efficiency of the spectrum and other resources and Internet content cache
multi-round allocation strategy. optimization models to maximize the user’s revenue in mobile
• In order to cope with the second case of obtaining edge computing systems and encourage users to combine the
historical data, this paper proposes a Prediction-based way of computing locally and offloading local computing
Optimal Algorithm (PA-OPT), which can achieve near- to the cloud [23]. In the choice of calculation method, liu
optimal performance. The PA-OPT algorithm predicts a et al. solved this selection problem by using the Markov
node’s computation capability and cost based on histor- decision model, and the order of scheduling is based on the
ical data and the data observed during task execution queue state of the task buffer [31]. Various machine learning-
through the Autoregressive Integrated Moving Average based approaches for MEC are also summarized in [32]. In
(ARIMA) technique. The actual parameters obtained [33], the joint task unloading and resource allocation problem
 3

in mobile edge computing is considered to be an NP-hard

problem, and finally solved by the idea of convex optimization
and transformation into Multiple Knapsack Problem (MKP).
In order to improve the QoS of the system, Pang et al.
proposed an effective task scheduling algorithm to make full
use of computing resources, coordinate computing resources
and balance the calculation of mobile edges and clouds
in the case of guaranteed delay [34]. In [35], the author
considers the heterogeneity of users, and also proposes an
optimization model of system overhead and delay, but the
model is proposed in the framework of balancing mobile edge
Fig. 1. Cloudlet Assisted Cooperative Task Assignment (CACTA) System.
and cloud resource allocation.

Resource allocation of the edge cloud: On the other hand,

the resource allocation of the edge cloud has also received
attention. Multi-access edge computing (MEC) [36] has been
proposed to move computing and storage out of the re-
mote cloud and closer to the user. Lin et al. studies the
allocation and scheduling of mobile computing tasks among
multiple Cloudlets to optimize overall cost efficiency [22].
Data transmission and computation are modeled based on the
relationship between currency and time cost, which achieves Fig. 2. System Architecture of a Cloudlet in CACTA System.
a higher admission rate than traditional methods of selecting
a single Cloudlet. Lo et al. explored the Authentic Operation
(AO) unit and dynamic network structure to enhance the use III. S YSTEM D ESIGN
of Deep Neural Network (DNN) [24]. They proposed using
A. Overview of a CACTA system
confidence level metrics to determine whether the input image
was analyzed by its built-in secondary network or transmitted A Cloudlet Assisted Cooperative Task Assignment (CAC-
to a remote server. Bi et al. proposed a computation offloading TA) edge computing system is introduced, as shown in Fig. 1,
method to improve the computation rate of all the wireless the architecture of a CACTA system consists of end users, a
devices by optimizing the computing mode selection and cloudlet [39], and mobile edge nodes [40]. Edge nodes nearby
system transmission time allocation [37]. De Maio et al. are connected to the cloudlet to provide their computational
proposed a heuristic-based approach to find a tradeoff solution resources to the system, and end users or clients in the
among application runtime, battery lifetime, and user cost system can utilize these computational resources. Fig. 2
[38]. Wang et al. considered the mobility of mobile users and show the architecture of a Cloudlet which consists of three
designs an online resource allocation algorithm that aims to modules: scheduler, prediction, and allocator (i.e., workload
minimize the sum of the four types of important overhead, assignment).
including static overhead and dynamic overhead [23]. The CACTA system’s work flow can be described as
follows. The scheduler module of the cloudlet receives request
Despite extensive research on resource allocation between information, such as workload and time requirement [41],
Cloudlet and edge nodes, only a few researchers have focused when a computation task request arrive in the system. It then
on the computational resource allocation of edge nodes be- broadcasts the request to the edge nodes geographically close
tween collaborative edges. Although the computing capability to the request and organizes them into a collaborative group
of mobile devices and cloud processing centers is very to work on the tasks. The cloudlet tracks the status of edge
different, potential mobile devices are expected to gradually nodes, such as their capabilities, costs, and available time.
reduce this gap and become a useful assistant for network The allocator or workload assignment module divides a task
data processing, such as tablets and personal computers. In into subtasks (for example, an image-processing application
[25], for the video analysis task, the method of reducing divides a collection of images into different sets of images),
the task completion time by cooperation with nearby user and assigns a set of those subtasks to edge nodes in a collab-
nodes is proposed, and the historical response time is analyzed orative cluster of edge nodes in multiple rounds according to
by regression analysis to improve the cooperation efficiency. a certain task assignment algorithm. Once the collaboration
In addition, Habak et al. proposed the system of marginal cluster completes the task, the cloudlet returns the result to
Femtoclouds [27]. The architecture is used as a controller the end user who requested the task.
to manage and coordinate users and tasks. Secondly, the task Considering whether the system can obtain long-term his-
management mechanism for dynamic mobile device groups is torical data, we propose two novel prediction-based workload
designed. The system is tested from a practical point of view allocation algorithms in this paper. The basic idea is as
and has achieved good experimental results. This shows the follows. System time is discretized into multiple rounds or
necessity and potential of cooperation between mobile users. time slots. In the first round, the allocator module assigns a
 4

certain amount of workload (e.g., a number of subtasks) to large enough that a task with a finite number of subtasks can
each edge node. Then, in the following rounds, it assigns a always be accomplished through the system. Here we assume
number of subtasks to edge nodes based on the prediction that Rm is sufficiently large so that a task with a finite number
or estimation of the capacities and available times of those of subtasks can always be completed by the system. Note that
edge nodes in future rounds. The predication is carried out via a CACTA system may allocate its resources to multiple end
the prediction module based on historical data. The predicted users. Without loss of generality, we focus on one end user
results will be utilized by the allocation module to determine and remove the notation m for the rest of our discussion.
the assignment of workload to edge nodes via some optimal We implement a total cost metric Ctotal as a joint equation
allocation strategy. of the cost and the latency of executing a task. It is defined
In contrast to the tightly managed and controlled computing as follows.
nodes in cloud computing, the edge computing nodes’ capac- Ctotal = αCalg + βTalg (1)
ities and their stay or presence times in the system are often
difficult to know in advance, due to the following reasons. Note that Calg includes the following costs: (1) Computation
They may be owned by different entities. They may have cost. (2) Storage cost. An edge node will ask the client for a
just joined the system at the time when the cloudlet attempts payment per unit task that it execute and keep in local storage,
to assign tasks to them, thus no historical data is available. and this payment might vary over time. (3) Data transmission
Even for an edge node that has stayed in the system for a cost. Transmitting a certain amount of subtasks from the client
long time, its fraction of CPU and memory that is available to an edge server node will incur some transmission cost.
for the system may fluctuate over time, because the edge Due to the above reasons, the system will need to make an
node is also concurrently utilized in other computing tasks appropriate allocation of subtasks to all edge nodes in order
(assigned by their owners) that are outside the system, or to save cost. A low value of Ctotal corresponds to a high
the node might simply leave the system due to the mobility system performance. Note that α(≥ 0) and β(≥ 0) are the
nature of its owner. Therefore, it is important for the system weighting factors to tradeoff cost and task’s completion time.
to predict or estimate the capacities of edge nodes in future If β > α, which means that the user put more attention on task
in order to assign appropriate workload to them. Details of completion time than on cost. Therefore, let W be the size of
our prediction technique will be presented later. data, we can characterize a user’s task J as J , (W, α, β).
Because of the fact that neither the system or the users can
fully know the capacities of all edge nodes at all future time
B. System Model slots, the system can only estimate or predict those values.
This section provides the description of the CACTA system Let b̂jt denote the estimated capacity of node j at time slot
model that we have used to implement our task offloading al- t. The number of subtasks that the system assigns to edge
gorithm. We consider a set of end users M = {1, 2, ..., UM }, node j is denoted by njt , which is upper bounded by b̂jt , i.e.,
where task requests are sent to a cloudlet. Each end user m njt ≤ b̂jt , and it is not necessarily true that njt = b̂jt because
has a computation task Jm . The mathematical variables used the system attempts to minimize the total cost metric, not just
throughout this work are summarized in Table I. Suppose to minimize task completion time. The number of subtasks
that the cloudlet of a CACTA system allocates the task actually completed in round t is denoted by b̃jt , which is the
Jm to a set of edge nodes or servers, which denoted by smaller one of njt and bjt , i.e., b̃jt = min{njt , bjt }.
S m = {1, 2, ..., Nsm }. Each edge node has a certain amount of If we know Talg , then Calg can be computed as follows
computation resources. When an end user sends a computing
task request to the cloudlet nearby, the cloudlet works as a ∑ ∑
Ns Talg

Calg = njt (ccomp

jt + ctran
jt + cstor
jt ) (2)
reliable platform that searches for available edge nodes near
j=1 t=1
the request, and then form the set S m of edge nodes to serve
the request. Note that Talg ({njt }) is a function of the vector of workload
The system time is discretized into multiple time slots or assignments {njt }, and hence Calg ({njt }) is also a function
rounds. The set of rounds or time slots to work on task Jm is of {njt }, which consists of three parts cost, i.e., computation
denoted by Rm = {1, 2, ..., Nrm }, assuming that the time slot cost, transmission cost and storage cost.
length is τ . Let cm
jt be the cost of each edge node j in a round
t with t ∈ Rm , which consists of computation cost, storage C. Optimization Formulation
cost, and data transmission cost. The cost can be interpreted
as the operating cost of the node or the payment that the node The objective of a user is to minimize Ctotal , and the
asks from the user. We assume that the computation capacity optimization is carried out by the CACTA system through
of edge node j in round t (i.e., the number of subtasks it some algorithm alg. That is,
can process) is bm m m
jt . Note that both cjt and bjt can vary over min Ctotal = α · Calg ({njt }) + β · Talg ({njt }) (3)
different rounds or time slots, so their values can be modeled {njt }
as stochastic processes. The sets of cm jt and bjt with ∀t, ∀j
m
s.t. 0 ≤ njt ≤ b̂jt , ∀j, ∀t (4)
are denoted by C and B respectively.
m m ∑Ns ∑Talg
We can characterize the CACTA system that works on task j=1 t=1 njt = Wm (5)
∑Ns
Jm through {S m , Rm , C m , Bm }. Here we assume that Rm is 0 ≤ j=1 njt ≤ Wm , ∀t (6)
 5

TABLE I 7500
PARAMETERS = 6, =3
7000 = 5, =5
Symbol Definition
M The set of end users 6500
Jm The computation task of end user m
Sm The set of edge nodes
6000
Rm

C total
The set of time slots work on task Jm
τ Time slot length
cm The cost of edge node j in a round t with t ∈ Rm 5500
jt
bm
jt The computation capacity of edge node j in round t
Cm The set of cm 5000
jt
Bm The set of bmjt
Ctotal The total cost metric 4500
The actual number of time slots to complete the task
Talg 4000
via algorithm alg
Calg The cost incurred by algorithm alg to complete the task 0 200 400 600 800 1000
bˆm The estimated capacity of node j at time slot t Time slot
jt Fig. 3. Ctotal varies along task completion time T , with 1000*1000 frames.
njt The number of subtasks the system assigns to edge node j
comp
cjt The computation cost
ctran
jt The transmission cost
cstor
jt The storage cost we find the T ∗ that corresponds to the minimum total cost
∗ ∗
Ctotal . Note that Ctotal is a lower bound for all algorithms
that do not have the system’s complete information.
where Calg ({njt }) is given by (2); and (4) shows that njt is We first assume that Tcost only and Ttime only is the upper
upper bounded by the estimated capacity; and (5) means the bound and lower bound of T ∗ . The upper bound Tcost only
total number of assigned subtasks is equal to the actual total is the time that minimizes the cost only in Eqn. (3) and the
number of subtasks. lower bound Ttime only is the minimum time to finish all
subtasks without considering cost. To find Tcost only , we only
Challenge. Note that Talg , i.e., the number of terms to
assign subtasks to the lowest cost node k in each round t by
add together in getting Calg , is essentially determined by
the following rule: nkt = bkt if ckt ≤ cjt , ∀j, (other nodes
decision variables {njt }, thus it is challenging to solve
with higher cost will not receive any subtask), till we finish
the above non-linear optimization problem, because Talg
the assignment of all subtasks. To find Ttime only , we let
(the number of summation terms) appears in the objective
njt = bjt , ∀j till all subtasks’s assignment is finished. Since
function. Furthermore a solution to the above optimization
T ∗ is the solution that minimizes the combination of comple-
problem might not complete all subtasks in practice, because
tion time and cost, we have Ttime only ≤ T ∗ ≤ Tcost only .
b̃jt may not match bjt the actual capacities of edge nodes,
Note that for any T ∈ [Ttime only , Tcost only ], Eqn. (5) can
i.e., b̃jt = min{njt , bjt }. always be satisfied and the feasible solution space of the
To solve the challenge, we initially propose an offline op-
optimization problem given by (3)-(6) is not empty. Thus,
timization algorithm to obtain the optimal solution n∗jt of (3)
for each T ∈ [Ttime only , Tcost only ], we can solve (3)-(6)
under the assumption of knowing the complete information,
to get a minimized Ctotal (T ) (which is a function of T ),
i.e., we know all cjt ’s and all bjt ’s. Then, in order to obtain
and then choose T ∗ that gives the lowest Ctotal (T ∗ ). It is
an online simple and effective solution, we develop an online
interesting to note that Ctotal is a smooth convex function of
algorithm based on prediction through the following steps. We
T in [Ttime only , Tcost only ], thus there must exist a unique
first utilize ARIMA technique to predict bjt and cjt in a real
T ∗ that minimizes Ctotal in the range [Ttime only , Tcost only ].
edge computing environment where the capacity and cost of
We next illustrate the above optimization procedure via an
edge node j in each time slot t are stochastic and unknown
example. To describe the example, we solve the optimization
in advance. Then back to the historical dataset, we find a
problem with complete system information by using edge
subset whose bjt ’s and cjt ’s have the highest similarity with
nodes’ computation capacities sampled from Google dataset
the bjt ’s and cjt ’s that we predict, which means that it has the
[28]. Fig. 3 shows an example where an image processing task
least square error among all the subsets in comparing with our
has W = 4×105 image frames or subtasks to be processed by
predicted values. Next, we derive an optimal solution of those
edge nodes. For different combination of constants α and β,
subsets, and let the system learn a regression model to make
there is always a minimum Ctotal . Take α = 6, β = 3 as an
the optimal task assignment n∗jt as the function of bjt and cjt .
example, we can observe that Ttime only is 4 and Tcost only
It is worth noting that the learned regression model, referred
is 860, then Ctotal is minimized when T is about 200.
to as an estimated optimal allocation strategy, is based on
the historical dataset that records observed bjt and cjt in the
past, which is then applied in our working system to allocate IV. P REDICTION - BASED DYNAMIC A LGORITHM
workload to each node at time t. For an dynamic edge computing environment where edge
nodes’ capacities are stochastic. The basic idea of a dynamic
D. Offline Optimization with Complete System Information algorithm is to assign workload to edge nodes in multiple
Consider an ideal case where the system has a full knowl- stages over time. A stage can include multiple time slots or
edge of bjt and cjt of all the edge nodes in all time slots. Then a single time slot. In each stage, the algorithm predicts all
 6

edge nodes capacities in the stage (which might include a Algorithm 1 PA-EMA
single time slot or multiple time slots), and decides workload Input: A task with size W ; Edge node set S = {1, 2, ..., Ns }
assignment via Exponential Moving Average (EMA) tech- Output: n∑ jt workload
∑ assigned to node j in time slot t, with
nique or a decision strategy that specifies the assignment W = j t njt j ∈ S, t ∈ R
as a function of the predicted capacities and costs of edge 1: while W assigned < W do
nodes. The algorithm keeps updating its prediction based on 2: for each round t ∈ {1, 2, ..., Nr } do
its observation of the actual capacities of the edge nodes in 3: for each edge node j ∈ {1, 2, ..., Ns } do
the past, and it keeps generating new workload assignment 4: AvailableTimeOfNodes()
of the remaining workload based on its most recent predic- 5: //Judge whether edge node is in collaborative
tion. Considering whether the system can obtain long-term group
historical data, we introduce two dynamic algorithms: PA- 6: Predict b̂j(t+1) using EMA model according to
EMA algorithm (Prediction-based Assignment based on EMA Eq.(8)
algorithm), and PA-OPT algorithm (Prediction-based Assign- 7: Let nj(t+1) = b̂j(t+1) and assign the workload of
ment Optimization algorithm), PA-EMA algorithm predicts nj(t+1) subtasks to edge node j
the nodes’ capacities in the next round by simply using 8: end for
the observed capacities in the current round based on EMA 9: end for
technique. PA-OPT algorithm is based on the prediction of 10: end while
edge nodes’ computation capacities/costs and an estimated
optimal allocation strategy learned from historical dataset.
The objective of a prediction is to minimize the error of B. Prediction-based Assignment Optimization Algorithm
estimating an edge node’s capacities in future rounds, given
This algorithm addresses a situation where edge nodes of
below:
a system have stayed in the system for a long time, and the
min |bjt − b̂jt |, ∀j ∈ S, ∀t ∈ R (7) system has a record of the historical data of those edge nodes’
computing capacities.
where b̂jt is predicted capacity.
Note that each edge node asks for a payment for a subtask
In the next, we first present two algorithms, and then that it works on. This payment can be regarded as the cost to
discuss the prediction and learning components about PA- the end user who submits the task. The cost of each edge node
OPT algorithm in details. varies over time in practice. For example, an edge node with
high available CPU capacity might ask for a high payment,
A. Prediction-based Assignment based on Exponential Mov- i.e., different tasks have different timeliness requirements, and
ing Average a stronger computing capability of the edge node means a
higher cost to the end user.
This algorithm addresses a situation where long-term his- The algorithm utilizes ARIMA technique [43] to predict b̂jt
torical data is not available to the system. Note that edge and ĉjt based on historical dataset, and decides its decision
nodes are highly mobile, and the time spent in a certain strategy via learning from historical data to make the optimal
area is uncertain, and it is possible to leave the area at any task assignment njt as the function of cjt and bjt , i.e., njt =
time. Secondly, an edge node may participate in multiple f ∗ (bjt , cjt ). Then the estimated strategy is denoted by
cooperation groups, and may also occupy part of the resources
in order to meet its own needs, so the computing performance n̂jt = fˆ∗ (b̂jt , ĉjt ) (9)
of a node is fluctuant with time. In addition, edge nodes have
PA-OPT Algorithm is shown in Algorithm 2, so it can do
high heterogeneity, which may be edge servers with strong
a workload assignment to edge nodes in multiple rounds in
computing capability, and may also be mobile terminals such
future. The basic idea of the algorithm is as follows.
as tablets, smart phones, etc., so their storage resources and
It predicts or estimates all edge server nodes’ capacities
computing resources are different.
in multiple future time slots using ARIMA model, i.e., b̂jt ,
In this case, the system in time t predicts the available
and then computes njt using regression model according to
capacity of an edge node j in the next round t + 1, denoted
Eq.(10) and assign the workload of njt subtasks to edge node
by b̂j(t+1) via EMA technique [42]. Based on the previous
j. Recall this algorithm will give us a minimal total cost
t rounds capacities of edge node j, i.e., b1 , b2 ,...,bt , We can
according to Eq.(3) and a task completion time T . Since all
get b̂t+1 , i.e., its expected capacity at time t + 1., as follows
edge nodes’ capacities are estimated, the task completion time
b̂t+1 = r · bt + (1 − r) · b̂t (8) is also estimated, denoted as T̃ , and within T̃ , the task might
not be completed in practice due to the difference between
where r is the smoothing index and adjusted empirically to actual nodes’ capacities and estimated capacities. Thus, at the
achieve the optimal accuracy. The term of which is closer to end of T̃ , if there are still subtasks left, we repeat the previous
the current time, the weight is greater. PA-EMA algorithm steps till all subtasks are completed. A formal description of
conducts workload assignment based on the predicted capac- the algorithm is given in Algorithm 2.
ity of each node, that is, it lets nj,t+1 = b̂t+1 , and it is shown In practice, the computation capacities of all edge nodes in
in Algorithm 1. all time slots are not able to be known by the system. Also,
 7

Algorithm 2 PA-OPT of these parameters will derive different models. Based on

Input: A task with size W ;∑ Edge∑node set S = {1, 2, ..., Ns }. the empirical dataset from Google [28], We found that the
Output: njt with W = j t njt j ∈ S, t ∈ R. The regression model performs quite well in calculating n∗jt based
completion time T and cost C of the input task. on bjt and cjt . For example, we randomly selected 100 edge
remaining
1: t ← 0, C ← 0, Wj ← 0, Wjassigned ← nodes from Google dataset, set the task amount W and the
f inished
0, Wj ← 0, j ∈ S weight parameters α and β, and ran the offline optimization
∑ assigned algorithm to get n∗jt . Then we use the regression model (10)
2: while j Wj < W do
3: for each edge node j ∈ {1, 2, ..., Ns } do to find an estimated strategy n∗jt = f ∗ (bjt , cjt ), and then the
4: Predict b̂jt using ARIMA model system assigns workload online at each time slot according
5: Compute njt using regression model according to to this estimated strategy, and the final Ctotal was nearly
Eq.(10) and assign the workload of njt subtasks to identical to the result of the offline optimization solution. Due
edge node j to space constraints, no detailed statistics are reported here.
6: C ← C + cstor Wjremaining + ctran njt + ccomp b̃jt In Section VI, we explained the process of getting bjt and
t t jt
f inished f inished cjt based on the Google dataset.
7: Wj ← Wj + b̃jt
8: Wjassigned ← Wjassigned + njt
9: Wjremaining ← Wjassigned − Wjf inished C. Prediction of edge node computation capacity and cost
10: end for In our system, we predict both the computation capacity
11: t←t+1 and cost of an edge node, i.e., bjt and ccomp . But for ease of
jt
12: end while exposition, we use the prediction of computation capacity bjt
13: T̃ ← t, Tj ← 0, ∀j. as an example to discuss our prediction method. We assume
14: for each edge node j ∈ {1, 2, ..., Ns } do that the observed time series of CPU capacity is denoted by
15: while Wjremaining > 0 do bt , which aims to predict bt+z , the z steps ahead from bt
16: C ← C + cstor T̃ +T
Wjremaining + ccompjt bj,T̃ +Tj [44], [45]. Let bt+z|t denote the z th step prediction from bt ,
j

17: Wjremaining ← Wjremaining − bj,T̃ +Tj i.e., bt+1|t ,...,bt+z|t . Through computing the one-step ahead
18: Tj ← Tj + 1 prediction with ARIMA iteratively, we can obtain the multi-
19: end while step prediction results bt+z .
20: end for The z th step prediction bt+z|t can be expressed as
21: T ← T̃ + max{Tj }, note that T can only be obtained
bt+z|t = F (bt+z−p|t , bt+z−p+1|t , ..., bt+z−1|t ) (11)
once all nodes finish the task together
22: return T , C and njt , ∀j, ∀t where function F is the prediction model, and p is the number
of lags. For bt+1 , if it is stationary for each t, which can be
described as ARMA (p, q), the time series is given by
a system can only get the information of all edge nodes in
bt+1 = φ0 bt + ... + φp−1 bt−p+1 +
a certain time slot. To assign tasks to the edge nodes in a (12)
certain time slot with the information of all nodes in this time at+1 − θ0 at+1 + ... + θq−1 at−q+1
slot, we analyze the data obtained by the offline algorithm, in where the φc , c = 1, 2, ..., p, and θh , h = 1, 2, ...q are the co-
which the system knows all the information of all nodes at the efficients estimated from a training set. The at is a white noise
beginning, and model the number of subtasks njt assigned to series, which is subject to normal distribution whose mean is
edge node j by a function which is a nonlinear combination zero and variance σ 2 is unknown. The parameter p and q are
of the computation capacity bjt and cost Calg . the order of AR (Autoregressive Model) and MR (Moving
To address the nonlinear optimization problem given in (3), Average Model) respectively. If the time series shows non-
we propose a functional relationship (as the optimal allocation stationary, we use ARIMA (p, d, q) model to describe the
strategy) between the optimal assignment solution n∗jt and the bt . Let η be the backward shift operator, which is defined as
system parameters bjt , cjt , shown below. η c b(t) = b(t−c) . Then the formula (1 − η)bt = bt − bt−1 can
be derived. By denoting the ∇ differential operator, an order
bjt
n∗jt = f ∗ (bjt , cjt ) = a1 + a2 cjt + a3 bjt + a4 (10) d difference series can be expressed as following
cjt
d
∇d bt = (1 − η) bt (13)
The optimal assignment is determined by capacity bjt and
cost cjt . Furthermore, it is related to the capacity per unit After the time series have been differenced d time to obtain
cost, i.e., bjt /cjt . the stationary time series, the ARIMA can be given by
We can find the optimal n∗jt given a set of system pa-
rameters bjt and cjt according to Section III-D. As the ∑
p−1 ∑
q−1

regression model is a mathematic model, njt computed by (1 − φc η c )(1 − η)d bt = (1 + θc η c )at (14)
c=0 c=0
it can be a negative number; if it happens, we replace the
negative number by zero. Note that the regression model is To measure the accuracy of the prediction using ARIMA
related to the parameters (W, α, β), thus different settings model, we introduce Mean Squared Error (MSE) to represent
 8

1.2
node 1
Assignment. dynamic online algorithms considered include
node 2 Short-Term Greedy Assignment and PA-EMA.
1 node 3 The basic idea of one-time static algorithms is that the
node 4 system divides a whole submitted task into multiple workload
node 5
node 6 shares, and assigns a workload share to each edge node before
0.8
MSE Deviation

node 7 those nodes start working.

node 8 Uniform Assignment Algorithm equalizes the task size of
node 9
0.6 node 10 all task shares (i.e., all shares have the same number of
subtasks). If we let timeList represent the set of all edge
nodes’ times of completing their workload shares, the com-
0.4
pletion time of the whole task is determined by the longest
completion times in timeList. Based on an assumption that
0.2 the system has a historical dataset that records all edge nodes’
computation capacities in a past history, we designed two
prediction-based one-time assignment algorithms: Random
0
0 50 100 150 200 250 300 Assignment and Fair Ratio Assignment. For Random As-
Time slot signment, the amount of workload assigned to each edge
Fig. 4. MSE deviation of each node. node follows a uniform distribution bounded by the minimum
and the maximum capacity in the historical dataset. For Fair
Ratio Assignment, the amount of workload assigned to each
the deviation between predicted capacities and actual capac-
edge node is proportional to the average of its historical
ities in T rounds. The formula is given by:
v computation capacities.
u Furthermore, PA-EMA algorithm conducts workload as-
u1 ∑ T
M SE j (T ) = t (bjt − b̂jt )2 , ∀j (15) signment based on the predicted capacity of each node, and
T t=1 the detail has been described in Section IV-A. In addition, the
which indicates the MSE value of the actual capapcity and idea of Short-Term Greedy Algorithm is as follows. At time
predicted capacity of edge node j in T rounds or time slots. t + 1, the algorithm assigns nj,t+1 = (b̃jt + max{b̃jk , 0 ≤
Then MSE Deviation, i.e., the ratio between ∑the MSE and the k ≤ t})/2 subtasks to node j. The greedy algorithm always
T
mean of actual capacities (i.e., Bj (T ) = T1 t=1 bjt ), is attempts to assign more subtasks in time t + 1 than what had
been completed in time t.
M SE j
Pj (T ) = , ∀j (16) V. TASK ONLINE ASSIGNMENT ALGORITHM BASED ON Q
Bj (T )
LEARNING
To illustrate this prediction error metric, we randomly choose
Resource management problems in systems and networks
ten nodes in Google dataset [28], and plot Pj (T ) in Fig.
are often challenging in the decision making of task online
4. Clearly, the prediction works well for most of the nodes.
assignment, where appropriate solutions rely on understand-
And for those nodes can be predicted well, we see that at the
ing workloads and environments. Motivated by the latest
beginning, Pj (T ) will increase first, but then it decreases and
advances in reinforcement learning in artificial intelligence,
tends to be stable when T increases to a certain number of
this section considers the design of algorithms that are
time slot. Note that one edge node has very high prediction
self-learning and enhanced from experience [46]. Because
error (starting with nearly 1.2 and oscillating between 0.6
reinforcement learning has strong learning ability in control
and 0.8 when T > 30). This shows that the Google dataset
and decision-making, many research works also adopt rein-
contains a mixture of highly predictable nodes and poorly
forcement learning methods. Among them, there are many
predictable nodes.
research works in resource management and task scheduling
that use the reinforcement learning technology to achieve
D. Other Algorithms good experimental performance [47][48][49]. However, some
In practice, a system cannot know the precise computation existing works are to regard the resource provider as a
capacities and costs of all edge nodes in future time slots resource pool, and they are not specific to assign to a certain
in advance. To address this practical system design issue, task processing machine. Some works are based on the
we will introduce several algorithms in this paper, which are assumption that workloads are initially allocated to a fixed
classified into two categories: one-time static algorithms and task processing machine. If the machine does not meet the
dynamic online algorithms. The former algorithms conduct calculation conditions, the workloads are assigned to other
a one-time allocation of all subtasks at the beginning of machines, which in turn generate communication overhead.
processing a submitted task. The latter algorithms conduct In fact, in the video analysis scenario, multiple cameras at
dynamic workload allocation over time in order to optimize the edge of the network may capture the activity of the
the system’s performance by dealing with the variations of target person in a certain period of time. After the scheduling
edge nodes’ capacities and costs. One-time algorithms include nodes obtain the information, the video is sent as a non-
Uniform Assignment, Random Assignment, and Fair Ratio splitable single task to the nearby available edge severs to
 9

execute the task. The reason for not splitting is that the There are three key elements in the reinforcement learning
video file is not too large and it is easy to retain information method, namely state space, action space, and reward. The
such as the geographical location. This section considers scheduling node is equivalent to the Agent, which can observe
the continuous arrival of tasks and the dynamic changes of the status of all edge nodes and tasks.
resource-providing nodes. A task online assignment algorithm State space: In this section, the state of the system, that is,
based on Q learning is proposed for video analysis tasks. the current allocation and task state of the collaboration group
Based on the above two algorithms, two main points are is represented by independent dicing, as shown in Fig. 5. The
considered and improved. squares in the figure represent the number of different node
• Not only the dynamic changes of available nodes over resources (mainly considering CPU) occupied by multiple
time are considered, but also the real-time impact on tasks in multiple time slots from the current time. Squares
node resources after task assignment is considered. of different colors represent different tasks. For example, the
• While performing the task assignment online, it is not green box on the far left indicates the resources of one unit
necessary to predict the computation capability and cost of CPU in the three slots of node 1. The rightmost task queue
of the edge nodes in the future to achieve the joint in the figure represents the waiting or upcoming task. To sim-
optimization of completion time and system cost like plify the model, only one task per slot is arrived. This section
PA-OPT algorithm. uses three levels to divide the computation capability bjt of
different nodes, which are low level (CL=1), intermediate
level (CL=2) and advanced level (CL=3), corresponding to
A. System Model the number of video frames processed per second is less than
Considering a collaboration cluster that has Ns edge nodes 5 frames, between 5 frames and 10 frames and greater than
at the edge of the network. Each node has different computing 10 frames, which can effectively reduce the space complexity.
resources, such as CPU and memory. The set of available edge Therefore, one state of the system s = (B ⃗ t , wi ) contains
nodes is represented as E = {1, 2, ..., m}. Here, the system two parts, namely the computation capability level vector
time is discretized into multiple time slots, and all tasks arrive B⃗ t = b1t , b2t , ..., bmt and the task size wi of all nodes in the
at the edge nodes in an online manner. This section assumes time slot t. The state space is represented as S and the size is
that only one task arrives in each time slot, which means that |S| = 3m · n. Incorporating the size of tasks into the state can
the scheduling strategy selects one task for scheduling in each better achieve adaptability and sensitivity to upcoming tasks.
time slot. The task set W = {w1 , w2 , ..., wn } represents the Action space: In each time slot, the scheduling node
size of each task wi . When the resources required for the task assigns a task to a node. Then an action is defined ∑m as a =<
computation exceed the available resources of node, the node aji >, ∀i ∈ 1, 2, ..., n, ∀j ∈ 1, 2, ..., m, and j=1 aji = 1,
will compute the task with the weakest computing capability ∀i, which means that a task can only be assigned to one node.
until the occupied resources are released. Consistent with the Therefore, the action space A contains all feasible actions,
above, the computation capability of edge node is heteroge- and the space size is |A| = mn , but since only one task is
neous. The computation capability here refers to the number allocated in one time slot, the action space is reduced to the
of processing video frames. The more the number of video linear space m.
frames processed per unit time, the stronger the computation Reward: The reward is developed to guide the Agent to
capability and the greater the computation cost of the node. find a good solution for the system goal: minimize the weight-
Assuming that the number of frames computed by the edge ed sum of completion time and system cost. Specifically,
node j in the time slot t and the computation cost per frame this paper defines ri = −(κ · Twi + (1 − κ) · Cwi ) as the
are bjt and cjt , respectively, the computation time Twi to reward for each time slot. The Agent will receive a reward at
∑Tw
complete task wi satisfies the equation wi = t=1i bjt and each time step, so this section simply defines the reward as
∑Twi the negative of the weighted sum of the computational time
the computation cost is Cwi = t=1 cjt · bjt . Similar to
previous paper, this section assumes that the size and the and the computational
∑T cost, i.e., maximizing the cumulative
resource required of each task is known when the task arrives. rewards t=1 rt .
Despite many assumptions, this simple model better describes
the scheduling between multitasking and multi-resources and B. Algorithm Description
proves the effectiveness of reinforcement learning in resource
Reinforcement learning is mainly divided into two types:
scheduling under not cumbersome settings. Therefore, the
model-based and model-free, in which Q-learning is model-
system goal of this section is to minimize the completion
free, without prior learning environment and modeling [50].
time and system cost of all tasks, and to introduce weight
The model-free type can be divided into an online algorithm
factor κ in order to balance the relationship between the two
and an offline algorithm. The offline algorithm may learn
parts. As shown below:
previous experience or other people’s experience, such as
∑
n
Q-learning. The Sarsa algorithm is an online algorithm that
M inimize (κ · Twi + (1 − κ) · Cwi ) (17) selects states and actions before the next time slot begins.
i=1
Simply, the computation capability of the node and task
where κ represents the weight of the task computation time amount observed by the Agent are taken as input, and the
and κ ∈ [0, 1]. action of the next time slot is performed by the Agent
 10

Algorithm 3 Q value Table

Input: Initialize Q(s, a), for s ∈ S, a ∈ A
Output:
1: for each round do
2: Initialize S
3: repeat
4: Choose the corresponding action a according to the
state s learning from Q table (ϵ greedy strategy)
5: Get the reward r and next state s′ after taking the
action a
6: Q(s, a) ← Q(s, a) + α[R + γmaxa Q(s′ , a) −
Q(s, a)]
Fig. 5. Example of system state representation [49]. 7: s ← s′
8: until s is the final state
9: end for
interacting with the environment. The system first trains and
learns the optimal strategy through a phase. At each stage,
a fixed number and size of tasks arrive at the system and in edge computing systems initially. Then, we evaluate the
are scheduled. When all tasks are completed, the episode performance of the proposed PA-OPT algorithm and compare
terminates. In order to train a generalized strategy, multiple it with other algorithms in edge computing environments
arrival sequences of task are considered during the training that range from settings where edge nodes capacities/costs
phase. In each training iteration process, the experiment are highly predictable to settings where they are poorly pre-
iterates N phases for a set of task sequences, thereby utilizing dictable. Finally, the performance of Q-learning based online
existing strategies to better explore the Q values of possible task assignment algorithm is evaluated in several aspects.
actions. Specifically, at each time slot t, the Agent observes The setting parameters are described in detail respectively
the state st of the environment in which it is located, and in the corresponding three parts. We choose Python as the
selects an action at . After this action, the environmental programming language in the PyCharm simulation platform
state is converted to st+1 and the agent receives the reward to perform our simulations.
rt . State transitions and rewards are random and assume a
Markov property, i.e., the state and reward of the next time
A. Evaluation Result of the PA-EMA Algorithm
slot depends only on the current state and the action taken.
In order to avoid the Agent always choose the action with As mentioned in Section IV, there are two conditions to
the largest Q value and fall into the local optimal solution, be considered. First, we address a situation where long-term
the greedy strategy is adopted here to let the Agent have the historical data is not available to the system as described in
probability of ε to randomly select an action. Section IV and we use the PA-EMA (Prediction Assignment)
It is important to note that in this process, the Agent to compare with uniform assignment.
can only control its behavior, and it does not have a priori For simulations, we consider that there are eight edge nodes
knowledge of which state the environment will change or in the collaborative cluster. The computing capability of each
how many rewards it can receive. By interacting with the device is variant, which is subject to normal distribution with-
environment, Agent can observe these states and learn the in a certain period of time. And the mathematical expectation
rules during training. The goal of Agent learning is to and the variance of each distribution are respectively selected
maximize the expected cumulative discount rewards: randomly from the intervals [30, 60] and [8, 19]. In this
∞
experiment, we assume that the time interval of each round is
∑
Q(s, a) = E[ γ k rt (st , at )|s0 = s, a0 = a] (18) 1 min, and the unit of the workload is MB. According to the
t=0
completion situation of the previous subtasks, we predict the
computing capability of each node at next minute. Moreover,
where γ < 1 is a discount factor, and it is the present value of
we use the Gaussian distribution algorithm to generate the
future rewards, indicating that the closer the state and action
residence time in the collaborative cluster of each node.
is to the current time, the greater the influence on the action
Even if the task is not completed, the node will leave the
selection of current time. The specific process of learning the
collaborative cluster once the time is up. Also, there is a
Q value table is shown in Algorithm 3, and the algorithm is
certain probability that there will be new nodes are added,
referred to as the QL algorithm.
maintaining collaborative cluster runs continuously. However,
for the one-time assignment strategy, the workload that per
VI. P ERFORMANCE E VALUATION node obtains is equal. Considering the mobility of users, we
In this section, numerical results are presented to illustrate define λ = Wtotal /Wassigned as effective transmission ratio
our proposed algorithms. First, we evaluate the PA-EMA (ETR), which indicates the utilization of the resource (e.g.,
algorithm and compare it with uniform assignment algorithm bandwidth). There is no doubt that the λ is closer to 1, a
to prove the efficiency of multi-round allocation algorithm higher utilization rate it shows.
 11

15 15
70
Uniform Assignment

PA-EMA

60 12

10

Frames/Second
50 9
Round

40
6
5
30
3

20
0 0
35 40 45 50 55 60 65 70 75 80 85 90 95 100 105
10 CPU Usage (%)

4000 6000 8000 10000 12000 14000 16000

Workload (MB)
Fig. 8. The computing capabilities of video analysis over CPU usage

Fig. 6. Computing time with different workloads.

video processing request to an edge server via Docker [46]

1.05
framework, which is a VirtualBox inside a Ubuntu Linux
Uniform Assignment

PA-EMA
computer. The data is sent through WiFi to the edge server.
The edge sever tested in this paper is Intel Core i7-6560U,
1.00
with a CPU of 2.20 GHz and a memory of 2048 MB.
Transmission efficiency

Then we changed the computation capacity of the sever

0.95 in the range of 100%, 90%, ..., 40% of its original capacity
(CPU utilization)). The OpenCV launched in docker container
0.90
is used to analyze the video recorded from the streets in
New York City. The purpose of this video analysis is to
identify the face of the target person from the short video.
0.85
In the face recognition experiment, we obtain the functional
relationship between the number of video frames processed
0.80 by the machine and CPU capacity by fitting the linear model,
4000 8000 16000 32000 64000 128000
as shown in Fig. 8.
Workload(MB)

In this experiment, a frame of video is regarded as a

Fig. 7. Transmission efficiency with different workloads. subtask. The sampling period of Google cluster data set
is 2 minutes [28]. For the convenience of calculation, this
experiment assumes that each time interval or round lasts
Here, the PA-EMA and uniform assignment are evaluated for 2 minutes. The normalized CPU utilization of each time
by comparing their computing time and the ERT, and the interval in the Google cluster dataset is then converted to
results are shown in Fig. 6 and Fig. 7 respectively. Given that the available CPU utilization, which is then converted to
the randomness of the results, we make ten calculations for the actual number of video frames processed per second.
each task of different amounts and draw the boxplot to show In other words, according to the empirical linear function
the results better. In Fig. 6, the MRA always shows better in face recognition application, the number of subtasks (or
performance than OTA. Especially, with the increasing of the video frames) bjt that can be processed by edge nodes j in
workload, the λ is stable which approximates to or equals 1. round t can be calculated.
Moreover, PA-EMA algorithm spends less time to finish the
The server capacity data (derived from Google dataset) in
task, which effectively reduces the computing time, as shown
this experiment is divided into two sets. One is the training
in Fig. 7. When the workload is 128000 MB, the PA-EMA
set (about 940 time intervals), which is the historical data
reduces nearly 20% compared with the uniform assignment
set. The other is working set, which is used in task allocation
and there is a trend that the gap between two algorithms is
experiment. Firstly, ARIMA model is built for the training set
widening.
data. By computing and comparing the values of Akaike In-
formation Criterion (AIC) and Bayesian Information Criterion
B. Evaluation of PA-OPT (BIC), the parameter p and q are determined to get the best
1) Experiment parameters: We let CACTA system running model. Then, the best ARIMA model is used to predict the
on the video processing application, and which generates computing capability of the nodes with a future time interval
useful system parameters to promote experimental research. of about 300 rounds. We let the value of computation cost per
Specifically, we considered an edge computing environment subtask on edge nodes follow three uniform distributions with
where a client (e.g., a Raspberry PI 3) sends via WiFi its three different average levels according to the edge nodes’
 12

computation capacities derived before. The computation cost

of a node is approximately proportional to its computation 14
PA-OPT

capacity, and its value is between 0.01 and 1. The storage cost 13 PA-EMA

Random Assignmnet
12
and data transmission cost follow the two uniform distribu-

{offline algorithm}
Short-term Greedy Assignment

11 Uniform Assignment
tions U (0.01, 0.6) and U (0.01, 0.3) respectively. Therefore, 10
FairRatio Assignment

the more powerful the computing capacity of edge nodes 9

is, the higher the computation cost is. This setting simulates 8
high-end servers charging high prices. 7

total
6
2) Predictability of edge nodes’ capacities and costs:

{alg}/C
5
We evaluate the performance in three different prediction 4
settings: 1) highly predictable setting, in which the computing 3

total
capacities and costs of edge nodes can be predicted fairly

C
2
accurately; 2) poorly predictable setting, in which it is almost 1
0
impossible to predict the capabilities and costs of nodes. 3) 1000 2000 4000 6000 8000 10000 12000
mixed setting, in which there are approximately the same Number of Frames(*1000)

number of poorly predictable and highly predictable edge

nodes. Fig. 11. Experiments with On-Off model of edge nodes.
For highly predictable setting, this paper considers an On-
Off model in which the computation capacity of edge nodes
alternates between non-zero and zero. In order to make the 14
PA-OPT

On-Off model comparable with the experiment driven by 13 PA-EMA

Random Assignmnet

Google data set, we makes the non-zero value of an edge 12

{offline algorithm}
Short-term Greedy Assignment

11
node take the average capacity value of this node in Google Uniform Assignment

FairRatio Assignment
10
data set. For poorly predictable setting, we let an edge node’s 9
computation capacity be a random number sampled from an 8
uniform distribution. Since the capacity that a node can take 7
total

will be equally likely to be any number within the range 6

{alg}/C

5
of min and max, it is difficult to predict its capacity in a
4
time slot. For mixed setting, we randomly sample a set of 3
total

100 nodes from the Google dataset, which includes some

C

2
highly predictable nodes and some poorly predictable nodes. 1
0
In addition, we choose another set of 58 nodes in Google 1000 2000 4000 6000 8000 10000 12000
dataset as a set of nodes with reasonably high predictability by Number of Frames(*1000)

calculating the Root Mean Square Error (RMSE) between bjt

and b̂jt , as their periodic CPU usage patterns can be predicted Fig. 12. Experiments with nodes of random bjt and cjt .
with reasonably high accuracy. The empirical CDFs of the
prediction errors of the two sets of Google nodes are shown
in Fig. 9 and Fig. 10. Ctotal,alg and Ctotal,of f line are the total costs of alg and
the offline optimization solution respectively. Competition
Empirical CDF Empirical CDF
1 1 ratio is a typical metric to evaluate the performance of
0.8 0.8
online algorithms. Figure 11 clearly shows that the prediction-
based optimization algorithm PA-OPT proposed in this paper
0.6 0.6
is significantly better than other algorithms in various task
F(x)

F(x)

0.4 0.4 sizes, and its cr,alg is very close to 1. When the task size
increases, the proposed PA-OPT algorithm is still close to
0.2 0.2
the optimal performance, while the performance of other
0
0 0.1 0.2 0.3 0.4 0.5
0
0 0.2 0.4 0.6 0.8 1 1.2 1.4
algorithms become worse. These results are not surprising,
x x
because using On-Off model, CACTA system can predict
the capacities/costs of edge nodes, and learn the estimated
Fig. 9. Empirical CDF of predic- Fig. 10. Empirical CDF of predic- optimal allocation strategy quite accurately. Here we use 100
tion errors of a set of 58 Google tion errors of a set of 100 Google
nodes. nodes. nodes and set α and β to be 6 and 3 respectively.
It is worth noting that PA-OPT algorithm still performs
3) Evaluation Results: Figure 11 shows the results of better than other algorithms in the setting where edge n-
On-Off model , where the x-axis represents the number of odes’ capacities/costs are poorly predictable. However if we
frames or subtasks in a video processing (face recognition) compare Figure 12 with Figure 11, we can find that the
application. Its y-axis represents the competitive ratio of performance of PA-OPT is worse in the setting with poor
the six algorithms. Note that the competition ratio of an predictability, which is reasonable, because the performance
Ctotal,alg
algorithm alg is defined as cr,alg = Ctotal,of f line
, where of PA-OPT depends on the prediction accuracy.
 13

when W = 5000. The experimental results illustrate the

importance of predictive accuracy in achieving near-optimal
3.5 PA-OPT
performance. Our proposed online prediction algorithm can
PA-EMA

3.0 Random Assignmnet

achieve near-optimal performance in the environment where
Short-term Greedy Assignment the edge node has high computation capability and cost
2.5 predictability, and its performance is not worse than other
algorithms in the setting of poor predictability. As a future
r, alg

2.0
work, we will explore the application of machine learning
C

1.5 algorithms to further improve PA-OPT algorithm.

1.0
C. Evaluation Result of QL Algorithm
0.5
According to the Q-learning based online task assignment
0.0 algorithm proposed in this paper, the performance of the
1000 2000 5000
Nunber of Frames(*1000)
algorithm is evaluated in several aspects. Firstly, we set
the environment of the system, assuming that there are 6
Fig. 13. Performance of a set of 58 Google nodes. available edge nodes in the system to provide computing
resources, and each node has different computation capability
and dynamically changes with time, that is, the computation
capability level will fluctuate in the above three levels, Here,
PA-OPT the computation capability of three nodes is stable at three
2.5 PA-EMA
Random Assignmnet different levels, respectively, and the other three nodes have
Short-term Greedy Assignment different amplitude fluctuations. In this experiment, it is
2.0
assumed that each time step is 5 min, that is, the system
receives a task request every 5 minutes, but records the
1.5
r, alg

change of the computation capability of the nodes every 1

minutes. Secondly, the value of ε in the Q-learning is set
C

1.0
to 0.9. When the random number is greater than 0.9, the
Agent will randomly select the action instead of selecting
0.5 the action that corresponds to the maximum Q value. Also,
the discount factor γ = 0.9, the learning rate α = 0.01,
0.0 and the initial values of the Q table are zeros. It is worth
1000 2000 5000
Nunber of Frames(*1000)
noting that the algorithm is aimed at video analysis tasks.
The computation capability and task size discussed in this
Fig. 14. Performance of a set of 100 Google nodes. chapter are all in frames. In the process of the experiment,
the relationship between the CPU utilization and the number
of processed frames in Fig. 8 is used as a reference for
This paper further conducts experiments based on the set of the conversion between the computation capability level and
58 Google nodes with fairly reasonable predictability of ca- the number of processed frames, and a regression equation
pacities/costs and the mixed predictability set of 100 Google is obtained. In addition, assuming that each task has the
nodes. The performance comparison of four algorithms using same request for resources, which is 0.5 computing units,
these two sets are shown in Fig. 13 and Fig. 14. Uniform and that is, 0.5 computation capability. When the computing
Fair Ratio algorithms are not shown here as their competitive resources are fully occupied, the node will run with the lowest
ratios are very high. computation capability. Similarly, the cost required per frame
First of all, it is noted that PA-OPT achieves the best is proportional to the actual computation capability.
competition ratio among all algorithms for a set of 58 nodes. As shown in Fig. 15, when there are tasks with six different
Interestingly, when the task size is 1000, the performance sizes coming in different time slots, all the task completion
of PA-EMA and random algorithm is not much worse than time and computation cost are obtained by adjusting the
that of PA-OPT. This is because that even the aggregate weighting factor κ. It can be clearly seen from the figure
prediction errors of the 58 nodes is low, there is still a that as κ increases, that is, the higher the delay requirement,
large predictability randomness among nodes. When task size the actual task completion time obtained according to the
gets larger (W = 5000), the performance of PA-OPT gets QL algorithm allocation result is continuously reduced, and
further improved. For a mixed environment (i.e., the set of the system cost is gradually increased. The completion time
100 nodes), the four dynamic algorithms show approximately and system cost are almost equal when κ = 0.4. It can be
the same competitive ratio when W = 1000, our PA-OPT seen that the Agent effectively learned the feedback from the
algorithm still performs better than other algorithms, but it environment and proved the rationality of the return function
performs not well when W = 2000 and W = 5000. The setting. Since the learning effect of the Agent is related to the
Greedy algorithm’s performance gets degraded significantly number of iterations of the training, this section compares the
 14

600 3500
QL Time
Completion Time
QL Cost
System Cost
3000

(k=0.5)
Random Time
500 Random Cost

2500 Round-Robin Time

Round Robin Cost

Objective Function Value

QL Algorithm

400
2000

1500
300

1000

200
500

100 0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 30 45 60 75 90 105 120 135 150
k Number of Tasks

Fig. 15. The completion time and the system cost vary along weighting Fig. 17. Objective function value of the three algorithms for different
factor κ in QL algorithm. number of tasks.

The experimental results are shown in Fig. 17. The height

190
of the bar of different colors in the figure indicates the target
value of the corresponding algorithm, and each column bar is
185
(k=0.5 )

composed of two colors, which represent the completion time

and system cost. Obviously, as the number of tasks increases,
180
Objective Function Value

the target value gradually increases. Here, the proportion

of completion time and system cost of the QL algorithm
175
are basically similar, while the system cost of the random
algorithm and the Round-Robin algorithm is close to 80%,
170
which is not reasonable. Most importantly, the QL algorithm
165
outperforms other algorithms in terms of the number of tasks,
especially when the number of tasks is 75, the QL algorithm is
160
optimized by nearly 25% compared to the random algorithm.
0 50 100 150 200 250 300 350 400 450 500 550 This shows that the QL algorithm is more fully learned under
Number of Iterations
this setting. The second set of experiments compares the
Fig. 16. Objective function value vary along the number of iterations.
performance of the three algorithms with different κ values
when the number of tasks is 90. The experimental results are
shown in Fig. 18. Since the allocation method of the random
algorithm and the round-robin algorithm is not affected by
objective function value obtained by changing the iteration
the value of κ, the two algorithms change linearly with the
number in the case of κ = 0.5, as shown in Fig. 16. As can
increase of κ. The change of QL algorithm is close to a
be seen in this figure, when the number of iterations is 100,
smooth curve, although it gradually decreases with the change
the target value drops to around 177 and tends to be stable,
of κ, but the value is slightly higher than the other two
so the number of iterations of the following experiments is
algorithms from κ = 0.6 to κ = 0.9, indicating that the setting
100. However, in this experiment, as the number of iterations
of the simulation experiment data and the return function need
increases, the change in the target value is not obvious, which
further improvement.
may be caused by insufficient learning or a unreasonable
setting of values.
Next, the QL algorithm proposed in this paper is compared VII. C ONCLUSION
with the other two algorithms, namely Random Algorithm In this paper, a CACTA edge computing system is pro-
and Round-Robin Algorithm. The random algorithm indicates posed. Based on the three characteristics of edge node high
that the selection of the nodes of the assigned task obeys a dynamics, high mobility and heterogeneity, an EMA-based
uniform distribution, and the round-robin algorithm indicates multi-round algorithm is designed. This is the first situa-
that the tasks are sequentially assigned to the respective nodes tion where the long-term historical data of system can not
in order, and the number of the computing nodes is one cycle be obtained. Secondly, this paper designs a dynamic task
of the round-robin algorithm. Both of the above algorithms do allocation algorithm based on prediction to provide near-
not consider the resource state and load balancing of the node. optimal performance to end users. Based on the assumption
The first set of experiments compares the performance of the that the computation capability and cost of all nodes are fully
three algorithms for different number of tasks when κ = 0.5. understood, the solution of nonlinear optimization problem is
 15

[6] Z. Li and J. Gui, “Energy-efficient resource allocation with hybrid

3000
tdma–noma for cellular-enabled machine-to-machine communications,”
QL Algorithm
IEEE Access, vol. 7, pp. 105 800–105 815, 2019.
Random Algorithm
[7] S. Wan, Y. Zhao, T. Wang, Z. Gu, Q. H. Abbasi, and K.-K. R.
2500 Round Robim Algorithm Choo, “Multi-dimensional data indexing and range query processing via
voronoi diagram for internet of things,” Future Generation Computer
Objective Function Value

Systems, vol. 91, pp. 382–391, 2019.

2000
[8] M. Wazid, A. K. Das, R. Hussain, G. Succi, and J. J. Rodrigues,
“Authentication in cloud-driven iot-based big data environment: Survey
1500 and outlook,” Journal of Systems Architecture, vol. 97, pp. 185–196,
2019.
[9] M. Huang, W. Liu, T. Wang, A. Liu, and S. Zhang, “A cloud-mec
1000
collaborative task offloading scheme with service orchestration,” IEEE
Internet of Things Journal, 2019.
500 [10] J. Luo, X. Deng, H. Zhang, and H. Qi, “Qoe-driven computation of-
floading for edge computing,” Journal of Systems Architecture, vol. 97,
pp. 34–39, 2019.
0 [11] J. Gui, X. Dai, and X. Deng, “Stabilizing transmission capacity in
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1
millimeter wave links by q-learning-based scheme,” Mobile Information
k Systems, vol. 2020, 2020.
[12] M. Chen, T. Wang, K. Ota, M. Dong, M. Zhao, and A. Liu, “Intelligent
Fig. 18. Objective function values of the three algorithm vary along resource allocation management for vehicles network: An a3c learning
weighting factor κ. approach,” Computer Communications, 2020.
[13] Y. Liu, Z. Zeng, X. Liu, X. Zhu, and M. Z. A. Bhuiyan, “A novel load
balancing and low response delay framework for edge-cloud network
based on sdn,” IEEE Internet of Things Journal, 2019.
derived. Then the paper introduces the PA-OPT algorithm, [14] X. Xu, Y. Xue, L. Qi, Y. Yuan, X. Zhang, T. Umer, and S. Wan, “An
which uses the ARIMA time series method to predict the edge computing-enabled computation offloading method with privacy
computational capabilities/costs of the edge nodes, and further preservation for internet of connected vehicles,” Future Generation
Computer Systems, vol. 96, pp. 89–100, 2019.
dynamically learns the laws of historical data to further [15] J. Wang, J. Hu, G. Min, W. Zhan, Q. Ni, and N. Georgalas, “Computa-
improve the PA-OPT algorithm. Then, an online assignment tion offloading in multi-access edge computing using a deep sequential
algorithm based on Q learning is proposed to minimize model based on reinforcement learning,” IEEE Communications Mag-
azine, vol. 57, no. 5, pp. 64–69, 2019.
the completion time and system cost. In this paper, the [16] B. Shojaiemehr, A. M. Rahmani, and N. N. Qader, “Automated negoti-
online learning allocation algorithm based on reinforcement ation for ensuring composite service requirements in cloud computing,”
learning is discussed and the effectiveness of the algorithm Journal of Systems Architecture, vol. 99, p. 101632, 2019.
is proved. In a word, this paper makes an in-depth study [17] S. Wan, Z. Gu, and Q. Ni, “Cognitive computing and wireless com-
munications on the edge for healthcare service robots,” Computer
on the scheduling strategy between edge nodes, and designs Communications, 2019.
multiple task allocation algorithms according to different [18] D. Zhang, L. Tan, J. Ren, M. K. Awad, S. Zhang, Y. Zhang, and
actual situations, and has achieved good experimental results, P.-J. Wan, “Near-optimal and truthful online auction for computation
offloading in green edge-computing systems,” IEEE Transactions on
in order to provide reference for future research work. Mobile Computing, 2019.
[19] D. Zhang, Z. Chen, M. K. Awad, N. Zhang, H. Zhou, and X. S.
Shen, “Utility-optimal resource management and allocation algorithm
ACKNOWLEDGMENT for energy harvesting cognitive radio sensor networks,” IEEE Journal
on Selected Areas in Communications, vol. 34, no. 12, pp. 3552–3565,
The authors gratefully acknowledge support from National 2016.
Natural Science Foundation of China projects of grant No. [20] E. Liu, X. Deng, Z. Cao, and H. Zhang, “Design and evaluation of
61772553, 61379058 and grant No.GCX20190884Y from a prediction-based dynamic edge computing system,” in 2018 IEEE
Global Communications Conference (GLOBECOM). IEEE, 2018, pp.
Central South University. This research was also supported 1–6.
in part by USA NSF grant CNS-1562264. The information [21] H. Zhang, B. Liu, H. Susanto, G. Xue, and T. Sun, “Incentive mech-
reported here does not reflect the position or the policy of the anism for proximity-based mobile crowd service systems,” in IEEE
federal government of USA. INFOCOM 2016.
[22] W. Lin, J. Lei, D. Kliazovich, and P. Bouvry, “Reconciling task
assignment and scheduling in mobile edge clouds,” in HotPNS, 2016.
R EFERENCES [23] C. Wang, C. Liang, F. R. Yu, Q. Chen, and L. Tang, “Computation
offloading and resource allocation in wireless cellular networks with
[1] C. V. Networking, “Cisco global cloud index: Forecast and methodol- mobile edge computing,” IEEE Trans. Wirel. Commun, vol. 16, no. 8,
ogy, 2016–2021,” White paper. Cisco Public, San Jose, 2016. 2017.
[2] P. Guan, X. Deng, Y. Liu, and H. Zhang, “Analysis of multiple clients [24] C. Lo, Y. Y. Su, C. Y. Lee, and S. C. Chang, “A dynamic deep neural
behaviors in edge computing environment,” IEEE Transactions on network design for efficient workload allocation in edge computing,”
Vehicular Technology, vol. 67, no. 9, pp. 9052–9055, 2018. in ICCD 2017.
[3] A. Yousefpour, C. Fung, T. Nguyen, K. Kadiyala, F. Jalali, A. Ni- [25] S. Yi, Z. Hao, Q. Zhang, Q. Zhang, W. Shi, and Q. Li, “Lavea: Latency-
akanlahiji, J. Kong, and J. P. Jue, “All one needs to know about fog aware video analytics on edge computing platform,” in ICDCS 2017.
computing and related edge computing paradigms: A complete survey,” IEEE, 2017.
Journal of Systems Architecture, 2019. [26] L. Wang, L. Jiao, J. Li, and M. M. Muhlhauser, “Online resource
[4] H. Shen, G. Bai, Y. Hu, and T. Wang, “P2ta: Privacy-preserving task allocation for arbitrary user mobility in distributed edge clouds,” in
allocation for edge computing enhanced mobile crowdsensing,” Journal ICDCS 2017, June 2017.
of Systems Architecture, 2019. [27] K. Habak, E. W. Zegura, M. Ammar, and K. A. Harras, “Workload
[5] Z. Kuang, G. Liu, G. Li, and X. Deng, “Energy efficient resource management for dynamic mobile device clusters in edge femtoclouds,”
allocation algorithm in energy harvesting-based d2d heterogeneous in ACM/IEEE Symposium, 2017.
networks,” IEEE Internet of Things Journal, vol. 6, no. 1, pp. 557– [28] C. Reiss, J. Wilkes, and J. L. Hellerstein, “Google cluster-usage traces:
567, 2018. format+ schema,” Google Inc., White Paper, 2011.
 16

[29] X. Chen, H. Zhang, C. Wu, S. Mao, Y. Ji, and M. Bennis, “Performance Xiaoheng Deng received the Ph.D. degrees in
optimization in mobile-edge computing via deep reinforcement learn- computer science from Central South University,
ing,” in 2018 IEEE 88th Vehicular Technology Conference (VTC-Fall). Changsha, Hunan, P.R. China, in 2005. Since 2006,
IEEE, 2018, pp. 1–6. he has been an Associate Professor and then a Full
[30] C. You, K. Huang, H. Chae, and B.-H. Kim, “Energy-efficient resource Professor with the Department of Electrical and
allocation for mobile-edge computation offloading,” IEEE Tran. on Communication Engineering, Central South Uni-
Wireless Communications, vol. 16, no. 3, 2017. versity. He is the Chair of RS Changsha Chapter, a
[31] J. Liu, Y. Mao, J. Zhang, and K. B. Letaief, “Delay-optimal compu- senior member of CCF, a member of CCF Pervasive
tation task scheduling for mobile-edge computing systems,” in 2016 Computing Council, a member of IEEE and ACM.
IEEE International Symposium on Information Theory (ISIT). IEEE, He has been a chair of CCF YOCSEF CHANG-
2016, pp. 1451–1455. SHA from 2009 to 2010. His research interests
[32] B. Cao, L. Zhang, Y. Li, D. Feng, and W. Cao, “Intelligent offloading in include wireless communications and networking, congestion control for
multi-access edge computing: A state-of-the-art review and framework,” wired/wireless network, cross layer route design for wireless mesh network
IEEE Communications Magazine, vol. 57, no. 3, pp. 56–62, 2019. and ad hoc network, online social network analysis, edge computing.
[33] I. Ketykó, L. Kecskés, C. Nemes, and L. Farkas, “Multi-user com-
putation offloading as multiple knapsack problem for 5g mobile edge
computing,” in EuCNC 2016. IEEE, 2016.
[34] A.-C. Pang, W.-H. Chung, T.-C. Chiu, and J. Zhang, “Latency-driven
cooperative task computing in multi-user fog-radio access networks,”
in ICDCS 2017. IEEE, 2017.
[35] X. Ma, S. Zhang, W. Li, P. Zhang, C. Lin, and X. Shen, “Cost-efficient
workload scheduling in cloud assisted mobile edge computing,” in
IWQoS 2017. IEEE, 2017. Jun Li is a M.Sc. student in school of Computer
[36] D. Sabella, V. Sukhomlinov, L. Trang, S. Kekki, P. Paglierani, R. Ross- Science and Engineering of Central South Universi-
bach et al., “Developing software for multi-access edge computing,” ty, Changsha, China. She received the B.Sc. degrees
ETSI White Paper, vol. 20, 2019. in electronic information engineering from Hunan
[37] S. Bi and Y. J. Zhang, “Computation rate maximization for wireless Normal University, Changsha, China, in 2018. Her
powered mobile-edge computing with binary computation offloading,” major research interests are wireless network and
IEEE Transactions on Wireless Communications, vol. 17, no. 6, pp. edge computing.
4177–4190, 2018.
[38] V. De Maio and I. Brandic, “First hop mobile offloading of dag
computations,” in Proceedings of the 18th IEEE/ACM International
Symposium on Cluster, Cloud and Grid Computing. IEEE Press, 2018,
pp. 83–92.
[39] J. P. A. Neto, D. M. Pianto, and C. G. Ralha, “Mults: A multi-
cloud fault-tolerant architecture to manage transient servers in cloud
computing,” Journal of Systems Architecture, vol. 101, p. 101651, 2019.
[40] B. Cai and K. Li, “Slo-aware colocation: Harvesting transient resources
from latency-critical services,” Journal of Systems Architecture, vol.
101, p. 101663, 2019.
[41] J. Real, S. Sáez, and A. Crespo, “A hierarchical architecture for time- Enlu Liu received the M.Sc. degrees in information
and event-triggered real-time systems,” Journal of Systems Architecture, and communication engineering from Central South
vol. 101, p. 101652, 2019. University, Changsha, China, in 2019. Her major
[42] S. Di, D. Kondo, and W. Cirne, “Host load prediction in a google research interests are wireless network and edge
compute cloud with a bayesian model,” in ICHPC, 2012. computing.
[43] D. Yang, J. Cao, C. Yu, and J. Xiao, “A multi-step-ahead cpu load
prediction approach in distributed system,” in CGC 2012.
[44] M. Farahat and M. Talaat, “Short-term load forecasting using curve
fitting prediction optimized by genetic algorithms,” IJEE, vol. 2, no. 2,
2012.
[45] Q. Zhang, M. F. Zhani, S. Zhang, Q. Zhu, R. Boutaba, and J. L.
Hellerstein, “Dynamic energy-aware capacity provisioning for cloud
computing environments,” in Proceedings of the 9th international
conference on Autonomic computing. ACM, 2012, pp. 145–154.
[46] docker. http://www.docker.com. Accessed: 2017-03-10.
[47] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G.
Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski
et al., “Human-level control through deep reinforcement learning,”
Nature, vol. 518, no. 7540, p. 529, 2015. Honggang Zhang holds a PHD in Computer Sci-
[48] T. Li, Z. Xu, J. Tang, and Y. Wang, “Model-free control for distributed ence (2006) from the University of Massachusetts,
stream data processing using deep reinforcement learning,” Proceedings Amherst, USA. He received his BS degree from
of the VLDB Endowment, vol. 11, no. 6, pp. 705–718, 2018. Central South University of China, and his MS
[49] H. Mao, M. Alizadeh, I. Menache, and S. Kandula, “Resource man- degree from Tianjin University of China. He also
agement with deep reinforcement learning,” in Proceedings of the 15th received a MS degree from Purdue University, West
ACM Workshop on Hot Topics in Networks. ACM, 2016, pp. 50–56. Lafayette, IN, USA. He is currently an Associate
[50] H. Mao, R. Netravali, and M. Alizadeh, “Neural adaptive video Professor of Computer Engineering in the Engi-
streaming with pensieve,” in Proceedings of the Conference of the ACM neering Department at University of Massachusetts
Special Interest Group on Data Communication. ACM, 2017, pp. 197– Boston, Boston, MA, USA. His research interests
210. span a wide range of topics in the area of computer
networks and distributed systems. His current research focuses primarily
on Edge computing, Mobile Computing, and Internet of Things. He was
a recipient of the National Science Foundation (NSF) CAREER Award in
2009.