Feature Article: DOI. No. 10.1109/MAES.2020.

3043072

Artificial Intelligence and Data Fusion at the Edge

Arslan Munir , Kansas State University, Manhattan, KS 66506 USA
Erik Blasch , Air Force Office of Scientific Research, Arlington, VA 22203 USA
Jisu Kwon and Joonho Kong , Kyungpook National University, Daegu 41566,
South Korea
Alexander Aved , Air Force Research Laboratory Information Directorate,
Rome, NY 13441 USA

the Vice-President (VP) of Special Projects in Arm’s

INTRODUCTION
Machine Learning Group [2]: “The edge is the next stage of
Advancements and miniaturization of electronic devices the evolution of AI technology because of the physical con-
that are not only able to sense and process data but can also straints, the cost constraints, and the practical constraints of
communicate with other devices have engendered an era of running all AI applications in the cloud. It simply does not
Internet of Things (IoT). These IoT devices are often make sense to send all the bits for things like video and audio
equipped with a multitude of sensors and generate zillions streaming to the cloud and back down for every situation,
bytes of data at the network edge. Many of the applications every endpoint.”
that need to utilize this data have embraced artificial intelli- In the age of AI, what is important is not the data alone
gence (AI) and machine learning (ML). However, transfer- but what we can do with this data and how to make sense of
ring this gigantic data to the cloud is often infeasible due to this data. As more and more data are being collected at the
limited network bandwidth and real-time constraints of network edge due to the proliferation of IoT and sensing
many applications including aerospace systems. Thus, there devices, the capability gap to make sense of this gigantic
is a desire to push AI frontiers to the network edge to utilize data, whether at the edge or the cloud, in a timely manner, is
the enormous amount of data generated by IoT devices also increasing. For instance, in surveillance applications,
nearer to the data source. This desire has led to the merger of the number of traditional sensors (e.g., ground radar and
edge computing (a novel trend in computing that pushes cameras), nontraditional sensors (e.g., IoT), and nonorganic
computing power away from the centralized nodes to the airborne platforms (e.g., unmanned aerial systems) has
logical extreme edges of a network [1]) and AI, resulting in a increased the opportunity to detect, track, and identify tar-
new discipline—AI at the edge or edge intelligence. In gets, as well as to counter threats; however, there is a lack of
the edge AI model, AI computations take place either on the processing capability to do so efficiently and effectively [3].
user device or somewhere in the network stack beneath the It is noted that for many applications, much of the collected
cloud, perhaps on an edge server. According to Steve Roddy, data is time sensitive and become useless if not utilized
timely. Hence, solutions such as data fusion are of para-
Authors’ current address: Arslan Munir, Department mount significance to enhance the effectiveness and usage
of Computer Science, Kansas State University, of sensed data in a timely manner. Data fusion is defined as
Manhattan, KS 66506 USA (e-mail: famunir@ksu. the process of combining data from multiple sources to pro-
edu). Erik Blasch, Air Force Office of Scientific duce more accurate, consistent, and concise information
Research (AFOSR) Arlington, VA 22203-1768 USA (e- than that provided by any individual data source.
mail: [email protected]). Jisu Kwon, Joonho The concept of both data fusion and AI has biological
Kong, School of Electronic and Electrical Engineering, origins. Data fusion is inspired from the capability of
Kyungpook National University, Daegu 41566, South advanced biological organisms to assimilate information
Korea (e-mail: [email protected], joonho.kong@knu. from multiple senses (e.g., sight, touch, smell, and taste)
ac.kr). Alex Aved, Air Force Research Laboratory
to make better sense of environment and increase their
(AFRL), Information Directorate, Rome, NY 13441
chances of survival. AI is also inspired from the cognition
USA (e-mail: [email protected]).
Manuscript received March 1, 2020, revised September displayed by advanced biological organisms (e.g.,
10, 2020; accepted November 14, 2020, and ready for humans). Both data fusion and AI are complementary to
publication December 3, 2020. enable machines to accomplish various intelligent tasks
Review handled by Chee-Yee Chong. and missions. Whether the AI computations are performed
0885-8985/21/$26.00 ß 2021 IEEE at the cloud or at the network edge, data fusion is needed

62 IEEE A&E SYSTEMS MAGAZINE JULY 2021

Authorized licensed use limited to: Beijing Normal University. Downloaded on August 06,2021 at 08:22:29 UTC from IEEE Xplore. Restrictions apply.
 Credit: Image licensed by Ingram Publishing

to provide more concise and consistent data to the AI both proposed framework, we provide a brief overview of AI
at the training and inference phases. This article focuses and ML and elaborate on how data fusion enhances AI. Con-
on AI at the edge and illustrates the benefits of integrating temporary AI is dominated by ML. Multitude of ML meth-
data fusion with AI at the edge. ods, which can be categorized under supervised learning,
Our main contributions in this article are as follows. semisupervised learning, unsupervised learning, and rein-
forcement learning (RL), help provide edge intelligence.
(1) We propose a framework for AI and data fusion at
Deploying an ML model requires first training the model
the edge.
and then using the trained model to perform inference. Since
(2) We provide a comparativ‘e discussion of different training is a more resource-intensive and time-consuming
data fusion and AI architectures. process; in the edge AI model, training is typically done on
(3) We discuss multiple levels of fusion and different the cloud and inference is performed at the edge. Although
types of AI, and how different types of AI relate to AI training can be performed at edge [4], given the limited
and align with different levels of fusion. resources of many edge devices, AI training will likely con-
tinue on powerful cloud-based computers, whereas infer-
(4) We highlight and elaborate the advantages of AI ence will be performed at edge devices. Among the existing
and data fusion at the edge including latency, ML methods, deep learning provides magnificent perfor-
energy efficiency, precision, security, privacy, cost, mance on various tasks. For example, convolutional neural
scalability, and sustainability. networks (CNN) have been utilized for image classification,
(5) We present experimental results for latency, energy object detection, and recognition. Recurrent neural networks
efficiency, and accuracy for AI and data fusion and (RNNs) find applications in natural language processing and
multitarget tracking. Deep reinforcement learning (DRL)
AI without data fusion to demonstrate the advan-
can be leveraged for determining optimal policy/strategy for
tages of AI and data fusion at the edge.
accomplishing various tasks, such as trajectory optimization
The remainder of this article is organized as follows. of autonomous vehicles. Regardless of any specific AI/ML
Section “Framework for Data Fusion and AI at the Edge” methods, edge computing provides a means for perfor-
presents a framework for data fusion and AI at the edge. mance- and energy-efficient execution of AI/ML algorithms.
Section “Models and Architectures for Data Fusion and AI” Figure 1 depicts our proposed framework for data
provides a comparative discussion of contemporary data fusion and AI at the edge. In the proposed framework,
fusion and AI architectures. Section “Alignment of Multi- data fusion, AI/ML processing, analysis, and decision-
level Fusion and AI” delineates multiple levels of fusion making are done at the following three levels:
and different types of AI, and also provides a mapping of dif-
(1) edge-of-network sensor/IoT nodes;
ferent AI and ML types to different fusion levels. Advan-
tages of data fusion and AI at the edge are discussed in (2) edge servers or fog nodes;
section “Advantages of Data Fusion and AI at the Edge.” (3) cloud servers.
Experimental results are presented in section “Experimental
Results.” Finally, section “Conclusions” concludes this AI at both the edge-of-network sensor/IoT devices and
article. edge servers (fog nodes) is referred to as edge AI. Edge AI
enables computations near the edge of the network and helps
in reducing the communication burden on the core network.
FRAMEWORK FOR DATA FUSION AND AI AT THE EDGE We distinguish the edge AI done at senor/IoT node level and
In this section, a high-level framework for data fusion and edge server (fog node) level to highlight the difference in
AI/ML processing at the edge is developed. To explain the compute capability of the two. In the proposed framework,

JULY 2021 IEEE A&E SYSTEMS MAGAZINE 63

Authorized licensed use limited to: Beijing Normal University. Downloaded on August 06,2021 at 08:22:29 UTC from IEEE Xplore. Restrictions apply.
 Artificial Intelligence and Data Fusion at the Edge

Figure 1.
Framework for data fusion and AI at edge.

AI is first performed at the lowest tier of the network edge network sensors/IoT devices. Since edge servers receive
comprising of senor nodes and IoT devices. These edge-of- data from many edge-of-network devices with diverse
network sensor/IoT devices can be sensor nodes sensing par- application requirements, performance requirements from
ticular features (e.g., temperature, humidity, and pressure), edge server are also diverse. When AI/ML processing with
smart phones, smart vehicles, video and imaging devices stringent latency requirements (e.g., for hard real-time sys-
(e.g., cameras including night vision imaging cameras and tems) is required from edge servers, edge servers are
thermal imaging cameras), and even airborne vehicles, such designed with high-performance stationary computing serv-
as unmanned aerial vehicles (UAVs), equipped with differ- ers with stable power source and high bandwidth network
ent sensors. These edge-of-network IoT devices typically connections. For mobile edge processing, edge servers can
possess limited computation and communication capabili- be power/energy constrained. For example, UAVs, which
ties. Given the increasing proliferation of AI-driven applica- are typically powered by batteries, can be used as mobile
tions, modern IoT devices can be outfitted with various ML edge servers or devices. These mobile edge devices often
accelerators to speed up the execution of ML algorithms in employ AI accelerators for fast and energy-efficient AI/ML
an energy-efficient manner as depicted in Figure 1. For processing [7], [8]. Moreover, edge servers or fog nodes are
example, IoT nodes equipped with cameras can execute more amenable for integration of ML accelerators, such as
hardware-accelerated lightweight CNN algorithms, such as CNN, RNN, and MLP accelerators as depicted in Figure 1.
MobileNet [5], to perform object classification. Each edge server manages a cluster of edge-of-network
Although modern and futuristic IoT devices can be sensors/IoT devices in its vicinity. Edge servers provide
equipped with AI accelerators [6] as depicted in Figure 1, applications, content, context, services, and storage to edge-
many of the contemporary edge-of-network sensors/IoT of-network IoT devices. For example, edge servers can
devices have limited computational capabilities that are not assist with complex image processing and storage tasks for
adequate to carry out complex AI tasks. In order to boost imaging data acquired from camera sensors at IoT devices.
the computational and AI capabilities at the network edge, Since edge servers are in close proximity to edge-of-network
edge servers are installed at the base stations in the edge IoT devices, the edge servers can perform the offloaded
computing paradigm. Edge servers possess much more tasks from IoT devices with much lower latency and much
resources and computational capability than the edge-of- lower communication overhead as compared to the cloud.

64 IEEE A&E SYSTEMS MAGAZINE JULY 2021

Authorized licensed use limited to: Beijing Normal University. Downloaded on August 06,2021 at 08:22:29 UTC from IEEE Xplore. Restrictions apply.
 Munir et al.

The edge servers in our framework are connected to the servers or fog nodes can offload their tasks to cloud. Since
top-tier centralized cloud server layer through the core net- many of the AI inference tasks are time sensitive, compu-
work. The core network consigns locally processed data tation offloading from an IoT device to another edge-of-
and information from the edge to the cloud for various pur- network device dE is advantageous if
poses, such as analytics, archival, and decision making at a
IoTdE
broader scale. At the cloud, high-performance servers are TpIoT þ TqIoT > TpdE þ TqdE þ Tt (1)
typically used because the cloud needs to service many
requests from the edge-of network devices and edge serv- where TpIoT and TqIoT represent the average processing
ers. Cloud servers are typically equipped with many high- latency and average queuing delay, respectively, at the
end central processing units (CPUs) (e.g., Intel Xeon pro- IoT device, TpdE and TqdE denote the average processing
cessor [9]) and graphic processing units (GPUs) for data- latency and average queuing latency, respectively, at dE ,
IoT-dE
base processing and AI/ML processing. For efficient and Tt denotes the transmission latency from an IoT
execution of AI tasks, the cloud is often outfitted with a device to another edge-of-network device dE for sending
variety of AI accelerators and neural processing units as the data for offloaded computation and receiving the
depicted in Figure 1. As an example, Google is currently results back from dE . Similarly, the computation offload-
deploying a proprietary AI accelerator called tensor proc- ing from an IoT device to an edge server SE is expedient if
essing unit (TPU) [10] to accelerate the AI/ML workloads IoTSE
TpIoT þ TqIoT > TpSE þ TqSE þ Tt (2)
in their data centers. The TPU is designed and implemented
as an application-specific integrated circuit (ASIC), which where TpSE and TqSE denote the average processing latency
is not reconfigurable after the chip fabrication. As another and average queuing latency, respectively, at SE , and
IoT-SE
example, Microsoft uses field-programmable gate arrays Tt denotes the transmission latency from an IoT
(FPGAs) for accelerating AI/ML workloads in their cloud device to an edge server SE for sending the data for off-
servers [11]. Since FPGAs are reconfigurable, the accelera- loaded computation and receiving the results back from
tor logic can be flexibly changed depending on users’ or SE . The auspiciousness of offloading from an edge server
service providers’ requirements. to the cloud can be expressed similar to (1) and (2) and is
In the proposed framework, data fusion plays an omitted for brevity.
important role along with AI as depicted by the data fusion
blocks at each hierarchical tier in Figure 1. Here, we pro-
vide a high-level discussion of data fusion at different tiers MODELS AND ARCHITECTURES FOR DATA FUSION
of the proposed framework, whereas the detailed discus-
sion of data fusion that occurs within data fusion blocks in
AND AI
Figure 1 is provided in Section “Models and Architectures There exist many models and architectures for data fusion
for Data Fusion and AI.” At the lowest tier, data fusion is to address plethora of issues surrounding human factors,
performed at the IoT node level to minimize the redun- AI, and IoT [19]. Table 1 presents seven major data fusion
dancy in the raw data acquired from sensors. The IoT models and/or architectures and compares them across dif-
devices then perform AI on this fused data to help improve ferent metrics, viz., ability to perform (static) data fusion,
accuracy, performance, and energy efficiency in carrying (dynamic) real-time sensing, capability of operating with
out the AI tasks. The IoT nodes transmit only the sanitized human-in-the-loop, applicability to IoT and/or cyber–
and fused data, and the analytics results on this fused data physical systems (CPS), handling of AI (including big
to the edge servers instead of sending the raw sensor data, data), and centralized or distributed processing. While
which enormously reduces the load on communication Table 1 highlights traditional data fusion models and
network and also conserves the communication/transmis- architectures with their current capabilities, it is to be
sion energy at IoT devices. The edge servers then fuse the noted that with new advances, these architectures can be
data received from multiple IoT devices. The edge servers redefined and enhanced toward recent methods, for exam-
also resolve the topological relationships between sensors ple, static to dynamic processing, centralized to distributed
and utilize the topological, contextual, and environmental processing, human–machine teaming, and small data (1/
information in data fusion. The edge servers perform AI 0s) to large data (streaming video).
on this fused data and then report the sanitized and fused Typically, the aerospace community focuses on two
data as well as analytics on this fused data to the cloud. main data fusion models or architectures. The first one lever-
The cloud performs the data fusion on the data received ages multiple filters in a centralized approach that has been
from the network edge and then performs AI on this fused widely extended to distributed methods and applied to object
data to obtain global analytics and insights. assessment for target tracking and distributed target recogni-
In the proposed framework, an IoT edge device can tion [20]. A good example of this type of data fusion archi-
computationally offload its data fusion and AI tasks to tecture is the distributed information graph (DIG). The
other edge devices or an edge server. Similarly, edge second one brings together big data to a user-defined

JULY 2021 IEEE A&E SYSTEMS MAGAZINE 65

Authorized licensed use limited to: Beijing Normal University. Downloaded on August 06,2021 at 08:22:29 UTC from IEEE Xplore. Restrictions apply.
 Artificial Intelligence and Data Fusion at the Edge

Table 1.

Comparison of Data Fusion and AI Architectures Symbols: @—Yes; $—Being Used; n—Been Adapted; t
u—Could
Be Adapted; •—Not Used
Model and/or Data Fusion Real-Time Human IoT/ AI Connected Reference
Architecture (Static) (Sensing) Centered CPS (Big (Distributed)
Data)
DFD @ $ u
t • n u
t Dasarathy[12]

DIG $ @ n $ u
t @ Hall et al.[13]

WSN $ @ • n u
t @ Chair and
Varshney[14]
ELF $ u
t • • @ $ Snoek et al.[15]

GAN $ • • • @ • Goodfellow et al.[16]

JDL/DFIG @ $ @ u
t $ $ Blasch et al.[17]

CPCG @ @ @ @ @ @ Cruise et al.[18]

“yes” indicates that (yes) research supports the directions/usage while “used” implies that it has been implemented in real systems.

operating picture (UDOP) in the Joint Directors of Laborato- 1, 2 and 3) and refinement (level 4, 5, and 6). System man-
ries (JDL)/Data Fusion Information Group (DFIG) model agement (level 6) incorporates contextual constraints
[17]. We note that the UDOP concept extends the common based on mission, objectives, and goals. The DFIG model
operating picture to help enable the human operator to super- describes different levels of fusion as follows.
vise the processing, exploitation, and dissemination of infor- Level 0—Data Assessment. Provides estimation and
mation for situation awareness. The UDOP enables prediction of signal/object observable states based on
rendering and visualization of data analytics services cus- pixel/signal level data association.
tomized to the human operator’s needs for efficient com- Level 1—Object Assessment. Provides estimation and
mand decision making for a given mission. While the DFIG prediction of entities/objects based on data association
model supports multiple sensors and distributed users, it still and (both continuous and discrete) state estimation.
relies on a common location for command and control. Other Level 2—Situation Assessment. Provides estimation
communities have been looking at fusion architectures that and prediction of relations between entities/objects.
emphasize fusion-focused data analysis, such as the data- Level 3—Impact Assessment. Provides estimation and
feature-decision (DFD) [12] model, wireless sensor net- prediction of impact on planned or estimated actions by
works [14], and the more recent efforts from the AI commu- the participants.
nity with the early and late fusion (ELF) methods [15] and Level 4—Process Refinement. An element of resource
generative adversarial networks (GANs). Since 2000, the management and provides adaptive data acquisition and
DFD and the JDL/DFIG have remained similar, while the processing to furnish estimation of sensing objectives and
WSN community has continually utilized the distributed prediction of impact on planned or estimated actions by
approaches that extend to IoT and CPS. While each of these the participants.
methods are important for the fusion community, there is a
desire for a data fusion architecture that is able to perform
distributed command and control amongst a variety of
human teams and distributed sensors. Hence, in this section,
we elaborate cyber–physical command guided (CPCG)
architecture that is able to perform distributed command and
control amongst various human teams and distributed sen-
sors and IoT devices. We further briefly discuss DFIG model
as it help provide an understanding of fusion levels.

DATA FUSION INFORMATION GROUP (DFIG) MODEL

Figure 2 depicts a JDL/DFIG model. The JDL/DFIG
model defines multiple levels of fusion where each level Figure 2.
exploits AI developments to support assessment (level 0, DFIG model for information fusion (IF).

66 IEEE A&E SYSTEMS MAGAZINE JULY 2021

Authorized licensed use limited to: Beijing Normal University. Downloaded on August 06,2021 at 08:22:29 UTC from IEEE Xplore. Restrictions apply.
 Munir et al.

Figure 3.
CPCG architecture.

Level 5—User Refinement. An element of knowledge distributed space. The AI agents of CPCG architecture are
management and provides adaptive determination of assisted with three different types of data fusion, viz., IF,
access control and display of information to support deci- OI, and CD. Organization of these three data fusion agents
sion making via human–machine interface. induces different AI architectures as depicted in Figure 3.
Level 6—Mission Management. An element of systems These fusion techniques can be characterized based on the
management and enables spatial–temporal control of data they provide and/or operate on.
assets (e.g., airspace operations), route planning, and goal
(1) Information Fusion (IF). The IF agent mines and
determination to support decision making while consider-
processes physical data/information that originates in
ing social, economic, and political constraints.
the external environment. The IF can be assisted with
different AI/ML approaches, such as symbolic, prob-
abilistic, connectionist, analogistic, evolutionary, and
CYBER-PHYSICAL COMMAND-GUIDED (CPCG) possibilistic (please refer to Section “Alignment of
ARCHITECTURE Multilevel Fusion and AI” for types of AI and ML).

As the name implies, the CPCG architecture combines the (2) Operator Infusion (OI). The OI agent assimilates
elements of the static (DFD) and dynamic (DIG) methods human-in-the-loop within CPCG architecture for
from the late 1990s with the user-focused updates from interpreting and assessing processed information/
the JDl/DFIG in the 2000s for “command guided” sys- data, specifying mission objectives, interacting with
tems, such as swarm of unmanned domain systems (UxS) CPCG agents for ML and fusion/diffusion augmen-
(where x can be space, air, ground, surface, or undersea). tation and refinement with social knowledge, and
The CPCG architecture seeks to not only utilize the dis- decision making. The UDOP enables the human
tributed data fusion but also the distributed diffusion of operator to direct his/her decision making at the
command to cyber–physical elements. Hence, the CPCG highest level of establishing or updating mission
architecture leverages the cloud and edge processing to be objective, or to expand his/her decision making to
able to collect data for IF, affords consumption and analyt- involve details of instantaneous coordination among
ics by operator infusion (OI), and then direct needs for engagement groups within CPCG architecture, or
control diffusion (CD). The CPCG architecture takes even to decision making at the level of individual
advantage of centralized command with distributed execu- sensors, weapons, actuators, and platform manage-
tion by expressing goals and having the contextual agents ment. This rich human operator access and interac-
develop the sensing and action strategy. tion with CPCG architecture at different levels
The AI agents in CPCG architecture mine data, pro- suggest that the human operator is infused into the
cess and fuse information, and store the results in a CPCG architecture.

JULY 2021 IEEE A&E SYSTEMS MAGAZINE 67

Authorized licensed use limited to: Beijing Normal University. Downloaded on August 06,2021 at 08:22:29 UTC from IEEE Xplore. Restrictions apply.
 Artificial Intelligence and Data Fusion at the Edge

(3) Control Diffusion (CD). The CD agent relates to the assets can be either edge devices (e.g., IoT) or edge servers.
planning side of AI. The CD dissects or decomposes The results from the machine-based IF-CD are sent to
high-level mission objectives that originate from the the human via OI interface, which then conveys the salient
human operator into specialized tasks or actions for information to the commander. Since this type is the
different engagement capabilities of CPCG archi- most distributed approach, it typically takes the longest to
tecture. The planning is an AI’s effort that generates execute as the bottleneck is the limited attention of the user-
a sequence of actions based on observations. The in-the-loop that has to manage the assets as well as provide
planning agent explores the space of all possible updates.
actions to select the optimal sequence of actions Left (Direction-Based)—DIG-Based. The commander
that meets the mission goals. Thus, the planning directs human-machine agents (OI/CD) that collect data
process diffuses or fans out the high-level mission (IF) to form decisions through a data refinement loop. The
objectives to the CPCG constituent systems termi- commander can provide directions to the OI, which then
nating in the lowest level control signals for individ- communicates those to the CD agent. Unlike the query-
ual sensors, actuators, and platforms. The control based (right), here the OI sends commands or directives to
theory’s duality between observation and control is the CD agent, which then decides how and when to collect
manifested in CPCG architecture as the duality data from edge devices, such as IoT and sensor systems
between IF and CD. The CD can be assisted with (e.g., cameras) so that the most important information is
AI techniques, such as statistical relational learning collected when needed. This type can be represented as a
(SRL) and Markov logic networks [3]. directed graph as the high-level OI directives are imparted
as objectives and the CD uses the operating constraints to
The CPCG architecture utilizes the principle of cen- determine the sensing and actuation needs based on the
tralized command with decentralized control. The CPCG graph. Since this type is graph-based, it is also machine
architecture has four processing types, which are robust for processing and analysis.
described in the following with reference to Figure 3. We demonstrate a CPCG architecture further through
Top (Planner)—AI-based. The top “operator node” a practical case study in the following.
subsumes the three intelligent agents (IF, OI, and CD). Case Study: Command-Guided Swarm (CGS).A practi-
The commander interacts with these intelligent agents to cal example of CPCG architecture is CGS. A CGS is a
simultaneously conduct operations and synchronize plans. multisensor, multidevice (devices can be weapons),
Since this processing type is most demanding, the IF, OI, multiplatform, single-human-operator system-of-systems
and CD leverage a cloud-based approach (see Figure 1) (SoS) [18]. The CGS comprises of multiple UxS, where x
with a UDOP to determine the needs in real time. For can be space, air, ground, surface, or undersea, under the
example, if a system is deployed and the scenario has mission-oriented tactical coordination of a single human
some common patterns and results, the information can be operator. The CGS utilizes advanced AI and human part-
utilized to develop/train semisupervised AI from labeled nering concepts to carry out fusion of information originat-
(20% from the OI) and unlabeled (80% from the IF) data. ing from the swarm’s multiple sensors and diffusion of
Right (Query-Based)—DFD-Based. The commander control out to the swarm’s multifarious platforms, sensors,
provides goals to the machine–human agents (IF/OI) from and devices (weapons). The CGS unifies a collection of
which decisions determine the sensor control loop of what semiautonomous intelligent agents operating in parallel
data to collect (CD). The commander queries results from that are neither tightly coupled through a built-in command
the IF agent that is getting updates from a variety of OI structure nor completely independent and autonomous.
analysis and reports. The OI provides an interface to the Complex behaviors may emerge from the coordination of
human that controls multiple edge units, such as IoT devi- semiautonomous agents in CGS, which entails collective
ces. If the IoT devices are collecting data from different intelligence or swarm intelligence of the CGS. The swarm
sources, then the results would be semireal time as the intelligence of CGS emerges from the distributed informa-
commander can query the information needed to make tion processing/fusion and engagement of control across
decisions, while the IF agent is processing data from mul- multiple AI agents of the CGS. The swarm intelligence lev-
tiple updates. erages active ML technologies and human–machine part-
Bottom (Human-Centered)—JDL/DFIG-Based. The nership that is enabled by edge computing.
commander coordinates with other field units and manages A commander of the CGS may desire to obtain multi-
other human agents (OI), who then interact with edge node perspective and multimodal observations of an object.
machine agents (IF, CD) sending organized results. In the Since CGS comprises different UxS each with different
human-centered approach, there is a direct communication types of edge devices (e.g., visual and infrared sensors), the
between the users. The OI works with a set of assets (e.g., goal for the swarm is to evolve such that the positions for
UAVs) that have on-board processing for IF- and CD-based the viewpoints of multimodal sensors observe an object/tar-
data collection, which essentially are “edge nodes” as the get from different perspectives and distances. The

68 IEEE A&E SYSTEMS MAGAZINE JULY 2021

Authorized licensed use limited to: Beijing Normal University. Downloaded on August 06,2021 at 08:22:29 UTC from IEEE Xplore. Restrictions apply.
 Munir et al.

multimodal sensors could have an overlapping or orthogo- management. The low-level IF deals with numerical data,
nal viewpoints. The same viewpoints (0 ) maximizes data such as locations, kinematics, and target attributes, inter-
registration whereas orthogonal viewpoints (90 ) offer dif- mediate-level IF handles objects/entities, whereas high-
ferent perspective of the object. The confluence of AI and level IF copes with abstract symbolic information, such as
data (e.g., image) fusion in CGS requires support from threat, intent, and mission objectives.
models, methods, and control [21]. Models assess theoreti- Alternatively, levels of fusion can also be character-
cal performance of task success based on the range/distance ized as [22]: (sensor, pixel) data fusion; knowledge/feature
of a device (e.g., camera) to an object. Methods enable fusion; and decision fusion.
empirical performance measurement of multimodal data
(1) Sensor/Pixel Data Fusion. At the lowest level, raw
fusion. Control enables coordinated positioning of UxS to
data produced by sensors and other sources is fused
obtain multiperspective data. Theoretical models relate
while comprehending the characteristics and rela-
object distance to the visual electro-optical and infrared
tions of the input. This low-level data fusion is also
image resolution for object detection and classification. The
known as sensor fusion because the data from dif-
resolution increases and classification accuracy improves as
ferent sensors is fused together. In case, sensors are
the distance of the sensor from the object decreases. Theo-
camera sensors (e.g., visual or infrared), sensor
retical models analyze the probability of success for an
fusion is also known as pixel fusion because pixels
object detection and classification task using a single
of images obtained from camera sensors are fused
modality (e.g., an electro-optical camera) versus multimo-
together. This sensor/pixel fused data provides an
dality (e.g., electro-optical and infrared cameras). Once the-
updated representation of data for further process-
oretical models indicate benefits of image fusion, AI
ing. Sensor fusion can be implemented as central-
methods for contextual analysis are then utilized. Context
ized or distributed [23]. In a centralized fusion
from the scenario includes lighting conditions and position
architecture, measurements of all sensors are avail-
of sensor as a function of range. Context analysis deter-
able during the fusion process and a batch method is
mines which multimodal sensor configuration would yield
used to fuse the sensor data. In a distributed fusion
successful results, that is, whether to use visual, infrared, or
architecture, different sensor measurements are
visual + infrared camera sensors. Section “Multimodal
fused with a separate fusion model. Then, during
Fusion and AI at the Edge” demonstrates how multimodal
the global fusion process, the fusion model informa-
fusion help improve AI precision in CGS.
tion of each sensor is available. The distributed
fusion architecture is more scalable with the
increasing amount of data as compared to the cen-
ALIGNMENT OF MULTILEVEL FUSION AND AI tralized architecture.
Data fusion and AI can be performed at multiple levels Depending on the sensor configuration, sensor
utilizing the three hierarchical tiers, viz., edge devices, fusion can be classified into three cases: competitive
edge servers, and cloud (see Figure 1). This section dis- sensor fusion; complementary sensor fusion; and
cusses different levels of data fusion, characterizes AI into cooperative sensor fusion.
different types and/or stages, and aligns types of AI with a) Competitive sensor fusion. In competitive sen-
different levels of fusion. sor fusion, either data from the sensor of same
modality are fused together or the data from the
sensors from multiple modalities are first trans-
MULTILEVEL FUSION formed to the same baseline and then fused.
Competitive sensor fusion is typically used to
A DFIG model defines seven levels of fusion: L0—L6
reduce noise and uncertainty of the sensor
(section “Data Fusion Information Group Architecture”).
measurements. For example, in case of a
Broadly, data fusion can be categorized as low level, inter-
surveillance application, multiple competitive
mediate level, or high level depending on the processing
camera sensors obtain (homogeneous) images
stage at which IF transpires. In the DFIG model, L0 can
of a target at the same time and fusing those
be termed as low-level data fusion as it combines raw data
images result in less noisy resultant images that
from multiple processes to produce new raw data. The
are more suitable for the surveillance or track-
intermediate-level data fusion in the DFIG model encom-
ing application.
passes L1 as it provides object assessment based on fusion
and extraction of features from raw data. Finally, L2 – L6 b) Complementary sensor fusion. In complemen-
(i.e., L2, L3, L4, L5, and L6) in the DFIG model can be tary sensor fusion, sensors measure different and
construed as high-level data fusion as these levels fuse distinct parts of the same event and the combina-
high-level information/features to assess situation and tion of these (heterogeneous) disparate measure-
impact, and help refine process, display, and mission ments results in a complete characterization of

JULY 2021 IEEE A&E SYSTEMS MAGAZINE 69

Authorized licensed use limited to: Beijing Normal University. Downloaded on August 06,2021 at 08:22:29 UTC from IEEE Xplore. Restrictions apply.
 Artificial Intelligence and Data Fusion at the Edge

an event. For example, a complementary set of AI. AI can be classified into four main types based on their
cameras in a surveillance application can pro- functionalities. Type I AI—reactive machines belong to
vide an extended picture of the scene which sim- the most basic type of AI systems that are purely reactive
plifies the subsequent tracking of a target. and do not have the ability to form memories or use past
experiences to inform current decisions. These machines
c) Cooperative sensor fusion. In cooperative sensor
can only be utilized for automatically responding to a lim-
fusion, a sensor is configured and/or positioned
ited set or combination of inputs. A famous example of a
based on the information from other sensors to
reactive AI machine is IBM’s Deep Blue, a supercomputer
generate more useful measurements. For cooper-
that beat chess Grandmaster Garry Kasparov in 1997 [24].
ative sensor fusion, either sensors can autono-
Type II AI—limited memory machines in addition to hav-
mously collaborate to configure each other or
ing the capability of reactive machines have the ability to
some input from human expert can be provided.
learn from historical data to make decisions though this
For example, a tracking task can require adapting
memory is limited and transient. Nearly all existing AI
the camera angles after observing behavior of a
applications (e.g., chatbots, virtual assistants, and autono-
target.
mous vehicles) fall under this AI category. The next two
(2) Knowledge/Feature Fusion. The fused sensor/pixel types of AI exist either as a concept or work in progress.
data provide a basis for feature extraction that devel- Type III AI—theory of mind is used to represent
ops a model for the underlying data to conceive pat- a machine (AI agent) that has the ability to form a predic-
terns in the data. The abstraction of fusion tive model of self and others and have the ability to repre-
components increases with the level of fusion. In the sent and discern the mental states of others, including
intermediate fusion levels, the data are available in their emotions, desires, beliefs, and intentions. Theory of
the form of models that represent knowledge from mind AI can provide intelligent machines/robots with
the observed event. The knowledge fusion can be powerful capabilities, in particular, social intelligence for
performed at model level or parameter level. In human–machine interaction [25]. Type IV AI—self-
model fusion, the knowledge is represented in form awareness is an extension of theory of mind AI and is
of different models, which are fused together. An often regarded as the ultimate objective of all AI research.
example of such a model is Gaussian model that pro- Self-awareness AI refers to an AI agent that has con-
vides information about the distribution of data. The sciousness and has the ability to form representation of
mixture of Gaussian distributions produces a Gauss- itself and others. Self-aware AI agents know about their
ian mixture model (GMM), which describes the dis- internal states and can predict the feelings and actions of
tribution of a dataset that is more complex than a others. This type of AI will not only be able to understand
unimodal Gaussian. The fused model contains more and induce emotions in those it interacts with, but also
precise knowledge about the overall data distribution. have emotions, needs, beliefs, and likely desires of its
CNNs and multiple kernel learning based ensemble own 24. Although self-aware AI can potentially boost our
methods are other examples of model fusion techni- progress as a civilization tremendously, it can also possi-
ques. In parameter fusion, parameters of different bly lead to catastrophe because self-aware AI would have
models are fused together [22]. the capability of developing ideas, such as self-preserva-
tion, and outmaneuver the human intellect to plot elabo-
(3) Decision Fusion. At the highest level of fusion, the
rate schemes to take over humanity 24. Consequently, AI
goal is to improve decision making and choice of
safety has been gaining traction in AI research and non-
actions. Decisions obtained based on different mod-
profit organizations [26].
els can be fused together to obtain better decisions.
An alternate system of classification that is more preva-
The decision fusion of multiple models/classifiers
lent in AI community is the classification of AI into different
can either consist of direct combination of the deci-
stages, viz., artificial narrow intelligence (ANI), artificial
sions from the individual models or can select a spe-
general intelligence (AGI), and artificial super intelligence
cific model/classifier for a given input. By observing
(ASI). ANI represents all the existing AI even the most
the impact of a chosen action, the entire fusion pro-
complicated ones including deep learning. ANI refers to
cess can be adapted for performance improvement
those AI systems that can only perform a specific task (e.g.,
and better decision making.
driving and speech recognition) with human-like capabili-
ties. AGI refers to the capability of AI to learn, perceive,
understand and function like humans. AGI will indepen-
TYPES AND STAGES OF AI dently build multiple competencies and generalizations
Since AI research profess to make machines emulate across various domains thus massively reducing the time
humans, the extent to which an AI system can imitate needed for training. AGI will make AI agents just as capable
human capabilities is used as a criterion to define types of as humans by replicating the multi-functional abilities of

70 IEEE A&E SYSTEMS MAGAZINE JULY 2021

Authorized licensed use limited to: Beijing Normal University. Downloaded on August 06,2021 at 08:22:29 UTC from IEEE Xplore. Restrictions apply.
 Munir et al.

Table 2.

AI Aligned With IF

Type of AI Focus Objective Information Fusion

Alignment

Type I Reactive machines Identify patterns from rules for L0 Data assessment
immediate action
Type II Limited memory Estimate response leveraging signal L1 Object assessment
processing
Type III Theory of mind Form representations of world L2 Situation assessment
and other agents L3 Impact assessment
Type IV Self-awareness Understand self conscious to interact L4 Process refinement
with prediction L5 User refinement
L6 Mission refinement

humans. ASI marks the apex of AI research as ASI agents rigid in the sense that logic statements are either
will exceedingly do better at everything than humans true or false with no possibility of compromise.
because of great memory, processing, analysis, and deci- Failure of logical systems lead to catastrophic fail-
sion-making. The development of ASI can potentially lead ures and thus more sophisticated ML approaches
to a scenario referred to as the singularity [27]. are needed for solving complex problems.

(2) Probabilistic ML. Probabilistic ML is based on

Bayesian statistics, conditional probabilities, and
MAPPING OF AI/ML TO FUSION LEVELS network/graphs of nodes. Probabilistic ML avoids
Table 2 provides a mapping of AI types to different fusion the rigidness of symbolic ML by modeling relat-
levels: Type I AI—reactive machines with rules support ionships as conditional probability distributions;
L0 processing; Type II AI—limited memory utilizes sig- however, probabilistic models lack the rich repre-
nal processing based methods to provide L1 functions; sentations and reasoning ability of symbolic ML.
Type III AI—theory of mind provides representations The ML for complex systems, such as CGS (dis-
about the world supporting L2/L3 functions; and Type IV cussed in section “Cyber–Physical Command-
AI—self-awareness supports prediction and interact with Guided (CPCG) Architecture” as a case study), uti-
L4/L5/L6 analyses. Although in Table 2, we have aligned lizes a novel hybrid of symbolic and probabilistic
Type III and Type IV AI with data fusion levels L2–L6, ML. This hybrid ML approach, which is referred to
this alignment is idealistic and implies that these AI types as SRL, combines Bayesian graphical models with
will be able to best perform the capabilities associated a first-order logic where logical symbolic represen-
with intermediate- and high-level fusion. Currently, all tations capture the underlying rich structure of the
fusion levels are assisted with Type I and Type II AI only problem domain, whereas the probabilistic methods
and a contemporary (not idealistic) alignment will map handle the uncertainty in data.
Type I and Type II to all fusion levels.
(3) Connectionist ML. Connectionist ML utilizes
Since AI is dominated by ML techniques, Table 3 pro-
computational models inspired by neural architec-
vides a mapping of ML methods to different fusion levels.
ture of the biological brain. Examples of connec-
It is to be noted that different ML methods can provide
tionist ML include artificial neural networks
different types of AI. For example, deep neural networks
(ANNs) and multilayer ANNs (i.e., deep learning).
(DNNs), either a single DNN or a connection of multiple
Table 3 also list some other ML methods that fall
DNN, can provide different types of AI, ranging from
under connectionist ML, such as CNN, RNN, long
Type I to Type III AI. Table 3 categorizes ML methods
short-term memory (LSTM), GAN, and RL. An
into six types: symbolic; probabilistic; connectionist; anal-
RNN is a class of ANNs where connection between
ogistic; evolutionary; and possibilistic.
nodes constitute a directed graph along a temporal
(1) Symbolic ML. Symbolic ML is based on first-order sequence that enables it to exhibit dynamic temporal
logic models and truth tables. Symbolic ML has behavior. RNNs utilize their internal state (memory)
been termed as good old-fashioned AI. It is excel- to process variable lengths of input sequences,
lent for implementing machine reasoning, but is which makes RNNs suitable for tasks such as

JULY 2021 IEEE A&E SYSTEMS MAGAZINE 71

Authorized licensed use limited to: Beijing Normal University. Downloaded on August 06,2021 at 08:22:29 UTC from IEEE Xplore. Restrictions apply.
 Artificial Intelligence and Data Fusion at the Edge

Table 3.

ML Aligned With IF
ML Methods Types of ML Information
Fusion
Symbolic Probabilistic Connectionist Analogistic Evolutionary Possibilistic Alignment
(Logic) (Bayes) (DL) (SVM) (GA) (Fuzzy)
Registration, • @ • @ • • L0 Data
Estimation Assessment
CNN, RNN, LSTM, • @ @ @ @ @ L1 Object
Estimation Assessment
CNN, Pattern @ @ @ • • @ L2 Situation
Matching Assessment
GAN • @ @ • @ • L3 Impact
Assessment
RL, Optimization, • @ @ • • @ L4 Process
Regularization Refinement
Active Learning @ @ @ • • • L5 User
Refinement
RL @ @ @ • • • L6 Mission
Refinement

connected handwriting recognition and speech rec- different sensors (multimodal registration), and/or
ognition. LSTM is a type of RNNs that have feed- from different viewpoints [29].
back connections and help overcome the vanishing
(4) Analogistic ML. Analogistic ML analyzes data anal-
gradient problem in RNNs. GAN is an ML frame-
ogies and similarities through distance computa-
work based on ANNs that learns to generate new
tions in feature hyperspace. Examples of analogistic
data with the same statistics as the training set. RL
ML include support vector machines (SVMs) and
is another area of ML that is concerned with deter-
nearest neighbors, such as K-nearest neighbor
mining an optimal set of actions in an environment
algorithm.
in order to maximize cumulative reward. RL is typi-
cally stated in the form of a Markov decision pro- (5) Evolutionary ML. Evolutionary ML utilizes compu-
cess (MDP), where dynamic programming provides tational models inspired by evolutionary competi-
a solution for the MDP [28]. DRL combines DNNs tion and survival. Examples of evolutionary ML
with RL algorithms (e.g., Q-learning) to solve previ- include genetic algorithms (GAs), genetic program-
ously unsolvable problems as DRL can learn from ming, and neuroevolution. We note that neuroevolu-
raw sensor data and/or images supplied as input. In tion is similar to genetic programming but the
active learning, an ML algorithm can interactively genomes represent ANNs by specifying structure
pose queries during the training process, usually in and connection weights.
the form of unlabeled data instances to be labeled
(6) Possibilistic ML. Possibilistic ML analyzes ambigu-
by a human user. Thus, active learning is an exam-
ous data using extension of classical logic to repre-
ple of human-in-the-loop learning. Connectionist
sent partial truths. Fuzzy inference systems and
ML often leverages regularization. Regularization
possibilistic logic systems are examples of possibil-
is a technique that makes slight modifications to the
istic ML.
ML algorithm to prevent overfitting to the training
data and to make the model more generalizable for
different test datasets. Connectionist ML also pre-
processes the input data through data registration.
ADVANTAGES OF DATA FUSION AND AI AT THE EDGE
Registration, often performed for images and thus Data fusion and AI at the edge can provide various advan-
known as image registration, is the process of trans- tages in terms of latency, energy, accuracy, security, pri-
forming different images of one scene into the same vacy, cost, scalability, and sustainability as discussed in
coordinate system. Registration is required in order the following.
to compare or fuse the data obtained from different Latency. Latency refers to the time spent in the whole
measurements. For example, images can be taken at AI inference process, including pre-processing, data
different times (multitemporal registration), by fusion, model inference, data transmission and post

72 IEEE A&E SYSTEMS MAGAZINE JULY 2021

Authorized licensed use limited to: Beijing Normal University. Downloaded on August 06,2021 at 08:22:29 UTC from IEEE Xplore. Restrictions apply.
 Munir et al.

processing [30]. Many edge devices and systems (e.g., sur- may be skipped due to limited resources of edge devices.
veillance systems, autonomous vehicles, and robots) have In such conditions, data fusion can help improve the accu-
stringent deadline requirements (in the order of microsec- racy by fusing a few adjacent frames of video and present-
onds to milliseconds) and missing those deadlines can ing the fused frames to the AI model for inference. The
result in catastrophes. According to Steve Roddy, the VP number of frames to be fused will depend on the applica-
of Special Project in Arm’s Machine Learning Group [2]: tion and available resources of an edge device. The fused
“Applications that people will engage within real-world frame will be able to capture the salient information in the
products such as controlling home devices or providing video frames and will provide better prediction accuracy
driver assistance in a car, all of those applications are run- instead of skipping the video frames if an edge device
ning on the edge and many will require real-time could not handle the feed rate.
responses. Any delay from bouncing information to the Security. Security is another advantage that is provided
cloud and back could be a problem.” Sri Chandrasekaran, by data fusion and AI at the edge. In case of fusion and AI
Senior Director of IEEE Standards Association, has also at the cloud, data need to travel from edge devices to the
emphasized the importance of speed/latency for AI infer- cloud and from the cloud to edge devices hundreds of miles
ence 2: “We can’t overlook the importance of latency. AI over multiple channels including wireless channels and
at the edge will allow for faster data transfer, which will, Internet thus exposing the data en route to attackers. The
in turn, benefit the many industries AI touches, especially data sent to and received from the cloud can be compro-
industrial IoT and automotive. These industries benefit mised over the wireless channels, wired channels, interme-
from AI at the edge because the machines and automobiles diate routers, or even the cloud computers itself. Data
must be able to understand many different aspects at once. fusion and AI at the edge devices minimizes the data trans-
Sending data to the cloud and back is not only inefficient, fer and thus alleviates the security issues associated with
but it is also less secure and much slower, ultimately lead- data transfer over multiple channels.
ing to a decrease in productivity and reliability.” Research Privacy. Privacy preservation is another advantage of
results also verify that data fusion and AI at the edge pro- data fusion and AI at the edge. Often the data requiring
vides much faster response as compared to sending data to inference is private and contains sensitive information
the cloud for fusion and inference [31]. (e.g., medical records, personal photos, financial reports,
Energy Efficiency. Data fusion and AI at the edge is and target information in defense applications). Sending
much more energy efficient than the data fusion and infer- this private data to the cloud for AI provides no privacy
ence at the cloud because it takes a large amount of power guarantees to the user. Microsoft Research has proposed a
to send data over the air, whereas it takes orders of magni- homomorphic encryption based solution referred to as
tude lesser power to do computations on the device when CryptoNets [32] that permits a data owner to send their
the data is available on-device. Since many of the edge data in encrypted form to the cloud service for inference.
devices are battery powered with no energy harvesting Since the cloud does not have access to the encryption
system (e.g., solar and thermal), sending data to the cloud key, the user data remain confidential. In CryptoNets, the
for fusion and inference and receiving the results back cloud service is able to provide inference on the encrypted
will expeditiously deplete these devices of the battery data and return the results to the user in an encrypted
power. form. However, the overhead of homomorphic encryption
Precision. AI precision or accuracy refers to the ratio limits the applicability of this solution on resource-con-
of the number of input samples that get correct prediction strained edge devices. Data fusion and AI at the edge ena-
to the total number of input samples [30]. Many edge bles inference at the edge and thus alleviates the privacy
applications, such as autonomous driving and face authen- issues associated with sending data to the cloud.
tication, require ultrahigh AI accuracy. Although increase Cost. Data fusion and AI at the edge also provide cost
in the number and type of sensors assist in covering large advantages. Due to advancements in semiconductor tech-
areas, the growing number of sensors have often resulted nology, the cost of system-on-chips (SoCs) is decreasing
in an increase in false alarm rates and have compounded with increasing capability to perform fusion and AI on
the target acquisition process in case of surveillance appli- these SoCs. These edge SoCs cost much lesser than build-
cations due to the fact that sensors can provide inaccurate, ing the apparatus and infrastructure required to perform
incomplete, or inconsistent data. Data fusion at the edge fusion and inference in the cloud.
can help filter the outliers and malicious readings, which Scalability. The number of edge devices is continu-
results in an improved AI/ML model with better precision ously on the rise approaching 100 billion in near future
than a model that is trained on outliers and malicious data. and producing hundreds of zettabytes of data. If each edge
Furthermore, in many edge applications, AI inference device has to send all the data back to the cloud data cen-
accuracy is also affected by the speed at which an applica- ter for fusion and AI inference, it will put an enormous
tion needs to process the input data. For a video analytics pressure on the network bandwidth likely causing the net-
application under a fast feeding rate, some input samples work to collapse (denial of service) as well as will require

JULY 2021 IEEE A&E SYSTEMS MAGAZINE 73

Authorized licensed use limited to: Beijing Normal University. Downloaded on August 06,2021 at 08:22:29 UTC from IEEE Xplore. Restrictions apply.
 Artificial Intelligence and Data Fusion at the Edge

huge investments on expensive data centers. Data fusion We consider the following three use cases for AI and
and AI at the edge imparts scalability to an intelligent data fusion on multiple hardware platforms suitable for
computing system because majority of the data fusion and edge computing:
AI computations are performed at edge devices and only
(1) data fusion and CNN model execution in Intel Xeon
limited fused and processed data need to be sent over the
CPU [9] (Xeon_CPU);
network to the cloud.
Sustainability. Data fusion and AI at the edge provide (2) data fusion and CNN model execution in Nvidia Jet-
a sustainable solution for emerging smart applications son TX2 GPU (JTX2 34) CPU (JTX2_CPU);
(e.g., autonomous vehicles, smart agriculture, surveil-
(3) data fusion in JTX2 CPU and CNN model execution
lance, and swarm intelligence) because increasing advan-
in JTX2 GPU (JTX2_CPU+JTX2_GPU).
ces in semiconductor will continue to make edge devices
more powerful to carry out inferences in real time in a We average the execution time of performing data
cost-effective manner. Moreover, edge AI will be able to fusion and CNN inference over 10 sets of 10 digits. We
meet AI needs for applications even in communication- have run our experiments on Ubuntu 18.04 operating sys-
denied environments or places where no infrastructure tem using CUDA 10.1 as a general purpose GPU frame-
exists for connection to the cloud. work, and Python 3.6.9 for implementation of data fusion.
For implementing AI with data fusion, first the 10 images
in each set are fused using data fusion in Python and the
EXPERIMENTAL RESULTS resulting fused image is then provided as input to the Dar-
knet framework [35] for CNN inference. We use two
In this section, we provide experimental results demon-
CNN models, i.e., LeNet [36] and AlexNet [37], for our
strating latency, energy, and precision advantages of
experiments. Regarding the training of CNN models, we
combined data fusion and AI at the edge. The experi-
utilize the trained weights provided by Krizhevsky et al.
mental results present latency and energy consumption
[38] for LeNet, whereas we train the weights of AlexNet
comparison between two CNN models with and with-
ourselves using MNIST training dataset comprising of
out data fusion. Furthermore, results demonstrate how
60 000 images. To smooth out any inconsistencies in
multimodal fusion help improve the precision of AI
latency due to operating system overhead and variations
for a CGS system.
in environmental parameters, we average the execution
time results over 10 independent measurements.

EXPERIMENTAL SETUP
We conduct two set of experiments to demonstrate
AVERAGE LATENCY FOR COMBINED DATA FUSION
latency, energy, and precision advantages of combined
data fusion and AI at the edge. The primary set of experi- AND AI AT THE EDGE
ments use handwritten digit datasets whereas secondary
Table 4 shows the speedup of AI (LeNet and AlexNet
experiments use different types of camera sensors, viz.,
CNN models) with data fusion over AI without data
visual (VI) and medium wavelength infrared (MWIR) out-
fusion. Results verify that AI with data fusion provides
fitted on UAVs.
significant speedups over AI without data fusion. For
In our primary set of experiments, we obtain experi-
example, AI with data fusion on Xeon CPU results in a
mental results for MNIST handwritten digit dataset 33. We
speedup of 2.6 and 9.7 for LeNet and AlexNet,
clarify that we have chosen this dataset for illustrating the
effectiveness of combined data fusion and AI; however,
Table 4.
experimental results for other datasets can be obtained
similarly. For practical relevance, we note that for social Speedup of AI With Data Fusion As Compared to AI
intelligence, often handwritten digits and text need to be
analyzed. We have randomly selected 100 handwritten Without Data Fusion
images from the dataset for each digit. We have made 10
sets of each digit image where each set contains 10 hand- Edge Computing CNN Model
written images. We then fuse the images in each set to
produce 10 fused images for each digit. Our data fusion Platform LeNet AlexNet
technique adds the pixel values at the same location of the
Xeon_CPU 2.55X 9.72X
10 images in the set. Thus, our primary set of experiments
demonstrates competitive pixel data fusion done explicitly JTX2_CPU 3.82X 9.76X
and model data fusion (knowledge/feature data fusion) JTX2_CPU + JTX2_GPU 9.57X 9.29X
implicitly by the CNN inference.

74 IEEE A&E SYSTEMS MAGAZINE JULY 2021

Authorized licensed use limited to: Beijing Normal University. Downloaded on August 06,2021 at 08:22:29 UTC from IEEE Xplore. Restrictions apply.
 Munir et al.

Figure 4.
Energy reduction imparted by data fusion and AI as compared to
Figure 5.
Accuracy of AI with data fusion as compared to AI without data
AI without data fusion.
fusion.

respectively. For JTX2 CPU, AI with data fusion provides same amount of input data can be processed much quicker
a speedup of 3.8 and 9.8 for LeNet and AlexNet, by the AI models as compared to the case without data
respectively, as compared to AI without data fusion. The fusion (see section “Average Latency for Combined Data
data fusion in JTX2 CPU and CNN model execution in Fusion and AI at the Edge”). This lower latency of AI with
JTX2 GPU lead to a speedup of 9.6 and 9.3 for LeNet data fusion also translates to lower energy consumption as
and AlexNet, respectively. compared to AI without data fusion.
Since GPUs serve as a decent platform for acceleration
of deep learning models, offloading the CNN inference
tasks to the JTX2 GPU engenders a speedup of 2.8 to
ACCURACY FOR COMBINED DATA FUSION AND AI AT
2.9 for AlexNet as compared to only using the JTX2
CPU. However, experimental results indicate that the THE EDGE
LeNet execution on GPUs results in lower performance as
AI with data fusion can obviously result in improved per-
compared to LeNet execution on CPU. This is due to the
formance as illustrated in section “Average Latency for
small size of LeNet model (i.e., LeNet model consists of
Combined Data Fusion and AI at the Edge” because infer-
only 60 000 parameters), which can be efficiently run on ence is performed on reduced data size. However, there
CPU without offloading. For LeNet inference on GPU, the may be a concern regarding accuracy of combined data
data transfer between CPU, memory, and GPU incurs fusion and AI because the fused data are different from
nonnegligible latency overhead that causes higher overall the original raw data. Hence, we measure the impact of
execution time of LeNet inference on GPU as compared to data fusion on accuracy using average softmax probability
CPU. Conversely, since AlexNet is a large CNN model as a metric by which the CNN model predicts a class. For
with 60 million parameters, AlexNet inference on GPU accuracy evaluations, we use Darknet framework [35]
provides better performance than CPU because the compu- with LeNet CNN model. The experimental setup is the
tation time for large data dominates the data transfer time. same as described in section “Experimental Setup.” We
generate one image with fusing 10 images by averaging
the pixel values, resulting in data size reduction by 90%.
ENERGY CONSUMPTION FOR COMBINED DATA FUSION For inference with data fusion, we use 100 fused input
images, whereas we use 1 000 original images in the case
AND AI AT THE EDGE
of inference without data fusion.
Data fusion also imparts energy benefits to AI. Figure 4 Figure 5 shows data fusion impact on softmax proba-
depicts the energy reduction furnished by data fusion and bility for CNN inference. Our CNN model has 10 output
AI as compared to AI without data fusion across three edge classes C_i where i; i 2 0; 1; 2; 3; . . . ; 9. C_i denotes
computing platforms and the two CNN models. Results class i that corresponds to the output for digit i (e.g., C_0
verify that the data fusion leads to huge energy savings for corresponds to digit 0 and C_9 corresponds to digit 9).
AI. For Xeon CPU and JTX2 CPU, the energy reductions Results indicate that data fusion does not adversely affect
furnished by data fusion and AI over AI without data fusion the softmax probability though there is a fluctuation in
are 56.11%–88.32% and 70.88%–88.42%, respectively. softmax probability depending on the digit class. Results
For the data fusion and AI execution on GPU along with show that data fusion imparts higher accuracy to AI on
the CPU in the JTX2 platform, energy reduction is average. For example, data fusion results in CNN infer-
88.00%–88.52% as compared to AI without data fusion. ence accuracy of 88.53% as compared to the accuracy of
These energy savings are attained because with data fusion, 87.51% without data fusion for all digit classes. Results

JULY 2021 IEEE A&E SYSTEMS MAGAZINE 75

Authorized licensed use limited to: Beijing Normal University. Downloaded on August 06,2021 at 08:22:29 UTC from IEEE Xplore. Restrictions apply.
 Artificial Intelligence and Data Fusion at the Edge

3 km, whereas using only visual imagery produces accept-

able precision when the sensor-to-object distance is less
than 3 km. We clarify that for CGS multimodal sensing,
control is asserted through user commands. The control
asserted through high-level user commands permeates to
different components of CGS through CD, which results
in routing of UAVs to positions through which overlap-
ping or orthogonal viewpoints can be obtained as required
by the deep learning methods for improving classification
precision.

Figure 6.
Precision versus distance for multimodal fusion and AI. CONCLUSION
In this article, we have discussed the emerging disci-
pline of data fusion and AI at the edge. As the price of
depict that for some classes, data fusion may lead to lower
computing continues to fall, edge fusion and intelli-
accuracy for AI than without using data fusion. For exam-
gence will continue to proliferate. In the foreseeable
ple, for C_5 in our experiments, softmax probability of
future, the cloud and edge AI will continue to coexist
CNN inference actually decreases, meaning that the data
where the cloud will be mostly used for training of AI,
fusion may lead to inaccurate prediction (or classifica-
whereas more and more inference will be performed at
tion). We observe that the softmax probability for C_5
the edge. In this article, we have proposed a hierarchi-
with data fusion decreases by 32% as compared to the
cal framework for data fusion and AI at the edge. The
softmax probability without data fusion. The reason being
article provides a comparative discussion of contempo-
that sometime for image classification, data fusion may
rary data fusion and AI models and architectures with
cause the image to become blurry, which may cause the
special emphasis on DFIG model and CPCG architec-
CNN model to misclassify the input. However, post-proc-
ture. This article also presents a case study of CGS as a
essing of fused images and/or applying different data
practical application of a CPCG architecture. This arti-
fusion techniques can help improve the accuracy. Results
cle aligns the AI techniques to different fusion levels.
also reveal that for all classes other than C_5, softmax
Finally, this article demonstrates the advantages of AI
probability with data fusion is either very close to or
and data fusion at the edge including latency, energy
higher than the softmax probability without data fusion.
efficiency, precision, security, privacy, cost, scalability,
For example, data fusion provides an improvement of
and sustainability. Experimental results have revealed
4.25% for softmax probability as compared to the softmax
that combining AI with data fusion can impart a
probability without data fusion for all digit classes except
speedup of 9.8 over AI without data fusion. Addition-
C_5. Results show that data fusion imparts maximum
ally, data fusion leads to energy savings of up to 88.5%
improvement in softmax probability for C_3, where the
for AI as compared to the AI without data fusion. Fur-
attained improvement is 15.81%. Hence, our results verify
thermore, results have demonstrated that data fusion
that data fusion significantly improves the performance
either maintains or improves the accuracy of AI in most
of AI while also improving or maintaining the accuracy of
cases. For our experiments, data fusion imparts a maxi-
AI in most cases.
mum improvement of 15.8% in accuracy to AI. Further-
more, experimental results for a CGS case study
demonstrate the advantage of multimodal data fusion
MULTIMODAL FUSION AND AI AT THE EDGE on precision of AI.
This experimental result focuses on the CGS case study
where a commander of the CGS may desire to obtain mul-
tiperspective and multimodal observations of an object
ACKNOWLEDGMENTS
using different types of camera sensors, viz., VI and This work was supported in part by the Air Force Research
MWIR, outfitted on UAVs. This experiment demonstrates Laboratory (AFRL) Information Directorate (RI), through
multimodal competitive data fusion and multimodal com- the Air Force Office of Scientific Research (AFOSR)
plementary data fusion. Figure 6 illustrates how and when Summer Faculty Fellowship Program, under Contract
multimodal image fusion benefits deep learning based FA8750-15-3-6003 and Contract FA9550-15-0001. Any
inference from images [18]. Figure 6 indicates that multi- opinions, findings, and conclusions or recommendations
modal image fusion (VI + MWIR) benefits deep learning expressed in this material are those of the author(s) and do
precision when sensor-to-object distance is greater than not necessarily reflect the views of the AFRL and AFOSR.

76 IEEE A&E SYSTEMS MAGAZINE JULY 2021

Authorized licensed use limited to: Beijing Normal University. Downloaded on August 06,2021 at 08:22:29 UTC from IEEE Xplore. Restrictions apply.
 Munir et al.

[17] E. P. Blasch et al., “High level information fusion (HLIF):

REFERENCES
Survey of models, issues, and grand challenges,” IEEE
[1] A. Munir, P. Kansakar, and S. U. Khan, “IFCIoT: Inte- Aerosp. Electron. Syst. Mag., vol. 27, no. 9, pp. 4–20,
grated Fog Cloud IoT: A novel architectural paradigm for Sep. 2012.
the future Internet of Things,” IEEE Consum. Electron. [18] R. Cruise, E. Blasch, S. Natarajan, and A. Raz, “Cyber-
Mag., vol. 6, no. 3, pp. 74–82, Jul. 2017. physical command guided swarm,” Defense Syst. Inf.
[2] B. Reese, “AI at the edge: A GigaOm research byte,” Jan. Anal. Center J., vol. 5, no. 2, pp. 24–30, 2018.
2019. [Online]. Available: https://gigaom.com/report/ai-at- [19] M. Bedworth and J. O’Brien, “The Omnibus model: a new
the-edge-a-gigaom-research-byte/ model of data fusion?,” IEEE Aerosp. Electron. Syst.
[3] E. Blasch, R. Cruise, S. Natarajan, A. K. Raz, and Mag., vol. 15, no. 4, pp. 30–36, Apr. 2000.
T. Kelly, “Control diffusion of information collection for [20] M. Liggins, C.-Y. Chong, I. Kadar, M. Alford,
situation understanding using boosting MLNs,” in Proc. V. Vannicola, and S. Thomopoulos, “Distributed fusion
21st Int. Conf. Inf. Fusion, Jul. 2018, pp. 1407–1414. architectures and algorithms for target tracking,” Proc.
[4] B. Gu, J. Kong, A. Munir, and Y. G. Kim, “A framework IEEE, vol. 85, no. 1, pp. 95–107, Jan. 1997.
for distributed deep neural network training with heteroge- [21] E. Blasch, R. Cruise, A. Aved, U. Majumder, and
neous computing platforms,” in Proc. IEEE Int. Conf. T. Rovito, “Methods of AI for multimodal sensing and
Parallel Distrib. Syst., Dec. 2019, pp. 430–437. action for complex situations,” AI Mag., vol. 40, no. 4,
[5] A. G. Howard et al., “MobileNets: Efficient convolutional pp. 50–65, 2019.
neural networks for mobile vision applications,” 2017, [22] S. Beddar-Wiesing and M. Bieshaar, “Multi-sensor data
arXiv:1704.04861. and knowledge fusion: A proposal for a terminology defi-
[6] K. Lee, J. Kong, and A. Munir, “HW/SW co-design of nition,” 2020, arXiv:2001.04171. [Online]. Available:
cost-efficient CNN inference for cognitive IoT,” in Proc. https://arxiv.org/abs/2001.04171
Int. Conf. Intell. Comput. Data Sci., 2020, pp. 1–6. [23] A. Munir, A. Gordon-Ross, and S. Ranka, “Multi-core
[7] Samsung, “What is the NPU in Galaxy and what does it embedded wireless sensor networks: Architecture and
do?,” Accessed: Dec. 2020. [Online]. Available: https:// applications,” IEEE Trans. Parallel Distrib. Syst., vol. 25,
www.samsung.com/global/galaxy/what-is/npu/ no. 6, pp. 1553–1562, Jun. 2014.
[8] N. Chen, Y. Chen, E. Blasch, H. Ling, Y. You, and X. Ye, [24] N. Joshi, “7 Types of artificial intelligence,” Jun. 2019.
“Enabling smart urban surveillance at the edge,” in Proc. [Online]. Available: https://www.forbes.com/sites/
IEEE Int. Conf. Smart Cloud, Nov. 2017, pp. 109–119. cognitiveworld/2019/06/19/7-types-of-artificial-
[9] Intel, “Xeon Processor E3-1230 v5.” Accessed: Dec. 2020. intelligence/#27e3a666233e
[Online]. Available: https://ark.intel.com/content/www/kr/ [25] A. F. T. Winfield, “Experiments in artificial theory of mind:
ko/ark/products/88182/intel-xeon-p% rocessor-e3-1230-v5- From safety to story-telling,” Front. Robot. AI, vol. 5, p. 75,
8m-cache-3-40-ghz.html Jun. 2018, doi: 10.3389/frobt.2018.00075.
[10] N. P. Jouppi et al., “In-datacenter performance analysis of [26] Future of Life Institute, Aug. 2020. [Online]. Available:
a tensor processing unit,” SIGARCH Comput. Archit. https://futureoflife.org/
News, vol. 45, no. 2, pp. 1–12, 2017. [27] J. Strickland, “What’s the technological singularity?”, Aug.
[11] J. Fowers et al., “A configurable cloud-scale DNN proces- 2020. [Online]. Available: https://electronics.howstuffworks.
sor for real-time AI,” in Proc. 45th Annu. Int. Symp. Com- com/gadgets/high-tech-gadgets/technological-singularity.htm
put. Archit., 2018, pp. 1–14. [28] A. Munir and A. Gordon-Ross, “An MDP-based dynamic
[12] B. Dasarathy, “Sensor fusion potential exploitation-innova- optimization methodology for wireless sensor networks,”
tive architectures and illustrative applications,” Proc. IEEE Trans. Parallel Distrib. Syst., vol. 23, no. 4,
IEEE, vol. 85, no. 1, pp. 24–38, Jan. 1997. pp. 616–625, Apr. 2012.
[13] D. L. Hall, C.-Y. Chong, J. Llinas, and M. Liggins II, [29] E. Kamoun and J. Joslove, “Image registration: From SIFT
Distributed Data Fusion for Network-Centric Operations. to deep learning,” Jul. 2019. [Online]. Available: https://
Boca Raton, FL, USA: CRC Press, 2012. medium.com/sicara/image-registration-sift-deep-learning-
[14] Z. Chair and P. Varshney, “Optimal data fusion in multiple 3c794d794b7a
sensor detection systems,” IEEE Trans. Aerosp. Electron. [30] Z. Zhou, X. Chen, E. Li, L. Zeng, K. Luo, and J. Zhang,
Syst., vol. AES-22, no. 1, pp. 98–101, Jan. 1986. “Edge Intelligence: Paving the last mile of artificial intelli-
[15] C. Snoek, M. Worring, and A. W. M. Smeulders, “Early gence with edge computing,” Proc. IEEE, vol. 107, no. 8,
versus late fusion in semantic video analysis,” in Proc. pp. 1738–1762, Aug. 2019.
13th ACM Int. Conf. Multimedia, Nov. 2005, pp. 399–402. [31] R. M. Shukla and A. Munir, “An efficient computation off-
[16] I. J. Goodfellow et al., “Generative adversarial nets,” in loading architecture for the Internet of Things (IoT)
Proc. Adv. Neural Inf. Process. Syst., Dec. 2014, devices,” in Proc. 14th IEEE Annu. Consum. Commun.
pp. 2672–2680. Electron. Netw. Conf., Jan. 2017, pp. 728–731.

JULY 2021 IEEE A&E SYSTEMS MAGAZINE 77

Authorized licensed use limited to: Beijing Normal University. Downloaded on August 06,2021 at 08:22:29 UTC from IEEE Xplore. Restrictions apply.
 Artificial Intelligence and Data Fusion at the Edge

[32] N. Dowlin, R. Gilad-Bachrach, K. Laine, K. Lauter, [35] J. Redmon, “Darknet: Open source neural networks in C,”
M. Naehrig, and J. Wernsing, “CryptoNets: Applying neu- 2013–2016. Accessed: Dec. 2020. [Online]. Available: http://
ral networks to encrypted data with high throughput and pjreddie.com/darknet/
accuracy,” Feb. 2016. [Online]. Available: https://www. [36] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner,
microsoft.com/en-us/research/publication/cryptonets- “Gradient-based learning applied to document recog-
applying-neural-networks-to-encrypted-data-with-high- nition,” Proc. IEEE, vol. 86, no. 11, pp. 2278–2324, Nov.
throughput-and-accuracy/ 1998.
[33] Y. LeCun, C. Cortes, and C. J. C. Burges, “The MNIST [37] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet
database of handwritten digits.” Accessed: Dec. 2020. classification with deep convolutional neural networks,”
[Online]. Available: http://yann.lecun.com/exdb/mnist Adv. Neural Inform. Process. Syst., vol. 25, pp. 1097–
[34] Nvidia, “Jetson TX2.” Accessed: Dec. 2020. [Online]. 1105, 2012.
Available: https://developer.nvidia.com/embedded/buy/ [38] T. Ashitani, “darknet_mnist,” 2020. [Online]. Available:
jetson-tx2 https://github.com/ashitani/darknet_mnist

78 IEEE A&E SYSTEMS MAGAZINE JULY 2021

Authorized licensed use limited to: Beijing Normal University. Downloaded on August 06,2021 at 08:22:29 UTC from IEEE Xplore. Restrictions apply.