Studies in Computational Intelligence 1072

Aboul Ella Hassanien

Deepak Gupta
Anuj Kumar Singh
Ankit Garg Editors

Explainable Edge
AI: A Futuristic
Computing
Perspective
Studies in Computational Intelligence

Volume 1072

Series Editor
Janusz Kacprzyk, Polish Academy of Sciences, Warsaw, Poland
The series “Studies in Computational Intelligence” (SCI) publishes new develop-
ments and advances in the various areas of computational intelligence—quickly and
with a high quality. The intent is to cover the theory, applications, and design methods
of computational intelligence, as embedded in the fields of engineering, computer
science, physics and life sciences, as well as the methodologies behind them. The
series contains monographs, lecture notes and edited volumes in computational
intelligence spanning the areas of neural networks, connectionist systems, genetic
algorithms, evolutionary computation, artificial intelligence, cellular automata, self-
organizing systems, soft computing, fuzzy systems, and hybrid intelligent systems.
Of particular value to both the contributors and the readership are the short publica-
tion timeframe and the world-wide distribution, which enable both wide and rapid
dissemination of research output.
This series also publishes Open Access books. A recent example is the book
Swan, Nivel, Kant, Hedges, Atkinson, Steunebrink: The Road to General Intelligence
https://link.springer.com/book/10.1007/978-3-031-08020-3
Indexed by SCOPUS, DBLP, WTI Frankfurt eG, zbMATH, SCImago.
All books published in the series are submitted for consideration in Web of Science.
Aboul Ella Hassanien · Deepak Gupta ·
Anuj Kumar Singh · Ankit Garg
Editors

Explainable Edge AI:

A Futuristic Computing
Perspective
Editors
Aboul Ella Hassanien Deepak Gupta
Scientific Research Group in Egypt (SRGE) Department of Computer Science
Cairo University and Engineering
Giza, Egypt Maharaja Agrasen Institute of Technology
New Delhi, Delhi, India
Anuj Kumar Singh
School of Computing Ankit Garg
University of Engineering & Technology School of Computing
(UETR) University of Engineering & Technology
Roorkee, Uttarakhand, India (UETR)
Roorkee, Uttarakhand, India

ISSN 1860-949X ISSN 1860-9503 (electronic)

Studies in Computational Intelligence
ISBN 978-3-031-18291-4 ISBN 978-3-031-18292-1 (eBook)
https://doi.org/10.1007/978-3-031-18292-1

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature
Switzerland AG 2023
This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether
the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse
of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and
transmission or information storage and retrieval, electronic adaptation, computer software, or by similar
or dissimilar methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
Preface

Explainable artificial intelligence (XAI) derives notions from different fields

including cognitive sciences, philosophy, and psychology to yield a range of proce-
dural methodologies that can produce explainable results for the end-users not having
enough background on artificial intelligence. The main emphasis has been placed
toward the progression of XAI that encompasses the human end-user in the cycle
and, therefore, becomes human-centric. XAI-based intelligent systems will facili-
tate enhanced prediction accuracy with comprehensible decision and traceability of
actions performed and have significant impact in the development of the society
at large. In the futuristic computing scenario, the goal of explainable edge AI will
be to execute the AI tasks and produce explainable results at the edge. The issues
of transparency, fairness, accountability, explainability, interpretability, data fusion,
and comprehensibility that are significant for the edge AI are being addressed here
through explainable models and techniques. The book has been organized in ten
chapters highlighting the different technological aspects of explainable edge AI.
Chapter 1 of the book provides a brief overview of explainable artificial intel-
ligence (XAI) by covering almost every aspect of XAI. It covers the principles,
techniques, current state of the art, benefits, and applications of XAI. It can be
inferred from the chapter that XAI is a great development in AI due to its transparent
nature. The chapter also addressed the challenges in XAI, as well as the field’s future
potential. Chapter 2 of the book serves as a learning platform for academics and
practitioners who are interested in essential parts of the new and quickly increasing
field of XAI. The need of trust and transparency in AI and the principal applica-
tion domain of XAI have been primarily discussed in this chapter. Chapter 3 epito-
mizes contemporary developments in explainable AI that describes explainability in
machine learning, constituting a fiction definition of explainable machine learning
that envelopes such prior conceptual propositions with a considerable focus on the
audience for which the explainability is needed.
A wide and insightful view of XAI and its application in various fields has been
presented in Chap. 4 of this book. This chapter also includes the future scope of this
technology and the need for the growth of this type of technology. Chapter 5 presents
recent challenges on edge AI and its numerous applications. This chapter provides

v
vi Preface

a thorough analysis of edge computing-related AI techniques and tools, sometimes

known as edge AI. The objective is to offer a hypothetical roadmap that may unite
important players and enablers in order to the development of edge AI.
Chapter 6 highlights the usage of XAI in healthcare which is a critical application
for the human community. ‘How XAI improves user trust in high-risk decisions’ has
been elaborated in this chapter. Significant AI recommendations, such as surgical
treatments or hospitalizations, requiring explanation from providers and patients have
been discussed here. Chapter 7 focuses on describing the role of explainable edge AI
to resolve real-time problems. Various issues in the applicability of explainable edge
AI for real-time implementations have been highlighted in this chapter. Chapter 8
focuses on different data models of fusion, discusses a framework for AI and data
fusion at the edge, and identifies potential challenges and possible solutions in this
regard. While doing so, the chapter intends to cover the fundamentals of explainability
in AI, the need to convert the black box system into a transparent one, and the
associated opportunities for explainable artificial intelligence.
In Chap. 9 of this book, a novel technique in data fusion model with security
and data optimization technique in edge computing has been presented. Here, the
proposed data fusion is carried out using secure sequential discriminant autoencoder
in which the improvement of data accuracy, as well as for the maximizing of edge
cloud-based sensor networks lifespan. The fusion of edge cloud data has been carried
out using discriminant autoencoder which is integrated with distributed edge cloud
users, where the security of the network has been enhanced using secure sequential
fuzzy-based trust model. The integration of edge computing framework with the
sensor network that can aid in the data collection, dissemination, and decision making
has been performed in Chap. 10 of the book. This chapter proposes a LSTM-based
strategy for an edge computing-enabled WSN to determine the status of the node
depending on the network parameters such as number of communications, number
of packets transmitted, and initial energy of the nodes. The proposed protocol is
implemented using TensorFlow and Keras libraries in Python language.
A wide range of topics related to the integration of XAI and edge AI have been
presented in this book. We are sure that the book will provide a comprehensive
knowledge about explainability in edge AI to the readers.

Giza, Egypt Aboul Ella Hassanien

New Delhi, India Deepak Gupta
Roorkee, India Anuj Kumar Singh
Roorkee, India Ankit Garg
Contents

1 Explainable Artificial Intelligence: Concepts and Current

Progression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Kirti Kangra and Jaswinder Singh
2 Explainable Artificial Intelligence (XAI): Understanding
and Future Perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Megha Gupta
3 Explainable Artificial Intelligence (XAI): Conception,
Visualization and Assessment Approaches Towards Amenable
XAI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Tasleem Nizam and Sherin Zafar
4 Explainable AI (XAI): A Survey of Current and Future
Opportunities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Meet Kumari, Akshit Chaudhary, and Yogendra Narayan
5 Recent Challenges on Edge AI with Its Application: A Brief
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Kapil Joshi, Harishchander Anandaram, Manisha Khanduja,
Rajesh Kumar, Vikrant Saini, and Yasmin Makki Mohialden
6 Explainable Artificial Intelligence in Health Care: How XAI
Improves User Trust in High-Risk Decisions . . . . . . . . . . . . . . . . . . . . . 89
Sheeba Praveen and Kapil Joshi
7 Role of Explainable Edge AI to Resolve Real Time Problem . . . . . . . 101
Ambeshwar Kumar, T. M. Rajesh, Manikandan Ramachandran,
and Deepak Gupta
8 Explainable Data Fusion on Edge: Challenges
and Opportunities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Shweta Sinha and Priyanka Vashisht

vii
viii Contents

9 Trust Model Based Data Fusion in Explainable Artificial

Intelligence for Edge Computing Using Secure Sequential
Discriminant Auto Encoder with Lightweight Optimization
Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
D. Prabakar, M. Sundarrajan, S. Prasath Alias Surendhar,
Manikandan Ramachandran, and Deepak Gupta
10 A Deep Learning Based Target Coverage Protocol for Edge
Computing Enabled Wireless Sensor Networks . . . . . . . . . . . . . . . . . . 161
Pooja Chaturvedi, A. K. Daniel, and Umesh Bodkhe
About the Editors

Prof. Aboul Ella Hassanien is Founder and Head of the Egyptian Scientific Research
Group (SRGE) and Professor of Information Technology at the Faculty of Computer
and Artificial Intelligence, Cairo University. Professor Hassanien is an ex-dean of the
Faculty of Computers and Information, Beni Suef University. Professor Hassanien
has more than 800 scientific research papers published in prestigious international
journals and over 40 books covering such diverse topics as data mining, medical
images, intelligent systems, social networks, and smart environment. Prof. Hassanien
won several awards, including the Best Researcher of the Youth Award of Astronomy
and Geophysics of the National Research Institute, Academy of Scientific Research
(Egypt, 1990). He was also granted a Scientific Excellence Award in Humanities from
the University of Kuwait for the 2004 Award and received the Scientific University
Award (Cairo University, 2013). Also, he was honored in Egypt as the best researcher
at Cairo University in 2013. He also received the Islamic Educational, Scientific and
Cultural Organization (ISESCO) Prize on Technology (2014) and received the State
Award for excellence in engineering sciences 2015. He was awarded the Medal of
Sciences and Arts of the first class by the President of the Arab Republic of Egypt,
2017.

Deepak Gupta received a B.Tech. degree in 2006 from the Guru Gobind Singh
Indraprastha University, Delhi, India. He received an M.E. degree in 2010 from
Delhi Technological University, India, and Ph.D. degree in 2017 from Dr. A. P.
J. Abdul Kalam Technical University (AKTU), Lucknow, India. He completed his
post-doc from the National Institute of Telecommunications (Inatel), Brazil, in 2018.
He has co-authored more than 207 journal articles, including 168 SCI papers and
45 conference articles. He has authored/edited 60 books, published by IEEE-Wiley,
Elsevier, Springer, Wiley, CRC Press, DeGruyter, and Katsons. He has filed four
Indian patents. He is the convener of the ICICC, ICDAM, ICCCN, ICIIP, and DoSCI
Springer conferences series. He is Associate Editor of Computer and Electrical
Engineering, Expert Systems, Alexandria Engineering Journal, Intelligent Decision
Technologies. He is the recipient of the 2021 IEEE System Council Best Paper Award.
He has been featured in the list of top 2% scientist/researcher databases worldwide.

ix
x About the Editors

In India, he secured Rank 1 as a researcher in the field of healthcare applications

(as per Google Scholar citation) and ranked #78 in India among Top Scientists 2022
by Research.com. He is also working toward promoting startups and also serving
as Startup Consultant. He is also a series editor of Elsevier Biomedical Engineering
at Academic Press, Elsevier, Intelligent Biomedical Data Analysis at De Gruyter,
Germany, and Explainable AI (XAI) for Engineering Applications at CRC Press. He is
appointed as Consulting Editor at Elsevier. He accomplished productive collaborative
research with grants of approximately $144,000 from various international funding
agencies, and he is Co-PI in an International Indo-Russian Joint project of Rs. 1.31 Cr
from the Department of Science and Technology.

Anuj Kumar Singh is working as Professor in School of Computing at University

of Engineering & Technology (UETR), Roorkee, India. He has more than 18 years
of teaching experience in technical education. He holds a Ph.D degree in the field of
Computer Science and Engineering from Dr. A.P.J. Abdul Kalam Technical Univer-
sity, Lucknow. He passed M.Tech degree with First Distinction from Panjab Univer-
sity, Chandigarh and B.Tech degree with First Honours from U.P.T.U Lucknow in
Computer Science and Engineering. In addition to these, he has also qualified UGC
NET. Having published more than 30 research papers in journals and conferences
including Scopus and SCIE, he has also authored one book and edited two. He has
also filed four patents. His areas of specialization include Intelligent Systems, Cryp-
tography, Network and Information Security, Blockchain Technology, and Algorithm
Design.

Ankit Garg is working as Associate Professor in School of Computing at Univer-

sity of Engineering and Technology, Roorkee, India. He has more than 13 years
of teaching experience in technical education. He holds a Ph.D degree in the field
of Computer Science from Uttarakhand Technical University, Dehradun, India. He
passed M.Tech degree from Uttarakhand Technical University, Dehradun, India and
MCA degree from HNB, Garhwal University, Chauras Campus, Srinagar, Garhwal,
India. In addition to these, he has published more than 30 research papers in jour-
nals and conferences including SCIE and Scopus. His areas of specialization include
image processing, computer graphics, and computer vision.
Chapter 1
Explainable Artificial Intelligence:
Concepts and Current Progression

Kirti Kangra and Jaswinder Singh

Abstract AI is a broad term used in computer science to mimic the human mind.
Machine Learning is also a part of AI and is used in various fields of decision-
making by using different algorithms. In the last few decades, AI has gained more
popularity in the field of Industry, Medicine, Education, Defense, etc. Based on the
algorithm AI is categorized into: Interpretable AI and Explainable AI (XAI). In inter-
pretable AI, experts have problems in understanding the working of the model. They
have no idea of working behind the algorithm so from here the term explainable AI
came into existence. The term Explainable AI is coined by DARPA. This is also
known as Explainable Machine Learning. The term came into existence to provide
transparency in the process of decision-making by different Machine learning algo-
rithms. In the current scenario, XAI is gaining popularity because of its transparent
working. XAI is used in every field where AI can be used but with some modifica-
tion or by adding some techniques of XAI as “SHAP (Shaply Additive exPlanations),
DeepSHAP, DeepLIFT, CXplain, and LIME”. The main goal of this chapter is to
provide a brief overview of XAI by covering almost every aspect of XAI. The chapter
is divided into sections, in the first section brief introduction is discussed. Related
work concerning medicine is stated in Sect. 2. Later other sections will cover princi-
ples, techniques, current state of art, benefits, and, applications. This chapter infers
that XAI is a great development in AI due to its transparent nature. The chapter also
addressed the challenges in XAI, as well as the field’s future potential.

Keywords XAI · Interpretability · Comprehensibility · Transparency · Privacy ·

SHAP · LIME

Abbreviations

CMGE Counterfactual multigranularity graph supporting facts extraction

LNN Logical neural network

K. Kangra (B) · J. Singh

Department of Computer Science and Engineering, GJUS & T, Hisar, Haryana, India
e-mail: [email protected]

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 1

A. E. Hassanien et al. (eds.), Explainable Edge AI: A Futuristic Computing Perspective,
Studies in Computational Intelligence 1072,
https://doi.org/10.1007/978-3-031-18292-1_1
2 K. Kangra and J. Singh

GNN Graph neural network

ADR Adaptive dimension reduction

1 Introduction

Various industrial areas have adopted emerging technologies, and Artificial Intelli-
gence (AI) is at the root of them [1]. While the development of AI can be addressed
back decades, there is general accord that smart systems with learning, reasoning, and
adaptive skills are of critical significance. AI approaches are attaining remarkable
standards of accomplishment by attempting to address extremely hard computations
as a result of these capacities, considering these approaches crucial for future growth
[2]. Furthermore, intelligent agents such as Google Assistant, Siri, and others are
technologies increasingly integrating into individual personal lives. AI models, on
the other hand, have become a black box whose internal workings are not completely
transparent due to the increasing complexity of its algorithms. The conceptions of
explainability and interpretability emerge as a result of this trade-off. Models before
2015 have been viewed as examples of Interpretability AI due to abstraction of their
whole internal working. XAI was introduced by DARPA in 2015 to provide a thor-
ough knowledge of how AI models work. The XAI initiative intends to develop a
set of machine learning approaches that generate more comprehensible models by
retaining excellent learning performance. AI is at the base of several activity areas
that have incorporated modern information technology [3].
Explainability in AI is a discipline that has applications in a range of areas,
including medical treatment, business operations, privacy, economic and ethical
considerations, automated driving, smartphones, and AI for developers [4]. All of
the applications will be explained later in detail in the application section.
The most frequently used terminologies in XAI are:
• Understandability: Describes a model’s ability to convey its function—how
it works—to an individual despite having to elucidate its inner core or the
computational means by which the model processes data internally [5].
• Comprehensibility: If deployed in tandem with machine learning techniques,
comprehensibility alludes to learning algorithm’s capacity to express its learned
facts in a way that is understood by individuals.
• Interpretability: It is described as the potential to express or convey information
to an individual in a comprehensible way.
• Explainability: Explainability is related to the concept of description as a connec-
tion among individuals and a selection provider. The selection provided by the
selection provider can be understandable to individuals [6].
• Transparency: A model is said to be transparent if it can be understood on its own.
1 Explainable Artificial Intelligence: Concepts and Current … 3

The word “machine learning (ML)” was coined from the term “artificial intel-
ligence”. Both of these are interconnected and can be used in the same applica-
tion. Even though their functioning models and algorithms are identical. ML has
increased in popularity in both research and industry, owing to the effectiveness of
its numerous algorithms, especially in the medical field. In certain applications, a
computer glitch or a misinterpretation has some impact, but in the medical profes-
sion, it is linked to human life. Early diagnosis of the disease is often crucial for
patients’ recovery or to reduce the risk of disease from progressing to more severe
stages. Explainable Machine Learning and XAI can be used interchangeably.
The main objective of this chapter is to provide a brief overview of XAI. The
chapter is divided into sections. Section 1 provides an introduction to XAI. Section 2
discussed the current finding of the researchers. Section 3 stated the principles of
XAI. Later sections discussed Models, Current progression, Applications, Benefits,
and Challenges of XAI.

2 Literature Review

Number of work has been done in the field of health care using the XAI model. Some
of the latest research work has been stated in the section to provide a precise overview
of XAI in healthcare. The Research work in the section mainly concentrates on the
medicinal sector.
In Quellec et al. [7] proposed a unique explanatory AI approach for the classifi-
cation of multi-label images for the prediction of diabetes retinopathy. For the clas-
sification, CNN and Pixel-level algorithms were used. Indian Diabetic Retinopathy
Image Dataset was used and Python was used for the experiment. The accuracy of
the proposed model was 93%.
In Wu et al. [8] propose a counterfactual multi-granularity graph supporting facts
extraction method to extract supporting facts from the irregular EMR. The approach
used by them was suitable for Lymphedema. Techniques used by the researcher
were GNN, Counterfactual reasoning. Dataset was acquired from the Local hospital
in china. The result of the anticipated model was 99% for each parameter (precision,
recall, and F1-score value).
In Kavya et al. [9] developed a model for the diagnosis of allergy. Techniques used
in the model were KNN, SVM, C5.0, MLP, AdaBag, RF, and Condition prediction.
Dataset was taken from the allergy testing centers of South India and the experiment
was performed using R-Studio. The result of the anticipated model was 86%, and
75% for Accuracy and Sensitivity respectively.
In Amoroso et al. [10] developed an EXI system based on adaptive dimension
reduction to predict breast cancer. For the classification clustering technique was
used. Dataset was taken from Italy.
In Dindorf et al. [11] design a pathology-independent classifier that presents expla-
nations for the categorization findings and projection probabilities. Two datasets were
used and acquired, the first dataset was part of the dissertation project of Friederike
4 K. Kangra and J. Singh

Werthmann and the second was part of the dissertation project of Claudia Wolf. One-
class SVM, binary RF, and LIME approaches were used. The result of the classifier
were F1-80 ± 12%, MCC-57 ± 23%, BSS-33 ± 28%.
In El-Sappagh et al. [12] designed a model for detecting Alzheimer’s disease
using XAI which was precise and explanatory. In the proposed model two layer
RF, SHAP, and Fuzzy techniques were used. Alzheimer’s Disease Neuroimaging
Initiative dataset was used. The result of the proposed model was at the First layer:
accuracy: 93%, F1-score: 93%, and at the Second layer: Accuracy: 87%, F1-score:
87% respectively.
In Peng et al. [13] developed an XAI model to predict Hepatitis. In the proposed
model LR, DT, KNN, SVM, RF, SHAP, LIME, and partial dependence plots were
used. Dataset was taken from UCI and the experiment was performed using Python
3.6.4. The result showed an accuracy of 91%.
In Sarp et al. [14] proposed a CNN-based XAI model to classify chronic wounds.
CNN and LIME techniques were used. Dataset was taken from ekare Inc. and python
3.6 was used for the experiment. The results of the anticipated model were Precision,
Recall, and F1-score scored 95%, 94%, and 94% respectively.
In Tan et al. [15] develop a deep learning model based on high-resolution computed
tomography scans of the temporal bone to diagnose fenestral otosclerosis. The tech-
nique in the proposed model was LNN. Dataset was taken from the institutional
committee of Eye and ENT Hospital, Fudan University. The result of the anticipated
model was AUC, Sensitivity, and Specificity scored 99%, 96%, and 98% respectively.
From the literature, it has been seen that XAI is used in healthcare to predict and
classify the number of diseases like Diabetes, Spine, Chronic wounds, etc. Different
XAI models such as LIME, SHAP, Fuzzy, Deep attributes, etc. are used to clas-
sify disease. By using the XAI model results of the traditional models have been
increased as seen in the literature. Using these models Accuracy has been increased
to a remarkable extent. So, it can be concluded that XAI has more potential and its
potential should be utilized in the field of health care.

3 Principles

The principles are the collection of rules that should follow by any XAI model. There
are three principles of XAI (see Fig. 1).

Fig. 1 Principles of XAI PRINCIPLES

Transparency Interpretability Explainability

1 Explainable Artificial Intelligence: Concepts and Current … 5

Principles are explained below:

• Transparency: If the system developer can demonstrate and explain how to obtain
model parameters from training data and produce labels from testing data [16].
• Interpretability: It is described as the capability to express or convey information
to an individual in a comprehensible way [17].
• Explainability: Explainability is related to the concept of description as a connec-
tion between individuals and a selection provider. The selection provided by the
selection provider can be understandable to individuals.
In response to the expansion of significance of this area, NIST outlined four
principles of XAI [18] in August 2020, which describe the following key principles
that an AI must adhere to be termed as XAI:
• Explanation: According to this principle, an AI system must provide proof,
support, or justification for whatever choice it makes.
• Meaningful: This principle argues that the AI system’s explanations must be
comprehensible and significant to the users. Because diverse communities may
have distinct interests and perspectives, the AI system’s interpretation must be
smooth to fit each group’s unique traits and requirements.
• Accuracy: This concept asserts that the AI system’s interpretation must properly
represent the system’s processes.
• Knowledge limits: This principle asserts that AI systems must recognize situations
in which they were not intended to work, and in that situation, their responses
may be unreliable.

4 Models

This section of the chapter covers XAI techniques to provide a precise overview. “XAI
is a research field on ML interpretability techniques whose aims are to understand
ML model predictions and explain them in human and understandable terms to build
trust with stakeholder”. To understand the ML model XAI comes with different
models.
To understand the XAI model user need to understand two terms as shown in
Fig. 2.

1. Model-Based: That develop interpretable ML models

2. Post hoc: Derive explanations for complex ML models
• Black Box Models (BB): models whose internal working cannot be easily
explainable.
• White Box Models (WB): models whose internal working can easily be
understood.

To understand the working of the BB, WB is used. XAI came into existence due
to trade-off between “interpretability and explainability. Two approaches: Integrated
6 K. Kangra and J. Singh

Fig. 2 Machine learning Machine Learning Models

model

Model Based Post Hoc

Black Box

White Box

Fig. 3 XAI classification

(transparency-based) and Post-hoc” models [19] came under “interpretability and

explainability Fig. 3. Provide an overview of the classification of XAI. Some terms
used in Fig. 3 are used further in this section.
• Integrated Methods
Integrated interpretability is a term that refers to the ability to interpret data,
Model’s best description is if it properly depicts itself and is straightforward
to comprehend [20]. This method is only applicable to model families with
a moderate level of complexity such as “linear models, decision trees, and
rules”. Other model communities, such as “artificial neural networks and support
vector machines, boosted trees, and random forests” are labeled as opaque
models because of their complexity, which makes it problematic for users to
realize the motivation underlying predictions.
Post-hoc interpretability retrieves information from a previously learned model
and is unaffected by the model’s behavior. The advantage of this method is that
it does not affect the model’s performance because it is considered BB. This
is familiar with how individuals justify their own decisions when they don’t fully
understand how their decision-making techniques work.
• Post-hoc Methods
The high potential benefits of utilizing complicated, opaque models are increasing
as hardware improves and data becomes more readily available. Interpretability
1 Explainable Artificial Intelligence: Concepts and Current … 7

SHAP

Feautre based concept

CAM
Model
Grad-
CAM
Global Attribute
Mapping
XAI Global Gradient based
Approaches Saliency Map
Deep attribute
map

Concept Activation Vector

Local Interpretable Model-

agnostic Explanations
Layerwise Relevance
Propagation

Fig. 4 XAI models

and explainability, on the other hand, are challenges that must be appropriately
addressed.
Interpretability: The transparent proxy model approach identifies an inter-
pretable model that estimates the black-box model’s predictions generally.
Explainability: For each prediction, these algorithms typically provide a
description in the form of feature significances for concluding. In [21], sensi-
tivity analysis and layer-by-layer relevance propagation were used to explain
deep learning model predictions in terms of input variables.
To make the BB model explainable XAI approaches came into existence as
shown in Fig. 4.

4.1 Features-Based Approaches

“Shapley Additive explanation (SHAP)” is a game-theoretic technique to expli-

cate machine learning predictions [20]. By modeling the features as participants
in an alliance game, SHAP aims to determine how much each feature influenced a
choice. The rewards of the game are an additive indicator of significance which is
known as the Shapley value that reflects the weighted average impact of a specific
feature for each and every feasible set of features. As a result, a model’s local and
global explanations are compatible, and the average prediction is evenly spread along
with all Shapley values, allowing for examinations of different explanations. More-
over, although predictive models may include non-independent pay-off splits, the
8 K. Kangra and J. Singh

explanation of the Shapley values is not always straightforward if the model is not
additive. Furthermore, while SHAP is model agnostic, optimizing the SHAP algo-
rithm for all model types is not always simple or effective.
CNN’s have their class activation maps (CAMs). The per-class weighted linear
sum of subjective features available at diverse geographical places in a picture is
represented by CAMs [22]. Global average pooling is applied to the last convolutional
feature map of a network on the output layer. The input features for a fully linked
layer, which is then output via a loss function, are then pooled with feature maps.
The portions in the input image that have a stronger effect on the CNNs’ choice are
emphasized per class and apparent through a heatmap visualization by projecting
the output weights back to the earlier convolutional layer. Pre-trained networks and
networks that do not follow the stated fully convolutional network design are not
eligible for CAMs. Additionally, the completely attached layer and map scaling
can cause spatial information to be lost. Grad-CAM and Grad-CAM ++ are two
expansions of the original CAM model that aim to improve the explainability of
CNNs.
Grad-CAM [23] generalizes class activation mapping (CAM) to any random CNN
architecture without training. Any target class’s gradients are delivered to the last
convolutional layer, which computes a significance score based on the gradients. A
heatmap illustration of the Grad-CAM, like other approaches, shows which portions
of the input image were most essential in the CNN’s judgments. GradCAM, on
the other hand, only creates coarse-grained conceptions and is unable to describe
numerous cases of identical objects in a single image.
To circumvent these limitations, Grad-Cam ++ [24] uses the weighted average
of the gradients. Feature-oriented methods reveal where a choice is being achieved
from the perspective of the input, but they lack at human-level description of how
and why the model arrived at those conclusions.

4.2 Global Approaches

Global attribution mappings techniques on a global scale [25] can interpret as neural
network’s (NN’s) predictions on a comprehensive level, beyond subpopulations, by
framing perception as weighted combined ranks for features with exact semantic
descriptions. The benefits include the ability to capture diverse subpopulations using
a granularity parameter that may be tuned. GAMs employ K-medoids clustering
technique to assemble similar local feature significances into clusters and determine
a pair-wise rank distance matrix among features. As a global attribution, the medoid
of the respective cluster describes the outline found in each cluster. As a result, this
method can be used to investigate features in distinct subpopulations of samples.
Gradient-based saliency maps [26] are mapping approach that displays the abso-
lute value of the dominant predicted class’s gradient (concerning the input features) as
a normalized heatmap. The pixels with the highest activation levels are emphasized,
and they are parallel to the most significant locations. The method’s description is the
1 Explainable Artificial Intelligence: Concepts and Current … 9

capability for an individual to sight which features in the image are being considered
in the categorization of the result. When propagating nonlinear layers, conversely, the
absolute value signifies that the gradient of neurons with negative input is inhibited.
Deep attribute maps are proposed by [27] as an approach for presenting the
generalization ability of gradient-based algorithms. The suggested layout also shows
comparisons between various saliencey-based explanatory techniques. The gradient
of the output is combined with the relevant input to give a heatmap that explains
a model’s prediction. The colors red and blue denote positive and negative depic-
tions of the final choice, respectively. Explanations are affected by noisy gradients
and input variability. Deep attribute is unable to justify what is the reason that two
models are providing similar or dissimilar results.

4.3 Concept Activation Vector Approach

In [28] concept activation vectors (CAVs) are the strategy for explaining the global
internal states of NN’s neurons to the NN’s high-level latent properties. As a result,
CAVs define the notch at which these conceptual attributes refer to a series of human
ideas that the user has chosen. Human bias is unavoidable, but by illuminating the
corresponding perception, any errors in the model’s decision-making process can be
addressed, such as if some features are erroneously considered important. Beyond
that, automatic concept-based explanations [29] extract CAVs without the need for
human intervention, eliminating human bias. Human intelligible perceptions are
split at numerous spatial dimensions from in-class visuals rather than being chosen.
Concept-based strategies, on the other hand, rely on the perceptions being unique to
the class, and the efficiency of description is impaired if the same perception is used
in various sessions.

4.4 LIME Approach

“LIME (Local interpretable model-agnostic explanations)” [30] is a model-agnostic

approach for producing nearby optimized ML model interpretations. LIME uses an
understandable surrogate model to acquire the local behavior of predictions from a
comprehensive “black box” model. For image classification, a picture is segmented
into chunks of continuous super-pixels, after which a penalized local classifier is
taught on a novel set of permuted samples of the original image. The concept is
to determine what the input delivered to every given class by modifying human-
understandable bits of the input data and learning the discrepancies between those
disruptions and the original image. On the other hand, if the parameters that determine
the perturbations are only dependent on preconceptions, these perceptions aren’t
always relevant or trustworthy on a human level.
10 K. Kangra and J. Singh

4.5 LRP Approach

“Layer-wise relevance propagation (LRP)” [31] explains the output of a multi-layered

NN in terms of its input using established propagation rules. The method produces
a heatmap that shows which pixels affect the model’s prediction and to what extent.
As a result, LRP places a premium on major impacts on a network’s conclusion.
While LRP can be used on a network that has already been trained, this is a post-hoc
approach that only gives a truncated consideration of the features’ contribution to
the decision.

5 Current Progression

AI started in 1950. After that, a new era of human intelligence started. In 1963, the
term Machine Learning was coined to train machines. By using ML automated era of
machines begins, machines can perform analysis tasks by themselves. ML increases
accuracy in analysis tasks by reducing human error. Both ML and AI are used in
different sectors. But in 1976, Deep learning (DL) came into existence, especially
for medical services. But AI, ML, and DL have problem that their inner working
can’t be explained. From there two terms as explained at beginning of this chapter
came into existence.
In 2015, the term XAI was coined to explain the inner working of different models
of AI. It uses different approaches explained above to make the model’s working
transparent. Now XAI is used in many sectors.
According to the current status of XAI, it is employed in a variety of industries like
finance, manufacturing, défense, stock, and healthcare. It is gaining popularity in the
field of health care due to its nature and characteristics. It is utilized by medical experts
in healthcare to gain greater precision in decision-making processes, as there is a need
for more accuracy in the medical area. Its various strategies combine with existing
artificial intelligence techniques to deliver more accurate results. It is mostly utilized
in medical image processing for further analysis. In the sector of image processing,
it helps by using saliency maps. For image classification, Deep attribute maps are
considered the best choice. These maps are important for determining which parts
of a picture are considered by the model for prediction and which are not to make an
accurate prediction. In the sector of Video threat detection, it is used to detect any
forgery in the video.
1 Explainable Artificial Intelligence: Concepts and Current … 11

In the current scenario, XAI models are used in various applications of

ML some glimpses are: SHAP is one of the most powerful tool of XAI. It
can be used in the field of medicine. By measuring the contribution of each
feature to the prediction, it may be used to explain the results of any machine learning
model. It is based on the ideas of game theory. LIME is used in Image analysis. All
the models which are explained above can be used with any of the ML models to
make their working explainable.

6 XAI in Medicines

The combination of XAI and healthcare is a powerful digital technology. The emer-
gence of AI-based diagnosis tools has sparked serious doubts regarding the accuracy
of AI-based outcomes. In the medical and healthcare sectors of the virtual sector,
trust is critical to AI’s success. If-else diagnostic models are organically explainable
since they are made up of feature value sets and deliver a score based on a feature
value of a specific case of disease detection. External symptomatic disease diag-
nosis is highly suited for if-else-based explainable medical diagnosis systems. If the
person’s symptoms list has a certain number of matching criteria using If-else-based
feature values, it can be determined if the person had a particular disease or not.
For instance, if the patient has a prior medical history. For example, if the patient
has a history of respiratory infection and cough, the odds of developing asthma are
increased. To cover a wide range of XAI, an explanation that is independent of the
AI model is required. Model agnostic are XAI techniques. A model agnostic method
is LIME. LIME is a framework for quantifying the weights of all aspects that go into
reaching a decision or making prediction. Other model-independent XAI approaches
exist, such as SHAPLEY [31]. Deep learning is a valuable tool for precise medical
diagnosis, but its black-box approach to prediction and conclusion limits it to a few
crucial areas of medicine. Some applications of XAI in medicine are shown in Fig. 5.

6.1 Seven Pillars of XAI in Medicine

There are 7 pillars of XAI in medicine which are discussed below [32]. These are
important in the implantation of the XAI models and will help to make any model
explainable in medicine.
• Transparency: Understandable by any user.
• Domain Sense: The explanation should make sense in the context of the
application and to the system’s user.
• Consistency: Across all models, the explanation should be consistent.
• Parsimony: It’s best if the explanation is as straightforward as possible.
12 K. Kangra and J. Singh

Fig. 5 XAI application in

medicine
Medical
Image
Anaylsis
Personlized
Medication Diagnosis
& care

XAI in
Medicine

Virtual R&D in
Monitoring Drug

Surgery

• Generalizability: Explainability and interoperability should apply to prediction

algorithms.
• Trust/Performance: The expectation that the related prediction algorithms for
explanation should function well.
• Fidelity: Both the implementation and the explanation should be in harmony.

7 Applications of XAI

In theory, the scope of XAI should be as broad as that of AI. Natural language
processing (NLP), health care, engineering, and military are just a few of the impor-
tant areas. NLP and engineering are used in banking, finance, digitization, and
automation [33].
• Data Security: The European Union and its data protection regulatory agency have
a “right to explanation” clause, that helps XAI algorithms to come to a decision
without breaching data security.
• Medical: Based on a patient’s medical history, XAI can provide a diagnosis. The
use of AI/ML techniques in the healthcare image processing area makes it quicker
for doctors to spot cancer tumors and other lung diseases in patients.
• Défense: As a result of computerized artillery and monitoring systems, XAI is
becoming more important in defensive activity. During battle mode training and
real conflict strategy, XAI also gives excellent second-hand help.
1 Explainable Artificial Intelligence: Concepts and Current … 13

• Banking: One of the most important financial industries with the largest impact
on people’s lives is banking. Every day, a large number of fraudulent transactions
and conns by cheaters take place. Well-trained XAI models can help with both
the investigation and decrease false positives in fraudulent transactions.

8 Benefits of XAI

The objective of XAI is to describe how certain outcomes or suggestions are made. It
explains why AI performs as it does and fosters trust among people and AI models.
The main advantages of XAI are [34]:
• Improved explainability and transparency: Businesses can have a deeper under-
standing of advanced AI models and they act in various ways under certain circum-
stances. Humans can utilize an explanation interface to comprehend how these
AI models arrive at particular results, even if it’s a black-box model.
• Faster adoption: As organizations expand their understanding of AI models, they
will be able to entrust them with more crucial judgments.
• Improved debugging: When a system behaves strangely, XAI can be used to
diagnose the problem and assist developers in troubleshooting.
• Allowing for audits to meet regulatory obligations.

9 Challenges in XAI

Explainability is both theoretically and socially compelling. XAI is an extremely

effective diagnostic mechanism that can deliver information well beyond what stan-
dard linear models could but to implement XAI researchers need to face challenges.
Some challenges of XAI are listed below [35]:
• Confidentiality: An algorithm may be proprietary and revealing it would be a
security risk. How can one be confident that the AI system hasn’t picked up a
biased view of the world due to flaws in the training data, model, or goal function?
• Complexity: Algorithms can be simple to grasp but complicated to deploy. As a
result, an inexperienced person’s conception is surreal, and this is an area where
XAI techniques may be effective. As XAI can provide more efficient and under-
standable algorithms.
• Irrationality: Algorithms that provide plausible, biased, or out-of-line conclusions
based on credible evidence. How can one be confident that an AI system’s choices
are fair? Rationality is logical, but it must be viewed in the context of the particular
data input given to AI systems.
• Injustice: To realize the working of the algorithm is easy but to know how the
system complies with a legal or moral norm.
• Lack of expertise: The majority of people will lack the necessary competence to
comprehend the explanation and rate the decision’s fairness.
14 K. Kangra and J. Singh

• Context-dependency: Algorithms can’t be discussed at a high level because the

results vary from person to person.
• Dynamics of data and decisions: Because data and decisions shift during time,
narratives must alter as well.
• Interference of algorithms: Often, a series of operations is required to collect and
process data from multiple sorts of sources, and many, often disparate algorithms
are employed.

10 Future of XAI

The global XAI market is expected to grow at a CAGR of 18.4% from 2022 to 2030,
from USD 4.4 billion in 2021 to USD 21.0 billion in 2030. Explainable AI is an
artificial intelligence method or technique in which the solution can be evaluated and
understood by humans. It differs from standard ML techniques, in which researchers
frequently fail to comprehend why the system has reached a particular conclusion.
Healthcare, retail, media and entertainment, aerospace, and défense are among the
areas that are embracing XAI. For example, in the retail industry, XAI is critical for
anticipating new fashion trends with reason, allowing retailers to stock and display
the latest products in their stores. Furthermore, in the e-commerce industry, the
notion of recommendation engines is an example of XAI, in which the system offers
things to the consumer based on his or her search history. Additionally, recommen-
dation engines promote accessories/additional products in addition to the one being
purchased. This is accomplished through the use of explainable AI, which allows the
system to recommend things that are beneficial to the customer [36].
Future investigations of medical XAI applications should include medical special-
ists in the design and development stages. Interdisciplinary collaboration is required
for a good medical XAI application. Medical experts, in particular, should provide
their prior medical expertise, as well as suggestions and feedback, to help enhance
the layout of AI algorithms. Medical XAI apps should be able to support doctors in
drawing explainable clinical inferences, according to AI experts and data scientists.
As a result, we expect that medical XAI models will gain acceptance in the medical
field.

11 Conclusion

XAI is, without a dispute, an important collaborative study in the AI system. The
ethical and quality-of-life considerations of utilizing AI in its existing situation in
real-world circumstances are problematic. This chapter delivers insight into XAI,
which has recently been identified as a critical necessity for ML methodologies to
be embraced in real-world applications. This chapter has focused on this topic by
highlighting the search for better interpretable ML algorithms, as well as discussing
1 Explainable Artificial Intelligence: Concepts and Current … 15

the principles that drive model explainability. In this chapter, various XAI techniques
such as LIME, SHAP, DEEPSHAP, etc. are covered. Every technique has its own
merits and demerits. In Shap, Shap values are calculated, based on those values
researchers will come up with a decision. If there is an error in the calculation of the
Shap value that will affect the decision of the researcher. This chapter also discussed
different applications and their need in the medical field and their future perspective
in every field especially concentrating on the medical field. By looking at the whole
scenario XAI is a powerful technique of AI. It can be concluded that this can act as
a game changer in the healthcare sector due to its transparent nature. Physicians can
easily make accurate decisions without the intervention of any technical expert.

References

1. D.M. West, The future of work: robots, AI, and automation. Futur. Work Robot. AI Autom.
1–205 (2018)
2. A. Barredo Arrieta et al., Explainable artificial intelligence (XAI): concepts, taxonomies, oppor-
tunities and challenges toward responsible AI. Inf. Fusion 58, 82–115 (2020). https://doi.org/
10.1016/j.inffus.2019.12.012
3. L.F. Huang, Artificial Intelligence, vol. 4 (2010)
4. Explainable AI—India | IBM. https://www.ibm.com/in-en/watson/explainable-ai. Accessed 12
June 2022
5. G. Montavon, W. Samek, K.R. Müller, Methods for interpreting and understanding deep neural
networks. Digit. Signal Process. A Rev. J. 73, 1–15 (2018). https://doi.org/10.1016/j.dsp.2017.
10.011
6. S.T. Mueller, R.R. Hoffman, W. Clancey, A. Emrey, G. Klein, Explanation in human-AI
systems: a literature meta-review. Def Adv Res Proj Agency 204 (2019)
7. G. Quellec, H. Al Hajj, M. Lamard, P.H. Conze, P. Massin, B. Cochener, ExplAIn: explanatory
artificial intelligence for diabetic retinopathy diagnosis. Med. Image Anal. 72(2018) (2021).
https://doi.org/10.1016/j.media.2021.102118
8. H. Wu, W. Chen, S. Xu, B. Xu, Counterfactual supporting facts extraction for explainable
medical record based diagnosis with graph network 1942–1955 (2021). https://doi.org/10.
18653/v1/2021.naacl-main.156
9. R. Kavya, J. Christopher, S. Panda, Y.B. Lazarus, Machine learning and XAI approaches for
allergy diagnosis. Biomed. Signal Process. Control 69, 102681 (2021). https://doi.org/10.1016/
j.bspc.2021.102681
10. N. Amoroso et al., A roadmap towards breast cancer therapies supported by explainable artificial
intelligence. Appl. Sci. (2021). https://doi.org/10.3390/app11114881
11. C. Dindorf et al., Classification and automated interpretation of spinal posture data using
a pathology-independent classifier and explainable artificial intelligence (XAI). Sensors
(Switzerland) 21(6323) (2021). https://doi.org/10.3390/s21186323
12. S. El-Sappagh, J.M. Alonso, S.M.R. Islam, A.M. Sultan, K. Sup, A multilayer multimodal
detection and prediction model based on explainable artificial intelligence for Alzheimer’s
disease (2021)
13. J. Peng et al., An explainable artificial intelligence framework for the deterioration risk predic-
tion of hepatitis patients. J. Med. Syst. 45(5) (2021). https://doi.org/10.1007/s10916-021-017
36-5
14. S. Sarp, M. Kuzlu, E. Wilson, U. Cal, O. Guler, The enlightening role of explainable artificial
intelligence in chronic wound classification. Electronics 1406 (2021). https://doi.org/10.3390/
electronics10121406
16 K. Kangra and J. Singh

15. W. Tan et al., The use of explainable artificial intelligence to explore types of fenestral otoscle-
rosis misdiagnosed when using temporal bone high-resolution computed tomography. Ann.
Transl. Med. 9(12), 969–969 (2021). https://doi.org/10.21037/atm-21-1171
16. D. Castelvecchi, The black box 2 0 |. Nature 538(7623), 20–23 (2016). [Online]. Available:
http://www.nature.com/news/can-we-open-the-black-box-of-ai-1.20731
17. L.H. Gilpin, D. Bau, B.Z. Yuan, A. Bajwa, M. Specter, L. Kagal, Explaining explanations: an
overview of interpretability (2019). [Online]. Available: https://paperswithcode.com/methods/
category/interpretability
18. P.J. Phillips et al., NISTIR 8312 four principles of explainable artificial intelligence (2021)
19. F.K. Dosilovic, M. Brcic, N. Hlupic, Explainable artificial intelligence: a survey, in 2018 41st
International Convention on Information and Communication Technology, Electronics and
Microelectronics MIPRO 2018—Proceedings, pp. 210–215 (2018). https://doi.org/10.23919/
MIPRO.2018.8400040
20. S.M. Lundberg, S.I. Lee, A unified approach to interpreting model predictions. Adv. Neural
Inf. Process. Syst. 2017(Section 2), 4766–4775 (2017)
21. W. Samek, T. Wiegand, K.-R. Muller, Explainable artificial intelligence: understanding,
visualizing and interpreting deep learning models (2017)
22. B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, A. Torralba, Learning deep features for discrim-
inative localization, in Proceedings IEEE Computer Social Conference on Computer Vision
and Pattern Recognition, vol. 2016, pp. 2921–2929 (2016). https://doi.org/10.1109/CVPR.201
6.319
23. R.R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, Grad-CAM: visual
explanations from deep networks via gradient-based localization, in Proceedings of the IEEE
International Conference on Computer Vision, vol. 2017, pp. 618–626 (2017). https://doi.org/
10.1109/ICCV.2017.74
24. A. Chattopadhay, A. Sarkar, P. Howlader, V.N. Balasubramanian, Grad-CAM++: generalized
gradient-based visual explanations for deep convolutional networks, in Proceedings—2018
IEEE Winter Conference on Applications of Computer Vision, WACV 2018, vol. 2018, pp. 839–
847 (2018). https://doi.org/10.1109/WACV.2018.00097
25. M. Ibrahim, M. Louie, C. Modarres, J. Paisley, Global explanations of neural networks 279–287
(2019). https://doi.org/10.1145/3306618.3314230
26. K. Simonyan, A. Vedaldi, A. Zisserman, Deep inside convolutional networks: visualising
image classification models and saliency maps, in 2nd International Conference on Learning
Representations ICLR 2014—Workshop Track Proceedings, pp. 1–8 (2014)
27. M. Ancona, E. Ceolin, C. Öztireli, M. Gross, Towards better understanding of gradient-based
attribution methods for deep neural networks (2018)
28. B. Kim et al., Interpretability beyond feature attribution: quantitative testing with concept
activation vectors (TCAV), in 35th International Conference on Machine Learning ICML 2018,
vol. 6, pp. 4186–4195 (2018)
29. A. Ghorbani, J. Zou, J. Wexler, B. Kim, Towards automatic concept-based explanations, in
NeurIPS (2019)
30. J. Dieber, S. Kirrane, Why model why? Assessing the strengths and limitations of LIME (iii)
(2020). [Online]. Available: http://arxiv.org/abs/2012.00093
31. P. Gohel, P. Singh, M. Mohanty, Explainable AI: current status and future directions, 1–16
(2021). [Online]. Available: http://arxiv.org/abs/2107.07045
32. C.E. Muhammad Aurangzeb Ahhmad, A. Teredesai, V. Kumar, K. Inc., Explainable-AI-in-
healthcare (2021)
33. XAI (Explainable AI) & top 5 use cases—GPU ON CLOUD. https://gpuoncloud.com/xai-exp
lainable-ai-top-5-use-cases/. Accessed 12 June 2022
34. Explainable AI (XAI) in 2022: Guide to enterprise-ready AI. https://research.aimultiple.com/
xai/. Accessed 12 June 2022
1 Explainable Artificial Intelligence: Concepts and Current … 17

35. Explainability of AI: The challenges and possible workarounds | by Rohitha Elsa Philip |
Medium. https://medium.com/@rohithaelsa/explainability-of-ai-the-challenges-and-possible-
workarounds-14d8389d2515. Accessed 12 June 2022
36. S.T. Mueller, R.R. Hoffman, W. Clancey, A. Emrey, G. Klein, Explanation in human-
AI systems: a literature meta-review (2021). https://www.nextmsc.com/report/explainable-ai-
market. Accessed 12 June 2022
Chapter 2
Explainable Artificial Intelligence (XAI):
Understanding and Future Perspectives

Megha Gupta

Explanations are critical for users to comprehend, trust, and

handle sophisticated artificial intelligence technologies
efficiently.

Abstract Currently, we are seeing a rapid and broad acceptance of artificial intel-
ligence (AI) in our everyday lives, which is contributing to the acceleration of the
transition towards a more algorithmic society. The availability of enormous datasets,
as well as recent advancements in deep learning methods, are allowing AI systems to
perform at or even beyond the level of human performance on an expanding variety
of challenging tasks. However, because of their layered non-linear structure, these
strong models have traditionally been regarded as “black boxes,” since they provide
no information about how they arrive at their predictions and hence cannot be used
trustily to generate predictions. They are often opaque, but they are capable of making
accurate predictions that cannot be explained in any other way. All of these choices
are having an increasing amount of effect on the lives of individuals. The distrust of
totally non-human, autonomous artificial intelligence systems is established as the
cornerstone of the movement. The root of distrust lies in a lack of knowledge as to
why intelligent systems make certain decisions in certain situations. As a result, this
problem has sparked a fresh discussion over explainable artificial intelligence (XAI).
Several lines of research work have since picked up on the definition, understanding,
and implementation of explainability, including expert systems, machine learning,
recommender systems, and approaches to neural-symbolic learning and reasoning,
all of which have occurred primarily during different periods in the history of artificial
intelligence. This chapter serves as an introduction for academics and practitioners
who are interested in learning about essential parts of the new and quickly increasing
field of research on XAI. The first section, titled “Introduction,” provides an overall
summary of the Explainable Artificial Intelligence. Section 2 describes the need of
trust and transparency in AI, which is what led to the development of the idea of XAI.
Section 3 discusses the many approaches that contribute to the functioning of XAI.
The principal applications domain of Explainable Artificial Intelligence is the topic

M. Gupta (B)
Department of Computer Science, MSCW, University of Delhi, New Delhi, India
e-mail: [email protected]

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 19

A. E. Hassanien et al. (eds.), Explainable Edge AI: A Futuristic Computing Perspective,
Studies in Computational Intelligence 1072,
https://doi.org/10.1007/978-3-031-18292-1_2
20 M. Gupta

of discussion in Sect. 4. In Sect. 5, the upcoming difficulties and potential directions

that this developing technology may take are discussed.

Keywords Explainable artificial intelligence · Transparency · Trust in AI · XAI

methodology · Perturbation · Backpropagation

Abbreviations

AI Artificial intelligence
ANN Artificial neural network
CAM Class activation mapping
DeConvNet Deconvolution neural network
HCI Human computer interaction
IG Integrated gradients
LIME Local interpretable model-agnostic explanations
LRP Layer-wise relevance backpropagation
ML Machine Learning
SHAP SHapley Additive exPlanations
SR Salient relevance
VQA Visual question answering
XAI Explainable artificial intelligence

1 Introduction

End users may better understand, trust, and control the next generation of AI systems
with the help of the XAI program’s new or updated ML approaches, which are
designed to develop explainable models. XAI is aimed for the end user who relies on
AI systems to make decisions or suggestions, or take action, and needs to understand
the system’s reasoning. If, for example, a big data analytics system recommends that
an intelligence analyst follow up on a given behavior, the analyst has to know why.
It’s also important for an operator to grasp the system’s decision-making process if
he or she wants to deploy an autonomous vehicle for future missions.
AI that is capable of producing explanations for how it works is dubbed XAI
(explainable Artificial Intelligence). The difference between present AI process and
the contribution of XAI to it could be easily understood through Figs. 1 and 2.
The purpose of explanations is to aid users in maintaining and effectively using the
system model, as well as to assist users in debugging the model to avoid and correct
wrong conclusions. Explanations, on the other hand, can be used for instructional
purposes as well as help individuals discover and grasp new concepts in an application
2 Explainable Artificial Intelligence (XAI): Understanding … 21

Input Data Learning Learned Output to

Process Function User

Fig. 1 Present AI process

Input Data Learning

Process

Explainable Output to
User
Model

Knowledge Decision Making/ User could

for Data Explainable Process Query

Fig. 2 Emerging XAI process

area. It’s important to keep in mind that explanations play an important role in
persuading users that the system’s decisions are best for them.
The goal of an explainable artificial intelligence (XAI) system is to make its
behavior more understandable to humans by offering explanations for its actions.
There are several broad ideas that may be used to the development of more effective
and human comprehensible artificial intelligence systems. When asked to describe
its capabilities and understandings, the XAI system should be proficient to describe
what it has done so far and what it plans to accomplish in future; it should also be
able to divulge the relevant information on which it is acting. While explainability in
artificial intelligence has lately garnered a great deal of attention, the beginnings of
this line of research may be traced back many decades to a time when AI systems were
primarily created as (knowledge-based) expert systems and other similar systems.
The authors of works [1–3] provide a summary of the increased activity in XAI
across a variety of fields and industries. The authors of [1] highlight how important
it is to support the output of a model, e.g., in precision medicine, where specialists
need significantly more information from the model to support their diagnosis than
a simple binary prediction. The author of study [2] stressed that there is a trade-
off between the performance of a model and its transparency, and that this is a
very important consideration. It is a common misconception to believe that if one
just concentrates on performance, the resulting systems would become opaquer.
22 M. Gupta

Comprehensibility, as described by the author, is the capacity of a learning algorithm

to express its learnt information in a human intelligible way; nevertheless, owing to
the difficulty of comprehensibility’s measurement, it is often related to the assessment
of the model complexity [3].
There is a substantial body of research and writing on explainable or inter-
pretable machine learning. An overview of interpretable machine learning methods
and approaches is offered in the work referred to as [4]. In addition, a taxonomy
of explanation models is provided, which divides them into three categories: global
techniques, local methods, and introspective approaches.
For a very long period, aspects of explanations’ understandability for lay users
were neglected to a greater or lesser extent. As it has been also mentioned in [5], the
majority of implementations do not concentrate on the end-users, who are the ones
who are impacted by the model; rather, they concentrate on the machine learning
developers, who employ explainability to debug the model itself. In actuality, there is
a disconnect between the purpose of transparency and the concept of explainability.
This is due to the fact that explanations mainly serve “internal” stakeholders rather
than “external” ones. In order to overcome this difference, reasons need to be human-
understandable while also being flexible to the various stakeholders [6]. For a system
to be considered trustworthy, its explanations need to be tailored to the specific needs
of the many kinds of users, with each user receiving information that is relevant to
their situation and carefully chosen. Because of this, an approach to explainable
artificial intelligence that centers on the needs of the user is required.
For a system to be considered trustworthy, its explanations need to be tailored to
the specific needs of the many kinds of users, with each user receiving information
that is relevant to their situation and carefully chosen. Because of this, a concept to
explainable artificial intelligence that centers on the needs of the user is required. In
this context, criteria underlying Responsible AI have been described in [7], which
state that justice, responsibility, and privacy (particularly in relation to data fusion)
should be evaluated while applying AI models in actual contexts.
Because the end user is going to be a human being, investigating the role that
explanations play in the interaction between humans and machines will be an essential
future study area. The research [8] looked at the role that humans play in AI that can
explain itself. A successful human–machine interaction requires, as a precondition,
the construction of explanations with the appropriate user emphasis, i.e., the asking
of the appropriate questions in the appropriate manner.

2 Need for Transparency and Trust in AI

Most of today’s applications now use AI black box systems. Transparency and
explainability are not essential if the overall performance of the system is good
enough with the machine learning model deployed. When these technologies fail,
the repercussions aren’t anything out of the ordinary, such as the mobile phone camera
not being able to identify a person or the translation service producing grammatical
2 Explainable Artificial Intelligence (XAI): Understanding … 23

errors. Thus, the openness and trust requirements for these AI systems are rather
low. The scenario is substantially different in critical applications. There may be a
problem with the lack of transparency of ML approaches in this case. It may not be
an option to rely on a data-driven system whose rationale is incomprehensible if a
single bad decision can result in threat to life and health of humans e.g., in case of
autonomous driving or in medical field or it may lead to huge financial losses e.g.,
in algorithmic trading etc. Because of this lack of transparency, machine learning is
being used cautiously in industries like healthcare, e-commerce, entertainment etc.
To build trust, an AI system must be able to explain its reasoning behind its actions.
This is crucial, not just in medical or safety-sensitive applications.
When compared to an incomprehensible intelligent system, one of the most
urgent goals of a XAI system is to help end users comprehend how the intelligent
system functions. Users’ mental models of the underlying intelligent algorithms are
improved by machine learning explanations, which provide understandable trans-
parency for the complicated intelligent algorithmic structures. Transparency in a
XAI system may also enhance user experience by making model outputs easier to
comprehend and interact with.
When utilized correctly, XAI may identify and dissuade hostile cases [9] while
also increasing transparency and fairness. For the sake of assessing the accuracy of
production projections and fending off competitors, openness is essential. Using an
adversarial example to deceive a classifier into thinking a phoney picture is in fact
genuine might have a negative impact on the accuracy of the classifier. The reliability
of AI algorithms to minimize threats [10] and give transparency in terms of model
comprehension, textual or visual reporting should be of primary relevance as we
increasingly depend on autonomous algorithms to assist our everyday lives.
In order to build confidence in the intelligent algorithm, XAI system provides
explanations to the end users. System dependability and accuracy may be assessed
and calibrated using a XAI system. Consequently, users’ faith in the algorithm leads
to their dependence on the system. Recommendation systems, autonomous systems,
and crucial decision-making systems are all examples of how XAI intends to increase
user trust via its open architecture.
The information and accessible answers to circumstances, as well as the trust we
establish, are the primary determinants of our social lives, choices, and judgments.
Explanations and rationales are preferable to blind faith in one’s own wisdom when
it comes to making decisions that aren’t ideal. In order to build trust with end-
users, including specialists, developers, legislators, and others, it is critical to explain
why a certain choice was taken. As we move towards a networked AI-driven socio-
economic ecosystem, stakeholders and government agencies need to be able to trust
the predictions of classifiers.
24 M. Gupta

3 Methods of Explainable AI

The explainable model’s core algorithmic principle may be divided into many subcat-
egories dependent on how it was implemented as shown in Fig. 3. XAI approaches
may be divided into two categories: those that concentrate on changes or adjust-
ments in input data, and those that focus on the model’s architecture and parameters.
Both perturbation-based and backpropagation-based approaches may be used. This
section summarizes the most important approaches to explainable AI.

3.1 Perturbation-Based

Perturbation-based XAI approaches are used to create explanations by repeatedly

probing a trained machine learning model with various input variations. These alter-
ations may be made at the pixel level by blurring, moving, or masking pixels,
replacing certain characteristics with zero or random counterfactual occurrences,

XAI Methods

Perturbation-Based Back Propagation-Based

DeConv Saliency Maps

LIME CAM

SHAP SR Maps

Prediction Axiomatic
Difference Analysis Attribution Maps

Randomized Input
Sampling PatternNet
Explanation

Randomization & PatternAttribution

Feature Testing

Fig. 3 Different methodology of XAI

2 Explainable Artificial Intelligence (XAI): Understanding … 25

or selecting a pixel or set of pixels (super pixels) for explanation. A perturbation-

based XAI approach may be thought of as a strategy that attempts to comprehend
neural activity and the influence of particular characteristics on a corresponding class
of outputs. It’s all about occlusion, filling operations, generative algorithms, filtering,
conditional sampling, etc. in the context of explainable algorithms. In most cases, the
attribution representations may be generated using simply the forward pass, without
the need of backpropagating gradients. The following are some perturbation-based
approaches.
(1) DeConv Nets by Zeiler et al. [11]: In this approach, the AlexNet model was
trained on the ImageNet dataset, and layer-by-layer filter visualizations were
performed as part of this research. Dataset bias and difficulties with few training
sets were discovered after investigating feature generalization. DeConv Nets
were used to visualize neural activity by occluding input instances.
(2) Local Interpretable Model-Agnostic Explanations (LIME) by Ribeiro et al. [12]:
In this approach, a point of interest was perturbed in order to create locally
accurate explanations. Using LIME as an explanation in human/user research
was shown to have a positive influence on an untrustworthy classification system.
Super pixels were found using Iterative perturbation to input data.
(3) SHapley Additive exPlanations (SHAP) by Lundberg et al. [13]: SHAP outper-
formed LIME time and time again. In user research, SHAP explanations were
shown to be in line with human explanations. Although SHAP values are useful
in producing explanations, several recent research contend that they do not assist
final decision making. In a game theoretic framework, the author investigates
feature correlations by eliminating characteristics.
(4) Prediction Difference Analysis by Zintgraf et al. [14]: In this work, authors
found a relevance value for each input feature, it was one of the initial steps in
determining whether or not an attribute’s positive or negative association with
the outcome was significant. The ImageNet dataset was used to train a variety of
models to better understand how different layers of deep nets support different
output classes. The positive and negative association between individual char-
acteristics and the output was discovered by the author via research of f and the
removal of individual features from x.
(5) Randomized Input Sampling for Explanation by Petsiuk et al. [15]: Authors
researched on saliency maps using a method called randomized masking of
inputs. ResNet50 and VGG16 were the networks that were used.
(6) Randomization and Feature Testing by Burns et al. [16]: This work presented
the examination of feature relevance via the use of hypothetical replacements
of features. The author used the neural networks BERT and Inception V3.
26 M. Gupta

3.2 Backpropagation- or Gradient-Based

Explainable algorithms that rely on backpropagation do many forward passes through

the neural network before using partial derivatives of the activations to create attribu-
tions during the backpropagation stage. Some examples include class activation maps
and saliency relevance maps. The preceding sub-section part discussed perturbation-
based approaches, which employ fluctuations in the input feature space to explain how
f is assigned to a certain class in the output class c. A neural network’s backward infor-
mation flow is used to analyze the effect and significance of the input x on the output
of the neural network. The bulk of gradient-based approaches, that are mentioned
below, concentrate on either visualizing activation of particular neurons with strong
effect or overall feature attributions moulded to input dimensions. Gradient-based
XAI algorithms have an inherent advantage in the development of visual explanations
that are intelligible to humans.
(1) Saliency Maps: Gradient-based XAI is based on a set of approaches that were
pioneered by the authors [17]. Gradients, neuronal activity of particular layers
using DeConv nets, etc., are shown as visuals by the authors. An important factor
in DeConvNet’s performance was the weight placed on gradient values during
backprop. Traditional CNNs would return zero results for negative gradients
if they used ReLU activation. However, the gradient value is not trimmed at
zero, in DeConvNets. This enabled for a more realistic depiction of what was
happening.
(2) Gradient Class Activation Mapping (CAM): For all pooling operations, most
saliency algorithms employ the global average layer instead of max-pooling.
CAM was used by Zhou et al. [18] to locate class-specific image areas on an
input picture in a single forward pass using a modified global average pooling
algorithm. In order to create deeper CNNs and better visualizations, Grad-CAM
[19] and GradCAM ++ [20] enhanced the CAM process. Class-discriminative
GradCAM may be used to locate the activity of neurons in a CNN network.
Additionally, it provides counterfactual explanations to show which parts of a
picture are detrimental to a given model’s output. Visual question answering
(VQA), picture categorization, and segmentation all benefit from the use of
GradCAM.
(3) Salient Relevance (SR) Maps: SR is a context-aware salience map based on the
LRP of the input picture presented by Li et al. [21]. In order to get started,
you’ll first need to discover the LRP relevance map for the picture of interest
that has the same dimensions as the input images. Individual pixels are given a
saliency value using a context-aware salience relevance map algorithm. Pixels
are said to be “salient” if they can be distinguished from other patches of pixels
on the same and various scales. This is done in order to distinguish between the
image’s foreground and background elements.
(4) Axiomatic Attribution Maps by Sundararajan et al. [22]: There were several
new axioms for gradient-based approaches. Saliency and gradient timings input
maps were improved and according to Ancona et al. [23], a gradient approach
2 Explainable Artificial Intelligence (XAI): Understanding … 27

in which output corresponding to input is multiplied by that input is beneficial

in giving interpretable explanations for model results. However, in [22], the
authors suggested Integrated Gradients (IG) and claimed that most gradient-
based techniques are lacking in certain ‘axioms’ which are desirable properties.
Deconvolutional networks (DeConvNets) [11], DeepLift [24], and guided back-
propagation [25] all include back-propagation logic that some authors found
contradicts basic assumptions of information theory.
(5) PatternNet and PatternAttribution by Kindermans et al. [26]: In this work, back-
projection of estimated data to input space using an LRP-based approach was
performed. Root point selection method is used for more accurate attributions.
Improved gradient-based approaches were presented by Kindermans et al. in
[26]. They developed PatternNet and PatternAttribution, which are able to esti-
mate the data component that triggered network activations. Similar to identi-
fying gradients, PatternNet uses a layer-by-layer back projection of the predicted
signal to the input space in order to detect gradients. Unlike LRP, PatternAt-
tribution provides neuron-level attribution of input signals to their associated
output classes.

4 XAI Application Domains

Nowadays industries are adopting the environment in which the demand for IoT
devices XAI, on the other hand, may have a big impact on a wide variety of industries
that depend on AI systems as shown in Fig. 4. In this section, we look at some of the
possible areas in which explainable models is proving to be more useful.

(1) Transportation: Automated cars have the potential to reduce traffic fatalities
and increase mobility, but they also present issues in terms of AI decision-
making that must be explained. Object classification is critical in autonomous

Transportation Healthcare

Defence
Finance
XAI

Academic
Research Construction

Architecture Law & Justice

Fig. 4 Different application domains for XAI

28 M. Gupta

cars because they must make split-second choices based on what they see. When
a self-driving automobile exhibits unexpected behavior due to a coding error.
The results might be disastrous. This isn’t science fiction; a self-driving Uber
in Arizona just killed a lady. In this case, it was the first reported death of a
completely automated car. Unidentified individuals said that the car’s software
detected an item in front of it but handled it as if it were a plastic bag or a piece of
tumbleweed blown by the wind as such, it is only via an understandable structure
that such an incident may become clearer and prevent it from occurring. It is
possible that the XAI might be used in transportation. There is still a long way
to go before self-driving car behavior can be explained.
(2) Healthcare: Human life depends on the medical diagnostic model. How can
we have faith in a black-box model’s instructions to treat a patient? About
two decades ago, a computer model called an artificial neural network (ANN)
was developed to help doctors decide whether or not to admit patients with
pneumonia to a hospital or treat them as outpatients. Neural networks were
shown to be significantly more accurate than traditional statistical approaches
in the early stages of research. In the end, it was discovered that the neural net
had incorrectly predicted that patients with asthma who had pneumonia would
have a reduced mortality rate and hence should not be hospitalized. Although
this may seem paradoxical, the training data showed that asthma patients with
pneumonia were admitted to the ICU, intensively treated, and survived [27]. It
was subsequently concluded that the AI system should be abandoned since it
was too risky to deploy in clinical practice. In order to avoid this critical issue,
we must read the model. Preliminary effort has been done recently to make
AI-based healthcare systems understandable [27, 28]. According to the rising
quantity of these papers, there is an increasing interest and difficulty for utilizing
XAI techniques in health care.
(3) Law and Justice: AI has the ability to improve recidivism risk assessments
and lower the costs of both crime and jail in the criminal justice system. It is,
nevertheless, important to ensure that the criminal decision model is fair, honest,
and non-discriminatory when it is used to estimate the likelihood of reoffending
in court. Mr. Loomis was sentenced to jail because of the use of proprietary
risk assessment software in Loomis v. Wisconsin [29]. Gender and ethnicity
are taken into consideration while using “Correctional Offender Management
Profiling for Alternative Sanctions: COMPAS,” the software at issue in this
case. It was unclear to the judge how the algorithms were employed and what
the causal audit procedure was. Despite the importance of being able to explain
the process by which a judgement is reached in the legal system, relatively few
attempts have been undertaken so far to do so [30–32].
(4) Finance: The use of artificial intelligence (AI) techniques in the financial
services industry may lead to advancements in wealth management, investment
advising, and client service. However, there are concerns about data security and
equitable lending with these new technologies. Loan providers have to adhere to
strict regulations in the financial business. Using AI-based algorithms in credit
scores and models presents a big problem since it is more difficult to offer
2 Explainable Artificial Intelligence (XAI): Understanding … 29

borrowers with the necessary “reason code”—the explanation of why they were
refused credit. Because of an opaque ML algorithm’s output, denial is much
more difficult. At Equifax and Experian, exciting research efforts are underway
to develop automated explanation codes and make AI credit score conclusions
more understandable and auditor friendly [33].
(5) Defense: Researchers in the military were the ones who started the present, well-
known XAI project [34], and the topic’s increased awareness is primarily due to
the call for study and the solicitation of DAPRA Projects. The AI explainability
dilemma is not exclusive to AI in the military. A paper from the MIT Technology
Review examines the limitations of depending on autonomous systems for mili-
tary operations, [35] Knight. This may lead to ethical and legal quandaries that
are reminiscent to those encountered in the healthcare industry. With the DAPRA
Ambitious XAI programme and a few other research efforts, the academic AI
research community is strongly represented in this application space. XAI’s
requirement is confirmed by the work done in each of the mentioned fields.
These kinds of studies, however, are still in their infancy, and much more study
has to be done. Other potential uses for XAI include cybersecurity, educa-
tion, entertainment, governance, and image identification, to name just a few.
Automated decision-making can cause harm in many different areas of life,
including employment, insurance, and social benefits, as well as housing and
differential pricing of goods and services. Future of Privacy Forum [36] illus-
trates the various areas where automated decision-making can do harm as well
as the areas where providing automated explanation can turn them into trustful
processes.

5 XAI Challenges and Future Prospective

How to develop more understandable models, how to build explanation interfaces,

and what are the psychologic criteria for successful explanations are all connected
research issues posed by this user-centered idea. To solve these difficulties, XAI
research teams are creating new ML approaches to construct explainable models, as
well as new HCI tools (such as visualization, language comprehension and gener-
ation) for successful explanations. Researchers are also focusing on describing,
expanding, and using psychologic theories of explanation to address the difficulties
of XAI working.
There have been major advances in the area of explainable artificial intelligence,
but obstacles remain in the methodologies and theory as well as in the way expla-
nations are applied. Using explanation approaches, we can learn more about how
the AI model works. However, there are still some limitations to these approaches.
For example, Heatmaps provide “first-order” information, i.e., which input elements
have been recognized as being relevant to predicting performance outcomes. But the
relationship between these qualities, such as whether they are relevant on their own
or only when they occur together, is still a mystery. There are several applications
30 M. Gupta

where this knowledge is critical, such as finding groupings of brain areas that work
together to solve a certain task (brain networks) rather than merely pinpointing the
most significant single voxels in each region.
In addition, explanations are too simplistic. In a heatmap, you can see which pixels
are most essential without having to think about more abstract ideas like what is in the
picture or what the scene looks like. To comprehend the model’s behavior, humans
must interpret the explanations. One of the trickiest parts of this process is interpreting
what you’ve seen. Model behavior should be explained on a more abstract, human
intelligible level by meta-explanations that collect information from these low-level
heatmaps. Low-level explanations have been gathered and the semantics of brain
representations have been quantified. Meta-explanations are a promising field of
study for future researchers.
Primary research areas in which enabling technologies and approaches for XAI
might possibly be found are:
(1) Data science: Algorithms based on AI/ML are data hoarders, always searching
for new sources of information to feed their insatiable appetite for data. This
data is crucial to the backward route that aims to create better explanations
and justifications. As a result, data science becomes a critical component in the
process of explaining things.
(2) AI/ML: To create an explanation, we believe that utilizing AI/ML as a
computational process to explain AI/ML is an intriguing work path to follow.
(3) Nature of Human Conduct: For the purpose of creating artificial explanations,
it is worthwhile to first simulate how people explain their actions and behaviors
to one another. As a result, using human science ideas may result in new and
better models that can be explained.
(4) Human Computer Interaction (HCI): The way a person interacts with a computer
may influence how well he or she understands and trusts the system. It is possible
to apply approaches from the discipline of Human–Computer Interaction (HCI)
to assist and build systems that are more transparent.
Research on the use of explanations in the human–machine interface is a promising
avenue for future development. Research on human elements in explainable AI
has already begun (e.g., [8]). Prerequisites for effective human–machine interac-
tion include constructing explanations with the correct user emphasis, which means
asking the right questions in a certain manner. In spite of this, additional research is
needed to optimize explanations for best human use. There is a lack of a formal and
widely agreed-upon definition of what constitutes an explanation in the notion of
explainable AI. The development of mathematically sound explanation techniques
was a first step in this direction. According to [37], for example, the explanation issue
is approached using Taylor decomposition as a theoretical framework. It’s another
potential avenue toward the objective of building a general theory of explainable AI
using axiomatic techniques [38].
In the end, the utilization of explanations that go beyond imagery is an
unsolved problem. To increase the model’s performance or minimize its complexity,
2 Explainable Artificial Intelligence (XAI): Understanding … 31

future work should demonstrate how explanations may be included into a wider
optimization process.

6 Conclusion

Human-centered research on competences and knowledge might extend the func-

tion of XAI beyond the job of describing a specific XAI system and helping its
users decide acceptable trust. XAIs have the potential to play important roles in
society in the not-too-distant future. Some of these roles could include not only
acquiring knowledge and imparting it to others, but also working with other agents
to communicate knowledge, develop inter-disciplinary insights and mutual under-
standing, collaborate with others to teach individuals and other agents, and draw
on previously known knowledge to speed up the discovery and application of new
information. Because of this new social perspective on knowledge understanding
and era, the future of XAI technology is just getting started.
In addition to serving as a bridge between artificial intelligence (AI) and the
general public, explainability is a valuable tool for identifying model and data biases,
verifying predictions, fine-tuning models, and acquiring new insight into the situation
at hand (e.g., in the sciences).
Because of this, it’s important to develop and choose XAI methods carefully as
the current research landscape shows. Findings have shown that relying solely on
typical explanation maps may not be helpful. Mission-critical applications may not
benefit from XAI because of human bias in visual explanations. The XAI assessment
landscape seems to be improving as a result of recent advances in human-centered
evaluations.

References

1. E. Tjoa, C. Guan, A survey on explainable artificial intelligence (XAI): towards medical XAI.
arXiv:1907.07374 (2019)
2. F.K. Dosilovic, M. Brcic, N. Hlupic, Explainable artificial intelligence: a survey, in 41st
International Convention on Information and Communication Technology, Electronics and
Microelectronics (MIPRO), pp. 210–215 (2018)
3. R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, D. Pedreschi, A survey of methods
for explaining black box models. ACM Comput. Surv. 51(5), 1–42
4. R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, D. Pedreschi, A survey of methods
for explaining black box models. ACM Comput. Surv. 51(5), 1–42 (2018)
5. U. Bhatt, A. Xiang, S. Sharma, A. Weller, A. Taly, Y. Jia, J. Ghosh, R. Puri, J.M.F. Moura, P.
Eckersley, Explainable machine learning in deployment, in Proceedings of the 2020 Conference
on Fairness, Accountability, and Transparency (Association for Computing Machinery, New
York, NY, 2020), pp. 648–657. Retrieved from https://doi.org/10.1145/3351095.3375624
6. M. Ribera, A. Lapedriza, Can we do better explanations? A proposal of user-centered explain-
able AI, in Joint Proceedings of the ACM IUI 2019 Workshops Colocated with the 24th ACM
Conference on Intelligent User Interfaces (ACM IUI 2019), vol. 2327 (2019). CEUR-WS.org
32 M. Gupta

7. A.B. Arrieta, N.D. Rodriguez, J.D. Ser, A. Bennetot, S. Tabik, A. Barbado, S. Garcia, S. Gil-
Lopez, D. Molina, R. Benjamins, R. Chatila, F. Herrera, Explainable artificial intelligence
(XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Inf. Fusion
58, 82–115 (2020). Retrieved from https://doi.org/10.1016/j.inffus.2019.12.012
8. I. Lage, E. Chen, J. He, M. Narayanan, B. Kim, S. Gershman, F. Doshi-Velez, An evaluation
of the human-interpretability of explanation. arXiv preprint arXiv:1902.00006 (2019)
9. I. Goodfellow, J. Shlens, C. Szegedy, Explaining and harnessing adversarial examples, in
Proceedings of the 3rd International Conference on Learning Representations, ICLR (2015)
10. A. Kurakin, I. Goodfellow, S. Bengio, Adversarial machine learning at scale, in Proceedings
of the 5th International Conference on Learning Representations, ICLR (2016)
11. D.M. Zeiler, R. Fergus, Visualizing and understanding convolutional networks, ed. by D. Fleet,
T. Pajdla, B. Schiele, T. Tuytelaars. Computer Vision—ECCV 2014 (Springer International
Publishing, Cham, 2014), pp. 818–833
12. T.M. Ribeiro, S. Singh, C. Guestrin, Why should I trust you? in Proceedings of the 22nd ACM
SIGKDD International Conference on Knowledge Discovery and Data Mining (New York,
USA, 2016), pp. 1135–1144
13. M.S. Lundberg, I.S. Lee, A unified approach to interpreting model predictions, in Advances in
Neural Information Processing Systems, pp. 4765–4774 (2017)
14. M.L. Zintgraf, S.T. Cohen, T. Adel, M. Welling, Visualizing deep neural network decisions
prediction difference analysis, in Proceedings of the 5th International Conference on Learning
Representations, ICLR (2017)
15. V. Petsiuk, A. Das, K. Saenko, RISE: randomized input sampling for explanation of black-box
models, in British Machine Vision Conference BMVC (2018)
16. C. Burns, J. Thomason, W. Tansey, Interpreting black box models via hypothesis testing. arXiv
preprint arXiv:1904.00045 (2019)
17. K. Simonyan, A. Vedaldi, A. Zisserman, Deep inside convolutional networks: visualising image
classification models and saliency maps, in Proceedings of the 2nd International Conference
on Learning Representations, ICLR—Workshop Track Proceedings (2013)
18. B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, A. Torralba, Learning deep features for discrim-
inative localization, in 2016 IEEE Conference on Computer Vision and Pattern Recognition
(CVPR) (IEEE, 2016), pp. 2921–2929
19. D. Batra, Grad-CAM: visual explanations from deep networks via gradient-based localization,
in Proceedings of the IEEE International Conference on Computer Vision (2017)
20. A. Chattopadhay, A. Sarkar, P. Howlader, V.N. Balasubramanian, Grad-CAM++: generalized
gradient-based visual explanations for deep convolutional networks, in 2018 IEEE Winter
Conference on Applications of Computer Vision (WACV) (IEEE, 2018), pp. 839–847
21. H. Li, Y. Tian, K. Mueller, X. Chen, Beyond saliency: understanding convolutional neural
networks from saliency prediction on layer-wise relevance propagation. Image Vis. Comput.
83–84, 70–86 (2019)
22. M. Sundararajan, A. Taly, Q. Yan, Axiomatic attribution for deep networks, in 34th
International Conference on Machine Learning, ICML 2017 (2017)
23. M. Ancona, E. Ceolini, C. Oztireli, M. Gross, Towards better understanding of gradient-based
attribution methods for deep neural networks, in 6th International Conference on Learning
Representations, ICLR 2018—Conference Track Proceedings (2018)
24. A. Shrikumar, P. Greenside, A. Kundaje, Learning important features through propagating acti-
vation differences, in Proceedings of the 34th International Conference on Machine Learning,
vol. 70(ICML17), p. 31453153 (2017). JMLR.org
25. J.T. Springenberg, A. Dosovitskiy, T. Brox, M. Riedmiller, Striving for simplicity: the all
convolutional net, in 3rd International Conference on Learning Representations, ICLR 2015—
Workshop Track Proceedings (2015)
26. P.-J. Kindermans, K.T. Schutt, M. Alber, K.-R. Muller, D. Erhan, B. Kim, S. Dahne,
Learning how to explain neural networks: patternnet and patternattribution, in 6th Interna-
tional Conference on Learning Representations, ICLR 2018—Conference Track Proceedings
(2018)
2 Explainable Artificial Intelligence (XAI): Understanding … 33

27. R. Caruana, Y. Lou, J. Gehrke, P. Koch, M. Sturm, N. Elhadad, Intelligible models for
healthcare: Predicting pneumonia risk and hospital 30-day readmission, in Proceedings of
the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,
pp. 1721–1730 (2015)
28. A. Holzinger, C. Biemann, C.S. Pattichis, D.B. Kell, What do we need to build explainable AI
systems for the medical domain? (2017). [Online]. Available: https://arxiv.org/abs/1712.09923
29. J. Lightbourne, Damned lies & criminal sentencing using evidence based tools. 15 Duke Law
Technol. Rev. Tech. Rep. 327–343 (2017). https://scholarship.law.duke.edu/dltr/vol15/iss1/16.
Accessed 6 June 2018
30. S. Tan, R. Caruana, G. Hooker, Y. Lou, Detecting bias in black-box models using transparent
model distillation (2018). [Online]. Available: https://arxiv.org/abs/1710.06169
31. C. Howell, A framework for addressing fairness in consequential machine learning, in
Proceedings of the FAT Conference Tutorials, pp. 1–2 (2018)
32. R. Berk, J. Bleich, Statistical procedures for forecasting criminal behavior: a comparative
assessment. Criminol. Public Policy 12(3), 513–544 (2013)
33. Equifax, Equifax launches NeuroDecision technology (2018). [Online]. Available: https://inv
estor.equifax.com/news-and-events/news/2018/03-26-2018-143044126. Accessed 6 June 2018
34. D. Gunning, Explainable artificial intelligence (XAI). Defense Advanced Research Projects
Agency (DARPA). Accessed 6 June 2018. [Online]
35. W. Knight, The U.S. military wants its autonomous machines to explain themselves, MIT
Technology Review (2017). [Online]. Available: https://www.technologyreview.com/s/603
795/theus-military-wants-its-autonomous-machines-to-explain-themselves. Accessed 6 June
2018
36. Future of Privacy Forum, Unfairness by algorithm: distilling the harms of automated decision-
making (2017). [Online]. Available: https://fpf.org/wp-content/uploads/2017/12/FPF-Automa
tedDecision-Making-Harms-and-Mitigation-Charts.pdf. Accessed 6 June 2018
37. G. Montavon, S. Lapuschkin, A. Binder, W. Samek, K.R. Muller, Explaining nonlinear
classification decisions with deep Taylor decomposition. Pattern Recogn. 65, 211–222 (2017)
38. M. Sundararajan, A. Taly, Q. Yan, Axiomatic attribution for deep networks, in International
Conference on Machine Learning (ICML), pp. 3319–3328 (2017)
Chapter 3
Explainable Artificial Intelligence (XAI):
Conception, Visualization
and Assessment Approaches Towards
Amenable XAI

Tasleem Nizam and Sherin Zafar

Abstract For last decade, due to the accessibility of huge databases and recent
advancements in deep learning methodology, machine learning systems have arrived
at or transcended tremendous performance in a spacious variety of tasks. One can
see this speedy development in speech analysis, image recognition, sentiment anal-
ysis, strategic game planning and many more, for e.g. in medical field, it’s used
for diagnosing different diseases, like breast cancer etc., based on their symptoms.
But many state-of- the-art models is facing lack of transparency and interpretability
which is a major hindrance in many applications, e.g. finance and healthcare where
visualization, interpretation and explanation for model’s decision is an obligation for
trust. This is an implicit problem of the current techniques carried by sub-symbolism
(e.g. Deep Neural Networks) that were not shown in the last hype of AI (specifically,
rule based models and expert systems). Models underlying this problem come within
the so-called Explainable AI (XAI) field, which is extensively acknowledged as a
racial feature for the practical deployment of AI models. As a result, explainable
artificial intelligence (XAI) has turned into scientific interest in last recent years. So,
this chapter epitomizes contemporary developments in Explainable AI that describes
explainability in Machine Learning, constituting a fiction definition of explainable
Machine Learning that envelopes such prior conceptual propositions with a consider-
able focus on the audience for which the explainability is needed. Except of this defi-
nition, this chapter starts a confabulation on its various techniques that are essentials
for analysing interpretability and explainability of Artificial Intelligence, and also
gives a comparison between two medical experiments, that are based on predicting
heart disease using disparate Explainable Artificial Intelligence techniques, which
can give a lead for researchers as well as practitioners or newcomers in the field of
Artificial Intelligence for selecting suitable methods with Explainable AI to grasp

T. Nizam (B)
Jamia Hamdard, New Delhi, India
e-mail: [email protected]
S. Zafar
Jamia Hamdard, New Delhi, India
e-mail: [email protected]

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 35

A. E. Hassanien et al. (eds.), Explainable Edge AI: A Futuristic Computing Perspective,
Studies in Computational Intelligence 1072,
https://doi.org/10.1007/978-3-031-18292-1_3
36 T. Nizam and S. Zafar

the advances of AI in their action sectors, without any previous bias for its dearth of
interpretability.

Keywords Explainable artificial intelligence · Machine learning · Interpretability ·

Explainability · Black box model · Post-Hoc method

1 Introduction

In the last decemvir, there were many tasks that were initially thought to be computa-
tionally impassable, but since 2012, artificial intelligence (AI) and machine learning
(ML) systems have achieved tremendous human performance in many fields.
Because of considerable hardware improvements combined with new optimiza-
tion algorithms these leadings in the AI field were achieved. At the same time, there is
too much growth in high quality open-source libraries also which allows developers
as well as researchers to quickly code and test models which also result in great work
in AI field [3]. With the upgrading in image classification, speech recognition, clas-
sical (board) games, object detection and many more have edged to their procreation
to the real-world applications outside research labs, mostly in the area of supervised
learning.
As the time passed, one can see advances in application demanding areas, such as
bioinformatics, government, medicine, churn prediction and content recommenda-
tion but in these applications also crucial trust-related problems are raised. In exten-
sions to additions in aforementioned areas, future applications will also combine
interpretable science, reliable ML and cognitive assistance [4]. Due to these changes,
many ethical challenges will occur that society will have to accommodate fast in
order to drive the developments to the positive directions and outcomes. More unfair
wealth distribution might be done when automation will substantially change the job
market. Generation of fake content and content recommendation teamed with other
technologies will significantly impact the social dynamics [5]. Those algorithms
will prescribe many things and that will alter human lives in ways, so humans will
desire to trust them in order to obtain those prescriptions. Instead of this, to sustain
trust [6], i.e. reliability, explanatory justifiability, unbiasedness (fairness), usability,
privacy etc, systems must amuse many criteria (assurances). Since humans are social
creatures addicted to life in human communities such affirmations are assumed due
to bias towards human decision making. Due to novel status of Artificial Intelligence
in our lives as well as being of the human formulation, it generates more disbeliefs.
At starting, AI interpretation was easy and the previous years have endorsed
the boost of opaque decision systems such as Deep Neural Networks (DNNs). The
experimental success of Deep Learning models such as DNNs branches from a
combination of adequate learning algorithms and their vast parametric space. DNNs
are considered as complex black-box models due to latter space which is consist
of hundreds of layers and millions of parameters. Transparency is reverse of black-
box-ness which means understanding or searching the technique by which model
3 Explainable Artificial Intelligence (XAI): Conception … 37

works. As black-box Machine Learning (ML) models are widely being used to give
essentials predictions in crucial contexts, the appeal for transparency is surging from
the different stakeholders in AI [5, 6]. If we use or create any decision which are not
legitimate, justifiable or that simply do not grant gathering accurate explanations of
their behaviour, then it might be dangerous for us [7]. All explanations that support
the output of a model are important, e.g., experts need many more information from
the model than a simple binary prediction for encouraging their diagnosis in precision
medicine. In general, people are restrained to follow techniques that are not precisely
tractable, trustworthy and interpretable, which results demanding for ethical AI [7,
8]. The systems will be progressively opaque if one only thinks about solely on its
performance. This is correct in the sense that there is a trade-off between model’s
transparency and its performance. However, if one works on the understanding of
a system then it can lead to the correction of its dearth. While developing a ML
model, its implementation can be improved by considering its interpretability as an
additional design driver. This interpretability basically helps in impartial decision-
making and can act as a guarantee that only useful variables produce the output
[9].
So, for removing the limitations of effectiveness of the recent generation of AI
systems, explainable AI (XAI) gives a suite of ML techniques that:
• Generates more explainable models while keeping a high level of learning
achievement.
• Empowers humans to appropriately trust, understand and properly balance the
imminent generation of artificially intelligent partners [10].
Previous surveys and overviews of interpretability in machine learning are given
in [2, 3, 14, 15, 17]. In this chapter we survey the overtures in the explainability and
interpretability of machine learning models using the supervised learning paradigm.
This chapter is composed as follows: in Sect. 2 we deal with the definitions and
preliminaries. In Sect. 3 we classify the techniques for interpretability and explain-
ability. Section 4 offers discussion and comparison of two research done using AI
techniques which also discuss the methodology that can be used in XAI for accurate
result with proper explanation. And at last, in Sect. 5, we conclude the chapter which
also discuss current state of research ideas.

2 Preliminaries and Definitions

Before processing with our literature survey, it is good to first build a common point
of understanding on which the term explainability stands for in the context of AI
and, more specifically, ML. This is important aim of this section, i.e. to pause at the
different definitions that have been used in regards to this concept.
38 T. Nizam and S. Zafar

Terminology description:
In some literature interpretability and explainability are interchangeable misused
which impedes the formation of common grounds. There are important differences
between these concepts. Expressing any procedure or action taken by a model with
the purpose of detailing or clarifying its internal functions is considered as explain-
ability [10, 11]. Explainability is an active characteristic of a model. By contrast,
interpretability denotes to a passive characteristic of a model implying to the level
at which a given model makes sense for a human watcher. This characteristic is also
considered as transparency.
To compile the most commonly used terminologies, in this section we simplify
the differences and sameness among terms often used in the XAI communities and
ethical AI.
• Intelligibility (or Understandability): It represents the characteristic of a model
to make a human grasp its functions—how the model works—without any
commitment for explaining its algorithmic means using which the model operates
data internally or one can say without discussing its internal structure [12].
• Comprehensibility: The capability of a learning algorithm to express any model’s
learned knowledge in a human knowable fashion is known as comprehensibility.
It is specially used for ML models [13].
• Interpretability: The aligning of abstract concept into a domain that humans
can make sense of is called interpretability. In other words. It is expressed as the
capability to describe or to give the meaning in knowable terms to a human.
• Explainability: An interface between a decision maker and humans is known as
explainability. It is considered as a notion for explanation. In other words, it works
as definite proxy of the decision maker as well as conceivable to humans [13, 14].
• Transparency: If a model is understandable by itself, then it is considered to be
transparent.
In all the above definitions, understandability appears as the most important
concept in XAI. Both interpretability and transparency are firmly tied to this concept,
while transparency attributes the characteristics of a model to be understandable for
people by its own whereas understandability measures the level to which a human
can understand a decision taken by a model. The potential of the audience to grasp
the knowledge contained in the model is considered as Comprehensibility.
None of the aforesaid definitions is restrictive or specific enough to facilitate
formalization. They implicitly build upon user’s preferences, expertise and other
circumstantial variables.
3 Explainable Artificial Intelligence (XAI): Conception … 39

3 Techniques for Explainability and Interpretability

There are two divisions of methods to explainability and interpretability: integrated

(transparency-based) and post-hoc. Interpretability can be enabled by using Trans-
parency. By analogy Transparency is ported to algorithmic matters such as discrim-
ination and unfairness and it was a traditional first step also for the preservation of
rights in human-based institutions. But, day by day models in AI are getting more
compound than human-based institutions and it becomes tough to get exact explana-
tion that users might be able to interpret. Also, human perspective, including our own,
is not transparent to everyone and details in the form of interpretations and explana-
tions may contradict from the real decision mechanism. In extension, transparency
and predictive performance are contrary objectives and they have to be cop-out in a
model [15, 16]. In [20], it is defined that it is not clear how much transparency needs
in the lengthy run. If the systems are self-contained and robust it may not be essential.
But, if they are component of other systems, then transparency can be needed for
debuggability.
Post-hoc interpretability documents information from already learned model and
it does not strictly depend on how the model works. The benefit of this method is
that it does not impact working of the model which is considered as a black-box
(BB). This is same mode to how people give justifications for their own outcomes,
without completely knowing the exact working of their decision-making techniques.
However, extra care must be considered in order to ignore systems that give plausible
but false explanations. Such explanations could gratify laws like GDPR, but there is
a problem of examining their accuracy [17, 18].

3.1 Integrated Interpretability

The best description of a normal model is the model itself; it perfectly depicts itself
and is simple to understand [21]. This method is narrowed to the model families
with lower flexibility or complexity, such as decision trees rules and linear models.
On the other hand, there are some model families such as support vector machines
(SVM), artificial neural networks (ANN), random forests and boosted tree, which
are considered opaque and their complexity block users from imaging the logic
behind predictions. The latter are most often taken as black-boxes and they are
handled with in a post-hoc manner. This arrangement between performance and
transparency is imaginary depicted in Fig. 1. The large part of task is done for
classification works. Different conditions can be implied on models in order to grow
their interpretability. Model sparsity, monotonicity and model’s size are some of
the properties that can be used for increasing interpretability. Even the selection of
model family (representation) can be treated a constraint on models that influences
the interpretability. Transparent models are both explainable and interpretable.
40 T. Nizam and S. Zafar

Fig. 1
Performance-transparency
trade-off [4]

User-based observations on interpretability of tree algorithms decision rules tables

in classification were executed in [24, 25]. In the recent chapter, decision tables
were found to be the simplest to use for inexperienced users and that the model
size normally has negative impact on interpretability, confidence and answer time.
Interpretability can be divided into two sub-approaches: pure transparent and hybrid.
In pure transparent method we are bound to use model families that are taken as
transparent. Evolutionary programming was used in [22] to look for sets of inter-
pretable classification rules with little number of rules and conditions. Interpretable
decision sets are collection of independent if–then rules. Since each rule can be
used independently, interpretation is simple. The model is constructed by optimizing
objective that considers both interpretability and accuracy. Region-specific predictive
models, Oblique treed sparse additive models were proposed in [23]. They accom-
plished competitive performance with kernel SVMs while providing interpretable
model.
Hybrid technique combines transparent model families with black-box techniques
in contemplation to get accurate trade-off between the predictive performance and
the model interpretability. Combination of SVMs and logistic regression was used
for credit scoring in [24] in contemplation to enhance accuracy of the initial inter-
pretable model. Multi-objective training of hybrid classifiers was practiced in [25]
to observe hybrid trees where certain leaves were replaced with black-box classifiers
for enhancing accuracy at the cost of interpretability.
3 Explainable Artificial Intelligence (XAI): Conception … 41

3.2 Post-Hoc Methods

Predictive performance increases the use of complex, opaque models. This is done
because of increased availability of data and hardware improvements. However,
explainability and interpretability are topics that have to be properly addressed.
Post-hoc explainability aims models that are not easily interpretable by design
by resorting to distinct means to improve their interpretability, such as visual expla-
nations, text explanations, explanations by example, feature relevance explanations
techniques etc. Each of these methods envelops one of the simplest ways humans
describe processes and systems by themselves. Furthermore, actual techniques, or
actual bunch of methods are defined to ease the future work of any researcher that
aims to search for a specific method that suits its knowledge. Not only this, the classi-
fication also involves the type of data in which the methods have been applied. There
is one point to be noted that many methods might be justifiable for many different
types of data, although the grouping only deals with the type used by the authors
that proposed such method [18–20]. Overall, post-hoc explainability techniques are
categorized first by the aim of the author (explanation technique e.g. Explanation by
simplification), then, by the technique used (actual technique e.g. sensitivity analysis)
and finally by the category of data in which it was practiced (e.g. images).
I. Text explanations accord with the problem of carrying explainability for a model
by means of learning to build text explanations that help explaining the results
from the model [26]. Text explanations also involve every technique generating
symbols that denote the working of the model.
II. Visual explanation techniques for post-hoc explainability intents at visualizing
the model’s action. Many of the visualization techniques existing in the literature
come along with dimensionality reduction methods that permit for a human
interpretable normal visualization.
III. Local explanations handle explainability by separating the solution space and
providing explanations to less complex solution subspaces that are admissible
for the whole model. These explanations can be composed by means of methods
with the differentiating attributes that these only describe part of the whole
system’s working.
IV. Explanations by example deals with the extraction of data examples that
express the result produced by a certain model, enabling to get an improved
understanding of the model itself. Similarly, to how people react when trying
to describe a given process, explanations by example are actually centred
in obtaining representative examples that catch the inner correlations and
relationships found by the model being analysed.
V. Feature relevance explanation methods for post-hoc explainability defines the
inner working of a model by computing an important score for its managed
variables. These scores compute the sensitivity (affection) a feature has upon
the output of the model. A comparison of the scores among distinct variables
42 T. Nizam and S. Zafar

discloses the effect provided by the model to every such variables when gener-
ating its output. Feature relevance methods can be considered to be an indirect
technique to explain a model [3].

4 Empirical Analysis

The growth in healthcare infrastructure of a nation plays a vital role in finding the
health-related condition of life of its public. The growing trends in artificial intel-
ligence has aided the service of computational resources to boost the healthcare
infrastructure globally [1–4]. One such situation that has captured a lot of attention
from research scientists is heart attack [5–8], controlling to the fact that 33% of
deaths annually is because of it. However, the decision-making process in this field
still relies upon the guesswork of the cardiovascular expert or upon the expertise.
This makes it deeply ambiguous to satisfy the quality of the decisions [9]. Therefore,
there is an urgency for a system to assist the process of cardiovascular risk judgement.
The growth in the number of artificial intelligence (AI) fanatics have helped in the
process by providing remarkable funding for AI-based research in medical decision
making. Nevertheless, the most important concern in the technique has endured the
explainability of the results produced by AI-based systems. Specially, in the case of
medical diagnosis, it is too much important for the decisions to have a conceivable
justification. There are many artificial intelligence methods that can help healthcare
decision support systems such as neural networks, genetic fuzzy systems and genetic
programming etc.
There are three important aspects of all AI systems need to be taken into consid-
eration: Reliability which refers to the accuracy of the results, Explainability which
considers for giving relevant explanations of the results and Understandability that
aims on the usability. With this perspective, in last decades, computer technolo-
gies with machine learning techniques have made medical aid software as a support
system for prior diagnosis of heart disease. Identification of any heart related diseases
at first stage can avoid the death risk. Different Machine Learning methods are used
in medical data to figure out the pattern of data and deciding prediction from them.
Healthcare data are normally enormous in volumes and complicated in structure.
Machine Learning algorithms are able to handle the massive data and mine them to
find the meaningful information. Machine Learning algorithms read from past data
and make predictions on real time data. This kind of Machine Learning framework
for coronary illness expectation can boost cardiologists in deciding quicker actions
so more patients can get medicines within a smaller timeframe, thus saving broad
number of lives.
3 Explainable Artificial Intelligence (XAI): Conception … 43

4.1 Machine Learning

Machine Learning is a branch of Artificial research [2] and has become a very trendy
facet of data science. The Machine Learning algorithms are designed to execute a
huge number of tasks such as classification, prediction, decision making etc. To learn
the Machine Learning algorithms, training data is needed. After the learning stage,
a model is generated which is taken as an output of Machine Learning algorithm.
After this, that model is tested and validated on a group of unseen real time test
dataset. The final accuracy of the model is then compared with the real value, which
advocates the overall correctness of predicted result.
There are many popular methods that are used with Machine Learning Techniques.
Some of them which are used in Chaps. 1 and 2 are as follows:

4.1.1 Support Vector Machine

Support Vector Machine [11] is a classification method of Machine learning, which

is used to evaluate data and detect patterns in classification and regression anal-
ysis. SVM is typically think over when data is classified as two class problem. In this
strategy, data is classified by getting the best hyper plane that segregates all data points
of one class to another class. The higher edge or separation between the two classes
is, the model is considered more better. Those data points that are lying on limit of the
margin are considered as support vectors. The actual base of SVM is mathematical
techniques/methods that are used to design complicated real-world problems. The
author has selected SVM for his experiment because his dataset—Cleveland Heart
Disease Dataset CHDD has many classes to predict based on different parameters. In
SVM, by using a function called kernel (Kernels of SVM), mapping of training data
is done. These are—quadratic kernel, linear kernel, Radial Basis Function kernel,
polynomial kernel, Multilayer Perceptron kernel, etc. Beside from the kernel’s func-
tionalities in SVM, few more techniques are available such as sequential minimal
optimization, quadratic programming and least squares. While creating the model
with SVM, most challenging task is method selection and kernel selection to avoid
the issue of underfitting and overfitting. So SVM requires that its model should be
tested and validated against actual data.

4.1.2 Decision Tree Decision Tree Algorithm

With the help of Machine Learning Algorithm, Decision Tree Decision Tree algo-
rithm is used to create the Classification models. Decision Tree classification algo-
rithm based on the tree-like structure. This is a type of supervised learning, where
it is already known what to achieve. Both the numerical and categorical data can be
applied on Decision tree algorithm. Decision tree has a root node, its branches and
leaf nodes. The evaluation of Data is done on the basis of accessing path from the
44 T. Nizam and S. Zafar

root to a leaf node. In [chapter index] they used a dataset—CHDD, which contains
total of 283 tuples that were assessed down the decision tree. For the heart disease
prediction, they possibly came to a positive or negative assessment. After that, these
were compared to the real parameters to check for the false negatives/false positives
which displays the accuracy, sensitivity and specificity of the model.

4.1.3 Naive Bayes Algorithm

Naive Bayes Algorithm, supervised machine-learning algorithm, is established on

the Bayes’ Theorem [27], which considers that all the features are statistically inde-
pendent to each other’s. In [28], the Naive Bayes Classifier is used with high dimen-
sionality of inputs data. For solving problems of computer vision application, Naive
Bayes method is mostly used. In specifically, it gives good results as a classifier.

4.1.4 Random Forest Classification

Random Forest [29] is an association of uncut classification-based trees. It provides

outstanding performance related to number of real-life situations, as there is no
effect of noise in the dataset of Random Forest Classification and there is less risk of
overfitting also. If we compare it to many other tree-based algorithms, it functions
faster than the other algorithms and generally enhances accuracy for testing and
validation data. In Random Forest Classification, there are many options to match
the performance of random forest while constructing a random tree.

4.1.5 Neural Network

Artificial Neural Networks are nearly clumsy electronic models based on the neural
structure of the brain.
The brain actually learns from experience. It is a proof given by nature that some
problems that are above the limit of current computers are actually handled by small
energy efficient packages. By using this brain modelling one uses a less technical
method to design machine solutions. These biologically encouraged techniques of
computing are considered to be the next vital advancement in the computer field.
Computers have problems recognizing even easy patterns. But now, enhancement in
biological research provides an initial understanding of the natural thinking mecha-
nism. This procedure of saving information as patterns, making use of those patterns,
and then solving problems encloses a new field in computing, which is called—Arti-
ficial Neural Network (ANN). ANN are those types of computers whose architecture
is created after the brain. They typically made of hundreds of normal processing units
which are wired together in a typical communication network. Each node or unit is a
simplified model of real neuron that sends off a new fires or signals, if it receives an
adequately strong Input signal from another node/nodes to which it is attached. ANN
3 Explainable Artificial Intelligence (XAI): Conception … 45

Fig. 2 Neural network [30]

is computational model or mathematical model, also can be called as an informa-

tion processing paradigm which is motivated by the way biological nervous system,
such as brain information system. ANN is consisting of interconnecting artificial
neurones. These neurons are programmed like to imitate the properties of n biolog-
ical neurons. These neurons work simultaneously to solve any specific problem.
ANN is applied/used for image analysis, speech recognition, adaptive control etc.
(Fig. 2).

4.1.6 Fuzzy Logic

The concept fuzzy denotes to things that are not understandable or are clear. In the
real world there are many situations where we face a situation when we can’t evaluate
whether the state is true or false, their fuzzy logic gives very important flexibility for
reasoning. In this way, we can think about the uncertainties and inaccuracies of any
situation.
In the Boolean system truth value, 1 denotes the absolute truth value and 0 shows
the absolute false value. But in the fuzzy logic, there is no concept for the absolute
truth and absolute false value. But in fuzzy system, it has an intermediate value that
represents which thing is partially true and partially false (Fig. 3).

Fig. 3 Fuzzy logic [31]

46 T. Nizam and S. Zafar

4.1.7 Genetic Algorithm (GA)

A genetic algorithm (GA) is a type of heuristic search algorithm which is used to solve
optimization and search problems. This algorithm is also a subset of evolutionary
algorithms, which are used in computation. Genetic algorithms apply the idea of
genetics and natural selection for giving solutions to problems. These algorithms
are taken as having better intelligence than random search algorithms because they
consider historical data to perform search to the best performing region within the
solution space. GAs is also built on the characteristics of chromosomes and their
genetic structure. There is fitness function which helps in giving the features of all
individuals within the population. The higher the function, the better the solution.

4.1.8 Adaptive Neuro Fuzzy Inference System (ANFIS)

Adaptive Neuro-Fuzzy Inference System (ANFIS) mixes properties/advantages of

both Fuzzy Logic (FL) and Artificial Neural Networks (ANNs) in a single framework.
It provides accelerated adaptive interpretation capabilities and learning capacity to
model complex features and captures nonlinear relationships. ANFIS has been prac-
ticed in various different domains and given solutions to commonly happening prob-
lems with improved space and time complexity. The key point of ANFIS is the
accuracy using the precise fuzzy interpretability and modelling, which enhances its
generalization ability. ANFIS has achieved importance amongst researchers for its
robustness in modelling fuzzy sets into crisp inputs and providing crisp outputs from
the fuzzy rules for reasoning purpose. In other words, ANFIS has to justice the
accuracy-interpretability trade-off [3].
In Chap. 2, researchers proposes ANFIS-GA algorithm takes the medical data
of a patient to predict the risk of having heart attack. The dataset used in Chap. 2
is the Cleveland dataset which have medical reports of 297 patients [16]. It shows
14 features including sex, age, resting blood pressure, chest pain type, fasting blood
sugar, cholesterol, maximum heart rate, resting electrocardiographic results, depres-
sion induced by exercise relative to rest, exercise induced angina, number of major
vessels, the slope of peak exercise, the heart status, and the diagnosis of heart disease.
136 out of 297 patients given in the dataset have some level of heart disease and 161
patients have no heart disease.
The identification of heart attack is divided into five different possibilities
including at no risk (level 0), slight risk (level 1), average risk (level 2), high risk
(level 3) and very high risk (level 4) of having heart attack. 40 instances are used
for testing and 257 instances of the dataset are used for training. First of all, using
Takagi Sugeno type fuzzy inference system, grid partitioning technique is applied to
create initial membership functions which is based on the training set. Using Gaus-
sian membership functions, if–then rules are derived. After making an initial FIS,
the algorithm is trained using ANFIS to recognize fuzzy parameters by processing
the dataset given to it. ANFIS used in the algorithm produces a single-output Sugeno
FIS and then integrates the least-squares and the backpropagation gradient decent
3 Explainable Artificial Intelligence (XAI): Conception … 47

techniques for training FIS membership function parameters. In Chap. 2, the system
is trained for 1000 epochs. This training process stops whenever the defined epoch
number is reached. In a novel technique and in next step, GA is used for optimizing the
membership function parameters. As a result of bringing together these three algo-
rithms (the neural network, the fuzzy logic, and the genetic algorithm), for predicting
heart attack a new classification algorithm is generated. GA is set to minimize the
cost function (CF) which is defined as root mean squared error (RMSE) between the
expected (desired) and the predicted heart attack level.
⎡
l n
l1 Σ
CF = RMSE = √ (D Oi − P Oi )2 a n (1)
n i=1

where, DO = Desired Output and PO = Predicted Output

N = Number of features provided in the dataset.
These predictions generated by the algorithm can be classified into four classes.
(1) YN = The case where a patient at a risk of having heart attack is predicted to
be at no risk
(2) YY = The case where the algorithm successfully predicts a patient being at a
risk of having heart attack
(3) NN = The case where the algorithm truly predicts no risk of having heart attack
for a patient at level 0
(4) NY = The case where the algorithm predicts being at a risk of having heart
attack for a patient who is healthy.
For evaluating the proposed evolutionary ANFIS-GA system, four evaluation
functions including specificity, sensitivity, precision and accuracy are defined to be
used along with RMSE-the evaluation functions for the training set acquired as the
specificity 79.1667%, the sensitivity 91.1504%, the precision 77.4436% and the
accuracy 84.4358%. Additionally, specificity, sensitivity, precision and accuracy for
the testing set achieved 81.25%, 79.1667%, 86.3636% and 80%, respectively. In this
way, researches proved in Chap. 2 that using ninefold cross validation, the above
proposed algorithm’s performance was quite satisfactory for predicting heart attack.

Specificity = NN NN + NY (2)

Sensitivity = YY YY + YN (3)

Precision = YY YY + NY (4)

Accuracy = YY + NN YY + YN + NN + NY (5)
48 T. Nizam and S. Zafar

In Chap. 1, researches have used an online dataset, i.e. UCI Repository, which
have The Cleveland Heart Disease Dataset. This dataset contains 1025 instances in
total, which have been used in this research work. Each data is considered based on
14 attributes. This dataset has missing values too, which is manipulated through data
pre-processing using some statistical techniques. The Class Value 1, is represented
as “tested positive for the disease”, and Class value 0 represents “tested negative for
the disease”. There is a division in this dataset into certain percentage, such as 20%
data has been taken as testing dataset and rest 80% data is taken as training dataset.
Different attributes of the original dataset have some missing values also which can
give an imprecise result and may degrade the accuracy of model also. To overcome
this problem, by using a method, “mean of column”, missing values are replaced.
This method basically replaces 0 with either mean values of neighbourhood values
or taking the average of neighbourhood values. Then accordingly it converts the
0 value with the newly calculated value. After that the values of the dataset were
changed from Numeric to Nominal so that it can be dataset compatible with the ML
Techniques used.
The model is made in Weka which is a Data Mining Tool. The Waikato Environ-
ment for Knowledge Analysis (WEKA) is an open-source machine learning software
which developed by Waikato University, New Zealand. The software easily executes
different standard data mining tasks such as clustering, data pre-processing, regres-
sion, classification feature selection and visualization. It provides an easy environ-
ment to load data in the form of URLs, files or databases. It examines and conceives
the true positive, confusion matrix, recall, precision and false negative etc. in a
convenient way.
Four accuracy measures have been taken for comparison of the four models, they
are as follows:
(1) Precision or Positive Predictive Value: It is the average probability of relevant
retrieval.

Precision = Number of true positives/Number of true positives

+ False positives (6)

(2) Recall: It is the average probability of complete retrieval.

Recall = True positives/True positives + False negative. (7)

(3) Accuracy: The accuracy of a classifier is given as the percentage of total correct
predictions divided by the total number of instances.
⎡ ⏋
Accuracy = Number of True Positives + True Negatives
/[Total Instances] (8)

(4) ROC (Receiver Operating Characteristic) Area: ROC depicts the performance
trade-off between the true positive rate (TPR) and false positive rate (FPR) of a
3 Explainable Artificial Intelligence (XAI): Conception … 49

classification model.

TPR = [Number of True Positives]/[Number of True Positives

⏋
+False Negatives (9)

FPR = [Number of False Positives]/[Number of False Positives

⏋
+True Negatives (10)

Classifier is generally taken as “GOOD” when ROC value is less than 0.80, “FAIR”
when ROC value is 0.77. The ROC value which is very close to 1 is considered as
best model with high accuracy.
At the end of experiment, researchers found that Random Forest and SVM
performed very well in comparison to Decision tree and Gaussian Naive Bayes.
Model developed with SVM produces 98% accuracy which is approximately 13%
greater than Decision tree and 8% greater than the Naïve Bayes. Similarly, model
build with Random Forest generates the best prediction result with 99% accuracy,
which is itself more accurate than our second best SVM model for heart disease
prediction. Unfortunately, decision tree is not suitable for this dataset.

5 Conclusion

By presenting this research, we have tried to analyse the different machine learning
techniques and to predict if someone in particular, given different individual
indications and attributes, will get coronary illness or not.
In Chap. 1, researchers used Cleveland dataset for heart diseases. This dataset
contains 1025 instances and used percent split to divide the data into two categories
which are testing and training datasets. They have also considered 14 attributes and
implemented four different algorithms, i.e. SVM, Random-forest, Decision Trees and
Naïve Bayes, to check the accuracy. By the end of this research, they have discovered
that maximum accuracy level is achieved by using Random Forest in dataset whereas
Decision Tree is playing out the least with an accuracy level.
In Chap. 2, a novel Adaptive Neural Fuzzy Inference System (ANFIS)-Genetic
Algorithm (GA) algorithm for predicting heart attack is proposed. By using the
explainablity of fuzzy rules, the optimization power of GA and training capability of
neural network, the algorithm was executed on the Cleveland dataset. After that, the
trained Fuzzy Inference System (FIS) obtained from ANFIS was given into GA to
optimize the membership function parameters. As a result, the Root mean Squared
Error (RMSE) between the predicted ones and the expected results obtained from
the algorithm reduced from 0.82754 to 0.75372. The efficiency of the proposed algo-
rithm was analysed by evaluation functions such as specificity, sensitivity, accuracy,
precision and RMSE.
50 T. Nizam and S. Zafar

The above results showed that the performance of the ANFIS-GA algorithm
in predicting heart attack was much satisfactory because this algorithm provides
explainable result. Providing explainable results are very useful for clinicians and
patients. This algorithm was designed in such a way that it gives matrices corre-
sponding to the predicted patients in different classes of having heart attack. Detecting
the attributes with highest significance and representing the satisfactory performance
of the algorithm on the new model may be taken as a good criterion for giving
explainable predictions based on the relevance of the features.

References

1. V. Sharma, S. Yadav, M. Gupta, Heart disease prediction using machine learning techniques, in
2020 2nd International Conference on Advances in Computing, Communication Control and
Networking (ICACCCN), pp. 177–181 (2020). https://doi.org/10.1109/ICACCCN51052.2020.
9362842
2. M. Aghamohammadi, M. Madan, J. Hong, I. Watson, Predicting heart attack through explain-
able artificial intelligence, pp. 633–645 (2019). https://doi.org/10.1007/978-3-030-22741-
8_45
3. A. Barredo Arrieta, N. Diaz Rodriguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado González,
S. Garcia, S. Gil-Lopez, D. Molina, V. Benjamins, R. Chatila, F. Herrera, Explainable artificial
intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI.
Inf. Fusion 58 (2019). https://doi.org/10.1016/j.inffus.2019.12.012
4. F. Došilović, M. Brcic, N. Hlupic, Explainable artificial intelligence: a survey (2018). https://
doi.org/10.23919/MIPRO.2018.8400040
5. S. Pouriyeh, S. Vahid, G. Sannino, G. De Pietro, H. Arabnia, J. Gutierrez, A comprehensive
investigation and comparison of machine learning techniques in the domain of heart disease, in
2017 IEEE Symposium on Computers and Communications (ISCC) (Heraklion, 2017), pp. 204–
207. https://doi.org/10.1109/ISCC.2017.8024530
6. C. Raju, E. Philipsy, S. Chacko, L. Padma Suresh, S. Deepa Rajan, A survey on predicting heart
disease using data mining techniques, in 2018 Conference on Emerging Devices and Smart
Systems (ICEDSS) (Tiruchengode, 2018), pp. 253–255. https://doi.org/10.1109/ICEDSS.2018.
8544333
7. Y.-X. Wang, Q.H. Sun, T.-Y. Chien, P.-C. Huang, Using data mining and machine learning
techniques for system design space exploration and automatized optimization, in Proceedings of
the 2017 IEEE International Conference on Applied System Innovation, vol. 15, pp. 1079–1082
(2017)
8. S. Chattopadhyay, A neuro-fuzzy approach for the diagnosis of depression. Appl. Comput.
Inform. 13, 10–18 (2017)
9. A.M. Sagir, S. Sathasivam, A novel adaptive neuro fuzzy inference system based classification
model for heart disease prediction. Pertanika J. Sci. Technol. 25(1) (2017)
10. D. Richard, D. Mala, GI-ANFIS approach for envisage heart attack disease using data mining
techniques. Int. J. Innov. Res. Adv. Eng. (IJIRAE) 5 (2018)
11. D. Bonanno, K. Nock, L. Smith, P. Elmore, F. Petry, An approach to explainable deep learning
using fuzzy inference. Next-Generation Analyst V. (2017)
12. B.W. Israelsen, ‘I can assure you ... that it’s going to be all right’—a definition, case for, and
survey of algorithmic assurances in human-autonomy trust relationships. ArXiv:170800495
Cs Stat, August 2017
13. T.T. Nguyen, P.-M. Hui, F.M. Harper, L. Terveen, J.A. Konstan, Exploring the filter bubble:
the effect of using recommender systems on content diversity, in Proceedings of the 23rd
International Conference on World Wide Web (New York, NY, USA, 2014), pp. 677–686
3 Explainable Artificial Intelligence (XAI): Conception … 51

14. K. Crawford, Opinion I Artificial Intelligence’s White Guy Problem,” The New York Times,
25-Jun-2016.
15. B. Goodman, S. Flaxman, European Union regulations on algorithmic decision-making and a
‘right to explanation’. ArXiv 1606.08813 Cs Stat, Jun 2016
16. S. Khedkar, V. Subramanian, G. Shinde, P. Gandhi, Explainable AI in healthcare. SSRN
Electron. J. (2019)
17. E. Tjoa, C. Guan, A survey on explainable artificial intelligence (XAI): toward medical XAI.
IEEE Trans Neural Networks Learn Syst 32(11), 4793–4813 (2021). https://doi.org/10.1109/
TNNLS.2020.3027314
18. M.T. Ribeiro, C. Guestrin, Why should I trust you? Explaining the predictions of any classifier,
pp. 1135–1144 (2016)
19. D. Wang, Q. Yang, A. Abdul, B.Y. Lim, Designing theory-driven user-centric explainable AI,
in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems—CHI
’19, pp. 1–15 (2019)
20. G. Marcus, Deep learning: a critical appraisal. ArXiv: 1801.00631 Cs Stat, Jan 2018
21. S. Lundberg, S.-I. Lee, A unified approach to interpreting model predictions. ArXiv:1705.07874
Cs Stat, May 2017
22. A. Cano, A. Zafra, S. Ventura, An interpretable classification rule mining algorithm. Inf. Sci.
240, 1–20 (2013)
23. J. Wang, R. Fujimaki, Y. Motohashi, Trading interpretability for accuracy: oblique treed sparse
additive models, in Proceedings of the 21th ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining (New York, NY, USA, 2015), pp. 1245–1254
24. T. Van Gestel, B. Baesens, P. Van Dijcke, J. Suykens, J. Garcia, T. Alderweireld, Linear and
non-linear credit scoring by combining logistic regression and support vector machines. J.
Credit Risk I (2005)
25. R. Piltaver, M. Lustrek, M. Gams, Multi-objective learning of accurate and comprehensible
classifiers—a case study (2014)
26. A. Bennetot, J.L. Laurent, R. Chatila, N. Diaz-Rodriguez, Towards explainable neural symbolic
visual reasoning, in NeSY Workshop IJCAI 2019 (Macau, China, 2019)
27. https://en.wikipedia.org/wiki/Bayes27_theorem
28. https://en.wikipedia.org/wiki/Naive_Bayes_classifier
29. https://towardsdatascience.com/understanding-random-forest-58381e0602d2
30. Images of neural network—Bing images
31. Fuzzy Logic | Introduction—GeeksforGeeks
Chapter 4
Explainable AI (XAI): A Survey
of Current and Future Opportunities

Meet Kumari, Akshit Chaudhary, and Yogendra Narayan

Abstract Artificial Intelligence is the technology that is being used to develop

machines that could work like humans or simply can have the intelligence relatable
to that of humans. But the development of this kind of technology model that mimics
humans involves a lot of complex calculations and complex algorithms that are diffi-
cult to explain and understand. For this problem, the concept of explainable artificial
intelligence (XAI) is developed and introduced. It is the technology that is developed
to ease the understanding process of machine learning solutions for humans. It is the
concept that is being developed for making it convenient for humans to understand
and interpret machine language. Black model machine learning (ML) algorithms
are very hard to understand for humans who have not developed them. AI models
that involve the methods like genetic algorithms or deep learning concepts are very
difficult to understand. It sometimes becomes a very hard task for the domain experts
too to understand the ML algorithms of the black block models, so the need for the
development of this type of technology was felt. Many times, results are developed
with very high accuracy are quite easy to understand for the domain experts. But
Explainable artificial intelligence has a great potential to make a change in domains
like finance, medicines, etc. It plays a vital role where it is important to understand
the results to build trustworthy algorithms. XAI can play a great role in “third-wave
AI systems” which include machines that can interact directly with the environment
and that can build explanatory models that allow them to develop the characteris-
tics of real-world phenomena. XAI has the potential to play a great role where the
organizations need to build trustworthy AI models and to make them trustworthy the
explainability of the AI models should be there for others as well. This technology
is developed primarily to make AI understandable to those who are practitioners.
This book chapter presents a wide and insightful view of XAI and its application in
various fields. This chapter also includes the future scope of this technology and the
need for the growth of this type of technology.

M. Kumari (B) · Y. Narayan

Department of Electronics and Communication Engineering, Chandigarh University, Mohali,
Punjab, India
e-mail: [email protected]
A. Chaudhary
Department of Computer Science and Engineering, Chandigarh University, Mohali, Punjab, India

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 53

A. E. Hassanien et al. (eds.), Explainable Edge AI: A Futuristic Computing Perspective,
Studies in Computational Intelligence 1072,
https://doi.org/10.1007/978-3-031-18292-1_4
54 M. Kumari et al.

Keywords Explainable artificial intelligence · Artificial intelligence · Machine

learning · Explainable methods · Deep learning

1 Introduction

Artificial intelligence (AI) can be aggrandized as Artificial Intelligence which means

the study of science that involves the invention of intelligent machines, computer
programs being the most important. It is about using computers to collect and know
about the intelligence that humans have, but it doesn’t restrict itself to biologically
observable methods only. It is a technology invented opposite to the natural things
using different tools or machines or machine learning codes. It was previously thought
that AI was developed just made to mimic human nature but with time its use and
importance are growing in various fields. The use of AI is growing in various fields
such as the web, self-driven cars, robotics, etc. It is being developed for various
purposes to reduce the human effort to do the hard task and also to reduce the time
taken to complete the heavy task. The year 1956 is usually considered to be as the birth
year of Artificial intelligence because of the famous conference held at Dartmouth
College. In the year 1955, first AI-based system was developed by Allen Newell, and
Herbert A Simon, which was known as Logic Theorist. Another approach to symbolic
AI appeared in the 1960s and is further named the Knowledge-Based approach. It
was derived during the project development process. This project was developed to
execute Dendral Expert system. Gradually, its usage and applicability are growing
in various tasks. AI plays a very important role in the field of robots. Various robots
are developed that mimic human activity. Many advancements are done every day
in search of solutions to various problems. Various kinds of logical algorithms are
developed and used to deal with various kinds of problems and develop solutions for
them [1].
AI is an imitation of human intelligence or knowledge in artificial machines
designed to make them behave and think the same as humans. The term can also be
used for any computer that displays features of human intelligence, such as learning,
thinking, and problem-solving. The ability to think and act with the best possible
chance of attaining a certain goal is the right quality of practical wisdom. Machine
learning is a subset of artificial intelligence that explains the idea that computer
programs can automatically learn and adapt to new data without human help. In-
depth learning algorithms allow this automatic learning by importing a lot of random
data, including text, images, and video [1, 2].

1.1 Understanding Artificial Intelligence

The first thing which comes to the mind of most people when they hear the word
intelligence is usually robots. The reason behind this is that high-quality movies
4 Explainable AI (XAI): A Survey of Current and Future … 55

often feature machines that behave like humans and that cause destruction in the
world. But nothing could be further from the truth. Artificial intelligence wishes that
human intelligence could be described in a way that makes it become easier for a
computer to solve complex problems and perform heavy calculations or tasks. Artifi-
cial intelligence aims to imitate processes in humans. When it comes to defining clear
processes, such as learning, thinking, and perception, researchers and developers in
the field make rapid progress unexpectedly. Some people think that designers will
soon be able to create better programs than people can read or understand right now.
Others, however, continue to hold onto this view because all psychological processes
involve quantitative judgment influenced by human knowledge. The terms used to
describe artificial intelligence in the past become obsolete as technology advances.
For example, machines that perform simple math or visualization of text are no longer
considered a combination of artificial intelligence because we now consider these
skills as part of any computer [3].

1.2 Advancement of the AI

With the advancements of AI in various fields, the major problem that comes out
is the lack of interpretability and trust. These factors are the major drawbacks that
make its applicability difficult in healthcare and finance where the need for a rational
model to make decisions is important for trust. AI has achieved increasing momentum
in its application in many areas to deal with increased complexity, scalability and
automation, which are also penetrating digital networks today. The rapid increase in
complexity and sophistication of AI-powered systems has evolved to such an extent
that people do not understand the complex mechanisms by which AI systems work
or how they make certain decisions—a particular problem when AI-based systems
calculate outputs that are unexpected or seemingly unpredictable. This is especially
true for opaque decision-making systems, such as those using deep neural networks
(DNNs), which are considered complex black-box models as shown in Fig. 1.

Fig. 1 Black box model illustrative diagram

56 M. Kumari et al.

1.3 Special Considerations

Artificial intelligence has received criticism from both the scientific community and
the general public since its inception. Another theory that emerges is that machines
will evolve to the point where people will not be able to keep up with them and will
move on its own, re-establishing itself in terms of power. Another is that technology
has the power to be armed and can attack people’s privacy. Some controversies have
focused on artificial intelligence and whether robots and other intelligent machines
should be granted the same human rights. Self-driving cars have caused controversy
because their cars are usually built considering the small number of accidents and
injuries. These vehicles will determine which route would cause the least damage if
given the opportunity to choose between colliding with one person at a time.

1.4 How Artificial Intelligence Can Affect Human Activities?

There are concerns that people may be forcibly fired from their jobs as more and
more businesses try to create certain jobs by using smart devices. Self-driving cars
may eliminate the need for taxis and car-sharing services, and manufacturers may
quickly switch workers into machines, further avoiding the need for human skills.
This concept will affect the life of the people who drive cars. The advancements in
the field of AI-based machines are done to reduce human efforts but at the same time,
it tends to reduce the job roles for humans. In the upcoming years, it is the tendency
that many decisions will be made by intelligent machines, and the human jobs role
in taking decisions would decrease.

2 Applications of AI

AI is used for performing a variety of tasks in various fields. It can be applied to

various processes in a variety of industries and areas. In the field of health care, AI
is evaluated and monitored for surgeries in the operating room, drug management,
and a variety of patient care. Other incidents of smart machines include self-driving
cars and computers playing chess. These machines have to consider all the factors
for making decisions because every action has an effect on the outcome.
Figure 2 shows the applications of AI in different fields. For self-driving cars
to work in a way that avoids collisions, a computer program should calculate all
external data and consider it. Fraudulent operations are used in the banking and
financial sectors to identify and signal suspicious behaviors, such as the use of an
unusual debit card and important accounting, that benefit the bank fraud department.
AI tools also help to simplify and facilitate the trading that is done by developing the
provision, demand, and price of securities easy to estimate. It can be used in several
4 Explainable AI (XAI): A Survey of Current and Future … 57

Fig. 2 Some of the major applications of artificial intelligence

fields without looking at its computer-related features. Some of these are listed below
[4], Fig. 2.
• Speech recognition: Computer speech recognition, automatic speech recognition
(ASR), or text-to-speech recognition skills are the skills that use a natural language
processing system (NLP) to convert a person’s speech into a text format. To
make voice search, many devices integrate such algorithms into their systems.
For example, Siri. Artificial Intelligence has played a very important role when it
comes to disability not only physically or partially or for those who are sometimes
unconscious or even learning. To get to the world where you are and where you are
known, the blind can just talk and Artificial Intelligence will do the necessities.
The illiterate get the edge that they do not need to learn to read or write as they
can speak their questions or instruct the software to perform certain tasks such as
calling someone and Artificial Intelligence can answer questions and make calls.
• Customer service: In today’s world, human agents are being replaced by tangible
agents when it comes to customer care. They respond well to frequently asked
questions (FAQs) for common questions, offer personal advice, suggest user sizes,
best-selling products, and change the way of thinking about customer engagement
across all websites and forums. Example Messages for bots, Slack, and Facebook
Messenger. While visiting e-commerce sites, we are confronted with a host of
visually impaired agents who ask, “How can I help you?”. These message bots
narrow the gap between Queries and Solutions and strengthen the sales process.
They save our time as we do not need to scroll down to read the description or
get out of the app to ask our questions via email.
58 M. Kumari et al.

• Computer view: This Artificial Intelligence feature helps computers and systems
to extract important information from digital images, videos, and other visual
inputs, and based on this audio and visual input, it can take action. It differs from
image recognition functions in its ability to provide recommendations. Computer
vision is developed using convolutional neural networks and it is used for tagging
images on social media, radiology image in health care, and self-driven vehicles
within the vehicles industry. It has made it easy to contact our friends on social
media and forward a photo or post to them.
• Automotive industry: The automotive industry has benefited greatly from this
new use of Artificial Intelligence. It has made it easier for beginners to drive cars
with this AI feature.
• Recommendation engines: Artificial Intelligence algorithms also assist compa-
nies in finding data trends using previous data management which can help in
the development of highly productive cross-marketing action plans. This can also
help in making additional relevant suggestions to customers while undergoing
the payment process for online shopping. It empowered businesses and consumer
goods as AI-assisted them in selecting what to recommend to the consumer so that
opportunities to purchase goods and services could be increased. For example, if
we search for a particular type of shirt on any e-commerce website, then there is a
high chance that we will encounter the same type in our standard online drives that
force us to think about it, often, which affects our way of thinking and organizing.
• Automatic stock trading: Automated stock trading is designed to improve the
portfolio of stocks. Artificial Intelligence shapes its future as these commercially-
driven AI platforms are trading billions of dollars a day without human interven-
tion. Robo Advisors analyze a lot of data, facts, problems, threats, etc. with high
accuracy and reduce risk. Only Artificial intelligence has made the industry faster
with rocket fuel. Many companies have introduced programs based on the end of
Artificial Intelligence after considering their growing demand and high success.
• Language Learning: Artificial intelligence has become an integral part of manu-
facturing, language learning, e-commerce, fintech, education, skills development,
medicine, electricity, and the automotive industry in our daily lives. Intelligence
startup in learning a foreign language has opened the way for personal education.
Integration of Artificial Intelligence into learning.
• Medicine: Artificial intelligence can be used in almost every part of medicine. It
helps in collecting patients’ data with great accuracy and minimal possibilities of
error. When physicians take the readings themselves, being a human tendency to
ignore some lesser important facts and figures, they make mistakes. Even getting
the medicine prescriptions written using Artificial Intelligence is highly appreci-
ated as many patients face adverse consequences including cases of death due to
doctors’ poor handwriting making it difficult for the pharmacists to figure out the
correct medicines and doses. Researchers in the field of AI has devised various
kind of tools to solve various problems using probability and other mathematical
operations. Most of the operations are done by various logic developed. The oper-
ations or the algorithms to develop artificial intelligence help in the innovation of
many new things. The researchers or the developers develop after examining a lot
4 Explainable AI (XAI): A Survey of Current and Future … 59

of possibilities. When the researchers or the developers develop the algorithms

of the solutions using different operations these developed algorithms are too
complex that it becomes a very difficult task for the researchers to explain them to
the other researchers or developers in the field. It becomes a very important task
to explain it or reproduce it to others for further development when a trustworthy
algorithm has to develop for a particular task. Healthcare is also being devel-
oped in a variety of areas with computer vision as it is easy to detect unwanted
diseases or changes in our body structures. To make the advancements more reli-
able and trustworthy these algorithms and logic should be developed in such a
way that they are easy to interpret by others. To deal with the problems due to the
complexity of the algorithms, the concept of Explainable Artificial Intelligence
was introduced. It is developed to make it easy for the researchers to explain their
developments or research work more easily as it is sometimes very difficult for
the field professionals also to understand the logic or the algorithms behind it.

3 Importance of AI in Health Industry

In recent decades large amounts of datasets, high computational powers, and machine
learning systems have developed for high-performance tasks. Artificial intelligence is
an approach toward the development of technology-based on computational learning
and intelligence. AI has the potential of analysing and solving the complex prob-
lems of the medical industry. These days AI is playing an important in the medical
field as the medical industry faces challenges in analyzing and application of a lot
of knowledge to solve various complex problems. The advancement of AI in the
field of medicine is to help clinicians to formulate the diagnosis, make complex
decisions, and in the prediction of the results. It is developed to assist healthcare
workers to perform their tasks easily and accurately. In the case of AI development
for the medical industry, the role of Explainable Artificial Intelligence becomes
more important to deal with complex predictions and therapeutic decisions. In the
medical field, the algorithms of the complex calculations need to be checked to make
trustworthy and for this, the complex calculations of the AI algorithms need to be
developed in such a way that they are easy to understand. It is very important to
deal with complex calculations. Artificial neural network (ANN) is one of the most
popular AI techniques used in the field of medicine. These computational tools are
the tools that are by biological systems. These systems consist of various computer
machines connected known as neurons that are capable of data representation and
processing simultaneously. Their various abilities to retain historical data or exam-
ples or analyzing capability and many others make them very attractive and important
analytical tools. But the development of AI tools needs precision and to make them
reliable the algorithms or the codes need to be easily explainable to others to make
them reliable [5, 6].
The use of AI is increasing day by day in the field of neuroscience. Different
Artificial intelligence models help in the interpretation of multimodal datasets that
60 M. Kumari et al.

have the potential to provide unbiased insights into brain function and detection of
disorders of the brain. The ability of AI to make precise classifications and predictions
is there but, in some cases, it is unable to explain how inputs and the results are
relatable to each other. One big challenge for humans is the inability to keep a check
and modulate neural activities in real-time. To deal with the complex problems in
the field of neuroscience, advanced AI solutions need to be developed so that the
resulting models are able to are capable of interpreting, modulation, and sensing from
a huge database. New approaches to deal with these complexities are developed using
machine learning. Explainable AI has developed a new way with techniques in which
AI and ML complex algorithms are combined to develop easily explainable solutions.
Data-driven and theory-driven models both have a contrast. The data-driven models
are more accurate in terms of prediction while the theory-driven models are more
explainable. In general, the models with high performance are less explainable, while
the explainable models are generally less accurate. These are the general issues that
the ML community face. These problems can be solved by using models that are
simple and easier to explain or by building appropriate models.

4 Introduction to Explainable Artificial Intelligence

Explainable AI is the form of AI in which the results are developed in such a way that
they are easy to understand by people other than the developers of that algorithm. If
humans have to accept any algorithm for a particular task, they need to have trust in
it, and to build trust the algorithms developed should be developed in such a way that
they are easily understandable and transparent. To make an algorithm trustworthy
transparency becomes a very important aspect for which the algorithm needs to be
easy to interpret or explain. For the purpose of extending transparency and trustwor-
thiness, the system was developed which was known as Truth Maintenance System.
This system tracks the reasoning and helps in the justification of the conclusions.
Latest AI approaches like deep learning and machine learning are very complex and
non-transparent in nature. To cope with this opaque nature and make the algorithms
easy to interpret various types of explainable models are developed like LRP or
Layer wise relevance propagation. In addition to this, work has also been done to
make glass-box models that are much more transparent for the purpose of inspection.
XAI models are developed having a primary focus on making it more understandable
to the people who practice AI rather than making it easier for the end-users. XAI
acts as a core for human-centric machine learning developments. There are various
types of machine learning (ML) models like rule lists, decision trees, simple naïve
Bayes Classifiers, etc. With the increase in complexity the understandability and the
explainability of the algorithms decrease as shown in Fig. 3.
Explainable AI (XAI) or Interpretive AI or Explainable device gaining knowledge
of (XML) is an AI wherein the outcomes of an algorithm are easy to understand with
the aid of people as shown in Fig. 3. It contradicts the belief of a “black box” in the
machine getting to know, which even its designers can’t provide an explanation for
4 Explainable AI (XAI): A Survey of Current and Future … 61

Fig. 3 Illustrative diagram

for explainable AI

why the AI got here to a positive end [2]. XAI guarantees to assist users’ paintings
more efficaciously by way of improving the intellectual models of users of effective
AI structures and doing away with their defective thoughts.
XAI may be an enforcement of a right of definition XAI is crucial even supposing
there aren’t any prison or regulatory requirements—as an instance, XAI can improve
the person enjoy of a service or product by assisting cease customers believe that AI
is appearing nicely. In this way, the cause of XAI is to give an explanation for what
has been completed, what has been carried out, what is going to occur subsequently
and to reveal the records on which the moves are primarily based. Those elements
make it feasible to (i) affirm existing knowledge (ii) question existing understanding
and (iii) generate new thoughts.
Algorithms that are used in AI can be divided into white machine getting-to-
know algorithms and black container (ML). White box fashions ML fashions offer
clean consequences for professionals within the subject. Black container models,
alternatively, are extraordinarily tough to outline and aren’t understood even by way
of specialists inside the subject. The procedures that extract model parameters from
the education information and generate the labels within the take a look at data
can be explained and stimulated. Translation method a possibility to recognize the
machine gaining knowledge of the model and lay the basis for choice making in a
way that humans can recognize. A definition of a idea that is considered crucial,
but coherent which means does no longer but exist. it is advised that the definition
in ML may be thought of as “a summation of the factors of an interpretative space
committed to a specific model for choice making”. while algorithms meet those
requirements, they offer the basis that are for validating and to track selections and
for that reason validating themselves, developing algorithms, and discovering new
information. Every now and then it is also feasible to attain the end result with high
accuracy with the white field ML interpretation set of rules itself. this is especially
vital in areas along with remedy, law, protection, and finance and in these cases it’s
important to understand choices and construct agree with algorithms [7, 8].
AI structures increase behaviours to fulfil mathematically defined goal structures
chosen by way of gadget designers because the command to “boom the accuracy
of trying out how nicely a film assessment is within the test database”. The AI can
research well-known policies which can be useful in a check set, including “updates
containing the word ‘horrible’ may be terrible”. but, “opinions with ‘Daniel Day-
Lewis’ are generally exact”; these guidelines may not be famous in the event that they
62 M. Kumari et al.

appear to be they cannot do matters normally outside of the train set, or if humans
view the regulation as “dishonest” or “wrong”. The rules in XAI may be researched
to get a concept of how the system has the potential to generate actual-international
information for the future without a check set. AI is undergoing a lot of developments
these days. Most of the time, every company has plans to install AI, use it actively,
or rename its old tech engines with AI-enabled technology. As more companies use
AI and has advanced statistics in their business process and automated decisions, it
demands a specific reason that how the decision-making models grow bigger and
bigger. XAI is based on advanced methodology and strong thinking that depends
on how the results are derived for the proposed algorithms so that AI technology
can be clinically validated. In addition, XAI systems are more focused to build AI
mechanism or algorithms that are easy to understand for AI users. Some researchers
have encouraged the use of natural-language mechanical learning models, rather than
the use of post-hoc definitions, in which a second model is developed to describe the
first. This is because post-hoc models increase complexity in the decision-making
process and in part because it is often unclear how a post-hoc definition can faithfully
replicate a completely different model calculation. The main aim of Explainable
Artificial intelligence is to make the AI models explainable.
• Interpretability
To enable interpretability transparency becomes a very important property that was
traditionally the first step to protecting the rights in institutions. But with the new
advancements in the field of AI, the models are becoming much more complex that
are hard to get meaning, and hence the interpretation may vary for the required mech-
anism. With the increase in the interpretability of ML models, it becomes easier to
predict the reason behind the decisions. Interpretability is the concept of the advance-
ments in AI models that makes it easier for humans to understand the reasoning behind
the models. The more the interpretability of the model, the easier is the ability to
understand the model, and the more it becomes trustworthy for its application.
• Explainability
It is the extent that involves the development of the models in that form that is
easy for the users or the researchers to understand. In the case of explainability,
fidelity is one of the most important terms. The models with low fidelity lack the
explaining ability of the ML models. Explainable artificial intelligence (sometimes
known as shorthand “XAI”) refers to the power of the algorithm or model owners
to understand how AI has achieved its findings by making AI technology as bright
as possible [9]. AI is capable of making decisions automatically. Like in case of
hiring, it is important for the decision-makers of the organization to understand how
the decisions are made by the AI models. Many organizations wish to use AI-based
mechanism but are uncomfortable allowing the model or AI to take decisions that have
a significant impact because they do not yet trust the model. The description helps in
this because it gives the details on how the models take decisions and hence helps to
figure out whether the decisions are favourable or not. Machine learning is growing
4 Explainable AI (XAI): A Survey of Current and Future … 63

at a large scale in research as well as in industrial applications. Interpretability and

explainability play an important role when the question of accountability.
Major challenges that are faced are that sometimes it becomes difficult to create
the right model, to get stakeholder purchases the model that could make decisions
better than the person. To make better decisions, the right input is must to get the
best answer. In many cases the authorities require how the decisions are made. They
need to see how the model feels free to make a healthy and transparent decision.

5 Background of AI and XAI

The initial developments in the field of AI were easily interpretable. It was found as a
research topic at the workshop of Dartmouth College, USA. At that workshop, many
people predicted that machines could be developed that would have intelligence like
that of humans, and to make this vision a reality, they were funded with millions of
dollars. Later on, the AI researchers underestimated the complications and difficulty
in the process of the development of this project. In 1974, due to the criticisms
from James Lighthill, pressure from the British Government as well as Congress,
the U.S., the funding for this was stopped. Seven years later, after getting inspired
by the Japanese Government, Billions of dollars were provided to them. By the late
1980s, the funding was withdrawn again. The interest in the field of AI increased
and funding improved in the early decades of the twenty-first century. During this
time period, machine learning concepts and algorithms were developed and were
successfully applied in various industries. In the early stage of the development of
AI, its applications were restricted to only specific tasks due to many reasons such
as the low computational powers of the processors. AI requires a lot of information
in the form of databases and in 70’s it was very difficult to build a large database. In
the 1980s, an attempt was made to deal with the common-sense knowledge problem.
A massive database was created that contained mundane facts remembered by an
average person. Douglas Lenat, who led this project, stated that no other shortcut is
possible, the only way is to teach the concepts of the machine one by one. Now AI
has achieved many of its old goals and now is used in many industries. Some of the
success of AI has been able to due to an increase in computer power. Knowledge
engineering helps AI to deal with real-world problems more efficiently. With the
increase in the use of Artificial intelligence in various fields, the complexities of the
algorithms have also increased with the increase in the use of the concepts like deep
learning and machine learning. With advancements, the need for the explainability
of the complex algorithms was felt and many researchers highlighted the importance
of explainability for maintaining transparency and increasing the trust factor [10].
Many mathematical structures are used to discover the ML mechanism. Math-
ematical knowledge generally helps in understanding the resulting pattern. When
these patterns are easy to understand it becomes easier to improve those patterns
to deal with the deep complexities. Integrating is one of the most commonly used
methodologies.
64 M. Kumari et al.

6 Challenges

To deal with big and complex problems, we need high-dimensional and large datasets.
To monitor brain functioning there is a need for development in the acquisition
system. There is a need to achieve more accuracy in the solutions developed using
the latest Explainable models. Next, there is a need for development in different
models as most of the models operate with single-level and multi-level modelling
is a hard task if classical methods are used. To manage these problems accurately,
a need for XAI models is felt. For this, we need to use the tools that can help us to
understand the transformation of the inputs and the resulting output that happen in
the brain network. Explainability is gaining great interest in radiology. In the medical
field, interpretability includes many factors that are sometimes not considered in other
fields, that include the responsibility and risk factor. It is a difficult task for scientists
to accept the validity and to rely on the developed solutions, without knowing how
they are derived. Users need to know the probability when a result can be wrong
and without knowing the working model this can’t happen. These days many Deep
learning libraries have started including their XAI libraries. The assessment criteria
like usability and reliability help the Machine learning community to keep a check
on how the algorithms are used and how they can be improved. AI and Machine
learning concepts have a great potential to transform the way of dealing with complex
problems in the future. It has the ability to create a great impact and solve the
complexities present in various fields. In many of its applications outside medicine.
Explainable AI allows the users to understand and interpret how the results are
derived. The demand for XAI in medical field is increased due to the requirement
of fair and ethical results. Giving an explanation to the practitioners allow them to
make the life- saving decisions more easily. Clinical Decision Support Systems are
in great need of Explainable models as these systems help medical practitioners to
make complex decisions. The developers are more interested in local explanations.
The increase in the complexities and the human need to solve the insight problems
has given an opportunity for AI to emerge and solve complex problems. To tackle the
real-world problems, the algorithms developed by AI should be easily understandable
[11, 12].
A sound research approach can handle the small issue of detecting study bias.
However, the intrinsic black-box structure, while improving predictive performance,
might cause problems in high-stakes decision-making if adequate justifications for
the choices made aren’t provided. It’s crucial to understand when a model will work
and when it won’t, as well as the reasons behind any specific predictions it makes and
the degree of its limitations. A common concern among potential users of artificial
intelligence is that it is often unclear how technology can reach their conclusions. If
AI algorithms are locked in a “black box” that prevents people from knowing how
the outcomes were achieved, it might be difficult to trust the developed algorithms
because human experts sometimes are unable to explain the findings of AI. Being
able to define AI can help organizations establish greater trust with customers, clients,
and investment partners. One of the most important benefits of being able to define
4 Explainable AI (XAI): A Survey of Current and Future … 65

artificial intelligence is that it can help technology owners determine whether human
bias has affected the model. This can be especially important when AI is called to
make life or death decisions, such as a hospital setting where medical professionals
may be required to explain to patients and their families why certain decisions have
been made. In order to move beyond interpretability and investigate the domain
of XAI that especially deals with deep learning models, various advancements are
undergoing day by day. According to the essay the earliest and most popular XAI
technique was developed by Ribeiro et al. in their paper Explaining the Predictions of
Any Classifier, and it is known as Local Interpretable Model-agnostic Explanations
(LIME). The use of LIME on photos to identify a husky as a wolf sheds light on
the importance of explainability in preventing poor learning. The study shows how a
model came to understand that the presence of snow in the image was a key criterion
for identifying the animal as a wolf. These explanations make it simple for us to spot
incorrect learning, which helps us avoid bias in both data and models [13–15].

7 Applications of XAI

The various XAI applications are shown in Fig. 4.

• Healthcare

In the healthcare sector AI can play a crucial role but the risk of using an unreliable AI
system is much greater. The choices made by AI models that help doctors categorize
serious diseases based on structured criteria or unstructured data such as medical
imaging naturally have a broad impact. If a system makes a prediction and if the

Fig. 4 The various application fields

66 M. Kumari et al.

doctor needs to spend time to check whether the AI system’s decision is accurate and
reliable then the model would be of no use. In that case, an AI system that predicts
with explains the reasons for its conclusion will be much more valuable. To save a life
in the healthcare sector the reason for taking a decision while undergoing treatment
is very important.
In the medical field, when a patient suffering from a disease undergoes diagnosis,
then, in that case, explainable AI can play a vital role to explain the process of
diagnosis. It can help the doctors to make a treatment plan, take the correct decisions
in complex situations, and also to explain their diagnosis process to the patients. This
can help in the development of trust between the doctors and patients. For example,
XAI can help in making decisions is the diagnosis of pneumonia. AI can also help
in the diagnosis of cancer in the healthcare sector where medical imaging is used.
Currently, many attempts are made to discover how explainable AI models can be
developed so that they could be applied to neuroscience. With explainability, these
models also need to have the capability to discover a data-driven manner [16].
Banking, financial services and insurance (BFSI)—BFSI has benefited greatly
from XAI’s potential to transform the sector. Credit risk assessment based on arti-
ficial intelligence is widely used in the banking and insurance industries. In indus-
trialized countries, multimodal premium estimation is becoming more popular with
pay-as-you-drive and pay-how-you-drive mechanisms in which machine learning
algorithms are used in their decision-making. Industry can implement AI solutions
more effectively if XAI systems are able to deliver superior results and clear expla-
nations while gaining adequate trust and meeting regulatory standards. With larger
financial stakes, actions such as loan denials, increases or decreases in health insur-
ance or auto insurance costs, or incorrect investment recommendations, XAI can
be seen as an invaluable tool in explaining these options to the parties who will be
affected.
• Automobiles
The future of the automotive industry is autonomous driving, which is evolving.
Self-driving or autonomous vehicles are exciting as long as they don’t make any
mistakes. In this risky use of AI, one mistake can lead to the loss of one or more
lives. Explainability is essential to know about the capabilities and limitations of AI
models prior to implementation. It is important to understand the limitations of driver
assistance systems (or autopilot) when consumers are using them so that the causes of
the limitations can be identified, clarified and addressed as soon as possible. Parking
assist and voice assistants are one of the attractive features that largely represent a
comparatively low-risk model choice. However, there are alternative situations such
as brake assist or self-driving vehicles.
• Judiciary
AI technologies are increasingly being used to make decisions in the legal systems of
Western countries. As with parole based on the likelihood of repeat offending, biases
in AI applications have far-reaching effects, and justice in them is fundamental as
they affect human rights and freedoms. XAI is undoubtedly a rapidly expanding field.
4 Explainable AI (XAI): A Survey of Current and Future … 67

It is also heavily researched at ZenLabs because it helps data scientists to understand

their models in a better way and remove any complexities or problems they may
inadvertently introduce.
Explainable AI systems can be useful in situations involving liability, such as
autonomous vehicles; if there is something wrong with the Explainable AI, one will
still have to reckon with one’s actions. Explainable AI models are trained using
the concepts of explainable techniques, using human-readable textual descriptions
to explain the thinking behind the model’s prediction. Today, Explainable tech-
niques are used in many different areas of artificial intelligence, such as natural
language processing (NLP), computer vision, medical imaging, health information,
and more. Other universities where Explainable AI research is being conducted are
MIT, Carnegie Mellon University, and Harvard [17–19].

8 Difference Between AI and XAI

The major difference between AI and XAI models are given in Table 1.
The main difference between the XAI and AI is that the XAI deals with the
problems with the practical aspects and it develops solutions that are easily under-
standable. The explainable methods used in Explainable AI are highly influenced by
the factors that a human needs for practical application. In recent studies or develop-
ments, it been has suggested that the pursuit of definition in AI strategies should be
taken as the secondary goal in pursuit of excellence in AI and that the promotion of
special XAI development may limit the overall performance of AI. XAI models are
based on advanced methodology and strong thinking from evidence-based medicine.
XAI systems are more focused on making AI systems understandable and more trust-
worthy or transparent to AI. Some researchers have encouraged the use of natural-
language mechanical learning models, rather than the use of post-hoc definitions, in
which a second model is developed to describe the first. This is because post-hoc
models increase complexity in the decision-making process and in part because it is
often unclear how a post-hoc definition can faithfully replicate a completely different
model calculation [19–22].
A typical AI application is comparable to explainable artificial intelligence, or
XAI, except that the processes and outcomes of a XAI algorithm may be described

Table 1 Difference between AI and XAI models [19–22]

S. No. AI models XAI models
1 Complex Easily understandable
2 Chances of reliability are less due to less More reliability
understandability
3 Understandable mostly by the developers Easily understandable by other developers
and users as well
68 M. Kumari et al.

so that people can understand them. Because of its complexity, artificial intelligence
makes judgments in real-time based on the knowledge it has learned from the data
it has been fed. We cannot completely optimise the AI application to be all that it
is capable of doing when we do not fully grasp how AI is making these judgments.
This is due to the fact that the bulk of AI using ML functions in a “black box,” or a
space that is unable to offer any obvious insights into how it generates judgments.
Transparency and explainability are not strictly required because many AI/ML appli-
cations are somewhat benign decision engines that are integrated with online retail
recommender systems. The stakes are substantially greater for other, riskier decision-
making processes including medical diagnosis in the healthcare sector, financial
investment decisions, and safety–critical systems in autonomous vehicles. As a result,
in order to be seen as trustworthy, dependable, and consistent, the AI employed in
those systems must be explicable, transparent, and intelligible. Applications for AI
and ML are widely used in business today to automate decision-making and to get
analytical insights [23–25].
An explainable AI model would allow a brand to enhance revenue by identifying
the real drivers of sales, whereas data models may be taught to anticipate sales based
on variable data. AI is a simulation of human intelligence processes by a computer
system. AI includes some special features such as natural speech recognition and
machine vision. AI is accelerating around the world in a very fast speed. AI requires
a set of specialized software and hardware for writing and training the machine to
get the desired work from the machine. Some of popular programing languages used
for AI are Python, R and Java. In short AI work by ingesting a large amount of data
(labelled training data) and analysing that data for correlation and patterns and using
that analysis for predicting the future states. AI focuses mainly on three skills i.e.
self-correction, reasoning and learning. artificial Intelligence is getting used across
many industries to offer the whole lot from personalization, automation, monetary
decisioning, suggestions, and healthcare. For AI to be depended on and ordinary,
people need to be able to recognize how AI works and why it comes to make the
decisions it makes. XAI represents the evolution of AI, and offers possibilities for
industries to create AI packages that are relied on, transparent, impartial, and justified
[26–29].
Sometimes while learning, AI models learn unwanted tricks that do a great job
while training by satisfying the pre-programmed goals. For example, in the year
2017, a program given an image recognition function learned to “cheat” by looking
at the copyright tag present with horse pictures, in case of learning to say that the
horse was actually photographed. In another 2017 edition, a surveyed AI given the
task of capturing objects in the physical world learned to cheat as the trick was to be
placed in between object and viewer in such a way that it seemed to be catching a
false object. Explainable AI is defined as AI programs that describe post-prediction
thinking. Explainable AI is a term that comes under artificial intelligence that is called
as “interpreting.” Interpreting allows us to understand what the model is learning,
what other information it provides, and the reasons for its judgments because of the
real-world story we are trying to fix. If model metrics are inadequate, interpretation
is required. Model interpretation enables us to predict how the model will perform in
4 Explainable AI (XAI): A Survey of Current and Future … 69

different test environments by comparing it with the training environment. With the
help of explainable AI, you can provide transparency about how AI systems make
decisions and make the developed models trustworthy. XAI has its use case in various
fields such as the military, manufacturing industry, and healthcare sector. Explainable
AI is a new domain in the field of research and there are many practical challenges to
explainable models that are developed till now. Another challenge to clarity may be
due to the accuracy of the model performance, as descriptive performance systems
tend to have lower performance compared to non-descriptive models or black-box
models. Explainable AI is an important part of the future of AI because descriptive
artificial intelligence models explain thinking through their decisions. This provides a
growing level of understanding between humans and machines, which can help build
trust in AI systems. Explainable AI improves the transparency, integrity, integrity, and
accountability of AI systems. Explainable AI Systems can be useful when trying to
understand the thinking behind a particular prediction or decision made by machine
learning models [30–32].
Without explaining the inner workings of an AI model and without knowing how
decisions are made, there is a risk factor to relying on the developed algorithms if
the user is unaware of what factors are taken into consideration and how the inputs
are processed. XAI gives the necessary clarity and helps to build transparency to
enable greater confidence in AI-based solutions. Thus, XAI is recognized as a key
feature for building artificial models and making them capable of making trustworthy
decision-making models and developing them in such a way that takes care of the
fundamentals of AI models as well as the fundamentals of the AI users. Trust is one
of the major factors for users to adopt AI-based models and solutions that incorporate
the decisions they make. However, there are some problems in the development of
explainable methods or models. It is an important factor that is achieving the easiness
of algorithm interpretability. Transparency influences the high-performance nature
of complex but opaque models [32, 33].

9 Conclusion and Future Scope

In this chapter, the current and future opportunities of XAI are surveyed. It is
concluded that XAI offers an opportunity where outcomes of AI algorithms can
be easily understood with the aid of people. With the development of complex algo-
rithms, the need for the development of explainable AI algorithms is also increasing
day by day. The need for explainability, to make a model trustworthy and transparent,
is increasing. But the questions that arise are whether the development of this tech-
nology will affect the job roles of the people? will the employment decrease? will
the problem of complexity of the algorithms be reduced, if yes then to what extent?
XAI can be used in various AI fields such as neuroscience, healthcare, BFSI,
automobiles, judiciary, etc. However, it has various challenges like the requirement
of high-dimensional and large datasets but the deep learning libraries can allow
XAI libraries to develop algorithms for real-world applications which can be easily
70 M. Kumari et al.

understood. Thus, in the future XAI will play an attractive role to build trustworthy
AI models.

References

1. D. Gunning, M. Stefik, J. Choi, T. Miller, S. Stumpf, G.Z. Yang, XAI—Explainable artificial

intelligence. Sci. Robotics 4(37), eaay7120 (2019)
2. A. Páez, The pragmatic turn in explainable artificial intelligence (XAI). Mind. Mach. 29(3),
441–459 (2019)
3. M. Langer, D. Oster, T. Speith, H. Hermanns, L. Kästner, E. Schmidt, K. Baum, What do we
want from explainable artificial intelligence (XAI)?–a stakeholder perspective on XAI and a
conceptual model guiding interdisciplinary XAI research. Artif. Intell. 296, 103473 (2021)
4. A.B. Arrieta, N. Díaz-Rodríguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, F. Herrera,
Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges
toward responsible AI. Inf. Fusion 58, 82–115 (2020)
5. J.M. Fellous, G. Sapiro, A. Rossi, H. Mayberg, M. Ferrante, Explainable artificial intelligence
for neuroscience: behavioral neurostimulation. Front. Neurosci. 13, 1346 (2019)
6. G. Yang, Q. Ye, J. Xia, Unbox the black-box for the medical explainable AI via multi-modal
and multi-centre data fusion: a mini-review, two showcases and beyond. Inf. Fusion 77, 29–52
(2022)
7. J. van der Waa, E. Nieuwburg, A. Cremers, M. Neerincx, Evaluating XAI: a comparison of
rule-based and example-based explanations. Artif. Intell. 291, 103404 (2021)
8. A.M. Antoniadi, Y. Du, Y. Guendouz, L. Wei, C. Mazo, B.A. Becker, C. Mooney, Current
challenges and future opportunities for XAI in machine learning-based clinical decision support
systems: a systematic review. Appl. Sci. 11(11), 5088 (2021)
9. Wolf, C.T. Explainability scenarios: towards scenario-based XAI design. in Proceedings of the
24th International Conference on Intelligent User Interfaces, (March, 2019), pp. 252–257
10. H. Hagras, Toward human-understandable, explainable AI. Computer 51(9), 28–36 (2018)
11. C. Panigutti, A. Perotti, D. Pedreschi, Doctor XAI: an ontology-based approach to black-box
sequential data classification explanations. in: Proceedings of the 2020 conference on fairness,
accountability, and transparency, pp. 629–639 (January, 2020)
12. T. Rojat, R. Puget, D. Filliat, J. Del Ser, R. Gelin, N. Díaz-Rodríguez, Explainable artificial
intelligence (xai) on timeseries data: a survey. arXiv preprint arXiv:2104.00950 (2021)
13. T.A. Schoonderwoerd, W. Jorritsma, M.A. Neerincx, K. Van Den Bosch, Human-centered XAI:
developing design patterns for explanations of clinical decision support systems. Int. J. Hum
Comput Stud. 154, 102684 (2021)
14. Y.S. Lin, W.C. Lee, Z.B. Celik, What do you see? Evaluation of explainable artificial intel-
ligence (XAI) interpretability through neural backdoors. arXiv preprint arXiv:2009.10639.
(2020)
15. F. Emmert-Streib, O. Yli-Harja, M. Dehmer, Explainable artificial intelligence and machine
learning: a reality rooted perspective. Wiley Interdisc Rev Data Min Knowl Discovery 10(6),
e1368 (2020)
16. E. Tjoa, C. Guan, A survey on explainable artificial intelligence (xai): toward medical xai. IEEE
Trans. Neural Netw. Learn. Syst. 32(11), 4793–4813 (2020), A. Adadi, M. Berrada, Peeking
inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE Access 6,
52138–52160 (2018)
17. Z. Papanastasopoulos, R.K. Samala, H.P. Chan, L. Hadjiiski, C. Paramagul, M.A. Helvie, C.H.
Neal, Explainable AI for medical imaging: deep-learning CNN ensemble for classification of
estrogen receptor status from breast MRI. in Medical imaging 2020: Computer-aided diagnosis,
vol. 11314, pp. 228–235 (SPIE, March, 2020)
4 Explainable AI (XAI): A Survey of Current and Future … 71

18. M. Saarela, S. Jauhiainen, Comparison of feature importance measures as explanations for

classification models. SN Appl. Sci. 3(2), 1–12 (2021)
19. J.J. Ferreira, M.S. Monteiro, What are people doing about XAI user experience? A survey
on AI explainability research and practice. in International Conference on Human-Computer
Interaction (Springer, Cham, July, 2020), pp. 56–73
20. C. Mencar, J.M. Alonso, Paving the way to explainable artificial intelligence with fuzzy
modeling. in International Workshop on Fuzzy Logic and Applications (Springer, Cham,
September, 2018), pp. 215–227
21. E. Dağlarli, Explainable artificial intelligence (xAI) approaches and deep meta-learning models.
Adv. Appl. Deep Learn. 79 (2020)
22. E. Puiutta, E. Veith, Explainable reinforcement learning: a survey. in International cross-
domain conference for machine learning and knowledge extraction (Springer, Cham, August,
2020), pp. 77–95
23. U. Ehsan, Q.V. Liao, M. Muller, M.O. Riedl, J.D. Weisz, Expanding explainability: towards
social transparency in AI systems. in Proceedings of the 2021 CHI Conference on Human
Factors in Computing Systems (May, 2021), pp. 1–19
24. F. Hussain, R. Hussain, E. Hossain, Explainable artificial intelligence (XAI): an engineering
perspective. arXiv preprint arXiv:2101.03613 (2021)
25. W. Samek, G. Montavon, S. Lapuschkin, C.J. Anders, K.R. Müller, Explaining deep neural
networks and beyond: a review of methods and applications. Proc. IEEE 109(3), 247–278
(2021)
26. https://www.datarobot.com/wiki/explainable-ai/
27. https://www.analyticsvidhya.com/blog/2021/01/explain-how-your-model-works-using-explai
nable-ai/
28. https://www.ibm.com/watson/explainable-ai
29. https://www.ericsson.com/en/reports-and-papers/white-papers/explainable-ai--how-humans-
can-trust-ai
30. M. El-Assady et al., Towards XAI: structuring the processes of explanations, in Proceedings
of the ACM Workshop on Human-Centered Machine Learning, vol 4 (Glasgow, UK, 2019)
31. C. Conati, O. Barral, V. Putnam, L. Rieger, Toward personalized XAI: a case study in intelligent
tutoring systems. Artif. Intell. 298, 103503 (2021)
32. U. Pawar, D. O’Shea, S. Rea, R. O’Reilly, Incorporating explainable artificial intelligence
(xai) to aid the understanding of machine learning in the healthcare domain. in AICS (2020,
December), pp. 169–180
33. J. Druce, M. Harradon, J. Tittle, Explainable artificial intelligence (XAI) for increasing user
trust in deep reinforcement learning driven autonomous systems. arXiv preprint arXiv:2106.
03775 (2021)
Chapter 5
Recent Challenges on Edge AI with Its
Application: A Brief Introduction

Kapil Joshi, Harishchander Anandaram, Manisha Khanduja,

Rajesh Kumar, Vikrant Saini, and Yasmin Makki Mohialden

Abstract Edge AI refers to the deployment of AI software on hardware throughout

the real world. Recently industry 4.0 adopted the cases on plant automation, factory
robot, edge computing and usage of application in AI domain. With higher bandwidth
and lower latency, edge computing promises to decentralize cloud applications. This
chapter provides a thorough analysis of edge computing-related AI techniques and
tools, sometimes known as edge AI. First section gives the introduction on edge
AI and 5G technology has been discussed in second section. Rest of section elab-
orated the importance and usage of edge AI with complete roadmap. In our vision,
Edge AI will improve network and radio access, provide adaption for data-driven
applications, and enable the development; distributed AI/ML pipeline deployment,
optimization, and implementation predetermined experience quality, faith, safety,
and privacy criteria. The Edge AI community, which spans several ICT, engineering,
and perform computer science subfields, researches unique machine learning tech-
niques for the edge computing environment. The objective is to offer a hypothetical
roadmap that may unite important players and enablers in order to the development
of edge AI. The outcomes of this chapter is to explain the usage of edge AI in industry
5.0 and will gives the better idea to development of new applications.

Keywords Edge AI · XAI · Application of EDGE AI · Roadmap

K. Joshi (B) · V. Saini

Uttaranchal Institute of Technology, Uttaranchal University, Dehradun, India
e-mail: [email protected]
H. Anandaram
Amrita Vishwa Vidyapeetham, Coimbatore, Tamil Nadu, India
M. Khanduja
USCS, Uttaranchal University, Dehradun, India
R. Kumar
Meerut Institute of Technology, Meerut, India
Y. M. Mohialden
Mustansiriyah University, Baghdad, Iraq

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 73

A. E. Hassanien et al. (eds.), Explainable Edge AI: A Futuristic Computing Perspective,
Studies in Computational Intelligence 1072,
https://doi.org/10.1007/978-3-031-18292-1_5
74 K. Joshi et al.

1 Introduction

Edge computing has the potential to decentralize using cloud applications simulta-
neously increasing bandwidth and lowering latency. These benefits are realized by
transferring application-specific calculations across the cloud, data-generating equip-
ment, and elements of the wireless and landline network infrastructure network edges.
However, existing Machine learning and artificial intelligence (AI) (ML) method-
ologies presuppose that calculations [1] are carried out in a robust computing envi-
ronment, also including datacenters with plenty of processing and storage for data
capabilities]. The recent Dagstuhl Seminar—“213,421” gathered community input
from a varied variety of professionals to provide light on the rapidly expanding
subject of Edge AI, which combines edge computing and AI/ML. This article is the
product of their efforts, and it examines demands for implementing AI techniques as
in context of edge computing are both technological and societal.
The phrase ‘Edge,’ as modern development and research progresses, remains a
nebulous term. Across communities and researchers, a shared understanding of what
the edge is, where it lives, and who supplies it is lacking. Similarity (latency and
topology), improved capacity of a network (effectively feasible data transfer rate),
lower processing strength, smaller size, and greater device heterogeneity are some of
its characteristics when compared to the cloud. When compared to end devices (the
last step), it offers more computing and storage resources. It is a fictitious entity that
allows you to outsource computation and storage without having to go to the cloud.
Present-day AI and ML techniques, which necessitate sophisticated computa-
tional structures, are becoming a growing source of friction [2] an improved request
supplied the data centre with adequate processing and storage for data capabilities.
Sending the required raw data to cloud, on the other hand other hand, places a strain
on the network in terms of bandwidth and throughput. Meanwhile, most businesses
are hesitant to share (possibly sensitive) data [3] with commercial cloud providers.
Edge AI, a rapidly emerging domain, addresses this tension. On the other end, appli-
cation of edge AI can be used in industry 5.0 like smart home, smart agriculture,
smart vehicles, smart education, smart critical industries, smart e-commerce, smart
transport and smart healthcare etc.
Edge AI has gradually made its entry into regular service areas that is linked
automobiles, instantaneous gaming, healthcare, and intelligent manufacturing, as
shown in Fig. 1.
Edge environments, from an infrastructure standpoint, create a distinct layer for AI
and provide possibilities for current technology like federated learning or embedded
AI, which aim to reduce RAM use on specific devices, increase maintaining data
privacy on a local device, and reduce communication needs between distributed
entities. These properties form the basis the usage cases depicted in Fig. 1. Our
Contribution is mentioned below:
• Discussed the use case if Edge AI in current era.
• 5G technology in industry 4.0.
• Elaborated the usage of cloud computing in next generation.
5 Recent Challenges on Edge AI with Its Application … 75

Fig. 1 Use case of edge AI in today’s era [4]

• Modification of current in edge AI.

• Designed the road map for edge AI.
This article intends to provide a plan that can bring people together essential
performers and facilitators who promote progress the territory of Edge AI, based also
on group’s contribution of the Dagstuhl Edge Intelligence Seminar. The viewpoints
of future cloud, AI/ML, and 5G beyond are covered in Sects. 2, 3, and 4, respectively.
The anticipated route plan and perspective are presented in Sect. 5.

2 Beyond Perspective with 5G

The transition from networks of the fifth generation (5G) to the sixth-generation (6G)
period shaping networking’s future is established. The advancements on this road
include network connectivity, including its speed, coverage, and durability, as well
as computer quality and latency. To realize the full future networks’ possibilities
with such a complex structure, several technologies must coexist and collaborate
both in terms of hardware and software. The establishment factors are generated
for edge computing and communications, for example or the use of Technologies
with dynamic network orchestration that are self-learning examples of this. Security
assurance techniques, secret communication, computation and learning are all key
viewpoints in holistic system trustworthiness. We go over the obstacles and opportu-
nities [4, 5] in this part. The following literature covers the efficiency of edge AI. V,
Mazzia proposed the structure micro perspective with training dataset. There is also
some literature which focuses on the privacy [4, 6–8] and security [9–11] issues.
76 K. Joshi et al.

2.1 Communication and Computation Process Using Human

Using the widespread use of smart phones or other portable electronic gadgets, impor-
tant compute, interaction, and sensor abilities have becoming useful in addressing
networking difficulties. Acquisition of local data, as used in federated learning,
decreasing reducing communication costs through device-to-device caching and
working together in the execution [5] of computationally complex tasks are just
a few examples. In the future, mobile devices will be able to select whether, the how,
when, and how much of such a task to assign a cloud or edge server. Furthermore,
individuals can assume the role of a computer worker, organize resource pools, and
split jobs according to their preferences in order to maximize their effectiveness and
usefulness.
While people or machines operated by people may contribute considerably Edge
AI, such participation comes with a number of drawbacks. Multi-agent systems can
be used to represent such problems. As a result, numerous mathematical methods
such as game theory and control can be used to handle problems. Furthermore, when
there is uncertainty and a lack of knowledge, ML and AI play a significant role. The
special qualities of people should be considered in all of the preceding procedures.
Humans, in particular, rely concerning heuristics and unjustified influences to make
decisions, while taking considerations include societal peer pressure and standards.
It’s also critical to think about reciprocal trust and respect each entity’s welfare while
interacting with self-interested entities. Finally, employing using people as either a
data source such as through GPS or body sensors raises privacy worries about are
inextricably linked to ethical and legal issues.

2.2 Applications in Scale

The future demands of the driving force behind societies produces using examples we
shift from present networks and systems for 5G and beyond. As a result, developing
novel technology to address societal use-cases develop critical. In a limited sense, it
contradicts present-day usage cases like as well as dependability, which are driven
by technology advancements instead of derived from society itself. Current vertical
trends include resource-efficient manufacturing, the production and distribution of
green energy, organic farming, and retail logistics optimization, to name a few.
To meet their diverse demands, heterogeneous entities are anticipated to create and
transfer a vast infrastructure for edge computing. Despite sharing certain common
objectives, such actors frequently have competing interests; for example, more advan-
tage for one attribute may diminish the benefit for the other (here the utility can
correspond to major sustainability, improved environmental factors). The finding
of a Pareto-optimal and [12] the most reliable answer to this issue is extremely
difficult due to the fact that different utility measurements are frequently at odds.
In fact, the problem is exacerbated since there are multiple decision-makers rather
5 Recent Challenges on Edge AI with Its Application … 77

than a single centralized figure. This is due to asymmetry of knowledge and varied
types/preferences, as well as self-interest. Artificial intelligence (AI) can help solve
this challenge by allowing using distributed systems communicate, acquire knowl-
edge, make choices, effectively turning Smart systems are interconnected chunks of
edge intelligence.

2.3 Edge Intelligence System and Latest Technologies

Beyond 5G Networks

Beyond Next-generation networks or 5G cover a variety of technologies, all of which

rely heavily on dependable cloud infrastructure and edge intelligence for successful
implementation. Campus networks, cooperative sensing and communication, the
Network for Open Radio Access (RAN), and intelligent reflecting surfaces (IRS)
are only a few examples. Joint communication and radar sensing technologies, for
example, can be implemented in 2 networking architectures: cloud RAN and small
cell networks. While the latter is compatible with cloud infrastructure, edge intelli-
gence is extremely beneficial to both implementations. Because combined Sensing
and communication are necessary fast processing of signals, and exact pattern
identification, which are both computationally demanding tasks, this is the case.
Campus networks are yet another case, which span a geographically confined
territory in order to meet the communication needs unique to that location. As a
result of the demand for safe, dependable, and ongoing, ultra-low latency industrial
communications, a manufacturing organization, for example, can integrate a campus
network. Building sites, medical facilities, and agricultural fields other similar appli-
cations can all benefit from campus networks? Campus networks are being driven by
5G technology, as well as AI and edge computing capabilities. Even in the absence of
precise knowledge, they make communication stable and secure, quick and failure-
free calculation, as well as reliable and efficient performance. Edge intelligence
is also required for the consistent effectiveness of various other technologies, like
IRS. IRS technique is based on finding the best [6] Beam arrangement that could
occur multiple times. Consequently, the edge can handle the requisite low-latency
computing.

2.4 Technology Meets Business

The debate over the future of 5G has recently been dominated by hypothetical use
situations, as well as overarching objective of developing real-time integration of
communication, AI, and edge computing services that adapt to applications’ changing
78 K. Joshi et al.

needs. These predicted solutions, which range from ordinary living smart transporta-
tion and medical benefits, are important, but they must be evaluated in the situation
with technology and new economic models.
There are only a few business cases, but we were able to find three instances
where AI technology is being used for whole new tasks and models are yet defined
for them:
• Interpreting the results of joint sensing and communication capabilities in
future networks. Future higher frequency Communications provides some degree
of “radar-like” environment recognition. This is an example of significant change
Regarding the actions or operations of present networks, and the security issues
involved are substantial.
• Optimal link-level communication details discovery through ML. Theoret-
ically, which is doable. The practicality within the technologies still is debat-
able, albeit with the required education expenses are covered by the optimization
benefits accrued.
• Interoperability and collaborative use of data and a technologies. The majority
of modern systems operate within single organizations However, we pose the
following query: “What are our options promote more interoperable technolo-
gies.” The need to identify as well as discuss pertinent security, privacy, and
ethical issues, as well as the instruments to address them, is shared by diverse
stakeholders the min trust worthy manner.

3 Future Perspective on Cloud Computing

The collaboration between future edge nodes and clouds shapes the future cloud
computing perspective. Latency, bandwidth, data localization, scaling, ease of use,
security, and fault tolerance are the key reasons for moving workloads from the
cloud to the edge. This collaboration encompasses not just technical obstacles, but
also business and stakeholder issues. The possibility conflict between the edges of the
cloud must work closely together to deliver the greatest assistance available clients, is
an important non-technical obstacle. We’ll go over the technology problems in detail
from a cloud perspective in the following sections. These issues, in our perspective,
are: Resource management, energy constraints and efficiency, security, trust, and
privacy and sporadic connectivity are the other four topics.

3.1 Resource Management

Even though the number edge servers could supply a significant number of resources,
required proximity [7] number edge servers severely limit the quantity of resources
available when in contrast to a cloud setting. Owing to hardware restrictions and task
execution lag, the quantity of different ML jobs that may be done at the same time is
5 Recent Challenges on Edge AI with Its Application … 79

Fig. 2 Various issues between devices [12]

limited. While execution and instantaneous loading of the associated models could
improve the amount of ML tasks that the edge can perform, this might lengthen the
latency of jobs it needs to be loaded. Thus, for tasks that require low latency, proactive
resource reservation may be required. Because resource reservations can substantially
reduce how many ML tasks there are supported, the need for resource reservations
should also be decided based on the possibility of failing and the repercussions of
such error. The administration of these resources in crucial situations can be difficult,
especially when resources are scarce. The issues are discussed in Fig. 2.
Importantly, edge AI operations are frequently embedded in broader settings,
such as a control workflow. As a result, various services provided via cutting-edge
resources must engage with one another. Not accounting for the notion data items
could need must be passed forward to services result in these services not being
ready to receive either data or not having the computational capacity to handle data
items. As a result, there may be a stream of data items. This is particularly true when
stream processing is used, such as when fresh data items must be processed and then
transmitted to various destination services based on the categorization. As a result,
it could be fascinating to watch and analyze how jobs are completed based [8] based
on overall priority level in light of the various loads at the edge.

3.2 Energy and Operational Constraints

Edge nodes are projected to be less cost and energy efficient compared to cloud data
centers in general. Economies of scale, therefore may act in opposition to edge data
80 K. Joshi et al.

centers by enabling greater cooling, for example. Renewable energy can be deployed
at both cloud data centers and edge nodes, however Unknown where these units are
produce greater energy efficiency. Large data centers, a single hand, offer greater
chances Using renewable energy, enhancing their persistence and environmental
but compatibility they also take in substantially more energy than edge nodes. The
opposite is true for edge nodes may be able to they generate their own power more
easily compared to cloud data centers because of modest geographic spread and
scale, despite their restricted Amount of renewable energy units available.
Furthermore, data must be transported over The Internet uses energy; hence it
increases the likelihood that edge nodes may outperform cloud data centers that
use less energy: Edge nodes can process data, for example, by detecting anoma-
lous instances and transmitting only pertinent information, by dynamically adapting
monitoring rates. Therefore, the amount of processing that can be done at the edge
determines how much energy can be saved. Another option is to shift part of the
training process to the edge; using techniques like federated education (see Sect. 4.3).
Only learnt parameters (such as gradients) must be communicated in this situation,
compared to raw data, which many various magnitudes bigger.
Another possible the benefit of edge nodes over general cloud server racks is the
more specifically constructed hardware, which is increasing common during market
(e.g., hardware accelerators) and makes better use of limited processing and graphical
capacities.

3.3 Security and Privacy Issues

Edge AI, as seen in Fig. 3, provides prospects for increased trust, security, and
privacy while also posing new obstacles. Because some threats may be discovered
early and countermeasures taken, security may be enhanced. However, the number
of potential targets and attack channels is substantially greater, resulting in a bigger
assault potential.

Fig. 3 Security and privacy in edge AI

5 Recent Challenges on Edge AI with Its Application … 81

Because of its dispersed structure and potential liability issues, Edge AI may
have a problem with trust in the system. Open implementation and specifications,
on the other hand, may help to boost edge server confidence. It may also be required
to distinguish between devices within the user’s control and resources offered by
providers (e.g., Azure, AWS, GCP). The implementation of a reputation system that
analyses and controls trust, as seen in other fields, is one viable option for building
trust. However, it will take time for new players to establish a reputation.
Another feature the data of Edge AI is what needed to complete the task of ML
is retained on servers at the edges. As a result, it’s widely assumed that edge servers
offer more privacy and trust than the cloud. As opposed to popular belief, Edge
cannot ensure privacy; rather, it creates additional obstacles for ensuring privacy.
That is, because edge node trust is more difficult to maintain and administer than
cloud provider trust, data privacy and security are harmed, even while the data’s
location prevents some assaults.

3.4 Intermittent Connectivity

Generally, we discuss utilizing edge servers with weak internet connections, i.e.,
edge servers in areas in which internet connection are bad. Edge intelligence has a
lot of potential in those places as well, and it can be a driving force for digitalization
in non-urban areas. Water and air pollution monitoring, as well as natural hazard
prediction (wildfires, floods, volcanic eruptions, and so on) are examples of such
systems. For example, in water pollution monitoring, edge resources could be used
to identify chemicals in the water.
Scenarios involving environmental monitoring where, streaming data must be
analyzed in almost real-time an absence of a solid network access in those places
could be the key driving reason. Monitoring systems are unable to continuously
transmit raw data processing on the cloud due to intermittent connectivity and energy
constraints, as opposed to less data-heavy control signals such as the result of machine
learning still possible to communicate algorithms despite disruptions and restricted
bandwidth. Furthermore, the participants who rely on the results Examples of ML
algorithms frequently located near the sources of data. By avoiding transferring
cloud storage for data, an entire processing stage may avoid, such as automatically
initiating countermeasures. In the prior a good example of monitoring water contam-
ination edge hence, services instruct opening or closing pumps or valves based on
the circumstance.

4 Evolving AI and ML

This paragraph discusses edge nodes from an AI perspective, and difficulties that it
raises. Edge servers, cloud servers, and mobile devices, Fig. 4 depicts the various
82 K. Joshi et al.

Fig. 4 Levels of AI with data connectivity

AI complexity levels and data availability: While mobile devices have the most data
at their disposal, their processing capabilities are often constrained. As a result,
preprocessed information can sent to community edge servers for tasks that mobile
devices can’t handle. The edge server offers extra capabilities and can do more
complex tasks than the mobile devices; however data access [9] is limited owing to
preprocessing. The inability to finished will be forwarded to a cloud server, capable
of execute even most sophisticated models, but only a fraction of the data is received.
Tasks that are unable to be performed will be forwarded a connection to the cloud
server, which execute most sophisticated models, but only is given a portion of the
edge server’s information.
The following are three challenges that have been highlighted and should be
addressed by the community: (1) the accessibility of accelerators for applications
using AI; (2) identifying a trade-off between precision and resource use; and (3)
Using Federated edge server learning is the three issues.

4.1 Accelerators for AI Usage

Although we anticipate that hardware accelerators and AI applications would also

be accessible at the edge, only modest hardware accelerators are anticipated to be
accessible there. In other words, compared to the cloud, capabilities at the edge are
5 Recent Challenges on Edge AI with Its Application … 83

used less elastically, which might lead to let hardware idle. It is anticipated that the
huge hardware accelerators would be put in the cloud due to business considerations.
To cooperate with them restricted resources, one current trend is to break down
smaller the model and AI algorithm parts that could be processed these tiny hardware
accelerators are used. The quality of service and the user experience of apps running
too close to the edge improved by combining the interference of these small hardware
accelerators. In general, huge models that require a lot of resources will need to be
moved to both the ground and the cloud.
Offloading and multi-tier architectures are also possible with machine learning.
Unlike classical workloads, which have intrinsically contradictory criteria for
expanding to higher tiers and nodes while holding the workload’s state consistent
across nodes, ML, particularly neural network training, has a distinct state distri-
bution synchronization models. Additionally, in contrast to conventional distributed
systems, overall consistency criteria easily possible reduced, resulting in a trade-off
of network bandwidth and replica count for relatively minor accuracy losses. As a
result, an essential question is what precision and delay constraints using the appli-
cation, as well as how to check [10] the model’s quality during its whole life-cycle
on the edge.

4.2 Trade off Between Accuracy and Resource Demand

Similar to before stated, edge servers face greater resource limitations than clouds
of servers. Due to their complexity, certain models may only be able to operate in
the cloud.
The common goal today’s AI research frequently aims to attain the greatest
feasible precision or to maximize the incentive mechanism. However, in the case
of Edge AI, the quality of the outputs is [11] not the sole criterion for success; other
factors such as energy usage and memory must also be addressed. As a result, models
must be flexible in order to change for the current state of memory and resources
at hand. There must be a way to adjust the dimensions of individual ML models
for this purpose. Miniaturization techniques such as quantization and approximation
can be used to achieve this. In general, these strategies provide a [13] comparison
of performance measures and model correctness. TinyML and Tensor-Lite are two
examples of Edge AI minimization. The evaluation of adjusting the ML model in
light of present accessible resources is an important research topic. Furthermore,
we address the prospect of offloading specific jobs to the cloud: if the precision of
the scaled-down model implemented near the edges is inadequate, offloading cloud-
based task may be necessary. The clouds can then offer very accurate results and
deliver them the edge once more [14].
84 K. Joshi et al.

4.3 Federated Learning

Then again to model downsizing, federated learning allows models to be trained on

several devices at the same time. This is particularly crucial for Edge AI, because the
dispersed the composition of edge servers makes federated learning more important.
The data for federated learning is maintained locally [15] and not made avail-
able to the cloud. When a complicated running on such an edge server, however,
the resources provided may not be adequate to execute or instruct the model. The
introduction of a sophisticated model exclusively mostly in cloud, which would be
trained collaboratively using fewer complicated models used in edge deployment, is
one research topic.
Federated education is also enables time-critical label [16] classification directly
while other labels are near the edge must be categorized in the cloud. The model gains
additional layer of flexibility as a result of this. Even if the model is reduced through
downsizing by combining we anticipate that the cloud model will classify all non-
time-critical labels, and that the time-critical labels will be categorized accurately to
remain high. As a result, one ongoing research challenge is how to [17] miniaturize a
model while maintaining high a subset of labels accurately, as well as how to decide
when a classification should be performed in the cloud. Furthermore, the informa-
tion gathered by edge servers and created by mobile devices may be considerably
different. Federated instruction can aid in the training of one model that uses a variety
[18] of sensor inputs, improving the precision of each model regardless arrangement
of sensors. The finding of fraudulent material that could jeopardize the process of
training is likewise a difficult task.
Another difficulty is the federated learning systems’ elastic adaptation to regularly
updated client partnerships while taking the fulfillment of collaboration criteria into
account. (e.g., the minimum needed of training partners) and keep improving the
model’s accuracy.

5 Outlook and Roadmap in Edge AI

Overview of the Roadmap: Fig. 5 depicts the Edge AI roadmap as it is currently

envisioned. Edge AI will advance in terms of data governance, service models, hard-
ware, and software in addition to the three key pillars of evolving AI/ML, future
cloud, and 5G beyond. The improvement in various Technology advancements and
expanding commercial interests will propel Hardware, service paradigms, software,
and data governance advancements for Edge AI.
The roadmap includes five main phases, starting with the current situation, which
is Scalable framework, reliable co-design, equal accessibility, deployment that is
sustainable, energy-efficient, and characterized by pervasive intelligent infrastructure
are led by companies that offer cellular, cloud, and AI/ML services. The order repre-
sented in the roadmap could be swapped or mixed as changes occur. Nonetheless,
5 Recent Challenges on Edge AI with Its Application … 85

Fig. 5 Road map of edge AI [19]

the combining their impacts technological enabling factors and non-tech require-
ments such user purchasing power, socioeconomic transformation behavior, as well
as commercial interests are reflected in this Edge AI roadmap.

5.1 Open Research Challenges

Despite its potential and potential, when deployed on a large scale, Edge AI may
encounter significant difficulties, such as energy optimization, trust worthiness,
ethical, security, and privacy concerns.
The power consumption of Edge AI needs to be improved as a crucial compo-
nent of sustainability. On Edge AI embedding infrastructures (such as roadside units
and micro base stations) to continuously enable sophisticated autonomous driving
and services for Extended Reality (XR) in the upcoming years. Through the pipe
line Edge AI has the option to exchange accuracy with efficiency of medical image
data, [20] data capture, transport, calculation, and storage reduced less time and less
power is used. For instance, it is possible to transport and selectively process noisy
signals from many sensors in order to conserve energy. Instead of being perfect and
totally accurate, a group of applications would indeed be pleased with “acceptable”
accuracy correct results. The optimization design can further increase energy effi-
ciency by adding this different dimension of accuracy. Concerning trust worthiness,
Edge AI gains from being so close to the end devices. But because of the scattered
deployment with profound insights in to personal circumstances, perceived safety,
86 K. Joshi et al.

and trust worthiness for Edge AI services have been causing stakeholders to express
their worries (e.g., end users, public sectors, ISP).
To succeed trust worthy Edge AI requires essential building components, such as
systems for verification and validation that guarantee transparency and explain ability,
particularly while training and deployment of Edge AI in distributed, unregulated
settings. The trust worthiness of Edge AI is a first step toward establishing appropriate
regulatory and governance frame work, which Edge AI’s potential can be built.

5.2 Safety and Privacy/Ethical Issues

When we talk about the security Regarding Edge AI’s privacy, there are two things
to think about, i.e.,
(1) Use of edge AI for privacy and security;
(2) Due to the deployment of Edge AI, there are data security and privacy risks, as
shown in Fig. 3.
Edge AI has the potential to be a key tool for ensuring network security and privacy.
Edge AI uses edge computing to process AI rather than using the cloud processes
the local info that is closer to that same user. That eliminates the need of transferring
eliminates data transfer between both the user and the cloud. The possibility potential
is fixed during the data transfer phase of the backhaul. Furthermore, local storage
and processing of data [21] can enhance privacy. Processing’s primary objective is
to move the interface of the AI workflow to the gadget while keeping data limited
to the device. The potential for a decentralized AI algorithm, which would remove
one single point of failure as in cloud AI system, is another security advantage of
edge AI. Edge AI might in some ways be is a son due to increased intelligence, to
reduce the impact of the attack on the immediate environment but also mitigate it
there already. There are growing concerns about privacy and safety a rising Edge AI
is used in this. Decentralization of Edge AI increases that number of access points
that AI system hackers and exposes previously undiscovered attack vectors. Edge AI
devices could lack the very same level of protection as the cloud, making them easy
entry points for an assault the AI system. Additionally, edge devices are reachable
physically, embedded Impact of AI on edge devices on capturing edge devices ever
than a non-AI edge device. For example, seizing control of Edge Ai technologies
might threaten or seize control of nearly all the local work services provided by a
certain place.

5.3 Conclusion and Future Scope

Edge AI’s promises are accompanied by brand-new difficulties and uncertainties.

This article is ours Endeavour Important players in the three key areas of 5G-beyond,
5 Recent Challenges on Edge AI with Its Application … 87

future cloud, and AI/ML are needed to capture the most recent technology advance-
ments for the envisioned roadmap. In terms of future advancement, AI applications
may be more elaborated for better environment and in this section, various issues has
been discussed on industry 4.0 and industry 5.0 with the usage of 5G technology in
future so here we expect the perspectives the ideas presented in this paper may offer
the community an alternative perspective and, in the long term, help to generate the
global adoption of Edge AI in future global world.

References

1. Y.L. Lee, P.K. Tsung, M. Wu, Techology trend of edge AI. in 2018 International Symposium
on VLSI Design, Automation and Test (VLSI-DAT) (IEEE, 2018, April), pp. 1–2
2. S. Greengard, AI on edge. Commun. ACM 63(9), 18–20 (2020)
3. X. Wang, Y. Han, C. Wang, Q. Zhao, X. Chen, M. Chen, In-edge ai: Intelligentizing mobile edge
computing, caching and communication by federated learning. IEEE Network 33(5), 156–165
(2019)
4. E. Li, L. Zeng, Z. Zhou, X. Chen, Edge AI: on-demand accelerating deep neural network
inference via edge computing. IEEE Trans. Wireless Commun. 19(1), 447–457 (2019)
5. T. Rausch, W. Hummer, V. Muthusamy, A. Rashed, S. Dustdar, Towards a serverless platform
for edge {AI}. in 2nd USENIX Workshop on Hot Topics in Edge Computing (HotEdge 19,
2019)
6. A.Y. Ding, E. Peltonen, T. Meuser, A. Aral, C. Becker, S. Dustdar, L. Wolf, Roadmap for edge
AI: a Dagstuhl perspective. ACM SIGCOMM Comput. Commun. Rev. 52(1), 28–33 (2022)
7. S. Yang, Q. Liu, X. Wan, J. Jiang, L. Ai, Edge-rich MoS2 nanosheets anchored on layered Ti3C2
MXene for highly efficient and rapid catalytic reduction of 4-nitrophenol and methylene blue.
J. Alloy. Compd. 891, 161900 (2022)
8. G.K. Agarwal, M. Magnusson, A. Johanson, Edge AI driven technology advancements paving
way towards new capabilities. Int. J. Innov. Technol. Manag. 18(01), 2040005 (2021)
9. M.K. Mudunuru, X. Chen, S. Karra, G. Hammond, P. Jiang, K.C. Solander, T.D. Scheibe,
EdgeAI: How to use AI to collect reliable and relevant watershed data (No. AI4ESP-1095).
Artificial Intelligence for Earth System Predictability (AI4ESP) Collaboration (United States,
2021)
10. C. Mwase, Y. Jin, T. Westerlund, H. Tenhunen, Z. Zou, Communication-efficient distributed
AI strategies for the IoT edge. Future Gener. Comput. Syst. (2022)
11. O. Vermesan, J. Bacquet, (eds.) (2019). Next generation Internet of Things: Distributed
intelligence at the edge and human machine-to-machine cooperation (River Publishers)
12. V. Mazzia, A. Khaliq, F. Salvetti, M. Chiaberge, Real-time apple detection system using
embedded systems with hardware accelerators: an edge AI application. IEEE Access 8,
9102–9114 (2020)
13. H. Yu, D. Yu, C. Wang, Y. Hu, Y. Li, Edge intelligence-driven digital twin of CNC system:
Architecture and deployment. Rob. Comput. Integr. Manufac. 79, 102418 (2023)
14. E. De Giovanni, F. Forooghifar, G. Surrel, T. Teijeiro, M. Peon, A. Aminifar, D. Atienza Alonso,
Intelligent edge biomedical sensors in the internet of things (IoT) era. in Emerging Computing:
From Devices to Systems (Springer, Singapore, 2023), pp. 407–433
15. Y. Singh, Z.A. Sheikh, On efficient and secure multi-access edge computing for internet of
things. in Proceedings of Third International Conference on Computing, Communications,
and Cyber-Security (Springer, Singapore, 2023), pp. 299–310
16. T.T.H. Kim, L. Lu, Y. Chen, ReRAM-based processing-in-memory (PIM). in Processing-in-
Memory for AI (Springer, Cham, 2023), pp. 93–120
88 K. Joshi et al.

17. A.K. Sharma, P. Singh, P. Vats, D. Jain, Deep learning and machine intelligence for opera-
tional management of strategic planning. in Proceedings of Third International Conference on
Computing, Communications, and Cyber-Security (Springer, Singapore, 2023), pp. 475–485
18. A. Bourbah, B. Meliani, Z. Madini, Y. Zouine, The next-generation 6g: trends, applications,
technologies, challenges, and use cases. in Proceedings of Seventh International Congress on
Information and Communication Technology (Springer, Singapore, 2023), pp. 761–770
19. L. Zanella, Y. Wang, N. Dall’Asen, A. Ancilotto, F. Paissan, E. Ricci, M. Pistore, Responsible AI
at the edge: towards privacy-preserving smart cities. in Ital-IA 2022 Convegno del Laboratorio
nazionale CINI-AIIS (2022, March)
20. M. Kirola, M. Memoria, A. Dumka, K. Joshi, A comprehensive review study on: optimized
data mining, machine learning and deep learning techniques for breast cancer prediction in big
data context. Biomed. Pharmacol. J. 15(1), 13–25 (2022)
21. M. Diwakar, A. Tripathi, K. Joshi, M. Memoria, P. Singh, Latest trends on heart disease
prediction using machine learning and image fusion. Mater. Today: Proc. 37, 3213–3218 (2021)
Chapter 6
Explainable Artificial Intelligence
in Health Care: How XAI Improves User
Trust in High-Risk Decisions

Sheeba Praveen and Kapil Joshi

Abstract Explainable AI (XAI) is a set of methodologies, design concepts, and

procedures that assist developers and organizations in adding a layer of transparency
to AI algorithms so that their predictions can be justified. AI models, their predicted
impact, and any biases may all be described using XAI. Human specialists can grasp
the forecasts generated by this technology and have trust in the results. Medical
AI applications must be transparent in order for doctors to trust them. Explain-
able artificial intelligence (XAI) research has lately gotten a lot of attention. XAI
is critical for medical AI solutions to be accepted and adopted into practice. Health
care workers utilize AI to speed up and enhance a variety of functions, including
decision-making, forecasting, risk management, and even diagnosis, by analyzing
medical pictures for abnormalities and patterns that are undetected to the naked
eye. Many health care practitioners already use AI, but it is frequently difficult to
understand, causing irritation among clinicians and patients, especially when making
high-stakes decisions. That’s why the health-care business requires explainable AI
(XAI). Significant AI recommendations, such as surgical treatments or hospitaliza-
tions, require explanation from providers and patients. XAI delivers interpretable
explanations in natural language or other simple-to-understand formats, allowing
physicians, patients, and other stakeholders to better comprehend the logic behind a
suggestion—and, if required, to dispute its validity.

Keywords Artificial intelligence · Machine learning · Deep learning · Explainable

artificial intelligence (XAI) · Diagnosis · Black box algorithm · Neural network ·
Medical and industrial applications

S. Praveen (B)
Integral University Lucknow, Lucknow, India
e-mail: [email protected]
K. Joshi
Uttaranchal Institute of Technology, Uttaranchal University, Dehradun, India

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 89

A. E. Hassanien et al. (eds.), Explainable Edge AI: A Futuristic Computing Perspective,
Studies in Computational Intelligence 1072,
https://doi.org/10.1007/978-3-031-18292-1_6
90 S. Praveen and K. Joshi

1 Introduction

In a Simple Word Explainable artificial intelligence (XAI) is “to understand how

a ML model has arrived to a particular decision” [1]. An enterprise must have a
thorough grasp of the AI decision-making processes, including model monitoring
and AI responsibility, and not blindly trust them. Explainable AI can help people
comprehend and explain machine learning (ML) and deep learning algorithms and
neural networks. Explaining the predictions is crucial for model credibility; yet, there
is a compromise between explainability and precision. Implementing traditional AI
is full of challenges, let alone its explainable and responsible version. Despite the
obstacles, it will bring relief to your employees, who will be more motivated to act
upon the system’s recommendations when they understand the rationale behind it.
Moreover, XAI will help you comply with your industry’s regulations and ethical
considerations. Explanations for machine decisions and predictions are thus needed
to justify their reliability. This requires greater interpretability, which often means we
need to understand the mechanism underlying the algorithms. One use case where
XAI might be beneficial is loan issuing. If a client’s loan application is denied, the
system will be able to defend its decision and provide a clear explanation [2].
When most people think of artificial intelligence, they see black box algorithms
that take large amounts of data as input, do their “magic,” and then provide unex-
plained answers that users are expected to believe. This kind of system is derived
straight from data, and even its developers are unable to explain its conclusion. Black
box systems, such as neural networks, perform very well on difficult prediction prob-
lems. No one can comprehend how the algorithms arrived at their predictions, despite
the fact that they provide very accurate outcomes. With explainable white box AI, on
the other hand, people can comprehend the reasoning behind the system’s actions,
making it more popular in corporate settings. Black box algorithms are more techni-
cally remarkable than these models. Nonetheless, its openness is a trade-off since it
provides a greater degree of dependability and is preferred in strict regulatory indus-
tries [3]. Regarding explainability, everything boils down to what you really want to
explain. There are two potential outcomes:
1.1. Describe the ancestry of the AI model: how the design was trained, which
data was utilized, the sorts of bias that are conceivable and how they may be
addressed.
1.2. Explaining the whole model is also known as “model interpretability.” There
are two ways to implement this strategy:
A. Proxy modelling: an easier-to-understand model, such as a decision tree, is
utilized to approximate a more complex AI model. Even while this approach
provides a brief summary of what to anticipate, it is still an estimate and actual
outcomes may vary.
B. Design towards interpretability: creating AI models such that their behavior is
clear and straightforward to explain. This method may result in less effective
models since it excludes complex tools from the developer’s toolset (Figs. 1
and 2).
6 Explainable Artificial Intelligence in Health Care … 91

Fig. 1 Explainable AI principles

Fig. 2 Relationship between

AI, ML, DL and XAI Artificial
Intelligence

Machine
Learning

XAI
Deep Learning

2 Explainable AI Principles

The US National Institute of Standards and Technology (NIST) developed four

explainable AI principles [4]. Four comprehensible AI principles were created by
the National Institute of Standards and Technology (NIST) in the US [4].
2.1. The system must be able to justify its results and provide evidence to back them
up (at least). There are several sorts of explanations:
92 S. Praveen and K. Joshi

a. Justifications for the end-user;

b. Justifications for building system trust;
c. Justifications for the end-user;
2.2. The explanation provided must be helpful in assisting users in completing their
responsibilities. The system must provide a variety of explanations that are
tailored to the different user groups if there are a variety of users with different
skill levels.
2.3. Unlike output accuracy, this explanation has to be accurate and understandable.
To guarantee a decent result, the system must function within the boundaries of
its specified knowledge.

3 Why Is Explainable AI Important?

For AI to be recognized in certain sectors, an explanation may be required. This

may be because of rules or because of people. Think about the categories of brain
tumors. No surgeon will feel confident doing a procedure because “the algorithm
says so.” What about the awarding of loans? Customers who had their application
rejected would want to know the why. There are much more tolerant use scenarios,
yes, where a justification is not necessary. For instance, even when using predictive
maintenance programmers, personnel would feel more comfortable understanding
why certain equipment could need preventative maintenance.
Although senior management often recognizes the benefits of AI applications,
they are not without reservations. There is always a “but,” claims Gaurav Deshpande,
vice president of marketing at Tiger Graph: “but if you can’t explain how you got at
the solution, I can’t utilize it. This is because to the possibility of bias in the black
box AI system, which might result in legal action, serious liabilities, and harm to
both the company’s reputation and financial sheet. The best XAI system will be
relatively accurate and able to communicate its findings to end users, executives, and
practitioners. Intelligent software using explainable AI ideas [5]:
• Helps people who use the system. They understand why decisions were made and
can agree with them. For example, a loan officer will feel better about telling a
customer that their loan application was turned down if he knows why.
• Ensures compliance. By checking the explanation given, users can see if the rules
of the algorithm are good and follow the law and ethics.
• It lets the system be improved. It’s easier for designers and developers to identify
and correct problems when they read the explanation.
• Eliminates prejudice. When users look at the explanation, they can see if the
system made a biased decision, override it, and fix the algorithm so that it doesn’t
make the same mistake again.
• Gives employees the power to act on what the system says. For instance, a XAI
might guess that a certain business customer won’t renew their software license.
The first thing a manager might do is give a discount. But what if the reason
6 Explainable Artificial Intelligence in Health Care … 93

Fig. 3 Example of XAI in loan processing

they left was because they got bad service? In its explanation, the system will say
this. Gives people the power to act. XAI lets people who are affected by certain
decisions, like granting a mortgage, challenge and possibly change the outcome
[6].

4 Explainable AI Example

XAI can give a detailed explanation at the model level of why a certain decision was
made. This explanation is in the form of a set of clear rules. Every person over the
age of 40 who has less than $433 in savings per month and who requests for credit
with a repayment term exceeding 38 years will be rejected a loan, according to the
simplified application below. The same is true for candidates under the age of 35
who have a monthly savings of less than $657 (Fig. 3).

5 Explainable Artificial Intelligence in Medical

and Industrial Applications

Explainable artificial intelligence (XAI) is all about “Understanding how a machine

learning model reached its decision.” A corporation must have a comprehensive
understanding of AI decision-making processes, model monitoring, and account-
ability, and must not blindly trust AI. Explainable AI might aid people in under-
standing and explaining machine learning methodologies, deep learning, and neural
networks. Due to the importance of model credibility, there is a trade-off between
explainability and precision. Implementing traditional AI is tough, much alone its
explicable and responsible form. Regardless of the obstacles, it will give relief to
your team, who will be more likely to accept the system’s suggestions if they have
a better understanding of the problem.
94 S. Praveen and K. Joshi

6 XAI in Finances

Finance is another area that has a lot of rules and needs to explain its decisions.
AI-powered solutions must be able to be checked, or they will have a hard time
getting into the market. AI can, among other things, help give out credit scores,
evaluate insurance claims, and optimize investment portfolios. But if the algorithms
give biased results, it could hurt their reputation and even lead to lawsuits. Not too
long ago, Apple got a lot of attention for its Apple Card, which was unfair to women
because it lowered their credit limits. Steve Wozniak, who helped start Apple, said
that this was true. He remembered that he and his wife don’t have separate bank
accounts or assets, but when they both applied for an Apple Card, his limit was
10 times greater than hers. New York’s Department of Financial Services examined
the firm after this incident. With AI that can explain its results, people can avoid
embarrassing situations like this. For example, XAI can help with the process of
giving out loans. If a client’s loan application is rejected, the system will be able to
explain why in detail and defend that decision. This lets people improve their credit
scores and apply again later [7].

7 Explainable AI in the Automotive Industry

Self-driving cars use a lot of data, which needs to be analyzed and made sense of
by AI. In the event of any problems, the system’s judgments must be accessible to
drivers, technicians, authorities, and insurance companies. Also, it’s very important
to know how vehicles will act in an emergency. Paul Appleby, a former CEO of
the data management software firm Kinetica, expressed his worry as follows: “If a
self-driving vehicle finds itself in a situation where an accident is likely, what steps
it should take? Put pedestrians at serious risk to safeguard the safety of the driver?
Avoid people on the street while putting the safety of the passengers at risk?” These
are hard questions to answer, and different people would have different ideas about
how to handle them. But it is important to give the algorithm rules it can follow
in these situations. This will help people decide if they are comfortable riding in a
car that can make certain decisions on its own. Also, the explanation given after an
incident will help developers make the algorithm better in the future [8].

8 Explainable Artificial Intelligence in Manufacturing

AI has many uses in manufacturing, such as predictive maintenance, managing inven-

tory, and improving logistics. This technology may enhance the “tribal knowledge”
of human workers thanks to its analytical skills. But it’s easier to make decisions
when you know why they’re good. Heena Purohit, Senior Offering Manager for
6 Explainable Artificial Intelligence in Health Care … 95

IBM Watson IoT, demonstrates how explainable AI is used in their AI-based mainte-
nance product. In order to fix a piece of equipment, the system gives human workers
a number of possibilities. The confidence interval for each choice is shown as a
percentage. Therefore, while making a decision, the user may still draw on their
“tribal wisdom” and experience. As well, each recommendation can show the output
of the knowledge graph along with the information used during the training phase
[9].

9 Achieving XAI in Health Care

AI can be used in many ways in healthcare. Numerous AI-powered medical solutions

may free up physicians’ time from tedious activities so they can concentrate on
providing direct treatment to patients. Additionally, algorithms may be taught to
recognize minute features that the human eye misses, making them effective in
diagnosing a variety of medical conditions [10]. However, clinicians are reluctant
to employ this technology and follow its suggestions when they cannot explain the
results. Duke University Hospital is one place that shows this. A group of researchers
set up a machine learning programme called Sepsis Watch on a patient’s device. This
application would transmit a warning if the patient was at danger of getting sepsis.
The researchers found that doctors didn’t trust the algorithm and didn’t follow its
advice because they didn’t understand it. Patients don’t want to be checked by AI
because they don’t trust it. In a study that was published in the Harvard Business
Review, people were asked to take a free stress test. 40 percent of the people who
signed up for the test did so because they knew it would be done by a real doctor.
When an algorithm did the diagnosis, only 26% of people signed up. When it comes
to figuring out what’s wrong and how to treat it, the choices you make can change
your whole life. It’s not surprising that doctors want to know what’s going on. This
is now possible, thanks to AI that can be explained [11]. For instance, Keith Collins,
the chief information officer (CIO) of SAS, said that his company is already working
on such a technology. He said, “Right now, we’re working on a case in which AI
analytics are being used to help doctors find cancerous lesions more accurately.” The
technology acts as the doctor’s “virtual assistant,” and it explains how, for example,
each variable in an MRI image helps the technology figure out which suspicious
areas are likely to be cancer and which are not.
AI is used by healthcare professionals to expedite and enhance a variety of func-
tions, including risk management, decision-making, and even diagnosis, by scanning
medical pictures to find abnormalities and patterns that are invisible to the human
eye. Many health care professionals now use AI as a necessary tool, but it is some-
times difficult to understand, which frustrates both patients and doctors, particularly
when making important decisions [12] (Fig. 4).
According to article [13] review, they used XAI to perform a study of the most
current developments in surgical and medical diagnostic applications between 2019
and 2021 from PubMed, IEEE Xplore, the Association for Computing Machinery,
96 S. Praveen and K. Joshi

Fig. 4 XAI process

and Google Scholar. In his article, they show how XAI can be used to diagnose
breast cancer through an experiment and show how XAI can be used in medical
applications. In the last part of the paper, a summary is given of the XAI methods
used in medical XAI applications, the problems that researchers have faced, and
where they think research should go in the future. The results of the survey show
that medical XAI is a promising area of research, and the goal of this study is to
give medical experts and AI scientists a place to start when making medical XAI
applications [14].

10 Why XAI is Important in Health Care

The majority of artificial intelligence (AI)-based diagnostic tools for healthcare work
as “black boxes,” which means that the findings do not provide an explanation of
why the computer believes the patient has a certain illness or ailment. Although
AI technologies are very strong, since physicians and regulators cannot indepen-
dently check their outcomes, adoption of these algorithms in healthcare has been
gradual. However, a brand-new kind of algorithm known as “explainable AI” (XAI)
is simple for people to comprehend. Therefore, it is probable that providers will
really apply the linked diagnoses since all indications point to XAI being quickly
accepted throughout the health care industry [11]. The black box feature of AI is
acceptable—and maybe even desirable—for many industries outside of healthcare
because it enables businesses to protect their valuable trade secrets. For instance,
deep learning in AI recognizes speech patterns to enable a user’s preferred voice
6 Explainable Artificial Intelligence in Health Care … 97

assistant to start their favorite movie. Deep learning algorithms identify connections
and patterns without ever requiring the operators to be aware of the data elements
that are most crucial to the outcome. For many uses of AI, there is no risk in relying
on it to continue to provide accurate results since the outcomes verify the algorithms.
However, the black box nature of AI makes it difficult for physicians and regulators
to trust it, maybe for good reason, in industries like health care where errors can
have fatal consequences. Doctors are largely taught to recognize outliers, or unusual
situations that don’t respond to conventional therapies. We cannot be certain that an
AI system would recognize such outliers or otherwise accurately diagnose patients,
for example, if it has not been trained correctly with the necessary data and we are
unable to comprehend how it makes its decisions.
The FDA, which now verifies AI algorithms by looking at what kind of data is
fed into the algorithms to make their judgments on the data, finds the black box
component of AI to be troublesome for the same reasons. A doctor stands between
the solution and the patient’s ultimate diagnosis or course of action, which is another
reason why many AI-related developments bypass the FDA [7].
For instance, the FDA continues to mandate that doctors be able to independently
verify the rationale behind the software’s recommendations in its most recent draught
guidance, which was published on September 28. This is done to prevent the software
from drawing increased scrutiny as a medical “device.” Because physicians can verify
the results of the algorithms, software is only loosely controlled. Think of a medical
imaging where clinicians may double-check worrisome lumps that the algorithm has
flagged. The difficulty for doctors with algorithms like deep learning, however, is
that they lack knowledge of the context in which a diagnosis was made.
Because of this, the XAI algorithms being created for use in the health care
industry can justify their findings in a way that people can comprehend. Many of the
XAI algorithms created so far are rather basic, such decision trees, and have very
specific applications. But as they advance, they will probably be the algorithms that
dominate the healthcare industry. Companies in the healthcare industry would be
advised to invest in its growth [11].

11 A Case Study “Life or Death: West Nile Virus”

This case study focused on how XAI assists a physician in saving the life of a
patient with West Nile virus. Case study demonstrates the Key characteristics of XAI
utilizing a key medical case of an early identification of a West Nile virus infection
in a person. Without AI and XAI, the patient’s life may have been jeopardized. There
are four characters in this story: the patient, the West Nile virus, the physician, and
the AI + XAI software. The doctor learns that AI and XAI just saved a life and
comes to trust AI via XAI at the conclusion of the story. On the long road ahead,
this symbolizes one of the first stages of collaboration between people and machines
[15].
98 S. Praveen and K. Joshi

12 XAI—Issues and Challenges

There are still several current concerns and obstacles at the junction of machine
learning and explanation [3].
• Starting with computers as opposed to humans. Should XAI systems person-
alize explanations for specific users? Should they consider the lack of expertise
among users? How can explanations be used to facilitate interactive and human-
in-the-loop learning, including allowing users to engage with explanations to offer
feedback and drive learning?
• Precision versus interpretability an important stream of XAI research on explana-
tion investigates interpretability’s strategies and constraints. Interpretability must
evaluate accuracy and fidelity tradeoffs and achieve a balance between precision,
interpretability, and tractability.
• Simplifying explanations by using abstractions. Patterns at the highest level
provide the foundation for defining large plans in large phases. It has long been
difficult to automate the discovery of abstractions, and today’s cutting-edge XAI
research is focused on understanding how abstractions are discovered and shared
in learning and explanation.
• Comparing the explanation of skills to that of choices. Reflecting on novel circum-
stances is an indication of expertise among highly skilled specialists. It is vital
to assist end users comprehend the competences of AI systems in terms of what
competencies a certain AI system has, how the competencies should be tested,
and if an AI system has blind spots, i.e., whether there are classes of solutions it
cannot discover.

13 Conclusion

This Chapter mainly focuses on the XAI versus AI Introduction. How XAI is a more
effective technology in the medical industry. For the most part, medical diagnostics
that make use of AI are “black boxes,” meaning that the findings don’t provide any
context for why the computer came to this conclusion about the patient. Although AI
technologies are exceptionally strong, their acceptance in health care has been gradual
due to the inability of physicians and regulators to validate their outcomes. However,
humans can readily comprehend XAI. All indications indicate to the quick adoption
of XAI throughout the health care industry, making it probable that providers will
use the accompanying diagnoses.
From the standpoint of human-centered research, study on competences and
knowledge might move XAI beyond the function of describing a specific XAI system
and assisting its users in determining acceptable trust. In the future, XAIs could play
significant societal roles. These tasks may involve learning and explaining, working
with other agents to link information, establishing cross-disciplinary insights and
common ground, cooperating in educating people and other agents, and relying on
6 Explainable Artificial Intelligence in Health Care … 99

previously acquired knowledge to expedite future discovery. From such a societal

standpoint of comprehending and producing information, XAI’s future is just starting.

References

1. D. Gunning, M. Stefik, J. Choi, T. Miller, S. Stumpf, G.Z. Yang, XAI—Explainable artificial

intelligence. Sci. Rob. 4(37), eaay7120 (2019)
2. A.B. Arrieta, N. Díaz-Rodríguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, F. Herrera,
Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges
toward responsible AI. Inf. Fusion 58, 82–115 (2020)
3. A. Das, P. Rad, Opportunities and challenges in explainable artificial intelligence (xai): A
survey. arXiv preprint arXiv:2006.11371 (2020)
4. D. Gunning, Explainable artificial intelligence (xai). Defense Adv. Res. Projects Agency
(DARPA), nd Web 2(2), 1 (2017)
5. J. Gerlings, A. Shollo, I. Constantiou, Reviewing the need for explainable artificial intelligence
(xAI). arXiv preprint arXiv:2012.01007 (2020)
6. R.M. Byrne, Counterfactuals in explainable artificial intelligence (XAI): evidence from human
reasoning. In IJCAI (2019, August), pp. 6276–6282
7. E. Tjoa, C. Guan, A survey on explainable artificial intelligence (xai): toward medical xai.
IEEE Trans. Neural Netw. Learn. Syst. 32(11), 4793–4813 (2020)
8. F. Bodendorf, S. Merbele, J. Franke, Deep learning based cost estimation of circuit boards: a
case study in the automotive industry. Int. J. Prod. Res. 1–22 (2021)
9. J. Liu, H. Tu, F. Xai, X. Yang, W. Zhang, A calculation model of equipment similarity on
manufacturing capability and its application. in 2010 8th World Congress on Intelligent Control
and Automation (IEEE, 2010, July), pp. 1686–1689
10. U. Pawar, D. O’Shea, S. Rea, R. O’Reilly, Explainable AI in healthcare. in 2020 International
Conference on Cyber Situational Awareness, Data Analytics and Assessment (CyberSA) (IEEE,
2020, June), pp. 1–2
11. M. Nazar, M.M. Alam, E. Yafi, M.S. Mazliham, A systematic review of human-computer inter-
action and explainable artificial intelligence in healthcare with artificial intelligence techniques.
IEEE Access (2021)
12. A. Adadi, M. Berrada, Explainable AI for healthcare: from black box to interpretable models.
In Embedded Systems and Artificial Intelligence (Springer, Singapore, 2020), pp. 327–337
13. Y. Zhang, Y. Weng, J. Lund, Applications of explainable artificial intelligence in diagnosis and
surgery. Diagnostics 12(2), 237 (2022)
14. M. Ahmed, S. Zubair, Explainable artificial intelligence in sustainable smart healthcare. in
Explainable Artificial Intelligence for Cyber Security (Springer, Cham, 2022), pp. 265–280
15. The XAI medical diagnosis timeline|Hands-On Explainable AI (XAI) with Python (pack-
tpub.com)
Chapter 7
Role of Explainable Edge AI to Resolve
Real Time Problem

Ambeshwar Kumar, T. M. Rajesh, Manikandan Ramachandran,

and Deepak Gupta

Abstract The growth of information technology (IT) has resulted in physical de-
vices being connected to the internet and having the ability to recognize other devices.
Artificial Intelligence algorithms are processed on edge or the devices of users. Edge
Computing based on the same premise, stores, processes, and manages data directly
at Internet of Things (IoT) endpoints. Edge artificial intelligence uses the device’s
hardware to process data and performs machine learning and deep learning proce-
dures. In the model, you can troubleshoot and improve model performance while
also assisting others in understanding the behavior of your models. To make the area
more real, explainable edge devices come in a wide range of costs and capabilities. A
decade ago, we couldn’t imagine that explainable edge artificial intelligence would
be at today’s level. Now it is a part of industries and even devices for customer service.
The best example of explainable edge AI is virtual assistants such as Alexa, google
assistant. They learn from the user’s world and phrases and can store them directly
on the device. These are just a few examples later, and we have possible applica-
tions in future works on the explainable edge artificial intelligence. Edge Computing
Platform facilitates the development and elastic operation of apps and services. Its
benefits the AI assisting in overcoming the technical obstacles that AI-enabled apps
experience. Combination of edge and AI is buzzwords within the industry to deliver
the performance and reduce the cost compared to state of arts XAI applications.
Moreover edge artificial intelligence are reducing the latency, improving user expe-
rience, and reducing the necessary bandwidth, consequently reducing the costs of
internet services. It surfs this movement since the need for data processing on the
device themselves also represents the increasing use of artificial intelligence. Arti-
ficial intelligence edge processing focused on model which trained them in central
data center using historical datasets. Compression techniques of data that enables

A. Kumar (B) · T. M. Rajesh

Dayananda Sagar University, Bangalore, Karnataka, India
e-mail: [email protected]
M. Ramachandran
SASTRA Deemed University, Thanjavur, Tamilnadu, India
D. Gupta
Maharaja Agrasen Institute of Technology, Delhi, India

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 101
A. E. Hassanien et al. (eds.), Explainable Edge AI: A Futuristic Computing Perspective,
Studies in Computational Intelligence 1072,
https://doi.org/10.1007/978-3-031-18292-1_7
102 A. Kumar et al.

squeezing large artificial intelligence models into small hardware form factors could
push some training to the edge over time.

Keywords Explainable artificial intelligence (XAI) · Edge computing · Deep

learning · Machine learning · Internet of things (IoT) · Neural network · Cognitive
science

1 Introduction

Explainable artificial intelligence (XAI) is a set of processes and methods that allows
human users to comprehend and trust the results and output created by machine
learning algorithms. Explainable AI is used to describe an AI model, its expected
impact and potential biases. It helps characterize model accuracy, fairness, trans-
parency and outcomes in AI-powered decision making. Explainable AI is crucial for
an organization in building trust and confidence when putting AI models into produc-
tion. AI explainability also helps an organization adopt a responsible approach to AI
development.
As AI becomes more advanced, humans are challenged to comprehend and retrace
how the algorithm came to a result. The whole calculation process is turned into
what is commonly referred to as a “black box” that is impossible to interpret. These
black box models are created directly from the data. And, not even the engineers or
data scientists who create the algorithm can understand or explain what exactly is
happening inside them or how the AI algorithm arrived at a specific result. There are
many advantages to understanding how an AI-enabled system has led to a specific
output. Explainability can help developers ensure that the system is working as
expected, it might be necessary to meet regulatory standards, or it might be important
in allowing those affected by a decision to challenge or change that outcome.

1.1 Definition of Explainable Artificial Intelligence

The explainable artificial intelligence (XAI) is a strong tool for addressing ethical
and legal concerns regarding AI systems by answering important how? and Why?
Questions. As a result, AI researchers have recognized XAI as a necessary component
of reliable AI, and explainability has recently gotten a lot of attention. Despite the
increased interest in XAI research and the requirement for explainability across a
wide range of areas, XAI still has a number of drawbacks. This blog post provides
an overview of the current state of XAI, as well as its advantages and disadvantages.
Interpretability aids in ensuring objectivity in decision-making by detecting and
correcting bias in the training dataset. It provides resilience by indicating potential
adversarial disturbances that modify the prediction. It also assure that only relevant
7 Role of Explainable Edge AI … 103

Fig. 1 Brain tumor MR

image and heat map showing
tumor region

variables are used to infer the result, i.e., that the model reasoning is based on genuine
causality.
Explanations can take many different forms depending on the goal and situation.
Human-language and heat-map explanations of model activities are shown in Fig. 1.
The machine learning model presented below is designed for doctors to utilize and
can diagnose brain tumor using MRI. Based on the MRI on the far left, the original
report gives a “ground-truth” report from a doctor. The Generated report includes an
explanation of the model’s diagnosis as well as a heat-map of the tumor locations
that influenced the conclusion. The Generated report offers doctors with an easy-to-
understand and validated explanation of the model’s diagnosis.

1.2 Edge AI Differ from Artificial Intelligence

Explainable AI is about making sense of black box models and AI is about making
sense of transparent AI models. XAI implements efficient techniques and methods
to ensure that every decision made during the process can be traced and explained
whereas in AI it often arrives at a result using the algorithms but the arrangements
of the systems do not fully understand how the algorithms reached to the results.
AI has achieved growing momentum in many fields to deal with the complexity.
Humans do not grasp the complex mechanisms by which AI systems work or how
they make certain decisions because AI-powered systems have progressed to the
point where humans do not understand the complex processes by which AI systems
work or how they make certain decisions. There’s a risk that an AI model won’t be
considered trustworthy or authentic if the core functionality and judgments it makes
aren’t explained. The role of domain knowledge interacting with an XAI system
in a realistic complex decision-making scenario, XAI brings the needed clarity and
openness to AI-based solutions, allowing for increased trust [1]. As a result, XAI is
recognized as a critical feature for the practical deployment of AI models in systems
and, more critically, for ensuring those AI users’ fundamental rights to AI decision-
making are respected. Our ultimate objective is to provide a comprehensive taxonomy
to newcomers in the field of XAI that can be used as a reference tool to encourage
future research advances, as well as to encourage experts and professionals from
other disciplines to embrace the benefits of AI in their respective activity sectors,
without prejudice due to its lack of interpretability.
104 A. Kumar et al.

1.3 Explainable Edge AI Can Assist Internet of Behavior

Effort

According to academics, applying XAI to Internet of Behavior approaches could

enable provide a more trustworthy and understandable foundation for modifying
human behavior. According to a study, the convergence of Internet of Things
devices, artificial intelligence, data analytics, and behavioral science can provide
both consumer and corporate benefits. The explainable technique also aids in over-
coming the ostrich effect, which is a major problem when using internet of Behavior.
“The ostrich effect,” she explains, “is a difficulty for the Internet of Behaviors because
people are essentially terrified”. “And, because it’s such a delicate subject, you may
find opposition when attempting to influence and modify people’s behavior” [2],
you’re dealing with sensitive information as well as their actions. To avoid this, as well
as to manage this resistance and other psychological aspects connected to comfort
and stress, XAI assists us in providing the user with the necessary understanding and
trust for the AI-based system.“
The internet behavior model refers the collection of data through emerging tech-
nologies. It generates useful knowledge about the customer behaviors, desire and
preferences. It is also useful for predicting user behavior. It is shown in Fig. 2. The
use of XAI approaches to interpret black-box generalization proved to be promising,
providing reliable outputs that could be compared to experts’ interpretations of the
same problem and prompting further studies in which it analyzed learning generaliza-
tion [3]. Given that it focuses on human nature, IoB can be a strong instrument. The
tendency to act is caused by the behavioral component of humans, not necessarily
other attributes such as cognition, emotion, personality, or communication, and is a
potent factor when paired with digital networks and technologies. In 2012, IoB was
first discussed. Professor Göte Nyman of the University of Helsinki’s Department
of Psychology is quoted as saying that if human behavior was assigned to devices

Fig. 2 Internet behavior model

7 Role of Explainable Edge AI … 105

with specific addresses, there would be an opportunity to benefit from the knowledge
gained by analyzing the history of patterns in many fields, including business, soci-
etal, health, political, and many others. According to a CISCO report cited by the
researcher, the IoT sensor commercial market is estimated to reach $22.48 billion
by 2023, with a predicted 29.3 billion linked devices accessible [4].

1.4 Organization of the Chapter

The rest of the chapter is organized as follow: Sect. 9.2 represents the concepts of
explainable artificial intelligence with benefits and challenges. Section 9.3 illustrates
the example of explainable artificial intelligence in real world with the future of XAI.
Finally, Sect. 9.4 discussed the future of XAI summarizes the work with conclusion
follow with references.

2 Concepts of XAI

XAI will develop a set of algorithms that will enable human users to com-prehend,
trust, and manage the next generation of artificially intelligent part-ners. This defini-
tion combines two notions (understanding and trust) that must be addressed before
proceeding. It describes a model’s ability to explain its function—how it works—to
a person without having to explain its internal structure or the computational means
by which the model processes data in-side. Trust can be defined as the belief that a
model will behave as planned, a specific issue although it should undoubtedly be a
feature of any explainable model [5], and this does not imply that every trustworthy
model is explainable on its own, nor does it indicate that trustworthiness is a property
of each explainable model a feature that is easily quantifiable.

2.1 Major Principles in XAI

The major principles in XAI deals with the Intelligence to interpret predictions from
machine learning models, resiliency, reliability, bias, and account-ability are some of
the other characteristics. In most cases, these phrases are defined as part of a larger set
of principles or pillars. The four principles of explainable artificial intelligence are
highly affected by the relationship of AI system with human recipient information.
Before proceeding to the principles of the XAI, the key term i.e. the output of an
AI system. [6] For an example grammar checking system, the output is grammatical
error and correction recommendation. For disease detection system, the output is
detected disease and suggested precautions. The XAI objective is to provide the
information with trust, confidence, informative and transparency, Fig. 3 provides the
106 A. Kumar et al.

Fig. 3 Principles in XAI

pictorial representation of principles in XAI.

The four principles are as follow:

• Explanation
• Meaningful
• Explanation accuracy
• Knowledge limits

2.1.1 Explanation

This principle obligates AI system to provide proof, support and logic for each output.
It does not require that the evidence be correct, informative, or understandable in and
of itself; it only implies that a system can provide an explanation. Currently, a corpus
of work is being done to build and validate explainable AI approaches. It imposes
no demands on those explanations in terms of quality. The ideas of Relevance and
Explanation Accuracy establish a framework for assessing explanations.

2.1.2 Meaningful

If the recipient understands the system’s explanations, it satisfies the Meaningful

principle. The user understands the explanation, if it is helpful in completing a task.
This idea does not imply that there is a one-size-fits-all answer. For a system, multiple
explanations may be required for distinct groups of users. Because of the Meaningful
principle, explanations can be adapted to each user group. Developers of a system
versus end-users of a system, Doctor/practitioner versus researcher and so on are
7 Role of Explainable Edge AI … 107

examples of wide groups. The goal of this principle may vary, e.g. what is mean-
ingful to a doctor may be different than what is meaningful for a researcher. So,
the confluence of the AI system’s explanation and a person’s existing knowledge,
experiences, and beliefs influences meaningfulness. The challenge in modeling the
interface between AI and humans is due to all of the factors that determine meaning-
fulness. Both computational and human variables must be taken into account when
developing systems that produce relevant explanations.

2.1.3 Explanation Accuracy

The Explanation and Meaningful principles only need a system to create explanations
that are meaningful to a user group when used together. This principle exposes
accuracy on a system’s explanations. In the same way as the Meaningful principle
allows for distinct explanation accuracy metrics for different groups and individuals,
this principle allows for different explanation accuracy metrics for different groups
and individuals. Some clients will require concise explanations that focus on the
important point(s) but lack the intricacies required to fully explain the algorithm’s
output generation process. However sometimes nuances data may be meaningful to
experts. Overall, if a system can create multiple types of explanations, it may be
deemed more explainable. Because of the various layers of explanation, the metrics
used to assess an explanation’s accuracy may not be universal or absolute.

2.1.4 Knowledge Limits

This principle asserts that systems detect situations in which they were not designed
or approved. To operate in which their responses are not trustworthy. This method
protects answers by identifying and disclosing knowledge boundaries, ensuring that
no judgment is made when it is not necessary. Two ways, through which system can
reach its knowledge limits, initially the question can be outside of the domain of the
system. For an example, system has to classify the wild animals but the user may
input the image of parrot. The system may produce an answer indicating that it was
unable to locate any wild animals in the input image and hence could not provide an
answer; this is a response as well as an explanation. Another ways depending on an
internal trust level, a knowledge limit can be reached, and the confidence of the most
likely solution may be too low, for an e.g. reptiles classification system, the input
image of a lizard is too blurry to identify. In this situation the system may recognize
the image of reptiles, but it is in low quality, explanation is found the image of reptiles
but the quality of an image is too low to identify it.
108 A. Kumar et al.

Fig. 4 Flow process of how XAI works

2.2 Working Scenario of XAI Principles

The description of XAI principles explained in Sect. 9.2.1, this section explains the
working environment of XAI. It examines each aspect and its impact on the final
result. Users can use this analysis to create new situations and see how different input
values affect the output. Users may see the factors that have a positive or negative
impact.
Success in machine learning has led towards artificial intelligence applications,
continuity to deliver better to the end client. Developing an autonomous system to
learn and decide automatically. To maintain high level of accuracy and prediction
explainable model work efficiently. It enables human user to understand, trust and
efficiently manage the artificially intelligent partners. In the Fig. 4 new machine
learning system will be able to justify their actions, identify their advantages and
disadvantages, and communicate how they will act in the future. It is shown in
Fig. 4.

2.3 Benefits of XAI

Number of advantage in XAI which put them in limelight and emphasizes the need
for faster large-scale adaptation around the world, the benefits of XAI can be found
across wide range of sectors and job activities. Some of the benefits of XAI are listed
below:
• Building user trust
• Reducing cost of mistakes
• Satisfying legal requirements
• Code confidence and compliance
7 Role of Explainable Edge AI … 109

2.3.1 Building User Trust

Using the XAI in the system the user can understand how the output is generated with
relevant inputs. When the error occurs and the procedure to reduce the error. It also
provides the risk assessment report for the risky posed by the automated decision
making system, so the user trust the system.

2.3.2 Reducing Cost of Mistakes

In the event of incorrect prediction, decision-sensitive professions such as healthcare

system, finance system, and law administrative system are severely impacted. Over-
sight of the findings decreases the impact of incorrect results while also identifying
the main problem, allowing the underlying model to be improved.

2.3.3 Satisfying Legal Requirements

AI model have shown most significance evidence of bias. It has proven a game-
changer for many users. User is seeing improved to increase productivity and
efficiency hence the legal requirements is satisfy using the XAI.

2.3.4 Code Confidence and Compliance

XAI provide the explanation knowledge with the accuracy tends to increase the
system’s confidence. Critical system such as banking and healthcare system re-quires
high code confidence from user for optimal solution. Increasing regulatory pressure
implies that companies should easily adapt and implement XAI in order to comply
with the government.

2.4 Challenges in XAI

Using black box strategies to make an unexplained conclusion might result in legal,
ethical, and operational issues. Because black-box models are not verifiable or trace-
ability prior to deployment making assurances about their behavior difficult, explain-
ability is an issue that is both scientifically and socially relevant, and it is at the core
of several areas of contemporary machine learning and AI research. The following
are some of XAI’s challenges:
110 A. Kumar et al.

2.4.1 Bias

It’s a difficult challenge to make sure that the AI algorithm doesn’t learn biased or
unbiased world views as a result of gaps in the training data, model, or objective
function.

2.4.2 Safety

It’s tough to tell whether AI is trustworthy or not without looking at how it came
to its conclusions. Because of the generalization in statistical learning theory, which
essentially shows how organizations cover the gaps in unseen knowledge, this is a
challenge.

2.4.3 Fairness

The impression of fairness is contextual and depends on the information provided to

the machine learning algorithms, therefore determining whether the choice made by
AI systems was fair or not is a difficulty for XAI.

2.4.4 Possible Way to Overcome the XAI Challenges Provides

Meaningful Explanations

Model Agnostic Technique

This strategy can be applied to any number of algorithms or learning methods. The
model agnostic approach will aid businesses in approaching XAI’s internal workings
as a mysterious black box.

Model-Specific Technique

This method can be used for a small number of algorithms or a specific collection
of algorithms. The internal workings of XAI are treated as a white box in the model
agnostic approach.

3 Example of XAI in Real World

XAI is a key research field. Our AI systems could become more reliable, legal,
efficient, fair, and robust with the help of explanations of their workings, which could
increase their acceptance and economic value. General opportunities and challenges
7 Role of Explainable Edge AI … 111

in XAI are analyzed potentially pros and cons from their perspective to overcome
the challenges in XAI [7]. Most of the organizations are working on their proprietary
XAI solutions. Real world implementation of XAI is discussed in this section.

3.1 XAI in Defense

Artificial general intelligence (AGI), a relatively new branch of AI research that

focuses on brain growth and, eventually, the development of sentient machines, might
give rise to robots. However, the use of machine learning a computer’s capacity to
learn without explicit programming and neural networks computer systems based
after the human brain and nervous system to improve human decision-making is
much more common. In order to enable human users to comprehend and trust (while
managing) the artificially intelligent agents are being built, a range of explainable
machine learning models must be developed while preserving their predictive accu-
racy [8]. The XAI programed selected 12 areas of focus for its varied areas of interest
after receiving hundreds of ideas. One of them is deep explanation, which is effec-
tively an inversion of deep learning in which the machine examines and then anal-
yses its own explanation using machine learning. It attempts to explain the inner
workings of the neural network’s own “black box” processes. Consider the hours of
drone-recovered video that you want to go through for crucial frames or sequences.
Figure 5 illustrates the framework of robot developed by Indian army with XAI
multi-layered architecture is capable of providing multitude of military applications.
Indian army has already built the snake robot, legged robot and wheeled robot etc.

Fig. 5 Multi agent robotics

framework
112 A. Kumar et al.

3.2 XAI in Autonomous Vehicle

The concept of explainable AI (XAI) is becoming increasingly popular. Users of

AI systems will almost probably seek an explanation and expect it. A machine-
produced explanation of what the AI has done or is doing will be in great demand
given the rapidly expanding number of AI systems [9, 10]. One such research area
is autonomous vehicles (AVs). It will gradually develop autonomous methods of
transportation with the aim of realizing the slogan “mobility for everyone. “There
will be a variety of autonomous vehicles available, including self-driving trucks,
motorcycles, submarines, drones, and planes. To be clear, fully autonomous vehicles
are those in which the AI operates without any help from a human while the vehicle
is being driven. While cars that require a human driver to share the driving effort are
typically classified as Level 2 or Level 3, these driverless vehicles are considered
Level 4 and Level 5. At Level 5, there isn’t a real autonomous vehicle—we don’t
even know if it will be possible or how long it will take to get there. While there
is debate about whether this testing should be permitted, the Level 4 attempts are
progressively attempting to gain traction by participating in very limited and selected
public roadway trials. One thing to mention right away is that the AI used in modern
AI driving systems is not sentient. In other words, the AI is mostly a collection
of computer-based algorithms and programming, and it is definitely not capable of
reasoning in the same way that people can work.
Figure 6 explain the framework of driverless vehicles feasible for all humans.
The XAI will charge of driving with all necessity equipment’s required in driving
a vehicle. The controller actions are limited by visual attention. The XAI features
vehicle is ready to handle the scenario.

Fig. 6 Autonomous vehicle framework

7 Role of Explainable Edge AI … 113

3.3 XAI in Fraudulent Activities

The developing field of explainable AI (or XAI) can give banks more clarity on
their AI governance while assisting them in navigating concerns of transparency and
trust [11]. The goal of XAI is to improve the explainability, usability, and compre-
hension of AI models for human users without compromising performance or fore-
cast accuracy. A strong XAI programmed can benefit firms in a variety of other
ways as well. Depending on the questions being asked and the modeling methods
being employed, explainability tools can reveal various forms of information about
a model. To lead explainability initiatives in conjunction with their AI teams and
business departments, several banks are forming specific task forces. The accuracy
of a company’s total fraud detection is impacted by the insights that XAI provides
to teams by explaining why models are performing or not performing. Teams cannot
be expected to know how to increase fraud detection rates without this important
understanding. The two fundamental elements of XAI—feature importance (which
is technically connected to XAI) and model evaluation and performance—deliver on
these promises. Features look over a period of time specified data, such as payment
type, transaction amount, location, and so forth. Before deploying the model, teams
can test the model’s performance. Prior to relying on a machine learning model for
fraud detection and prevention, organizations should test the model’s performance
to determine its accuracy and false positive rate.

3.4 XAI in Marketing

Artificial intelligence (AI) marketing makes judgments automatically using data

collecting, analysis, and further observations of audience or economic trends that
could have an impact on marketing activities. One of the most important steps
in starting a XAI marketing programmed is choosing the appropriate platform or
platforms. Marketing professionals need to be astute in spotting the holes that the
platform is attempting to fill and wise in choosing solutions based on skills. This
will depend on the objective that marketers are attempting to fulfill. For instance,
products used to increase speed and productivity will need to have different function-
ality than those used to raise XAI consumer satisfaction levels generally [12, 13].
XAI Reduces the Latency and Increases Precision & Relevancy in the Campaign
Process of product. The current method is simply too sluggish, too inaccurate, and
too irrelevant to the customer in a one-to-moment environment where the relevance
of marketing messages and customer experiences are crucial for establishing trust
and keeping hold of existing clients. Through similarity-based XAI, not in a narrow
sense but in a broader sense, it is possible to reduce latency by 50% (for those
processes that bookend the campaign window itself) and increase precision and rele-
vance by 20–100% or more. This technology can be activated across the workflow
in a phased approach for maximum impact. By drawing the most important insights
114 A. Kumar et al.

from their datasets and acting on them in real time, marketers can use XAI to alter
their entire marketing campaign if it is used properly. Customers can receive individ-
ualized messages at the right times throughout the consumer life cycle with the aid of
XAI. Additionally, XAI can assist marketers in identifying at-risk clients and deliv-
ering information to them that will encourage them to re-engage with the business.
When compared to humans, XAI can analyses tactical data more quickly and apply
machine learning to reach quick decisions depending on campaign and customer
context. Team members now have more time to concentrate on strategic projects
that will later guide XAI-enabled campaigns. With AI, marketers can use real-time
data to choose better media instead of waiting until the conclusion of a campaign
to make selections. Consumers react differently to communications across channels;
some may be moved by comedy, others by logic, and others by an emotional appeal.
XAI and machine learning can track which messages customers have interacted with
and build a more thorough user profile. From there, marketing departments can send
users more tailored communications based on their choices.

3.5 Summary

This chapter defines the briefly introduction of explainable artificial intelligence and
explain about the internet behavior efforts with real world examples of XAI. The
major points to recall are as follow:
• Explainable Artificial Intelligence is useful in building trust and confidence when
putting AI models into production.
• AI is differing from XAI in terms of black box decision.
• XAI approaches to interpret black-box testing.
• Behavioral efforts assist through XAI
• XAI deals with the Intelligence to interpret predictions from machine learning
models, resiliency, reliability, bias, and accountability are some of the other
characteristics.
• Users can use this analysis to create new situations and see how different input
values affect the output.
• In defense XAI to improve human decision-making in more effective to deal with
the real time problems.
• Autonomous vehicle is the rapidly expanding on road with effective features and
ease of access.
• XAI helps in fraudulent activities to detect the intruder and block their hacking
activities through blocking the card immediately.
• In marketing sector XAI achieves a tremendous success to deliver appropriate
messages to customer and establishing trust and keeping hold of existing clients.
7 Role of Explainable Edge AI … 115

4 Future of XAI

Since 2000, there have been 14 times as many active AI startups. Recent develop-
ments in deep learning have enabled AI to handle a variety of marketing and sales
choices as well as search engines, virtual assistants, and online translators. The level
of autonomous driving technology is quickly rising to that of fully automated driving.
Despite Elon Musk’s claim that “over a million robo taxis would be on the road for
sure by next year,” Modern GPUs are now effective enough to be utilized for activ-
ities other than visual rendering, such machine learning or crypto currency mining.
Data scientists found that these are repeated parallel jobs even if CPUs are typically
employed to do them. GPUs are therefore commonly employed in AI models to facil-
itate effective learning. A type of autonomous supervised learning is self-supervised
learning (also known as self-supervision). This method, unlike supervised learning,
handles the labeling process on its own and does not rely on humans to categories
the data.

5 Conclusion

The objective of this chapter has been contributed towards understanding the concepts
of explainable artificial intelligence to stores, processes, and manages data directly at
Internet of Things (IoT) endpoints. The internet behavior efforts promising to provide
reliable outputs that could be compared to experts’ interpretation, the explainable
artificial intelligence is growing rapidly in every sector of organization to complete
the task in easiest and user friendly. It will provide the steps through which end user
trust the system. The overall structure of this chapter provides an efficient knowledge
to the user to understand the concept of XAI.

Consent for Publication No any Consent for Publication.

References

1. Dikmen, T.L. Webb, B.P. Chang, Y. Benn, ‘The ostrich problem’: Motivated avoidance or
rejection of information about goal progress. Soc. Pers. Psychol. Compass, 7(11), pp. 794–807
(2013)
2. T.L. Webb, B.P. Chang, Y. Benn, ‘The ostrich problem’: Motivated avoidance or rejection of
information about goal progress. Soc. Pers. Psychol. Compass 7(11), 794–807 (2013)
3. L.A. de Souza Jr, R. Mendel, S. Strasser, A. Ebigbo, A. Probst, H. Messmann, J.P. Papa,
C. Palm, Convolutional neural networks for the evaluation of cancer in Barrett’s esophagus:
Explainable AI to lighten up the black-box. Comput. Biol. Med. 135, p. 104578
4. G. Nyman, J. Peltonen, M. Nelson, J. Karjalainen, M. Laine, T. Nyberg, H. Tuomisaari, On the
behavioral theory of the networked firm. In Big data and smart service systems, (pp. 179–192).
Academic press (2017)
116 A. Kumar et al.

5. A.B. Arrieta, N. Díaz-Rodríguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. García,

S. Gil-López, D. Molina, R. Benjamins, R. Chatila, F. Herrera (2020). Explainable artificial
intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible
AI. Inform. Fusion, 58, 82–115 (2020)
6. P.J. Phillips, C.A. Hahn, P.C. Fontana, D.A. Broniatowski, M.A. Przybocki, Four principles of
explainable artificial intelligence, Gaithersburg, Maryland (2020)
7. S.N. Payrovnaziri, Z. Chen, P. Rengifo-Moreno, T. Miller, J. Bian, J.H. Chen, Z. He, Explain-
able artificial intelligence models using real-world electronic health record data: a systematic
scoping review. J. Am. Med. Inf. Assoc. 27(7), 1173–1185 (2020)
8. D.W. Gunning, D. Aha. DARPA’s explainable artificial intelligence program. AI Mag
9. S. Gupta, B. Amaba, M. McMahon, K. Gupta, The Evolution of Artificial Intelligence in
the Automotive Industry. 2021 In Annual reliability and maintainability symposium (RAMS)
(pp. 1–7). IEEE (2021)
10. H. Mankodiya, M.S. Obaidat, R. Gupta, S. Tanwar, XAI-AV: Explainable Artificial Intelli-
gence for Trust Management in Autonomous Vehicles. In 2021 international conference on
communications, computing, cybersecurity, and informatics (CCCI). IEEE, pp 1–5 (2021)
11. X. Zhu, X. Ao, Z. Qin, Y. Chang, Y. Liu, Q. He, J. Li, Intelligent financial fraud detection
practices in post-pandemic era. Innovation 2(4), 100176 (2021)
12. M. Bellucci, N. Delestre, N. Malandain, C. Zanni-Merk, Towards a terminology for a fully
contextualized XAI. Proc. Comput. Sci. 192, 241–250 (2021)
13. Y.K. Dwivedi, E. Ismagilova, D.L. Hughes, J. Carlson, R. Filieri, J. Jacobson, V. Jain, H.
Karjaluoto, H. Kefi, A.S. Krishen, V. Kumar, Y. Wang, Setting the future of digital and social
media marketing research: Perspectives and research propositions. Int. J. Inf. Manage. 59, 102
(2021)
Chapter 8
Explainable Data Fusion on Edge:
Challenges and Opportunities

Shweta Sinha and Priyanka Vashisht

Abstract Advancements in technology and the availability of small electronic

devices have led to the availability of big data. This humungous amount of data
is nowadays available in nearly every field, be it science or social domain. As much
of these collected data are time-sensitive, they need to be utilized timely to give
an effective outcome. Data fusion is of paramount significance in enhancing the
collected data’s effectiveness. But, today, as AI has gained significant momentum,
machine learning (ML) techniques, mainly deep neural network (DNN), are widely
utilized to deliver what is expected from the data. Edge intelligence tries to fulfil
the promise of giving efficient and timely output. Deep learning, undoubtedly being
robust, also possesses some disadvantages, and one of these relates to the explain-
ability of ML methods that help solve any problem. With dense layers in DNN and
millions of parameters, it is difficult to interpret the model creating a black box.
The chapter aims to outline the explainability of data fusion at the edge. It high-
lights different data models of fusion, discusses a framework for AI and data fusion
at the edge and identifies potential challenges and possible solutions in this regard.
While doing so, the chapter intends to cover the fundamentals of explainability in AI,
the need to convert the black box system into a transparent one, and the associated
opportunities for explainable artificial intelligence.

Keywords XAI · Explainable AI · Data fusion · Edge computing · Intelligent

machine learning

1 Introduction

Artificial Intelligence (AI) has achieved tremendous growth in many activity sectors.
In the last few years, the existence of AI at the core of almost all developments can
be easily observed. The inclusion of learning, reasoning and adaptation capabilities
has helped AI enable systems to perform better than any human being [1]. Besides,

S. Sinha · P. Vashisht (B)

Amity University Haryana, Gurugram, India
e-mail: [email protected]

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 117
A. E. Hassanien et al. (eds.), Explainable Edge AI: A Futuristic Computing Perspective,
Studies in Computational Intelligence 1072,
https://doi.org/10.1007/978-3-031-18292-1_8
118 S. Sinha and P. Vashisht

AI-based systems have been deployed in several susceptible decision-making tasks

that range from biometric surveillance systems and criminal justice systems to areas
that have diverse social and political connections [2]. Research and innovation based
on AI techniques in the area of voice, automotive and healthcare have continuously
seen an increasing growth rate, and it is projected to expand at the rate of 38.1%
from 2022 to 2030 [3]. The more these AI systems have started affecting human life,
the more the quest to understand the decision-making mechanism of these systems
is increasing.
The traditional AI methods are typically hard-coded rules to execute step by step
and are easily interpretable. But, in the last few years, with the increased availability
of a massive quantum of data AI market has witnessed frameworks that leverage
the power associated with the enormous volume of data. In ML methods, logic
and inference provide interpretation of any problem and give solutions. To further
improve the performance, a combination of efficient learning algorithms with huge
parametric space was utilized, and DNN came into existence. Deep learning is a
range of algorithms using neural networks with deep layers. These DNNs comprise
millions of parameters and hundreds of layers making the complete system opaque.
This opacity further leads to naming it the Black-box model [4]. Their vast parametric
space allows them to easily model any complex problem [3]. Several tools are now
available that can increase insight into the inner working of DNN, but transparency
is still a problem.
In general, humans don’t readily adopt techniques that are not trustworthy or are
not tractable and interpretable. The explanations for supporting outputs are generally
necessary to better understand the system and its behaviour. And sometimes they will
be even helpful in removing deficiencies from the system. With the employment of
AI in critical application areas like precision medicine, autonomous vehicles etc.,
the demand to provide transparency is increasing among AI stakeholders [5]. On the
other hand, the focus is always on increasingly improving the system performance,
and it will, in turn, increase the opacity. So, there must always be a trade-off between
a model’s transparency and performance [6]. Besides, interpretability is also one of
the key design drivers for any ML system for its easy implementability. It can help
in detecting and correcting bias from the datasets. Further, interpretability can act as
insurance that the inference is only due to meaningful variables.
Lack of insight into the internal mechanism of work limits the effectiveness of
AI methods. Particularly the ML models. To address this issue, explainable Artifi-
cial intelligence (XAI) [7] proposes to create more transparent AI, particularly ML
techniques. The aim is to generate a suite of methods that produce more explanation
of the inside working while maintaining a high level of performance. The need for
XAI (Fig. 1) can well stem from the reasons mentioned below.
• Explanation to identify weaknesses; a model that can be understood and explained
can easily be improved for its shortcomings and perform as per users’ needs.
• An explanation for monitoring the system so that no wrongs can be done. Under-
standing system behaviour closely gives better control over vulnerabilities and
helps quickly identify flaws.
8 Explainable Data Fusion on Edge … 119

Fig. 1 Need for Explainable

AI

Explain to
Identify
Weakness

XAI

Explain to Explain to
Justify Monitor
Decision Performance

• An explanation is needed to justify the decision taken. This helps in building trust,
mainly when unexpected outcomes are observed. XAI systems will provide the
required information to explain the results.
With the tremendous growth in information sources, data fusion approaches are
being utilized to exploit them simultaneously for solving any learning task. Undoubt-
edly, using the fusion of heterogeneous data sources has drastically improved ML
node performance. Different data fusion techniques can also enrich the explainability
of ML models besides maintaining the privacy of training data used by ML models.
Data fusion can be done at a centralized or distributed level, where it can be either
knowledge fusion, data fusion or model fusion. Either of these is going to impact
the ML models and help provide results. But, the acceptability requires explanation,
and hence the fusion approaches need to be exploited to achieve explanation while
maintaining privacy.
The objective of this chapter is to
• Provide an overview of the essential concepts and terms used in XAI-related
research.
• Define diverse approaches to post-hoc explainability.
• Elaborate on different data fusion models and discuss the need for XAI for data
fusion.
• Highlight some of the application domains that demand XAI.
• Enumerate a series of challenges XAI, especially in the context of edge
intelligence.
120 S. Sinha and P. Vashisht

2 Data Fusion Architecture and Models

Data fusion is the combined examination of various interconnected datasets that offer
different perspectives on the same phenomenon. In general, more accurate inferences
can be drawn from the correlation and fusion of data from multiple sources than from
the examination of a single dataset. In the context of combining multiple data sets
for analysis, this section frames the primary Data Fusion Models and their levels.
Both in the research and business arenas, many data fusion frameworks have been
created. By choosing the best method for the given challenge, these frameworks have
been applied in several projects to assist in the creation of fusion systems. Conven-
tional data fusion models are discussed in this section, along with their existing capa-
bilities. It should be highlighted, however, that these structures can be redefined and
enhanced in order to use more modern techniques, such as switching from static to
dynamic processing, from human–machine teaming, from centralized to distributed
processing, and from the insignificant amount of data to enormous real-time data.

2.1 JDL/DGIF Model

The Data Fusion Subpanel, later known as the Data Fusion Group, was established
in the middle of the 1980s by the Joint Directors of Laboratories. Data fusion
now includes sensor and information fusion because of the World Wide Web. The
JDL/DFIG announced a data fusion model that separated the various processes. The
model specifies the number of stages of fusion, each of which takes advantage of
recent advancements in AI to facilitate evaluation at a lower level (tier 0–3) and refine-
ment at mid-level (tier 4–6). However, contextual restrictions are based on mission,
objectives, and goals in system management (tier 6). The Data Fusion Information
Group (DFIG) model currently has seven tiers, and they are depicted in Fig. 2 and
are discussed as follows:
• Tier 0: Subject Evaluation, based on the relationship between the pixel and signal
levels, provides approximation and forecast the signal and object observable
states.
• Tier 1: Object Assessment, is constructed on data and their relationship and state
estimation, which offers valuation and prediction of entities and objects.
• Tier 2: Situation Assessment offers assessment and forecast of relationships
between objects and entities.
• Tier 3: Threat Refinement, estimates and predicts the impacts of participant’s
planned or anticipated activities.
• Tier 4: Process optimization is an aspect of resource management that offers
adaptive information collection and processing to give an assessment of sensing
targets and impact forecasting for planned or estimated actions by participants.
8 Explainable Data Fusion on Edge … 121

Fig. 2 The JDL model

• Tier 5: User Refinement is an aspect of managing knowledge that offers adap-

tive access to control and information display to help decision-making through
human–machine interfaces.
• Tier 6: Mission Management is an aspect of managing systems that permits
the spatial–temporal management of resources, preparation, and target setting
to aid in taking decision while taking into account societal, financial, and political
constraints.

The model is useful for general understanding, but it does not direct a developer
in determining the methods that should be utilized [8]—as a result, it is ineffective
for creating an architecture for a practical system. In addition to broadening the func-
tional model and connecting the taxonomy to fields outside of the original military
focus, Steinberg, Bowman, and White proposed expanding and revising the JDL
model [9]. These changes included integrating a data fusion tree architecture model
for system description, design, and development.

2.2 Waterfall Model

The Waterfall model is a three-tiered hierarchal model and is presented in Fig. 3.

From the data level to the decision-making level, data flow is operational. Feed-
back data from the decision-making module is continuously updated by the sensor
system. The feedback component gives the multi-sensor system advice on recali-
brating, reconfiguring, and data collection issues. The description of each tier of the
Waterfall model is as follows:
Level 1: In order to offer the necessary information about the environment, the
raw data is suitably converted at this level. Models of the sensors and, if possible,
122 S. Sinha and P. Vashisht

Fig. 3 The waterfall model

of the detected phenomena are required to complete this task. These models might
draw from the experimental analysis.
Level 2: The feature extraction and feature fusion processes make up level 2. These
procedures are used to draw conclusions about the data on a symbolic level. While
maximizing information given, they seek to minimize the data content. Lists of
estimates with associated probabilities (and beliefs) are the result of this level.
Level 3: The association of object and events are drawn at this level. The infor-
mation that has been obtained, the libraries and databases that are available, and
the interactions between people are used to assemble potential courses of action.
The waterfall approach has the same flaws as the JDL model since they are iden-
tical. The waterfall model’s main drawback is the absence of any feedback data flow,
despite being more accurate than other models in its analysis of the fusion process.
Although not widely utilized abroad, the waterfall approach has been used in the
defence data fusion sector in Great Britain [10].

2.3 Omnibus Model

In 1999, Bedworth and O’Brien introduced the Omnibus model which is shown in
Fig. 4 [10]. This model combines the Boyd loop, Dasarathy, and Waterfall models.
The Boyd control loop is described by the authors as an iterative process with four
elements (observe, orient, decide, and act) acting in a close loop. The Dasarathy
model, in contrast, is made up of the three fundamental stages of data fusion: data,
feature, and decision.
The model was developed after examining the advantages and disadvantages of
previous models, and it incorporates the majority of the strong points of other method-
ologies. Although the model offers a much more fine-grained organization of the
8 Explainable Data Fusion on Edge … 123

Fig. 4 The omnibus model [10]

processing layers, it exhibits a cyclic nature similar to the Boyd loop. The concept
is designed to be applied recursively at two different abstraction levels numerous
times within the same application. The model is first used to describe and structure
the entire system. Second, the system’s individual subtasks are modelled using the
same structures. In the Omnibus model, highly sophisticated hierarchical separa-
tion of the sensor fusion jobs does not permit a horizontal division into tasks that
reflect distributed sensing and data processing. Decomposition into modules that may
be independently implemented, independently tested, and independently reused for
other purposes is therefore not supported by the paradigm.
Data fusion is a complex idea with apparent benefits but also many difficulties,
especially with new paradigms with artificial intelligence integrated applications.
The development of artificial intelligence (AI) has led to a meteoric rise in AI-based
services. AI technology, such as machine learning (ML) and deep learning (DL),
achieves state-of-the-art performance in a variety of sectors. Additionally, present
applications frequently use centralized data management, which necessitates user
uploading of data to the central data centre. But there is an enormous amount of
data generated and gathered by billions of mobile users and Internet of Things (IoT)
devices that are dispersed at the network edge. Such a large amount of data requires
enormous bandwidth resources to upload to the distributed environment such as the
cloud, which may also cause users to experience intolerable latency [11]. With the
development of cloud computing, edge computing [12–16] brings cloud services
closer to end users. Edge computing provides platforms for computer resources
that are typically found at the edge of networks, such as networking, storage, and
processing. Edge servers, which can include IoT gateways, routers, and microdata
centres in mobile network base stations, on automobiles, among other locations, are
the devices that provide services for end devices. Edge intelligence is the result of
combining edge computing and AI to address the significant issues facing AI-based
applications.
124 S. Sinha and P. Vashisht

The discussion of edge intelligence is provided in the next part, along with exam-
ining several fundamental unresolved problems apart from potential theoretical and
technical directions.

3 Edge Intelligence

An era of the “Internet of Things” has emerged as a result of technological advance-

ments and the downsizing of electronic gadgets that can not only sense and process
data but also connect with other devices (IoT). These Internet of Things (IoT) devices
frequently have a large number of sensors, which generate enormous amounts of data
at the network edge. Many of the applications that need to use this data have embraced
artificial intelligence (AI) and machine learning (ML). Due to constrained network
capacity and real-time requirements of numerous applications, such as medical,
aerospace, data analytics etc., moving this enormous amount of data to the cloud
is frequently not viable. Consequently, there is a drive to push AI boundaries to the
network edge to use the massive amount of data generated by IoT devices closer to
the data source. Due to this requirement, edge computing moves computing capacity
away from the centralized nodes to the logical extreme edges of a network [17],
and AI have combined to create a new field known as Edge Intelligence. In order to
fuel AI applications without solely relying on the cloud, edge intelligence makes use
of the widely available edge resources. There is yet no official definition for edge
intelligence, despite the early stages of research. Most organizations currently refer
to Edge Intelligence as “the paradigm of running AI algorithms locally on an end
device, using data that are created on the device.” [18]. Edge intelligence is supported
by a wide range of ML techniques, which fall under the categories of unsupervised
learning, supervised learning, semi-supervised learning, and reinforcement learning
(RL).
An ML model must first be trained before it can be deployed, and it must then
be used to make inferences. Since training requires more time and resources than
inference, in the edge AI model, training is frequently carried out in the cloud while
inference is carried out at the edge. While AI inference will be carried out at edge
devices, AI training will probably continue on robust cloud-based computers due to
the resource limitations of many edge devices [19]. Deep learning, one of the ML
techniques now in use, excels in many tasks. Convolutional neural networks (CNN),
for instance, have been used for object detection and recognition as well as image
classification. Recurrent neural networks (RNNs) are used in multitarget tracking and
natural language processing. An example of a task that can be accomplished using
deep reinforcement learning (DRL) is the trajectory optimization of autonomous
vehicles. Edge computing offers a solution for the performance- and energy-efficient
execution of AI/ML algorithms, independent of any particular AI/ML techniques.
8 Explainable Data Fusion on Edge … 125

3.1 Challenges and Resolutions of Edge Intelligence

In edge intelligence, there are certain outstanding problems that remain unsolved. It is
essential to recognize, examine, and seek out fresh theoretical and technical answers
to these problems. In this article, we examine several significant edge intelligence
challenges and potential solutions. Scarcity of data at the edge, data consistency
on edge devices, poor adaptability of statically trained models, privacy and security
concerns, and incentive mechanisms are some of these difficulties [20].

3.1.1 Scarcity of Data at the Edge

The majority of machine learning methods mainly supervised machine learning,

and depend on having enough adequate high-quality training instances to achieve
good performance. However, when the acquired data is sparse and unlabelled, it
frequently fails in edge intelligence applications, such as speech recognition and
HAR. In contrast to conventional cloud-based intelligent services, which pool all
of the training instances into a single database, edge devices create models using
data that they either generate themselves or that they collect from their surroundings.
There aren’t many high-quality training examples in these datasets, such as good
image features. Since most existing works assume that the training instances are
of high quality, they neglect this problem. The training dataset is frequently unla-
belled as well. To address the issue of unlabeled training examples, which necessi-
tate human annotation, a few articles [21, 22] propose using active learning. Only
scenarios with a limited number of occurrences and classifications would be suitable
for such an approach. Federated learning strategies make use of data decentraliza-
tion to efficiently address the issue. However, federated learning is only appropriate
for collaborative training as opposed to the solitary training required for customized
models.
Following are some of the issues that are explored and their potential solutions:
• Use shallow entities can be trained on a small amount of data. In general, a machine
learning algorithm will learn from tiny datasets more effectively the simpler it is.
In some cases, a straightforward model, like Naive Bayes, a linear model, or a
decision tree, is sufficient to solve the issue, as opposed to more complex models,
like neural networks, as they are essentially trying to learn less. Consequently,
when addressing practical issues, it is important to select an appropriate model.
• Adopt techniques based on incremental learning. Edge devices could incremen-
tally retrain a widely-used pre-trained model to take into account their new data.
In this way, a customized model that may be created with just a few training
knowledge from other models is leveraged to improve the performance of a
related model. The cold-start issue is often avoided, and the amount of training
data needed is decreased. This technique is known as transfer learning. Transfer
126 S. Sinha and P. Vashisht

learning may therefore be an option when there is a lack of target training data,
and there are some similarities between the source and target domains.
• A model can be made more reliable through data augmentation by having more
data during the training process [23]. For instance, expanding the number of
photos while maintaining the semantic meaning of the labels through flipping,
rotating, scaling, translating, cropping, etc. The network would become invariant
to these deformations and perform better with unseen data through training on
enhanced data.

3.1.2 Consistency of Data on Edge Devices

Applications that rely on edge intelligence, such as voice recognition, activity recog-
nition, emotion recognition, etc., typically gather information from a significant
number of sensors that are dispersed throughout the edge network. The informa-
tion collected might not, however, be reliable. Different sensing contexts and sensor
heterogeneity are two variables that contribute to this issue. The environment (such
as the street and library) and its weather (such as rainy, and windy) introduce back-
ground noise to the sensor data that is collected, which may have an effect on the
model’s accuracy. The unexpected difference in the acquired data from sensors may
also be due to the heterogeneity of those sensors (such as hardware and software).
Various sensors, for instance, have various sensitivities, sampling speeds, and sensing
efficiency. Even the sensor data that was gathered from the same source may differ
between sensors. As a result, the model training would vary depending on how the
data were different, for example, the parameters of the features [24]. Existing sensing
applications still struggle with this variation.
If the model is trained centrally, this issue might be readily resolved. The ability
to learn the invariant characteristics to the variations is ensured by the centralized
huge training set. However, edge intelligence does not cover this ground. Future
research into this issue should concentrate on finding ways to counteract the varia-
tion’s detrimental impact on model accuracy. To achieve this, representation learning
and data augmentation are two potential research approaches that may be taken into
consideration. In order to make the model more resistant to noise, data augmentation
could be used to supplement the data during model training. For instance, in speech
recognition software for mobile devices, several types of background noise can be
added to mask environmental variance.
The performance of models is strongly influenced by data representation. In order
to develop models with more useful characteristics, representation learning focuses
on learning how to represent data [25], which could also be used to disguise hardware
disparities. The model’s performance would be much enhanced if we could “trans-
late” the representations between two sensors that are using the same data source
for this issue. As a result, representation learning offers a promising way to lessen
the effects of inconsistent data. Future efforts could be made in this regard, such as
developing processing pipelines and data transformations that are more efficient.
8 Explainable Data Fusion on Edge … 127

3.1.3 Bad Compliance of Statically Trained Model

Most edge intelligence-based AI applications train the model on a central server

before deploying it on edge hardware. Upon completion of the training process,
the trained model will not need to be retrained. Low performance and a poor user
experience are the results of these statically trained models’ inability to deal with
the unknown new data and jobs in unfamiliar situations. For models trained using a
decentralized learning approach, only local knowledge is utilized. As a result, these
models may only become authorities in their limited geographic regions. A wider
serving area results in a decline in service quality.
Two options may be taken into consideration to deal with this issue: sharing
knowledge and lifetime machine learning. An advanced learning paradigm called
lifetime machine learning (LML) [26] permits ongoing knowledge building and
self-learning on new tasks. Instead of being taught by humans, machines are to
independently acquire new knowledge based on previously acquired knowledge.
Meta-learning [27], which enables machines to automatically learn new models, and
LML are slightly dissimilar. LML could be used by edge devices with a variety of
learnt tasks to adjust to changing environments and deal with unknowable data. It
is important to remember that the LML is not primarily intended for edge devices;
therefore, powerful computing hardware is anticipated for the machines. Therefore,
if LML is used, model design, model compression, and offloading techniques should
also be considered. Communication between many edge servers is made possible
through knowledge sharing [28]. When an edge server receives a job for which
it lacks the knowledge necessary to deliver quality service, it may send knowledge
requests to other edge servers. Since the knowledge is distributed around various edge
servers, the server with the necessary knowledge answers the query and completes
the task for users. In such a knowledge-sharing paradigm, a method for knowledge
appraisal and a system for knowledge querying are needed.

3.1.4 Security and Privacy Concerns

Heterogeneous edge devices and edge servers must cooperate to offer processing
power in order to realize edge intelligence. This technique involves sending locally
cached data and computation tasks (training or inference tasks) to unfamiliar
machines for additional processing. The data could include personally identifiable
information about users, such as tokens and images, which raises the possibility
of privacy breaches and attacks from hostile users. If the data is not encrypted,
malevolent people could easily get access to confidential information. A few initia-
tives [29–31] propose performing some local preparatory processing, which could
conceal sensitive data and cut down on the amount of data provided. The processed
data can still be used to extract personal information, albeit [32]. Additionally, by
adding malicious code, users might attack and take over a computerized device by
128 S. Sinha and P. Vashisht

controlling computer tasks being infected by a virus. It is the main obstacle that insuf-
ficient security and privacy protection measures methods to safeguard the security
and privacy of users attacked.
A viable solution is the credit system. The credit system employed by banks, which
verifies each user’s participation in the system and examines their credit information,
is comparable to this. The system would remove users with a history of poor credit.
As a result, everyone who uses computing equipment can trust them, and they are
all secure. Privacy could be protected via encryption, which is currently employed
in several works [20, 33]. However, before the training or inference tasks are carried
out, the encrypted data must be decrypted, which increases the amount of computing
required. Future work might focus more on homomorphic encryption to address the
issue [20]. A form of encryption known as homomorphic encryption enables direct
computation on ciphertexts and produces encrypted results. Once the data has been
decrypted, the outcome is identical to what would have been obtained by computing
the unencrypted data. Therefore, by using homomorphic encryption, the training or
inference process might be directly done on encrypted data.

3.1.5 Mechanism for Incentives

The two most crucial phases of edge intelligence are model training and inference
and data collection. Securing the accuracy and usefulness of the information from
the gathered data during data gathering is difficult. The time, battery, and bandwidth
that data collectors use to perceive and gather data are all of their own resources.
It is unrealistic to believe that every data collector will be willing to contribute,
let alone for pre-processing data cleaning, feature extraction, and encryption, which
further uses resources. Each member must put aside personal interests in order to
cooperate for the purpose of collaborative model training/inference. For instance, one
master and multiple employees make up the architecture suggested in [34]. Workers
identify objects in a certain mobile visual domain and supply training instances for
masters via pipelines. Such an architecture is effective in private settings, such as at
home, where all of the equipment is driven to work together to develop a superior
intelligent model for their owner or master. It would not, however, function well
in public settings where the master initializes a job and assigns subtasks to new
players. In this situation, extra incentive issues develop that are not normally taken
into account in smart settings when all devices are not owned by a single master.
Participants must be motivated to collect data and complete tasks.
Future initiatives should consider reasonable incentive mechanisms. Participants
have a variety of missions, such as collecting data, processing it, and analyzing it, all
of which require different resources. Every participant wants to receive the greatest
potential prize. The operator, on the other hand, seeks to maximize model accuracy
at the lowest cost. The difficulties in creating the best incentive mechanism are
determining how to quantify the workloads of various tasks to match commensurate
rewards and how to jointly optimize these two competing objectives. Overcoming
these difficulties could be the focus of future work.
8 Explainable Data Fusion on Edge … 129

4 Explainable Artificial Intelligence

Although the deep learning tools available today can generate reliable results, they
are tough to comprehend due to their opaque nature. As the demand for trustable AI
worldwide is increasing, attempts are being made to integrate ethical standards during
design and implementation. Explainable AI, one of the components of trustable AI,
tries to dive deep into the rationale of the decision taken by the AI system. It is
considered a new research field that tries to make AI systems such that their behaviour
can be explained and the outputs are understandable to humans [35]. The primary aim
of XAI is to explore the logic involved in the decision-making process and outline
the merits and drawbacks of the system. This information can be utilized for further
improvements. Any XAI-powered AI system removes the confusion related to the
output of the system, making it a trustable system.

4.1 Terminology in XAI

Better understanding of XAI can be achieved if there is a common point of under-

standing for explainability. The desired characteristics of XAI and terminologies
used in research and the public domain need to be adequately explained. These
terminologies are mentioned below in Table 1.
In all these characteristics of XAI, Interpretability, also referred to as understand-
ability, is an essential feature of XAI [36]. The terms interpretability and transparency
are strongly tied to this concept. Justifiability is also connected in the sense that a
clear understanding of the system can easily identify the decision as right or wrong.
Understandability can refer to either model understandability or human understand-
ability. With the importance of the understandability of the system, the presence of
a human is the cornerstone of XAI.

Table 1 Characteristics of XAI

Term Description
Intelligibility It refers to the easy understandability of the internal working of a model for
any human, without any need for explicit demand for an explanation
Justifiability It supports the outcome of the AI system by understanding its decision but
may not always give an explanation for the same
Interpretability The ability of the system to showcase its functionality to humans in an easily
understandable term. It is closely related to explainability
Explainability The ability of the system to provide an explanation about the decisions being
taken by the system. It refers to the description of the internal working of the
system
Transparency Refers to the self-explanatory aspect of the system that leads to
understandability without any external interventions
130 S. Sinha and P. Vashisht

Table 2 Summary of XAI purposes

Purpose Discussion
Informativeness The ML models are used to provide a solution that matches human
performance. But, the problem being solved by ML methods is not exactly the
same as humans in their environment. It is a mapping from one domain to the
other but not an exact mapping. The information regarding this mapping and
other aspects of the problem solving is required to understand the system
working and its decision making. Hence, explainability here targets to capture
the information
Trustworthiness Building trust is the primary aim of the XAI model. It gives confidence
regarding the model’s performance in an adverse situation. Also, as almost all
explainable models are trustworthy, the vice-versa may not be true. Out of the
significant limitations associated with trustworthiness is its fuzziness which
makes it difficult to quantify
Causality Causality refers to the relationships between the variables. As the black box
models are generally implemented in complex systems with a huge number of
variables or parameters, explainability helps define the causal relationship
between the variables and helps better understand them [39]
Accessibility One of the essential goals of explainable AI is accessibility [40]. It helps end
users to be more involved during the ML model development phase.
Accessibility also eases the understandability of non-expert users of the
system
Interactivity Researchers [41] have highlighted interactivity with the user as one of the
goals behind XAI. This can be beneficial when the user’s response or
interactions influence system performance
Fairness One of the goals sought by researchers of XAI is fairness. Explainability tries
to capture each detail of the black box model, leading to a clear visualization
of the inside process. This guarantees fairness in ML methods

4.2 Purpose of XAI

Several research activities around XAI have been carried out [37, 38] to date. Most
of this research highlights the goals that the explainable model possesses or should
possess, but the purpose behind achieving these goals is different from most of the
research carried out in XAI. A summary of some of the purposes of XAI is presented
in Table 2.

5 The How of Explainability in AI

There are a few intriguing queries when the demand for XAI is generated. They start
with understanding the contextual aspects of XAI, i.e. what exactly is XAI, followed
by covering the identification of the purpose for which XAI has been demanded,
i.e. Why needed. And lastly, when one is conceived about exploring the inside of
the black-box models, they query how it can be done. The explainability of any AI
8 Explainable Data Fusion on Edge … 131

system can be considered not only after the black-box development has been done
rather, it can be done at all stages of development, namely, at the Pre-modelling stage,
at the model development stage known as Explanation modelling and also at Post-
modelling stage. At each stage, many ways of explainability exist, which depend
upon factors contributing to the AI model operations. Figure 5 shows the stages of
explainability of AI system [42].
• Pre-modelling Explainability—A suite of methods diverse in nature but having
a similar goal of understanding exists for pre-modelling explainability that works
mainly at the data level. These methods can be categorized as dataset description
standardization, exploratory data analysis, data set summarization and feature
engineering [43].
In real-time, the datasets usually come with incomplete documentation. This
makes the interpretation difficult. Standardization can make this interpretation
consistent, and smooth communication between users and creators regarding the
data can be done. Some specification standards are recommended as data state-
ments and data nutrition labels [42]. The exploratory data analysis aims to extract
a statistical summary that ranges from mean and standard deviation to the dimen-
sionality of the dataset. Dataset summarization provides a prediction on the given
dataset with the help of already used training data. As these training data are too
huge, posing storage challenges, a possible way is to store only a relevant subset
of the training data set. A good mechanism should be employed for selecting
the representative dataset. Explanation of any AI system can be as good as the
selection of features used to explain the predictions of model. Feature engineering
helps in identifying the relative importance of input features.
• Explainable Modelling—Explainable AI approaches can be applied at a local
or a global level which is either inherent or, Post-hoc. With the choice of specific
AI models that are inherently explainable, we can avoid the problem of explain-
ability due to the black box model. Linear models, decision trees etc., are some

Pre-Modelling Explainability
(Understand data used to develop model)

Explanable Modelling
(Develop models that are intrinsically explainable)

Post-Modelling Explainability
(Extract explanation from surrogate models to describe
pre-developed opaque models)

Fig. 5 Stages of AI explainability

132 S. Sinha and P. Vashisht

of the models that are inherently transparent and have low-performing power.
But, for high dimensional input, these simple, inherently explainable models also
do not guarantee explainability. Like, the linear regression model, when applied
to high dimensional data, may not be simulatable or, we can say, explainable.
It is customary to use an inherently simple or explainable model approach with
the black box method to obtain a high-performing explainable model. To better
understand, deep K-nearest neighbour is the amalgamation of K-nearest neigh-
bour inference applied on opaque training set learnt through deep hidden layers.
Self-explanation neural network and deep weighted averaging classifier (DWAC)
are a few more models obtained from the combination of the traditional transparent
approach and the black box model.
• Post-modelling Explainability—After the explainability, attention to data and
model next comes the explainability after the fact. The basic idea behind this
is to provide an explanation of the simplification or, Feature level explanation.
The post-hoc explainability techniques can be grouped based upon their target,
i.e. what they aim to explain about the model. The drivers are the cause for the
explanation, the explanation family that communicates with the user about the
driver and the target and finally, the estimator that deals with the overall computa-
tional process of obtaining the explanation. The local interpretable model-agnostic
explanation tool (LIME) [44] explains an instance of a complex deep learning
model, which is the model’s target. Input features serve as the driver, and the
importance score of each feature is communicated to the user that is computed
through the local evaluation of the model input termed as an estimator. The targets
are objects of an explainability method. They vary in their scope and complexity
and can be of inside or outside type. Inside targets are bothered about the details
internal to the model, and outside targets refer to the functionality or output of any
model. Drivers are a set of factors that impact the target, and explainability deals
with the causal relationship between the two. Training samples, input features and
choice of optimization algorithms can affect AI model development; hence, they
are the explanation drivers.
The post-hoc explainability methods aim at providing information content that
is easily understandable by a human. The impact of the driver on the target should
be communicated in an easy and understandable way, ensuring interpretability and
completeness. The importance score of each of the drivers, if–then based on decision
rules and decision trees, are some of the ways to explain the relationship between
the target and the drivers. In the fourth category, the estimators have a wide range,
and they vary in terms of their applicability and the underlying mechanism. The
estimation methods are either model specific, which works only for a specific method,
or model-agnostic, which can be used for any black box model. These models can
be used at the local level that offers interpretation for a particular instance or can
be applied to a global scenario where surrogate representation can be developed to
approximate any black box model.
Even though post-hoc explainability provides an understanding of the black-box
models, there are certain limitations to it [43]:
8 Explainable Data Fusion on Edge … 133

• These methods do not precisely match; instead only approximate the underlying
models while creating a surrogate white box model to explain the decisions made
by the system. But since it is only based on approximation, it cannot truly rely on
critical applications.
• These methods work for sample data collected randomly instead of the complete
data set. Due to randomness in the sampling process, the post-hoc explainability
suffers instability.
These shortcomings highlight the instability and low fidelity of post-hoc explain-
ability methods and simply points out that the post-hoc explainability methods are
helpful for model design and development but cannot be relied upon for regulatory
use.

6 Application of XAI for Data Fusion

A brief description of XAI highlights that it can bring significant benefits to an

extensive range of application domains that rely on AI. Most AI applications are
data-centric, and nowadays, for creating a robust and reliable system, the input data
is taken from several sources that may be heterogeneous. Data fusion is one of the key
players in the creation of such a system. This section presents some critical domains
that rely on AI, and XAI may find opportunities in these domains. A few mentioned
here have witnessed exponential growth in XAI applicability and research in the last
few years.
• Transportation
Vehicles running on the road without human intervention are the latest buzz in
the technology world. These automated vehicles promise a reduced mortality rate
with increased mobility features. As the motor vehicle runs on the designated path,
it must take decisions regarding objects coming into its way. These decisions
must be taken in fractions of seconds. The decisions are dependent upon the
efficiency and effectiveness of the object classification algorithms. Challenges
lie in obtaining accurate results and are due to the explainability of intelligent
systems. Uber killed a woman in Arizona. The AI systems are mostly opaque;
their internal working and decision-making criteria are difficult to understand.
Only a simple explainable system can explain the decision taken and can open
the door for further improvements. A few pieces of research in the direction of
XAI for transportation have been initiated [45]. Still, owing to the importance and
criticality of the domain, explainability for self-driven vehicles are crucial.
• Finance
In financial sectors, AI tools can be used to predict risks associated with the loan
and can also assess other financial risks. It can provide customer interactions
and solve domain-related queries throughout the day without any manual inter-
ventions. The ML-based intelligent system also helps in fraud detection. The AI
tools used for the purpose possess security risks that can be related to data theft. It
134 S. Sinha and P. Vashisht

is needed that the tools used here must be explainable so that the reasons behind
the decisions taken and selections made should be clear, making the tool/system
trustable.
• Military
The need for XAI was first felt in the military domain, and after a call for research
by DARPA [7] project, it widely spread in other domains too. AI has several
application areas in the military where it can showcase its potential. It can be used
in automated surveillance for tracing suspicious activities. Machine learning is
being applied for internal security and used to forecast and protect unauthorized
intruders. All these need to be a trustable and reliable system. They require XAI
to get better understandability as they all involve life and death.
• Legal
The law is the force to maintain decorum in society. All aspects of business, be
it merger, acquisition, selling or purchase, require a legal contract. As a powerful
abiding force in everyday’s life, the law is also influenced by advanced techno-
logical developments. In the legal domain, AI is leveraged for contract analytics,
criminal justice, litigation predictions, lawyer bots, etc. All these use ML power,
so complex models can be easily presented in a simple form. As the impact of all
these legal services critically influences human life, understanding their working
and decision-making is very much needed. XAI is essentially required to obtain
transparency, and recently, a few works have been initiated for the same [46, 47].
• Healthcare
The continuous growth in medicine and other healthcare monitoring aspects is
credited to the technological developments in recent years. The advanced devices
available for patient monitoring continuously capture data, and their analysis can
give much information regarding patient conditioning that is impossible even for
humans to explore. With this large amount of collected data, the information
provided by traditional AI techniques and even ML methods is sometimes insuf-
ficient. Recently, these enormous amounts of data are being used by deep learning
methods to identify the association between different types of patient data and
use them effectively. Even though the DNN-based AI techniques lead to a new
era of digital healthcare, the challenges still persist. With the number of parame-
ters approximating millions in number in the deep learning model, it is difficult
to exactly comprehend what the model portrays of the data, for example, inter-
preting the radiological images [48]. From the black box of DNN computations,
expressing intuitive interpretations that can clarify model uncertainty, support
model results, or provide new clinical insights is difficult. Explainability and
understandability are very important for the success of these models. Recently,
AI-based visualization methods have been developed to describe the AI models.
Some widely used models include occlusion maps [49], salience maps [50], and
class activation maps [51]. A few studies [52–54] have explored explainability in
medical AI in recent years.
The work in each of the domains supports the necessity for XAI. Such study is,
however, still in its early stages, and significant research efforts must yet be made
8 Explainable Data Fusion on Edge … 135

in this regard. Moreover, cybersecurity, education, entertainment, government and

image recognition etc., also find exciting applications in XAI.

7 Conclusion

The conventional “Black Box” models that are traditionally being used in AI systems
are often marked for their lack of transparency and the way they hide potential biases.
XAI in fields like criminal justice, medical decisions and search engine outputs can
be helpful in removing potential biases while giving the proper explanation for the
decisions taken by the systems. Use of XAI leads to fair and ethical practices for
development of intelligent and responsible AI. They help in building trust for the
system. In future, XAI may find itself deep rooted in the field of medicine where it
provides the advantage of explanation that helps in easy understanding. The intent of
this chapter was to provide the brief about XAI that can help in better understanding
of the decisions made. In this chapter, the authors have discussed the emerging
discipline of edge intelligence. With the tremendous increase in the availability of data
and decreasing cost of computing devices, the rise of data-driven applications will
continue to proliferate. A Survey of available research highlights the long existence
of cloud and AI. With the increasing applicability of intelligence in real-time data-
centric applications, the cloud will be used to provide storage for AI training data,
and the edge will be utilized for inference and decision making. This chapter presents
a brief discussion on data fusion models and edge intelligence. With the increasing
performance of intelligent systems, the opacity of the system has increased. The
article takes a deep dive into the explainability of artificially intelligent systems.
The fundamentals of XAI are presented here in brief. At the same time, reviewing
the application domain of AI, the chapter details the opportunities in data-centric
applications and outlines the challenges and their solutions in data fusion and edge
intelligence.

References

1. D.M. West, The future of work: Robots, AI, and automation. Brookings Institution Press (2018)
2. A.B. Arrieta, et al. Explainable artificial intelligence (XAI): concepts, taxonomies, opportuni-
ties and challenges toward responsible AI. Inf. Fusion 58, 82–115 (2020)
3. G. Yang, Q. Ye, J. Xia, Unbox the black-box for the medical explainable AI via multi-modal
and multi-centre data fusion: A mini-review, two showcases and beyond. Inf. Fusion 77, 29–52
(2022)
4. D. Castelvecchi, Can we open the black box of AI? Nature News 538(7623), 20 (2016)
5. Preece, D. Harborne, D. Braines, R. Tomsett, S. Chakraborty, Stakeholders in Explainable AI,
(2018)
6. F.K. Došilović, M. Brčić, N. Hlupić, Explainable artificial intelligence: A survey, in 2018 41st
International Convention On Information And Communication Technology, Electronics And
Microelectronics (MIPRO) (IEEE, 2018)
136 S. Sinha and P. Vashisht

7. Gunning, David. “Explainable artificial intelligence (xai).“ Defense advanced research projects
agency (DARPA), nd Web 2.2 (2017): 1.
8. J. Llinas, D. L. Hall, An introduction to multi-sensor data fusion, in Proceedings of the
International Symposium on Circuits and Systems, vol 6537–540 (1998)
9. Elmenreich, Wilfried, A review on system architectures for sensor fusion applications, in
IFIP International Workshop on Software Technolgies for Embedded and Ubiquitous Systems
(Springer, Berlin, Heidelberg, 2007)
10. M. D. Bedworth and J. O’Brien, The omnibus model: A new architecture for data fusion?
In Proceedings of the 2nd International Conference on Information Fusion (FUSION’99),
(Helsinki, Finnland, 1999)
11. D. Xu et al., Edge intelligence: Empowering intelligence to the edge of network, in Proc.
IEEE 109(11), 1778–1837 (2021)
12. W. Shi, J. Cao, Q. Zhang, Y. Li, L. Xu, Edge computing: Vision and challenges. IEEE Internet
Things J. 3(5), 637–646 (2016)
13. T.X. Tran, A. Hajisami, P. Pandey, D. Pompili, Collaborative mobile edge computing in 5G
networks: New paradigms, scenarios, and challenges. IEEE Commun. Mag. 55(4), 54–61
(2017)
14. P.G. Lopez et al., Edge-centric computing: Vision and challenges. ACM SIGCOMM Comput.
Commun. Rev. 45(5), 37–42 (2015)
15. Y. C. Hu, M. Patel, D. Sabella, N. Sprecher, and V. Young, Mobile edge computing—A key
technology towards 5G, ETSI White Paper, vol 11, no 11 (2015), pp. 1–16
16. F. Bonomi, R. Milito, J. Zhu, S. Addepalli, Fog computing and its role in the Internet of Things,
in Proceedings of the first edition of the MCC workshop on Mobile cloud computing (MCC),
(2012), pp. 13–16
17. R, P. Kansakar, S. U. Khan, IFCIoT: Integrated fog cloud iot: a novel architectural paradigm
for the future internet of things. IEEE Consum. Electron. Mag. 6(3), 74–82 (2017)
18. Edge Intelligence in 2022—Deep Learning And Edge Computing—viso.ai
19. A. Munir et al., Artificial intelligence and data fusion at the edge. IEEE Aerosp. Electron. Syst.
Mag. 36(7), 62–78 (2021)
20. D. Xu et al., Edge intelligence: Architectures, challenges, and applications. arXiv:2003.12172.
(2020)
21. T. Xing, S. S. Sandha, B. Balaji, S. Chakraborty, and M. Srivastava, Enabling edge devices that
learn from each other: Cross modal training for activity recognition, in Proceedings of the 1st
International Workshop on Edge Systems, Analytics and Networking (ACM, 2018), pp. 37–42
22. F. Shahmohammadi, A. Hosseini, C. E. King, M. Sarrafzadeh, Smartwatch based activity recog-
nition using active learning, in Proceedings of the Second IEEE/ACM International Conference
on Connected Health: Applications, Systems and Engineering Technologies (IEEE Press, 2017),
pp. 321–329
23. Mathur, T. Zhang, S. Bhattacharya, P. Velickovic, L. Joffe, N. D. Lane, F. Kawsar, P. Lio,
Using deep data augmentation training’ to address software and hardware heterogeneities in
wearable and smartphone sensing devices, in 2018 17th ACM/IEEE International Conference
on Information Processing in Sensor Networks (IPSN) (IEEE, 2018), pp. 200–211
24. Mathur, A. Isopoussu, F. Kawsar, R. Smith, N. D. Lane, N. Berthouze, On robustness of cloud
speech apis: An early characterization, in Proceedings of the 2018 ACM International Joint
Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and
Wearable Computers (ACM, 2018), pp. 1409–1413
25. Y. Bengio, A. Courville, P. Vincent, Representation learning: A review and new perspectives.
IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1798–1828 (2013)
26. Z. Chen, B. Liu, Lifelong machine learning. Synth. Lect. Artif. Intell. Mach. Learn. 10(3),
1–145 (2016)
27. R. Vilalta, Y. Drissi, A perspective view and survey of meta learning. Artif. Intell. Rev. 18(2),
77–95 (2002)
8 Explainable Data Fusion on Edge … 137

28. Soller, J. Wiebe, A. Lesgold, A machine learning approach to assessing knowledge sharing
during collaborative learning activities, in Proceedings of the Conference on Computer Support
for Collaborative Learning: Foundations for a CSCL Community. International Society of the
Learning Sciences, 2002, pp. 128–137
29. H. Li, K. Ota, M. Dong, Learning IOT in edge: Deep learning for the internet of things with
edge computing. IEEE Netw. 32(1), 96–101 (2018)
30. A. E. Eshratifar and M. Pedram, “Energy and performance efficient computation offloading for
deep neural networks in a mobile cloud computing environment, in Proceedings of the 2018
on Great Lakes Symposium on VLSI. (ACM, 2018), pp. 111–116
31. Y. Kang, J. Hauswald, C. Gao, A. Rovinski, T. Mudge, J. Mars, and L. Tang, Neurosurgeon:
Collaborative intelligence between the cloud and mobile edge, in ACM SIGARCH Comput.
Archit. News 45(1), (ACM, 2017)
32. W. Yang, S. Wang, J. Hu, G. Zheng, J. Yang, and C. Valli, Securing deep learning based edge
finger-vein biometrics with binary decision diagram, IEEE Trans. Ind. Inf. (2019)
33. J. Konecnˇ y, B. McMahan, D. Ramage, Federated optimiza- ‘ tion: Distributed optimization
beyond the datacentre. arXiv:1511.03575, (2015)
34. Y. Huang, Y. Zhu, X. Fan, X. Ma, F. Wang, J. Liu, Z. Wang, Y. Cui, Task scheduling with
optimized transmission time in collaborative cloud-edge learning, in 2018 27th International
Conference on Computer Communication and Networks (ICCCN). IEEE, (2018), pp. 1–9
35. M. van Lent, W. Fisher, and M. Mancuso, An explainable articial intelligence system for small-
unit tactical behaviour, in Proceedings of the 16th International Conference on Innovative
Application of Artificial. Intelligence, (2004), pp. 900–907
36. A. Adadi, M. Berrada, Peeking inside the black-box: A survey on explainable artificial
intelligence (XAI). IEEE access 6, 52138–52160 (2018)
37. D. Doran, S. Schulz, T.R. Besold, What does explainable AI really mean? A new conceptual-
ization of perspectives. arXiv:1710.00794, (2017)
38. Gilpin, Leilani H., et al., Explaining explanations: An overview of interpretability of machine
learning, in 2018 IEEE 5th International Conference on data science and advanced analytics
(DSAA). IEEE, (2018)
39. P. Rani, C. Liu, N. Sarkar, E. Vanman, An empirical study of machine learning techniques for
affect recognition in human–robot interaction. Pattern Anal. Appl. 9(1), 58–69 (2006)
40. A. Chander, R. Srinivasan, S. Chelian, J. Wang, K. Uchino, Working with beliefs: AI trans-
parency in the enterprise, in Workshops of the ACM Conference on Intelligent User Interfaces,
(2018)
41. P. Langley, B. Meadows, M. Sridharan, D. Choi, Explainable agency for intelligent autonomous
systems, in AAAI Conference on Artificial Intelligence, 2017, pp. 4762–4763.
42. Online Source: https://towardsdatascience.com/the-how-of-explainable-ai-post-modelling-
explainability-8b4cbc7adf5f. Accessed on 12 jul 2022
43. Online Source: https://www.infoworld.com/article/3634602/explainable-ai-explained.html.
Accessed on 14 Jul 2022
44. M.T. Ribeiro, S. Singh, C. Guestrin, “Why should I trust you?” Explaining the predictions of any
classifier, in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining, 2016, pp. 1135–1144
45. Haspiel, Jacob, et al. “Explanations and expectations: Trust building in automated vehicles.
In Companion of the 2018 ACM/IEEE international conference on human-robot interaction.
2018.
46. C. Howell, A framework for addressing fairness in consequential machine learning, in Proc.
FAT Conf., Tuts., (2018), pp. 12
47. R. Berk, J. Bleich, Statistical procedures for forecasting criminal behavior: A comparative
assessment. Criminol. Public Policy 12(3), 513–544 (2013)
48. J.R. England, P.M. Cheng, Artificial intelligence for medical image analysis: a guide for authors
and reviewers. Am. J. Roentgenol. 212(3), 513–519 (2019)
49. M.D. Zeiler, R. Fergus, Visualizing and understanding convolutional networks, in European
Conference on Computer Vision, (Springer, 2014), pp. 818–833
138 S. Sinha and P. Vashisht

50. K. Simonyan, A. Vedaldi, A. Zisserman, Deep inside convolutional networks: Visualizing

image classification models and saliency maps, in Workshop at International Conference on
Learning Representations, (2014)
51. R.R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, Gradcam: Visual
explanations from deep networks via gradient-based localization, in Proceedings of the IEEE
International Conference on Computer Vision, (2017), pp. 618–626
52. Z. Zhang, Y. Xie, F. Xing, M. McGough, L. Yang, Mdnet, A semantically and visually inter-
pretable medical image diagnosis network, in Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, (2017), pp. 6428–6436
53. S. Tonekaboni, S. Joshi, M.D. McCradden, A. Goldenberg, What clinicians want: Contextu-
alizing explainable machine learning for clinical end use, in Proceedings of the 4th Machine
Learning for Healthcare Conference, Proceedings of Machine Learning Research, vol. 106,
PMLR, Ann Arbor, Michigan, pp. 359–380, (2019), eds. by F. DoshiVelez, J. Fackler, K. Jung,
D. Kale, R. Ranganath, B. Wallace, J. Wiens
54. E. Khodabandehloo, D. Riboni, A. Alimohammadi, Health XAI: Collaborative and explainable
AI for supporting early diagnosis of cognitive decline, future gener. Comput. Syst. 116, 168–189
(2021)
Chapter 9
Trust Model Based Data Fusion
in Explainable Artificial Intelligence
for Edge Computing Using Secure
Sequential Discriminant Auto Encoder
with Lightweight Optimization
Algorithm

D. Prabakar, M. Sundarrajan, S. Prasath Alias Surendhar,

Manikandan Ramachandran, and Deepak Gupta

Abstract The amount of data generated, collected, and processed through computer
networks has increased exponentially in recent years. Network attacks have also
become an inherent concern in complex networks as a result of this increase of data.
The practise of assessing trust using attributes that influence trust is known as trust
evaluation. It is confronted with a number of serious challenges, including a shortage
of critical assessment data, a requirement for big data processing, a call for a simple
trust relationship expression, and the expectation of automation. Machine learning
(ML) has been applied to trust evaluation in order to overcome these issues and intel-
ligently and automatically evaluate trust. This research propose novel technique in
data fusion model with security and data optimization technique in edge computing.
Here the proposed data fusion is carried out using secure sequential discriminant
auto encoder in which the improvement of data accuracy, as well as for the maxi-
mizing of Edge-cloud based sensor networks lifespan. The fusion of edge cloud
data has been carried out using discriminant auto encoder which is integrated with

D. Prabakar
Department of CSE, SRM Institute of Science and Technology, Ramapuram Campus,
Chennai 600089, India
M. Sundarrajan
Department of CSE, SRM Institute of Science and Technology, Ramapuram Campus,
Chennai 600089, India
S. Prasath Alias Surendhar
Department of Biomedical Engineering, Aarupadai veedu Institute of Technology(AVIT),
Chennai, India
M. Ramachandran (B)
School of Computing, SASTRA Deemed University, Thanjavur, India
e-mail: [email protected]
D. Gupta
CSE Department, Maharaja Agrasen Institute of Technology, Delhi, India

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 139
A. E. Hassanien et al. (eds.), Explainable Edge AI: A Futuristic Computing Perspective,
Studies in Computational Intelligence 1072,
https://doi.org/10.1007/978-3-031-18292-1_9
140 D. Prabakar et al.

distributed edge cloud users, where the security of the network has been enhanced
using secure sequential fuzzy based trust model. The data optimization has been
established using Genetic swarm lightweight optimization algorithm. The experi-
mental analysis has been carried out based on data fusion as well as network security
model. The parametric analysis is carried out in terms of network security analysis,
throughput, Coverage fraction, Delay time, Energy consumption, Storage efficiency.

Keywords Data fusion · Security · Data optimization · Edge computing · Trust

evaluation

1 Introduction

Cloud computing (CC) has evolved into a viable alternative for providing an on-
demand platform for processing as well as sifting large amounts of data. Cloud
computing is being used in various fields, including but not limited to education,
finance, manufacturing, and healthcare [1]. As more methods rely on CC, novel-
methods that utilise benefits of CC while remaining lightweight and free of cloud’s
complexity are required to meet the properties of lightweight systems like IoT
devices. Although IoT devices can perform some essential activities like regulating,
actuating, and sensing, they cannot perform complex as well as sophisticated oper-
ations like controlling massive smart transportation methods or smart medical treat-
ments. Numerous IoT applications are time-critical, requiring fast decisions in order
to provide best possible performance. Examples include automotive networks as
well as aided healthcare systems [2]. Furthermore, CC is too sophisticated for these
devices to operate, and it lacks support for some of the most basic characteristics
of IoT methods, such as location awareness as well asbandwidth constraints. Fog
computing and edge computing are two novel technologies that can provide cloud
computing benefits while also addressing the unique peculiarities of IoT systems.
Although these new methods open the door to IoT system innovation as well as
growth, they also pose significant security and privacy concerns that may hinder IoT
system implementation and use [3].
With growth of IoT, a slew of smart applications are being developed that take
use of the ability to connect a variety of devices to internet. These apps will produce
a large volume of data, which will require to be analysed quickly in order to produce
useful as well as actionable data. EI refers to capacity to bring ML tasks from a remote
cloud closer to IoT devices, either partially or completely. Data fusion [4], also called
as data fusion or multi-sensor data fusion, refers to multisensor data resources. To
achieve goal of improving method stability, the system is able to obtain more precise
data. Data fusion is classified into three types based on the level of information
representation: datalevel fusion, featurelevel fusion, and decisionlevel fusion. The
datalevel fusion, which is the lowest level of fusion, immediately fuses collected
raw data, then extracts feature vector from fused data before being rated as well as
recognised.There is no risk of data loss as well as obtained result is most precise. In
9 Trust Model Based Data Fusion … 141

greenhouse WSNs, data fusion is a critical method for resolving low precision as well
as poor stability. It removes redundant dataas well as minimises data transmission
from various time and space multi-source data, meeting the goal of improving data
collecting accuracy as well as system stability.
Numerous activity areas that have adopted new data technologies have placed
Artificial Intelligence (AI) at the centre [5]. AI methods attains unprecedented levels
of performance while learning to solve gradually difficult computational tasks as
a result of these characteristics. AI-powered methods have lately advanced to point
where their design as well as implementation practically need no human intervention.
When such methods make decisions that affect people’s lives, there is a growing need
to comprehend how AI technologies deliver such data. While original AI methods
were simple to understand, opaque decision approaches such as DNNs (Deep Neural
Networks) have gained popularity in recent years. DL (Deep Learning) methods
like DNNs are empirically successful due to a combination of efficient learning
methods as well as their large parametric space [6]. However, a better understanding
of a method might lead to its shortcomings being corrected. Interpretability as an
additional method driver can enhance implementability of a machine learning model
for three reasons:
• Interpretability aids in ensuring objectivity in decision-making by detecting and
correcting bias in training dataset.
• Interpretability aids resilience by revealing potential adversarial disturbances that
could cause prediction to shift.
• Interpretability can serve as a guarantee that only relevant variables are utilized
to infer the result, i.e., that method reasoning is based on true causation.
All of this means that, in order to be regarded realistic, the system interpreta-
tion should provide either a knowledge of the model operations as well as predic-
tions, a visualisation of methods discrimination rules, or clues on what can disrupt
method. To prevent limiting effectiveness of today’s AI methods, eXplainable AI
(XAI) recommends developing a set of machine learning approaches. (1) build more
explainable models while preserving high learning performance (for example, predic-
tion accuracy), and (2) assist people in comprehending, suitably trusting, and effec-
tively managing the next generation of AI partners. XAI also analyses psychology
of explanation and draws on insights from Social Sciences [7].
Contribution of this chapter is as follows:
1. To propose novel technique in data fusion model with security and data
optimization technique in edge computing
2. To develop data fusion using secure sequential discriminant auto encoder in which
the improvement of data accuracy, as well as for the maximizing of Edge-cloud
based sensor networks lifespan
3. To enhance security using secure sequential fuzzy based trust model
4. To optimize the data transmission using Genetic swarm lightweight optimization
algorithm.
142 D. Prabakar et al.

2 Background

Statistical methods, hidden Markov methods, ANNs, and fuzzy logic are some of
solutions presented for IDS. Support vector machines (SVMs) were shown to be
highly accurate in creating IDS in recent studies [8]. However, its fundamental
drawback is that it necessitates a lengthy training period, limiting its use. Classic
data mining methods, such as association rule mining, are also used in IDS. For
example, to design and create IDS, [9] used rule-based approaches, where expert
knowledge is considered a rule set. As an analytical model, association rules were
utilised instead of human specialists. The extraction of a high number of association
rules, which improves model’s complexity, is a limitation of such methods. Feature
selection is one of the most important parts of a supervised classification model [10].
The computing time of the algorithms will be reduced by selecting essential features.
Many IDSs have been designed with feature selection because of the significant
aspects of network data [11]. In creating an IDS, work [12] classified main aspects
that are critical for real-world intrusion detection. In [13], the author used a feature
selection method to build a lightweight IDS. In big data framework for IDS, work
[14] created a PCA (principal component analysis) feature selection method. Work
[15] produced an IDS method that used a gain ratio as a feature selection strategy as
well as two classification methods, namely SVM as well as rule-based classification,
to identify class label as well as displayed greater accuracy levels for DoS attacks.
One disadvantage of this method is the high computational cost of using a separate
gain ratio and classification algorithm. They do, however, take a different approach
from the one taken in this SLR. In [16], the authors propose as well as assess a
framework based on distributed edge model that uses Docker method to create a
very lightweight as well as significant virtualization solution. Work [17] focuses on
relationship between DL as well as edge, whether it’s to utilise DL to optimise edge
or to execute DL algorithms on the edge. The research is separated into five sections:
edge DL applications, inference, edge computing and optimization. In [18], author
took a comprehensive look into edge intelligence from perspectives of edge caching,
training, inference and offloading [19] take a broad look at edge intelligence from
perspectives of edge caching, training, inference and offloading [20] created a multi-
agent method for boosting monitoring as well as security in cloud IoT environment.
Security is provided by intelligent agents sharing trust as well as certificates for
authentication.

3 System Model

This section discuss novel technique in data fusion model with security and data
optimization technique in edge computing. Here the proposed data fusion is carried
out using secure sequential discriminant auto encoder in which the improvement of
data accuracy, as well as for the maximizing of Edge-cloud based sensor networks
9 Trust Model Based Data Fusion … 143

lifespan. The fusion of edge cloud data has been carried out using discriminant auto
encoder which is integrated with distributed edge cloud users, where the security of
the network has been enhanced using secure sequential fuzzy based trust model. The
data optimization has been established using Genetic swarm lightweight optimization
algorithm.

3.1 Secure Sequential Discriminant Auto Encoder (SSDAE)

Based Data Fusion

Edge servers have far more resources and computing power than sensors and IoT
devices at the network’s edge. Because edge servers receive data from a variety of
edge of network devices with varying application needs, edge server performance
needs are similarly varied. Edge servers are created with high-performance fixed
evaluating servers with steady power sources as well as high bandwidth network
connections when AI/ML processing with severe latency constraints is required. Edge
servers are power/energy restricted for mobile edge computation. UAVs are utilised
as mobile edge servers or devices because they are often powered by batteries. For
quick and energy-efficient AI/ML computation, these mobile edge devices frequently
use AI accelerators. Furthermore, as shown in Fig. 1, edge servers or fog nodes are
more agreeable to the incorporation of ML accelerators such as CNN, RNN, and
MLP accelerators.
Each edge server is in charge of a cluster of sensors and IoT devices at network’s
edge. Edge servers offer edge-of-network IoT devices with apps, content, context,
services, and storage. Edge servers, for example, can help with complicated image

Fig. 1 Framework for data fusion and AI at edge

144 D. Prabakar et al.

processing as well as storage responsibilities for imaging data collected from IoT
devices’ camera sensors. Our framework’s edge servers are linked to a top-tier
centralised cloud server. The core network sends locally processed data from edge to
cloud for a variety of reasons, including analytics, archiving, and large-scale decision-
making. High-performance servers are commonly employed in cloud since the cloud
must fulfil a large number of requests from network edge devices and edge servers.
As shown by data fusion blocks at every hierarchical layer in Fig. 1, data fusion
plays an essential role in the proposed system alongside AI. We present a high-level
overview of data fusion at several tiers of proposed method here, whereas Section
III provides a thorough study of data fusion within data fusion blocks in Fig. 1. IoT
devices then employ AI to improve accuracy, performance, and energy efficiency
of AI jobs by fusing data. The data obtained from different IoT devices is then
fused by the edge servers. The edge servers also use topological, contextual, and
environmental data in data fusion to determine topological links between sensors.
The edge servers employ AI to process the fused data as well as then send sanitised
and fused data, analytics, to cloud. Data obtained from network edge is merged in
the cloud, and then AI is used to the merged data to generate global analytics and
insights. Edge servers and fog nodes can also offload duties to the cloud.
Utilizing2 blocks: an encoder as well as decoder, we aim to learn probability
distribution (PD) of data: X in the VAE architecture. The encoder uses q(Z/X) to map
X . Decoder block will work in same way as encoder, transferring latent variables to
a new set of specifications that define a new set of related PDs: p(Xb/Z), from which
we will gather samples once more. Our network’s output will be these final samples:
Xb. The end goal is to resemble the network’s input and output, X and Xb, as closely
as feasible. However, to achieve that goal, we must map data’s internal structure to
PDs q(Z/X) and p(Xb/Z). Conditional PD p(Xb/Z) and q(Z/X) model likelihood of
Xb and Z but are dependent on their certain inputs: Z and X are the letters Z and
X, respectively. We train the PD: q(Z/X) and p(Xb/Z) in a VAE by employing a
variational technique, which converts learning method to a reduction method, which
can be simply expressed in terms of SGD in a NN. Method Specifications: θ and
ϕ are utilized to reflect architecture as well as weights of NN employed in Fig. 2.
These parameters are fine-tuned during the VAE training process as well as treated
as constants.
We strive to maximise the chance of getting the desired data as an output in the
variational approach by maximising a number known as Evidence Lower Bound
(ELBO) [22]. ELBO is made up of 2 parts: (1) a measure of distance between
PD of q(Z/X) and some similar reference PD, where distance is usually Kullback–
Leibler (KL) divergence; and (2) log likelihood of p(X) under PD p(Xb/Z), which is
probability of obtaining desired data (X) with final PD that generatesXb. All learnt
PDs are parameterized PD, which means that a set of parameters entirely defines
them. This is critical for the model’s operation because we use these parameters,
which are obtained as network node values, to represent corresponding PD: p(Xb/Z)
and q(Z/X).
Parts of loss function to be minimised using SGD for ID-CVAE method are
shown in Fig. 2. As previously stated, the loss function is divided into 2 parts: a KL
9 Trust Model Based Data Fusion … 145

Fig. 2 Comparison of SSDAE with a typical VAE method

divergence as well as log likelihood portion. Second portion uses the distribution
p(Xb/Z, L), which is a distance between X and Xb, to determine how likely it is to
generate X. By decreasing this distance, we are effectively preventing q(Z/X) from
deviating too far from its previous, and therefore functioning as a regularisation
term. This regularisation term has the advantage of being automatically modified,
therefore cross-validation is not required to adjust a hyper-parameter connected with
regularisation, as it is in other methods.
We utilize multivariate Bernoulli distribution for distribution p(Xb/Z, L). Because
the output specifications that describes distribution is mean, which is same as likeli-
hood of success, the Bernoulli distribution does not require a final sampling. Ground
truth X, which has already been scaled to [0–1], can be regarded as a [0–1] scaled
value for this likelihood. The distributions chosen for q(Z/X) and p(Xb/Z, L) are in
line with ones selected; they are simple as well as produce decent results. Boxes in
lower half of loss function’s certain choice. This is a specific choice for generic loss
function shown in Fig. 2.
Before entering following section, denote sample set {xi , yi }i=1 n
by a matrix form
X = [x1 ; x2 ; . . . ; xn ] ∈ R and Y = {y1 ; y2 ; . . . ; yn } ∈ R . Here, yi is an r-
nx n×r

dimensional one-hot row vector with a value of 1 in ith dimension and 0 in other
dimensions, and r is number of fault categories.
DIAE encoding method is same as AE method discussed in Sect. 3.1, in that it
uses Eq. to translate the input sample xi into
[ a hidden layer (1).] While introducing
an extra dimension to sample x i as: xi = xi1 , xi2 , . . . , xin−1 , 1 . The effect will be
the same as if you used bias term in Eq. (1).
Unlike traditional AE paradigm, DIAE decoding method is divided into two
stages. One method is to use the hidden representation h i Eq. (1) to reconstruct
input data:
( )
x̂i = g h i WxT (1)
146 D. Prabakar et al.

The other element is the input data prediction using h i Eq. (2):

ŷi = g(h i WT ) (2)

The discriminant weight matrix from hidden layer to output yi is WT ∈ R m×r .

In the DIAE model, Eq. (4) can be used to introduce discriminant information.
Furthermore, we include a symmetric matrix L ∈ R r ×r into the DIAE method to
capture structural data among various fault types. Degree of relatedness between each
pair of fault categories is indicated by each element of L. The input data prediction
can then be rewritten as Eq. (3):

ỹi = ŷi L = g(h i WT )L (3)

We introduce structural information as well as discriminant information among

different fault kinds from Eq. (4). The loss function of DIAE is constructed as follows
using Eq. (5):

α Σ 1 Σ ( ( ))
n n
J= (xi − x̃i )2 + −κσ yi , ŷi L
2n i=1 n i=1
β λ( )
+ ∥L − L T ∥2F + ∥Wx ∥2F + ∥WT ∥2F (4)
2 2
where α, β, λ are regularization parameters. Determine the optimal W X , Wr and L
by minimising J, because the matrices Wx∗ , WT: and L − are all randomly initialised.
Every term in Eq. (6) is described as follows for a better understanding:
• Reconstruction error is 2n 1 Σ n
i=1 (x i − ẋ i ) , which is the same as the traditional
2

AE. By reducing the use of this phrase. The ability of features to represent input
data is enhanced.
• Discriminant
( ( information
)) is represented by the second term
1 Σ n
n i=1 −κ σ yi , ŷi L . The extracted features’ discriminant ability would
be improved if this term was reduced. Here κσ (yi , yi L) is Gaussian kernel by
Eq. (5):

( ) (√ )(√ ) ( ( )2 )
κσ yi , ŷ L = 1/1 2πσ 2πσ exp − yi − ŷi L /2σ 2 (5)

Σ n
As pointed by n1 i=1 κσ (a, b) are viewed as an approximate evaluation of
correntropy Vσ (A, B) Eq. (6):

Vσ ( A, B) = E(κθ (A, B)) = ∫ κθ (A, B)d FAB (a, b) (6)

where A = [a1 , a2 , . . . , a N ]T and B = [b1 , b2 , . . . , b N ]T is two stochastic variables,

E(κσ (A, B)) is expectation of κσ (A, B), κσ (·, +) is Mercer kernel, and FAB (a, b)
9 Trust Model Based Data Fusion … 147

is joint PDF. The PDF of finite samples is uncertain, as it is in many real-world

situations, hence the above approximate estimate of correntropy is frequently used.
Minimizing 2nd term in Eq. (8) is thought to be capable of enhancing discriminant
ability of extracted features.
We simplify reduction of Eq. (7) by utilising matrix form, as follows:

min J1 + J2 + J3 + J4 (7)
wx ,w1 L

where
( ( ))
J1 = −tr K σ Ŷ L , Y
α
J2 = ∥ Ẋ − X ∥2F
2
β (8)
J3 = ∥L − L T ∥2F
2
λ( )
J4 = ∥W X ∥2F + ∥W I ∥2F
2
To solve Eq. (9), we use a different optimization strategy. Fix Wγ and L first, then
optimise Wx using the gradient descent algorithm: Second, fix Wx and L then opti-
mise W I using the gradient descent approach. Finally, fix W X and WT , then optimise
L using the gradient descent approach. These three phases run in order until conver-
gence, starting with initialising W X , Wr , L. The following equations can be used to
calculate the values of W X , WT , L:

∂J
Wτ = W T + η
∂ WT
(9)
∂J
L = L +η
∂L

where η is learning rate. Partial derivatives of J based on Wx , WT , L, i.e. d

Wx
, d
Wτ
, πA
by Eq. (10)

∂J ∂ J1 ∂ J2 ∂ y ∂1 ∂ WT
= + = +
∂ WX ∂ WX ∂ W X ωT WT WT
c T (( ( ) ) )
= 2H D Ỹ L − Y L T ⨀ dg(H WT ) + λWT (10)
σ
∂J ∂ J1 ∂ J4 C ( ) ( )
= + = 2 Ŷ T D Ŷ L − Y + β L − L T
∂L ∂L ∂L σ

H = [h 1 ; h 2 . . . . . . , h n ] ∈ R n×m is output of hidden layer, ⨀ is Hadamard product.

We reduce the often appearing formulas in Eq. (11) to make it easier to show the
derivative results:
148 D. Prabakar et al.
( )
B = − Ŷ L − Y (Ŷ L − Y )T /2σ 2
√
C = −1/ 2π σ D = 1 ⨀ exp(B)
( ( ) ) (11)
G = D Ŷ L − Y L T ⨀ dg(H WT )
( ) ( )
E = X̂ − X ⨀ dg H W XT

Algorithm: 1—SSDAE:
{ }n
Input: Training sample set {X, Y } = x i: y i i=1 , the number of hidden layers K , neuron
{ }K
number in every hidden layer m, j −1 , maximum iteration number max_iter
{ }K
Output: Weight matrix of every hidden layer W x = W ix i=1 , and output of every hidden
{ }K
layer H i i=1
1. Evaluate H 0 = X
2. for i ← 1, K do
3. Start randomly W Xi and WTi ; Start L as a unit diagonal matrix
4. Construct DlAE network with m i hidden neurons:
5. Set iter = 1;
,
6. Utilize an alternative optimization technique to update W X , WTl , L ,
7. while iter < max iter do
8. If difference of loss function J in 2 consecutive iterations is smaller than threshold, break loop:
9. iter ← iter + 1:
10. end while
11. Build i - th hidden layer of SDIAE utilizing Wxi :
( )
12. Evaluate H i utilizing H i = g H i−1 W Xi :
13. X i+1 ÷ H i
14. end for
15. return W X

3.2 Secure Sequential Fuzzy Based Trust Model (SSFTM)

Resource constraints in the edge computing environment frequently result in large

disparities in number of requests as well as responses to services. Some devices
refuse to reveal identity data out of concern for privacy, but still want to interact;
some require a high-level security method due to real-time services; and some only
expect to benefit from resource interaction. As a result, establishing a targeted trust
evaluation strategy is difficult. Furthermore, relying on a single factor to assess
trust is unreliable. The authentication process, for example, normally has only two
outcomes: pass or fail. To highlight the complexity of edge devices in executing
9 Trust Model Based Data Fusion … 149

resource interaction or application service, we must analyse many trust metrics with
multiple critical attributes. The proposed approach for assessing trust is based on
Fig. 4. If there have been multiple contacts between ECUs clustered by common
tasks in past period of time, the corresponding trust degree of ECX is anticipated
using a capsule network to examine trust value of every ECU. As a result, we can
determine the overall level of confidence in the edge computing ecosystem.
The trust property of ECUs is described as Eq. (12):

Vt =< Vi, Vb, Vc > (12)

where Vi, Vb, Vc represents identity trust, behavior trust and capacity trust.
(1) Identity Trust: When assessing a device’s identification trustworthiness, privacy
concerns must be taken into account. Exposing a device’s ID, for example,
exposes it to risk of identity theft, fraudulent data production, and so on. In
edge computing, public key infrastructure (PKI) has been widely employed
as a trustworthy authentication technique that can readily implement one-to-
one interaction in a distributed context. Identification trust Vi is defined as the
believability of an ECU’s identity during the resource request procedure. To
compute Vi, we assume that ECUs’ identity trust is assured by an anonymous
authentication mechanism based on ECC, which is utilized to evaluate each
ECU’s key. ECC generates a discrete elliptic curve E first, then generates public
as well as private keys based on one point O(x, y) on E in following 2 steps:
Step 1: Creating a Server Key The local edge server generates a key pair (Kp,
Ks) as shown in Eq. (13):

K p = H (Il , Ie )

( )
Ks = M K p (13)

The hash function H calculates the public key Kp based on the ID Il of the
edge server and the ID Ie of the ECUs. According to Kp, the private key Ks is
calculated using the encoding function M. Following the calculation of Kp and
Ks, the local edge server provides identity data D = {Kp, E, O} to ECUs.
Step 2: ECUs Key Generation: After the ECUs get D as Eq. (14), three keys
Kvi, Kep, and Kes are calculated:

K vi = R1 (O(x, y))K ep = R2 (O(x, y))

K ep = R2 (O(x, y)) (14)

where virtual key is used Kvi is used to safeguard ECUs’ genuine IDs, whereas
Kep and Kes are ECUs’ public and private keys, respectively, and R1() and R2()
150 D. Prabakar et al.

are PRNG functions. Then ECUs sends a message to the local edge server with
the format F = {IDe, Kv, Kep, a}, where an is a nonce integer. The public key
is used to contact edge servers and ECUs in the later authentication step, while
private key is used to complete identity matching. Vi is set to 1 if the ECUs pass
the authentication; otherwise, Vi is set to 0.
The proposed framework would calculate trust values utilizing fuzzy rules,
with fuzzy inputs being objective and subjective trust values as well as fuzzy
outputs being trust values.Low, medium, and high are the 3 states of fuzzy
outputs, and their ranges are 0 to 1. CSPj trust value maps to distrust if the
obtained trust value is low. CSPj trust value maps to trust if the acquired trust
value is high, and uncertain if the obtained trust value is low.
We define behaviour trust of ECUs as Vb as a weighted total of three terms by
Eq. (15), taking into account the process of service request as well as response:

Vb = wbc Vbc + wbe Vbc + wbr Vbr (15)

where Vbc stands for behaviour constraint, Vbe for behaviour experience, and
Vbr for behaviour change rate. wbc+wbe+wbr = 1 are weight coefficients that
can be modified depending on the task at hand.
(2) Behavior Constraint: Varying ECUs observe different numbers of behaviour
restrictions or interaction specifications, which are found by certain tasks using
Eq. (16):
Vbe = w1 ξ1 + w2 ξ2 + · · · + we ξc (16)

Σ c
where ξi is constraint or specification that ECUs follows, i=1 wi = 1 are
weight coefficients.
Behavior Experience: Because the edge computing environment is always
evolving, it is vital to evaluate trust changes on a regular basis.ECUc is used to
define the ECUs that will be examined.Behavior experience Vbh indicates the inter-
active credibility of the recent past for interacting behaviours of ECUc with many
ECUs, where minimum engagement can also lead to effective trust development.
Assume that ECUs have interacted with k other ECUs, namely ECU1, ECU2, and
ECUk. Let’s see what happens if we use Eq. (17)
{ }
s j (Δt) = τ j1 , τ j2 , . . . , τ js , . . . , τ jt (17)

where 0 ≤ τ js ≤ 1 is the interaction observation between itself and ECUj during

time unit s. Assign a threshold to ECUj based on the size of interactive resources
α j ∈ (0, 1). ( )
Σ t
H j+ (Δt) = i=1 1 τ ji ≥ α j if the condition is true, the 1(.) function returns 1;
otherwise, it returns 0. Equation (18) can be used to define Vbe
9 Trust Model Based Data Fusion … 151

+
1 Σ H j (Δt) Ri (t)
k
Vbe = (18)
k j=1 t F̂i

Rate of Behavior Change: Because behaviour varies over time, Vbr, which indicates
the real change in behaviour trust, is defined as the rate of behaviour change. There
are last time series Δ(t − 1), mainly consider t − H j+ (Δ(t − 1)) of ECU j with
ECUc . For the time series Δ(t), the basic exponential smoothing factor is H j+ (Δt)
of ECU j with ECUc , λ.
Because ECUs typically do not keep a lot of previous data due to restricted
resources, the rate of behaviour change is based on two time series: Δ(t − 1), and
Δ(t). We believe historical data that is closer to the present moment to be more
suggestive, so recent data is given more weight Δ(t).
Capability Trust is described as degree of confidence in a device’s capability
property, such as accessibility, available bandwidth, response speed, and so on.
L = {e1 , e2 , . . . , en } is a set of ECU gives an ECUs . C = {c1 , c2 , . . . , cm } is
capability
{ property set of } ECU. Value of m of any ECU under an ECUn . is identical.
Cs = cs1 , cs2 , . . . , csm indicates any cak ∈ Cn requires based
⟨ on ck ∈ C, where cak⟩
is standard value preset based on task requirements. Vs = V⟨s = Cs1 , Cs2 , . . . ,⟩ Csn
notes a vector of ECUs which contains Cs of ECU , and V = C 1 , C 2 , . . . , C n is a
vector where C is actual C s when ECU is working.
The correlation coefficient of Vs and V is described as ρ. We propose that value
of denotes capacity trust of ECU s, which can be expressed as coefficient vectors
with the values of Vc · ωvT and ωv, T
.
CCA [12] method is used to evaluate ρ as follows:
{ }K
Wx = Wxi i=1 .

3.3 Data Optimization Using Genetic Swarm Lightweight

Optimization Algorithm
The gateways are relay nodes that must be monitored to ensure that the load on edge
servers is balanced. An appropriate Gateway-Edge setup is necessary for this. For an
in-depth analysis of network traffic, we use a variety of Key Performance Indicators
(KPIs) to determine the best configuration. The transmission path for each gateway
is determined by an SDN controller. It checks the load on each server and raises an
alarm if it exceeds a certain threshold value, requiring the existing Gateway-Edge
(GE) connection to be re-configured. Determining ideal balanced GE configuration
at time t + 1 is a key.GE configuration at a given time t can be given by a vector G
t = {G t 1, G t 2,… , G t n}, where G t n ∈ {1, 2,…, N}. G t n = m, for example,
indicates that at time t, the nth gateway is communicating to mth server.We use two
KPIs to solve Gateway-Edge configuration issue: the network’s Average Residual
Energy (KPIARE) and the servers’ Load Fairness Index (KPILFI). To maximise NP
at time t, the normalised weighted sum of these two KPIs is used, as stated in Eq.
(19).
152 D. Prabakar et al.

Max(N P) = α K P I A R E + β K P I L F I (19)

NP is an optimization problem’s principal objective function, α and β are weights

allocated to every KPI. As indicated in Eq. (20), these weights gives priority level of
every KPI in goal function.

( ) (Σ Σ )2
M N
1 Σ n=1 In,i ϕn (t)\ϕmax
Nn
Ri (t) 1 i=1
NP = α + β( ( )2 (20)
Nn i=1 E M Σ M Σ N
i=1 n=1 I n,i ϕn (t)\ϕmax

Ri (t)
where Ê
is residual energy of a sensor node I at time t and is described as ratio of
node i’s remaining energy (Ri(t)) to each node’s initial energy (Ê) at time t. At the
moment of deployment, Ê is the same for all nodes in the network. We examine load
fairness at the edge servers for the second KPI. In, i is a binary indication, meaning
it is 1 if an nth gateway sends φn packets to ith server at time t, and 0 otherwise.
With an enhance in number of M and N, number of alternative configurations grows
exponentially. We use evolutionary methods, such as GA and DPSO, to address GE
configuration as an optimization issue. To reach an optimal setup as well as balanced
load, these algorithms go through the processes below.
1. Produce random population R 0 of size Δ. Best possible position for every
particle, i.e., Gateway, is started such that Pbest i0 = ri0 , ∀1 ≤ i ≤ Δ
2. Determine each particle’s fitness value for DPSO and every chromosome’s fitness
value for GA in R 0 and its global best position Gbest0 by Eq. (21)
( )
Gbest0 = arg max F Pbestli (21)
1≤i≤Δ

3. If best candidate solution for GE configuration is found or maximum number of

generations is achieved, search for GA is completed; otherwise, Step 4 is carried
out. If best candidate solution is found in DPSO, particle velocities in current
population must be updated utilizing Eq. (22).
( ) ( )
viI = jw viI −1 + a1 r1 PbestiI − xil + a2 r2 GbestiI − xil . (22)

xI Igives particle I’s current position at the I th iteration, r1 and r2 are random
variables in (0, 1) range, a1 and a2 are acceleration constants utilized to pull
particles toward best position, and jw reflects inertia effect of the preceding
particle’s velocity.
4. Best accessible γ chromosomes from the present population are then extracted
for GA. Present population is R I, and probability of selection is Ps. Iteration
number is simply updated in DPSO. i.e., I = I + 1.
5. In case of GA, crossover as well as mutation are done on γ. If best candidate
solution for GE configuration is found in DPSO, search is completed; otherwise,
Step 6 is performed.
9 Trust Model Based Data Fusion … 153

6. All of the processes from Step 2 are repeated for GA. In DPSO, each particle’s
personal best position is updated utilizing Eq. (23).
⎧ ( )
PbestiI −1 , i f F riI ≤ F PbestiI −1 )
PbestiI = (23)
riI , otherwise.

7. The global optimum position for DPSO is updated using Eq. (24).
⎧ ( ) ( ) ( )
⎨ arg max F Pbestli , i f F Pbestli > F Pbestl−1
i
Gbestli = 1≤i≤Δ
(24)
⎩ Gbestl−1 , otherwise.
i

8. For DPSO, repeat all steps from Step 1.

Based on NP as well as KPI values, SDN controller continuously configures
the Gateway-Edge. The above technique can be applied to edge servers that are
homogeneous or have the ability to estimate at nearly identical times. It does not
take into consideration dynamic scenarios in which some data is processed ahead
of others. As a result, a service migration method is suggested to further balance
load—which is doable because most edge servers run Linux-based OS.

Algorithm: 2—GSLOA
Input: dynamic CPU and memory utilization threshold
values i.e.U t c and U t m , respectively—computed periodically using Eq. (12); channel
condition C t
Output: migration list map → input for load optimization
module in Fig. 3
1. compute CPU utilization level of the edge node (E Nuc );
2. compute memory utilization slevel of the edge node (E Num );
3. compute channel condition (Cc )
4. For each node edge do
5. Select application m from edge node
6. Choose edge node n as destination node
7. Map = m,n
8. End if
9. End for
10. Return map

Fig. 3 Details on loss function elements for SSDAE method

154 D. Prabakar et al.

Fig. 4 Illustration of edge computing architecture

4 Performance Analysis

Performance of trust evaluation with data fusion as well as data optimization

approaches is addressed in this section. The experimental infrastructure for analysing
the suggested trustworthy data gathering algorithm was sbuilt using MATLAB
R2018a. The experimental setup consists of randomly placing 100 nodes in a 300
× 300 m area as well as selecting 30 nodes to serve as CH nodes. Experimental
setup includes two IoTs, one cloud, and one BS, with each IoT having its own edge
platform as well as cloud sitting on top of IoT. Users are positioned in lowest layer of
the cloud, where underlying nodes are clustered independently as well as adjacent to
the BS nodes. The data transfer rate is 2 × 106 bit/s, and the transmission time from
edge node to cloud is sixteen milliseconds. Moving edge node is considered to move
at a constant speed from the chosen beginning point, speed as well as communication
radius are changed. Following creation of initial path without crossover, a path with
highest trust value within required moving distance is constructed.
Tables 1, 2, 3, 4, 5 and 6 shows the comparative analysis between existing and
proposed technique in terms of network security analysis, throughput, Coverage
fraction, Delay time, Energy consumption, Storage efficiency. Here comparison has
9 Trust Model Based Data Fusion … 155

been carried out for enhancing data fusion model with security and data optimization
technique in edge computing. The proposed technique obtained network security
analysis of 90% for 500 epochs since the analysis has been carried out based on
number of epochs in NN as represented in Fig. 3. Then the throughput analysis of
proposed technique has been carried out and it obtained 98% for 500 epochs as
shown in Fig. 4. The analysis of coverage fraction obtained of 81% for 500 epochs
as shown in Fig. 5. The proposed technique obtained as 65% for 500 epochs of
energy consumption as shown in Fig. 6. Delay time of proposed technique is 55%
as shown in Fig. 7 for 500 epochs. Comparative analysis has been carried out for
storage efficiency of 92% for 500 epochs as shown in Fig. 8. From the above analysis,
proposed technique obtained optimal results (Figs. 9 and 10).

Table 1 Comparative analysis of network security analysis

Number of nodes SVM PCA_IDS SSDAE_SSFTM-GSLOA
100 65 71 75
200 69 75 79
300 71 79 83
400 75 81 85
500 78 83 90

Table 2 Comparative analysis of throughput

Number of nodes SVM PCA_IDS SSDAE_SSFTM-GSLOA
100 75 81 85
200 79 83 89
300 81 86 91
400 83 89 95
500 85 93 98

Table 3 Comparative analysis of coverage fraction

Number of nodes SVM PCA_IDS SSDAE_SSFTM-GSLOA
100 61 65 68
200 63 69 71
300 65 73 75
400 69 75 79
500 71 79 81
156 D. Prabakar et al.

Table 4 Comparative analysis of energy consumption

Number of nodes SVM PCA_IDS SSDAE_SSFTM-GSLOA
100 71 65 55
200 73 68 59
300 75 69 61
400 79 72 63
500 80 75 65

Table 5 Comparative analysis of Delay time

Number of nodes SVM PCA_IDS SSDAE_SSFTM-GSLOA
100 61 51 45
200 63 53 49
300 64 55 51
400 65 59 53
500 69 61 55

Table 6 Comparative analysis of storage efficiency

Number of nodes SVM PCA_IDS SSDAE_SSFTM-GSLOA
100 66 75 81
200 69 79 83
300 75 83 85
400 79 85 89
500 81 89 92

Fig. 5 Comparison of network security analysis

9 Trust Model Based Data Fusion … 157

Fig. 6 Comparison of throughput

Fig. 7 Comparative analysis of coverage fraction

5 Conclusion

This research proposes novel technique in data fusion model with security and data
optimization technique in edge computing. Here the proposed data fusion is carried
out using secure sequential discriminant auto encoder in which the improvement of
data accuracy, as well as for the maximizing of Edge-cloud based sensor networks
lifespan. The fusion of edge cloud data has been carried out using discriminant auto
158 D. Prabakar et al.

Fig. 8 Comparative analysis of energy consumption

Fig. 9 Comparative analysis of delay time

encoder which is integrated with distributed edge cloud users, where the security of
the network has been enhanced using secure sequential fuzzy based trust model. The
data optimization has been established using Genetic swarm lightweight optimization
algorithm. The experimental analysis is carried out based on data fusion and network
security model. The parametric analysis is carried out in terms of network security
analysis of 90%, throughput of 98%, coverage fraction obtained of 81%, Energy
9 Trust Model Based Data Fusion … 159

Fig. 10 Comparative analysis of storage efficiency

consumption of 65%, Delay time of 55%, Storage efficiency of 92%.The limitation

of this research is to enhance security as well as enhance data transmission rate.
In future this technique can be implemented for industrial based application and
medical application with enhanced security and storage efficiency.

References

1. W. Mao, W. Feng, Y. Liu, D. Zhang, X. Liang, A new deep auto-encoder method with fusing
discriminant information for bearing fault diagnosis. Mech. Syst. Signal Process. 150, 107233
(2021)
2. M. Lopez-Martin, B. Carro, A. Sanchez-Esguevillas, J. Lloret, Conditional variational autoen-
coder for prediction and feature recovery applied to intrusion detection in iot. Sensors 17(9),
1967 (2017)
3. C. Jia, K. Lin, J. Deng, A multi-property method to evaluate trust of edge computing based
on data driven capsule network. In IEEE INFOCOM 2020-IEEE Conference on Computer
Communications Workshops (INFOCOM WKSHPS), (IEEE, 2020), pp. 616–621
4. M.A. Jan, M. Zakarya, M. Khan, S. Mastorakis, V.G. Menon, V. Balasubramanian, A.U.
Rehman, An AI-enabled lightweight data fusion and load optimization approach for Internet
of Things. Futur. Gener. Comput. Syst. 122, 40–51 (2021)
5. B. Mahbooba, M. Timilsina, R. Sahal, M. Serrano, Explainable artificial intelligence (xai)
to enhance trust management in intrusion detection systems using decision tree model.
Complexity (2021)
6. A.B. Arrieta, N. Díaz-Rodríguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. García, S.
Gil-López, D. Molina, R. Benjamins, R. Chatila, F. Herrera, Explainable Artificial Intelligence
(XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Inf. Fusion
58, 82–115
160 D. Prabakar et al.

7. A. Krajna, M. Brcic, M. Kovac, A. Sarcevic, Explainable artificial intelligence: An updated

perspective. Under Rev (2022)
8. F. Hussain, R. Hussain, E. Hossain, Explainable artificial intelligence (XAI): An engineering
perspective. arXiv:2101.03613 (2021)
9. G. Vilone, L. Longo, Notions of explainability and evaluation approaches for explainable
artificial intelligence. Inf. Fusion 76, 89–106 (2021)
10. T. Rojat, R. Puget, D. Filliat, J. Del Ser, R. Gelin, N. Díaz-Rodríguez, Explainable artificial
intelligence (xai) on timeseries data: A survey. arXiv:2104.00950 (2021)
11. A. Holzinger, M. Dehmer, F. Emmert-Streib, R. Cucchiara, I. Augenstein, J. Del Ser, N.
Díaz-Rodríguez, Information fusion as an integrative cross-cutting enabler to achieve robust,
explainable, and trustworthy medical artificial intelligence. Inf. Fusion 79, 263–278 (2022)
12. C.K. Leung, F. Jiang, Y. Zhang, Explainable machine learning and mining of influential patterns
from sparse web. In 2020 IEEE/WIC/ACM International Joint Conference on Web Intelligence
and Intelligent Agent Technology (WI-IAT). (IEEE, 2020), pp. 829–836
13. N. Mehdiyev, P. Fettke, Explainable artificial intelligence for process mining: A general
overview and application of a novel local explanation approach for predictive process
monitoring. Interpretable Artif. Intell. Perspect. Granular Comput. 1–28 (2021)
14. I. Kakogeorgiou, K. Karantzalos, Evaluating explainable artificial intelligence methods for
multi-label deep learning classification tasks in remote sensing. Int. J. Appl. Earth Obs. Geoinf.
103, 102520 (2021)
15. D. Thakker, B.K. Mishra, A. Abdullatif, S. Mazumdar, S. Simpson, Explainable artificial
intelligence for developing smart cities solutions. Smart Cities 3(4), 1353–1382 (2020)
16. A. Anguita-Ruiz, A. Segura-Delgado, R. Alcalá, C.M. Aguilera, J. Alcalá-Fdez, explainable
Artificial Intelligence (XAI) for the identification of biologically relevant gene expression
patterns in longitudinal human studies, insights from obesity research. PLoS Comput. Biol.
16(4), e1007792 (2020)
17. S.R. Islam, W. Eberle, S.K. Ghafoor, A. Siraj, M. Rogers, Domain knowledge aided explainable
artificial intelligence for intrusion detection and response. arXiv:1911.09853 (2019)
18. C. Dindorf, J. Konradi, C. Wolf, B. Taetz, G. Bleser, J. Huthwelker, F. Werthmann, E. Bartaguiz,
J. Kniepert, P. Drees, U. Betz, M. Fröhlich, Classification and automated interpretation of spinal
posture data using a pathology-independent classifier and explainable artificial intelligence
(Xai). Sensors, 21(18), 6323
19. Y. Xu, Y. Wu, H. Gao, S. Song, Y. Yin, X. Xiao, Collaborative APIs recommendation for
artificial intelligence of things with information fusion. Futur. Gener. Comput. Syst. 125, 471–
479 (2021)
20. S. Atakishiyev, H. Babiker, N. Farruque, R. Goebel, M.Y. Kima, M.H. Motallebi, J. Rabelo, T.
Syed, O.R. Zaïane, A multi-component framework for the analysis and design of explainable
artificial intelligence. arXiv:2005.01908 (2020)
Chapter 10
A Deep Learning Based Target Coverage
Protocol for Edge Computing Enabled
Wireless Sensor Networks

Pooja Chaturvedi, A. K. Daniel, and Umesh Bodkhe

Abstract The sensor networks have attracted a numerous research attention due to
its diverse applications ranging from surveillance and monitoring applications. The
sensor nodes are usually characterized as having scarce resources; hence energy effi-
cient mechanisms which can enhance the resource utilization are of great significance.
The integration of edge computing framework with the sensor network can aid in the
data collection, dissemination and decision making. Scheduling approaches which
divide the nodes into a number of set covers and monitor the given points of interest
with the desired confidence level along with the objective of maximizing coverage
and network lifetime have been proved a prominent approach. The determination of
set covers is a NP hard problem and is dependent on different network parameters
such as node contribution, trust values and coverage probability. In this scheme, the
node has to monitor the neighboring node parameters at regular intervals, which
incurs a huge number of communication overhead. The nodes in sensor network can
employ the learning strategy to determine its best possible action to enhance the
network coverage as well as network lifetime. The chapter proposes a LSTM based
strategy for an edge computing enabled WSN to determine the status of the node
depending on the network parameters such as number of communications, number of
packets transmitted and initial energy of the nodes. The proposed protocol is imple-
mented using tensor flow and keras libraries in the python language. The keras tuner
package has been used to determine the best parameters such as number of hidden
layers and number of neurons in each layer. The obtained parameters are used to
construct a hyper model and the efficiency of the model is evaluated in terms of
the loss function. The explainability of the proposed model is investigated using the

P. Chaturvedi (B) · U. Bodkhe (B)

Institute of Technology, Nirma University, Ahmedabad, Gujarat, India
e-mail: [email protected]
U. Bodkhe
e-mail: [email protected]
A. K. Daniel
Madan Mohan Malaviya University of Technology, Gorakhpur, India
e-mail: [email protected]

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 161
A. E. Hassanien et al. (eds.), Explainable Edge AI: A Futuristic Computing Perspective,
Studies in Computational Intelligence 1072,
https://doi.org/10.1007/978-3-031-18292-1_10
162 P. Chaturvedi et al.

Local Interpretable Model-agnostic Explanations (LIME) framework and the effect

of all the features on the status prediction have been determined.

Keywords Target coverage · Network lifetime · LSTM · Deep learning · Energy

efficiency · Edge enabled WSN

1 Introduction

WSNs are extensively applicable in different environmental investigation and

reporting applications due to their minimal energy need, lesser cost and diverse
applications. The different issues arises in the sensor network incorporating a large
number of nodes such as increased energy consumption, increase latency, low QoS
and more communication overhead.
Apart from these issues, ensuring the coverage to the environmental parameters
is also a major issue. The node scheduling strategy is often used in the network
design which ensures the significant observation of the environment and energy
optimization. The scheduling strategy is based on the approach of keeping a small
subset of nodes in active state to monitor the required target region, as compared to
the activation of all the nodes[1, 2]. One such strategy is shown in [3], in which the
nodes are activated according to the network parameters such as coverage probability
detection, node trust worthiness and contribution in monitoring of the targets by the
nodes. The nodes are scheduled to be in the active state depending on their capability
to be included in the set cover. The larger the number of set covers, the larger will
be network lifetime.
Predictive analysis based techniques aid in the network applications such as
reducing the redundant data transmission, predicting the network conditions and
forecasting the tangible multistep information. The paper presents a prediction based
approach in which the status of the nodes is predicted on the basis of historical data
using the LSTM based neural network. The LSTM based neural network consist of
the memory element which can improve the learning capability of the neural network.
The main contributions of the paper are as follows:
1. The deep learning based prediction model is proposed for the target coverage
problem in the wireless sensor network to determine the status of the node to be
considered as active or sleep.
2. The different parameters of the network are considered such as probability of
coverage, trust value of the nodes.
3. The objective of the approach is determining the number of set covers consisting
of minimum number of nodes.
4. The determination of set cover is considered as the NP hard problem and is
computationally complex, so machine learning based strategy can be adopted to
predict the node status on the basis of various parameters.
The organization of the paper is: the different machine learning based approaches
have been discussed in the Sect. 2, network model in Sect. 3, the proposed protocol
10 A Deep Learning Based Target Coverage Protocol for Edge … 163

in Sect. 4, the simulation setup and parameters is described in Sect. 5, simulation

results and analysis is discussed in the Sect. 6 and Sect. 7 consist of conclusion and
future scope.

2 Machine Learning in WSN

The machine learning based approaches are considered as efficient because it requires
less complexity. The different applications in which machine learning approaches can
be applied are: localization [3–6], coverage and connectivity [7–13], anomaly detec-
tion [14–18], fault detection [19–22], routing [23–25], MAC [26–31], data aggrega-
tion [32–35], synchronization [36–38], congestion control [39–43], and quality of
service [44–46]. The survey on the applications of machine learning technique in
different applications in WSNs can be found in [47–50].
The benefits of employing machine learning based approaches for the coverage
problem are as:
(i) The set cover consisting of minimum number of nodes can be determined
quickly and easily.
(ii) The nodes can be classified as active/idle in the dynamic environment without
losing the data.
In [51], authors have proposed a regression based interface design approach
which maintains the connectivity of the network. In [52], SVM based communi-
cation approach has been proposed which reduces the communication complexity
and hence improves the connectivity of the network. In [53], random forest coverage
and connectivity based approach is proposed which improves the coverage area by
changing the attributes of the network. In [54], reinforcement learning based coverage
optimization approach is proposed which enhances the network lifetime.

3 Network Model

Figure 1 represents the network model considered for the proposed prediction based
strategy in edge computing based environment. At the lowest level the number of
sensor nodes is deployed to monitor the given environment. The sensor nodes collect
the information from the environment and transmit it to the edge server where it is
used for further processing and analysis. The advantage of the edge server is that
the data is processed near the source of the data generation. The edge server then
forwards the data towards the application layer which usually comprise of different
applications which may utilize the sensed information.
164 P. Chaturvedi et al.

Fig. 1 The edge enabled network model for WSN

4 Proposed Protocol

The steps of the proposed protocol are as shown in the Fig. 2:

1. Identification and definition of the problem

In this paper we have considered the problem of determination of the node status
on the basis of various network parameters such as coverage probability, trust,
node degree, number of communications, number of packets transmitted, CH or
not and energy consumed. The problem is considered as a binary classification
problem which classifies the nodes into one of the two states as active or sleep.
The machine learning based prediction model is used so that the communication
overhead will be reduced and the energy conservation is achieved. We have
deployed the network consisting of 30 nodes and 10 targets in the square region
of dimension of 100*100. The parameter values for the network are obtained by
executing the simulation for the 100 iterations. The results of the simulation are
stored in .csv file.
2. Identification of the input and output parameters
In the next step we have identified the output parameter as Status and the
remaining attributes are considered as the input parameters.
3. Importing the required packages
In the next step, the necessary package required for the simulation and analysis
purpose are imported.
(i) pandas—It is used to load, visualize and analyze the dataset. The dataset
is read using the read_csv method available in this package.
10 A Deep Learning Based Target Coverage Protocol for Edge … 165

Fig. 2 Flow chart for the Start

proposed approach
Identification and definition of problem

Identification of input and output

parameters

Import the required packages

Understanding the dataset by performing

the Exploratory Data Analysis

Transformation of dataset using different

encoding mechanism

Split the dataset into training and

validation set and train the model

Make the prediction and determine if the

optimal parameters are achieved?
Yes No
Construct the hypermodel

Make the prediction using the hypermodel

and determine the efficiency

Explain the prediction of the model using the

Local Feature Importance on LIME

Boost the model performance using LGBM

Classifier

Identify the global importance of features

End
166 P. Chaturvedi et al.

(ii)numpy—It is used to perform the mathematical operations on the dataset.

(iii)
matplotlib.pyplot—This package is used to plot the different figures and
plots related to the dataset.
(iv) label encoder from sklearn.preprocessing—The label encoder method
from sklearn.preprocessing is used to encode the categorical attributes
such as node staus, CH or not into numberical values.
(v) seaborn—This package is used to plot the correlation matrix for the
different parameters available in the dataset.
(vi) tensorflow keras—The tensorflow package is used to build using the
deep learning based models. The keras package is also imported from
tensorflow library.
(vii) kerastuner—This package is used to perform the hyper parameter tuning
of the LSTM based neural network.
(viii) kerastuner.tuners random search—The random search from the kerastuner
is used to generate the different models by varying the number of hidden
layers and neurons in the layer.
(ix) train_test_split from sklearn.model_selection—The train_test_split
method from sklearn.model_selection is used to divide the original
dataset into training and testing dataset.
(x) lime and lime_tabular—The lime package is used to achieve the explain-
ability of the machine learning model. It explains the reason behind the
particular output of the designed model. The lime_tabular method is used
to visualize the impact of different parameters on the positive and negative
classes in the notebook.
(xi) lightgbm—Gradient descent based boosting mechanism is a collection
of ensemble machine learning methods which aims to minimize the loss
function. In the boosting method the hard to classify data instances are
given more weights. There are different implementations available for
the gradient descent based boosting methods such as Extreme Gradient
Descent boosting (Xgboost) and Light Gradient Descent boosting model.
In this paper we have used light gradient boosting model to improve the
prediction model which is based on decision trees. The advantages of the
light gradient descent boosting model are:
a. The memory requirement is low.
b. Accuracy of the model is more.
c. It supports the GPU learning, parallel and distributed modeling.
d. These methods are capable of handling large number of datasets.
e. The efficiency of the model is high.
f. The training of the model is faster.
4. Understanding the dataset by performing the Exploratory data analysis
The Exploratory data analysis phase consists of following steps:
a. Identifying the number of attributes having missed or zero values.
b. Identifying the number of duplicate entries.
10 A Deep Learning Based Target Coverage Protocol for Edge … 167

Fig. 3 Count of active and

sleep nodes

c. Identifying the size of the data set. The considered data set consist of 100
rows and 9 columns. The column names are Node id, coverage probability,
trust, no of communications, no of packets transmitted, whether the node is
cluster head or not, degree of the nodes, energy consumed, no of targets and
the status of the node as either active or sleep.
d. Determining the number of nodes in the active and sleep state using the
pandas library in python language. The number of active and sleep nodes
based on the simulation of 100 nodes is as shown in the Fig. 3. The results
show that there are 60 active and 40 sleep nodes, so we can say that the data
set is balanced.

5. Transformation of dataset using the different encoding mechanisms

a. The next step is preprocessing of the data set. Since the column CH or not
and Status contains the categorical data, so we have used the label encoding
mechanism to transform the data into numerical values.
b. In the next step, the correlation plot is determined to identify the most influ-
ential parameter on the node status. From the Fig. 4 it is clear that coverage
probability, CH or not has the most significant impact on the node status.

6. Spiting our data to train and validation data

In the next step, the dataset is divided into training and test dataset using sklearn’s
inbuilt train_test_split method. The test data set is considered as 30%. For
determining the efficiency of the proposed approach we have used the validation
dataset size as 30%.
7. Using keras tuner to determine the optimal parameters for the LSTM based
neural network
In this step the keras tuner is used to determine the optimal number of layers
and neurons in each layer. The random search method is used to evaluate the
loss function of the model for the varying number of layers and neurons.
168 P. Chaturvedi et al.

Fig. 4 Correlation plot

8. Designing the hyper model

The hypermodel is designed using the optimal parameters obtained in the step
7. The hyper model is then evaluated by executing the model for 1000 iterations.
9. Making the prediction and determining the efficiency
The efficiency of the hyper model is determined in terms of the loss function
by executing the model for 1000 iterations.
10. Explaining the model with Lime
The local impact of the various parameters on the node status prediction is
determined using the LIME package which is available in the python language.
The explain method is used to display the impact of the different parameters on
the node status as active or sleep.
11. Using extreme gradient boosting machine learning model (Lightgbm) for
prediction
The gradient boosting machine learning model is used to predict the node status.
The efficiency of the model is determined for the validation dataset of size 30%.
The metric method is used to visualize the training and testing accuracy of the
model.
12. Identifying the global importance of the features
The global importance parameter is used to identify the significant parameters
using the feature importance method available in lightgbm.
10 A Deep Learning Based Target Coverage Protocol for Edge … 169

Table 1 Parameter values

Parameter Values
No. of nodes 20
No. of targets 5
Sensing range 5
Communication range 10
No. of layers 2–20
No of neurons in each layer 32–512
Step size 32
Learning rate 0.001, 0.0001, 0.00001
Activation function at hidden layer Rectified linear unit
Activation function at output layer Linear
Loss function Mean absolute error
Performance metric Mean absolute error
Test data set size 30%
Max trial 100
Execution per trial 5
No. of iterations 1000

5 Simulation Setup and Parameters

The dataset obtained through the simulation of the sensor network is provided as
input to the LSTM based neural network. The keras tuner package is used to obtain
the optimal parameter such as number of hidden layers and number of neurons in
each layer. The number of layers is varied from 2 to 20 and number of neurons is
varied between 32 and 512 with a step size of 32. The learning rate is considered
as 0.001, 0.0001 and 0.00001. The other simulation parameters are as shown in the
Table 1.
The hypermodel is constructed based on the parameter values obtained and then it
is trained for 100 trials with 5 executions per trial. The hypermodel is then executed for
the 1000 iterations to analyze its efficiency. The results of the training and validation
are discussed in the next section.

6 Simulation Results and Discussion

The performance of the proposed prediction based model is evaluated using the
simulation setup and parameters defined in the previous section as discussed in the
subsections:
170 P. Chaturvedi et al.

6.1 Model Analysis

The network parameters obtained through simulation are provided as input to LSTM
based neural network model. To determine the optimal number of layers and neurons
the keras tuner package is used. The keras tuner regularly vary the number of hidden
layers, number of neurons and learning and train the model for the different learning
rates. The mea absolute error is considered as the loss function. The validation split
is considered as the 30% and the loss function for the validation data set is obtained
as val_mean_absolute_error: 0.179. The results of 10 best iterations are summarized
in the Table 2. It can be seen from the results that the best loss function is obtained
as 0.1604 for the number of layers as 4 and learning rate at 0.001.
The number of neurons in each layer for the 10 best trials is as shown in the
Table 3.
The score of the 10 best iterations is shown in the Fig. 5.
Using the best parameters thus obtained, a hyper model is constructed and the
prediction is done for the test dataset. The hypermodel is trained for the training
dataset and tested for the test dataset by executing the model for the 1000 iterations.
The results show that after 1000th iteration, the loss function is almost stable. The
loss function using the hypermodel for the training and test dataset is as follow:
[test loss, train loss]: [0.1099, 0.1099].
The comparison results of the loss function for the training and validation dataset
is as shown in the Fig. 6. The results show that the hypermodel accurately predicts
the node status for the testing as well as the validation dataset.

Table 2 Result of 10
Trial Num_layers Learning rate Score
iterations
1 4 0.01 0.16047
2 6 0.001 0.1706
3 6 0.01 0.1752
4 3 0.001 0.1778
5 3 0.01 0.1790
6 4 0.01 0.1791
7 4 0.01 0.1793
8 7 0.001 0.1798
9 2 0.001 0.1833
10 5 0.01 0.1853
Table 3 Number of neurons in each layer in each trial
Trial/union 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
1 224 352 192 96 64 192 288 352 384 448 384 288 320 384 64 320 352 256 224 288
2 224 96 384 448 320 480 352 32 320 288 160 32 96 384 384 160 160 96 352 480
3 352 32 192 96 160 128 480 320 192 256 128 256 224 416 192 256 256 224 416 288
4 128 192 288 32 352 320 416 416 64 32 416 256 448 96 448 256 416 320 32 448
5 224 480 320 224 256 96 480 480 160 256 224 416 320 352 128 192 128 128 96 384
6 384 192 96 192 256 320 448 32 512 416 480 480 448 64
7 160 448 512 512 384 480 256 384 448 192 320 480 160 160 384 448 416 256 416 384
8 288 320 32 160 480 192 352 32 192 96 64 160 192 320 256 256 96 96
10 A Deep Learning Based Target Coverage Protocol for Edge …

9 128 384 416 320 352 448 256 256 384 416 224 64 352 416 192 288 96 256 224 192
10 320 416 320 256 320 352 480 192 352 352 256 128 512 96 384 448 320 64 384 32
171
172 P. Chaturvedi et al.

Fig. 5 Score of the 10 best

iterations using hypermodel

Fig. 6 Model loss versus Epoch

6.2 Explainability of the Proposed Model Using Lime

The machine learning models are usually considered as the black box which only
produces the results in the form of prediction and the impact of the different param-
eters are usually hidden from the outside world. So the LIME package is used to
understand the most significant parameter in the prediction of the node status.
a. Positive
Figure 7 represents the local explanation of the various features on the node
status being as active. The green color represents the positive correlation with
10 A Deep Learning Based Target Coverage Protocol for Edge … 173

Fig. 7 Impact of network parameters on the positive class

the node status and the red color represents the negative correlation on the node
status. It can be inferred from the figure that the attributes CH or not, energy
consumed, number of packets and number of targets have positive correlation on
the node status, whereas the coverage probability, node degree, trust and number
of communications have negative correlation on the node status.
b. Negative
Figure 8 represents the local explanation of the various features on the node
status being as sleep. The green color represents the positive correlation with
the node status and the red color represents the negative correlation on the node
status. It can be inferred from the figure that the attributes CH or not, energy
consumed, number of packets and number of targets have positive correlation on
the node status, whereas the coverage probability, node degree, trust and number
of communications have negative correlation on the node status.

Fig. 8 Impact of the network parameters on the negative class

174 P. Chaturvedi et al.

Fig. 9 The impact of

network parameters in the
list form on both the classes

The correlation of various features on the node status is shown in the list form in
the Fig. 9. The active status is represented as 0 and sleep status is represented as 1.
Row 0
The local importance of various features for the first row in the dataset has been
analyzed for the node status as active or sleep as shown in the Fig. 10. The parameter
values for the row are as shown in the Table 4.
Row 1
The local importance of various features for the first row in the dataset has been
analyzed for the node status as active or sleep as shown in the Fig. 11. The parameter
values for the row are as shown in the Table 5.

6.3 LGBM Classifier

In this section we have implemented the LGBM classifier for the considered dataset
to improve the training efficiency and faster convergence. The other simulation
parameters are:
Learning_rate = 0.09, max depth = 5 random state = 42, Verbose = 20,
eval_metric = ‘logloss’.
The results of the model prediction for the training and validation dataset are as
shown in the Table 6.
The binary logloss function is used to evaluate the efficiency of the LGBM clas-
sifier as shown in the Fig. 12. The loss value is approaching to minimum in 200
iterations for the training as well as the validation, which proves the efficiency of the
proposed model.
 10 A Deep Learning Based Target Coverage Protocol for Edge …

Fig. 10 Local impact of feature on the node status for the row 0
175
176 P. Chaturvedi et al.

Table 4 Parameter values for

Parameter Intercept Prediction_local Right
the row 0
Value 0.180 0.230 0.235

The accuracy of LGBM classifier is obtained as:

Training accuracy: 1.0000
Testing accuracy: 0.9667

6.4 Global Feature Importance

The global feature importance on the node prediction is plotted as shown in the Fig.
13. The results show that the most dominant features for the node status prediction are
Column 0–coverage probability, Column 1–trust, column 3–CH or not and Column
4–node degree.

7 Conclusion and Future Scope

The paper presented a LSTM based data prediction approach which can be used
to determine the node status in the scheduling approach to monitor a given set of
targets. The keras tuner package has been used to determine the optimal number
of neurons and number of hidden layers. The results are then applied to construct
the hypermodel which is then used to predict the node status. The loss function for
the model is determined for the different learning rate, number of hidden layers and
number of neurons. The lime package is used to determine the local feature impact
on the target attribute. The LGBM based classifier is used as a boosting algorithm
to improve the prediction model. The simulation results show that the LGBM based
boosting strategy achieves the accuracy of 100% and 97% for the training and testing
dataset.
As a future work, we aim to determine the different performance parameters for
the proposed prediction model such as network lifetime, energy consumption and
other QoS parameters.
 10 A Deep Learning Based Target Coverage Protocol for Edge …

Fig. 11 Local impact of feature on the node status for the row 1
177
178 P. Chaturvedi et al.

Table 5 Parameter values for

Parameter Intercept Prediction_local Right
the row 1
Value 0.188 0.236 0.251

Table 6 Loss values for the LGBM classifier for training and validation dataset for 100 Epochs
Epochs Training loss Validation loss
0 1 0 1
20 0.1118 0.1118 0.1277 0.1277
40 0.0356 0.0356 0.0618 0.0618
60 0.0118 0.0118 0.0554 0.0554
80 0.0038 0.0038 0.0637 0.0637
100 0.0014 0.0014 0.0747 0.0747

Fig. 12 Loss function for the training and validation datasets

Fig. 13 Feature importance

of the model using LGBM
classifier
10 A Deep Learning Based Target Coverage Protocol for Edge … 179

References

1. J. Yick, B. Mukherjee, D. Ghosal, Wireless sensor network survey. Comput. Netw. 52(12),
2292–2330 (2008)
2. R. Mulligan, H.M. Ammari, Coverage in wireless sensor networks: a survey. Netw. Protoc.
Algorithms 2(2), ISSN: 1943–3581
3. P. Chaturvedi, A.K. Daniel, An energy efficient node scheduling protocol for target coverage in
wireless sensor networks. Paper presented at the 5th international conference on communication
system and network technologies (CSNT-2015), 2015
4. J. Kuriakose, S. Joshi, R.V. Raju, A. Kilaru, A review on localization in wireless sensor
networks. Adv. Sig. process. Intel. Recogn. Syst. 599–610 (2014)
5. P. Cottone, S. Gaglio, G.L. Re, M. Ortolani, A machine learning approach for user localization
exploiting connectivity data. Eng. Appl. Artif. Intell. 50, 125–134 (2016)
6. M. Bernas, B. Paczek, Fully connected neural networks ensemble with signal strength clustering
for indoor localization in wireless sensor networks. Int. J. Distrib. Sens. Netw. 11(12), 1–10
(2015)
7. I. Ullah, Y. Shen, X. Su, C. Esposito, C. Choi, A localization based on unscented Kalman filter
and particle filter localization algorithms. IEEE Access 8, 2233–2246 (2020)
8. S M. Mohamed, H.S. Hamza, I.A. Saroit, Coverage in mobile wireless sensor networks (M-
WSN): a survey, Comput. Commun. 110(C), 133–150 (2017)
9. W. Fang, X. Song, X. Wu, J. Sun, M. Hu, Novel efficient deployment schemes for sensor
coverage in mobile wireless sensor networks. Inf. Fusion 41, 25–36 (2018)
10. H. Li, S. Wang, M. Gong, Q. Chen, L. Chen, IM2DCA: Immune mechanism based multipath
decoupling connectivity algorithm with fault tolerance under coverage optimization in wireless
sensor networks. Appl. Soft. Comput. 58, 540–552 (2017)
11. M. Abo-Zahhad, N. Sabor, S. Sasaki, S.M. Ahmed, A centralized immune-voronoi deploy-
ment algorithm for coverage maximization and energy conservation in mobile wireless sensor
networks. Inf. Fusion 30, 36–51 (2016)
12. W. Wang, H. Huang, Q. Li, F. He, C. Sha, Generalized intrusion detection mechanism for
empowered intruders in wireless sensor networks. IEEE Access 8, 25170–25183 (2020)
13. C. Miranda, G. Kaddoum, E. Bou-Harb, S. Garg, K. Kaur, A collaborative security framework
for software-defined wireless sensor networks. IEEE Trans. Inf. Forensics Secur. 15, 2602–2615
(2020)
14. Y. Gao, F. Xiao, J. Liu, R. Wang, Distributed soft fault detection for interval type-2 fuzzy-
model-based stochastic systems with wireless sensor networks. IEEE Trans. Ind. Informat.
15(1), 334–347 (2019)
15. J. Tan et al., An efficient information maximization based adaptive congestion control scheme
in wireless sensor network. IEEE Access 7, 64878–64896 (2019)
16. F. Hajjej, M. Hamdi, R. Ejbali, M. Zaied, A distributed coverage hole recovery approach based
on reinforcement learning for wireless sensor networks. Ad Hoc Netw. 101 (2020). 102082
17. Z. Sun, L. Wei, C. Xu, Z. Lv, An event-driven mechanism coverage algorithm based on sensing-
cloud-computing in sensor networks. IEEE Access, 7, 84668–84679 (2019)
18. B. Khalifa, A.M. Khedr, Z. Al Aghbari, A coverage maintenance algorithm for mobile WSNs
with adjustable sensing range. IEEE Sens. J. 20(3), 1582–1591 (2020)
19. A. Farhat, C. Guyeux, A. Makhoul, A. Jaber, R. Tawil, A. Hijazi, Impacts of wireless sensor
networks strategies and topologies on prognostics and health management. J. Intell. Manuf.
1–27 (2017)
20. P. Oluwasanya, Anomaly detection in wireless sensor networks. arXiv preprint arXiv:1708.
08053.
21. A.D. Paola, S. Gaglio, G.L. Re, F. Milazzo, M. Ortolani, Adaptive distributed outlier detection
for WSNs. IEEE Trans. Cybern. 45(5), 902–913 (2015)
22. M. Wazid, A.K. Das, An efficient hybrid anomaly detection scheme using K-means clustering
for wireless sensor networks. Wireless Pers. Commun. 90(4), 1971–2000 (2016)
180 P. Chaturvedi et al.

23. H.H. Bosman, G. Iacca, A. Tejada, H.J. Wörtche, A. Liotta, Spatial anomaly detection in sensor
networks using neighborhood information. Inf. Fusion, 33, 41–56 (2017)
24. C. O’Reilly, A. Gluhak, M.A. Imran, S. Rajasegarar, Anomaly detection in wireless sensor
networks in a non-stationary environment. IEEE Commun. Surv. Tutorials 16(3), 1413–1432
(2014)
25. T. Muhammed, R.A. Shaikh, An analysis of fault detection strategies in wireless sensor
networks. J. Netw. Comput. Appl. 78, 267–287 (2017)
26. H. Geng, Y. Liang, F. Yang, L. Xu, Q. Pan, Model-reduced fault detection for multi-rate sensor
fusion with unknown inputs. Inf Fusion 33, 1–14 (2017)
27. R.R. Swain, P.M. Khilar, S.K. Bhoi, Heterogeneous fault diagnosis for wireless sensor
networks. Ad Hoc Netw. 69, 15–37 (2018)
28. R. Palanikumar, K. Ramasamy, Effective failure nodes detection using matrix calculus
algorithm in wireless sensor networks. Cluster Comput. 1–10 (2018)
29. M. Hammoudeh, R. Newman, Adaptive routing in wireless sensor networks: QoS optimisation
for enhanced application performance. Inf. Fusion 22, 3–15 (2015)
30. X. Liu, Routing protocols based on ant colony optimization in wireless sensor networks: a
survey. IEEE Access 5, 26303–26317 (2017)
31. M. Asif, S. Khan, R. Ahmad, M. Sohail, D. Singh, Quality of service of routing protocols in
wireless sensor networks: a review. IEEE Access 5, 1846–1871 (2017)
32. K.-L.A. Yau, H.G. Goh, D. Chieng, K.H. Kwong, Application of reinforcement learning to
wireless sensor networks: models and algorithms. Computing 97(11), 1045–1075 (2015)
33. J. Kabara, M. Calle, MAC protocols used by wireless sensor networks and a general method
of performance evaluation. Int. J. Distrib. Sens. Netw. 8(1), 1–11 (2012)
34. I. Mustapha, B.M. Ali, A. Sali, M.F.A. Rasid, H. Mohamad, An energy efficient reinforcement
learning based cooperative channel sensing for cognitive radio sensor networks. Pervasive Mob.
Comput. 35, 165–184 (2017)
35. S. Kosunalp, Y. Chu, P.D. Mitchell, D. Grace, T. Clarke, Use of Q-learning approaches for
practical medium access control in wireless sensor networks. Eng. Appl. Artif. Intell. 55,
146–154 (2016)
36. M. Rovcanin, E. De Poorter, I. Moerman, P. Demeester, A reinforcement learning based solu-
tion for cognitive network cooperation between co-located, heterogeneous wireless sensor
networks. Ad Hoc Netw. 17, 98–113 (2014)
37. M. Rovcanin, E. De Poorter, D. van den Akker, I. Moerman, P. Demeester, C. Blondia, Exper-
imental validation of a reinforcement learning based approach for a service-wise optimisation
of heterogeneous wireless sensor networks. Wireless Netw. 21(3), 931–948 (2015)
38. M. Ambigavathi, D. Sridharan, Energy-aware data aggregation techniques in wireless sensor
network. Adv. Power Syst. Energy Manag. 165–173 (2018)
39. k. xie, L. Wang, X. Wang, G. Xie, J. Wen, Low cost and high accuracy data gathering in WSNs
with matrix completion. IEEE Trans. Mob. Comput. 17(7), 1595–1608 (2017)
40. H. Lin, D. Bai, Y. Liu, Maximum data collection rate routing for data gather trees with data
aggregation in rechargeable wireless sensor networks. Cluster Comput. 1–11 (2017)
41. E. Kanjo, E.M. Younis, N. Sherkat, Towards unravelling the relationship between on-body,
environmental and emotion data using sensor information fusion approach. Inf. Fusion 40,
18–31 (2018)
42. D. Capriglione, D. Casinelli, L. Ferrigno, Analysis of quantities influencing the perfor-
mance of time synchronization based on linear regression in low cost WSNs. Measurement
77(Supplement C), 105–116 (2016)
43. J.J. Prez-Solano, S. Felici-Castell, Adaptive time window linear regression algorithm for accu-
rate time synchronization in wireless sensor networks. Ad Hoc Netw. 24 (Part A), 92–108
(2015)
44. G. Betta, D. Casinelli, L. Ferrigno, Some Notes on the Performance of Regression-based Time
Synchronization Algorithms in Low Cost WSNs (Springer International Publishing, Cham,
2015)
10 A Deep Learning Based Target Coverage Protocol for Edge … 181

45. J.J. Ṕerez-Solano, S. Felici-Castell, Improving time synchronization in wireless sensor

networks using Bayesian inference. J. Netw. Comput. Appl. 82, 47–55 (2017)
46. Pau, A.V. Bobovich, A fuzzy data fusion solution to enhance the QoS and the energy
consumption in wireless sensor networks. Wireless Commun. Mob. Comput. 1–10 (2017)
47. E. Alpaydin, Introduction to Machine Learning (MIT Press, Cambridge, MA, USA, 2020)
48. D. Praveen Kumar, T. Amgoth, C.S.R. Annavarapu, Machine learning algorithms for wireless
sensor networks: a survey. Inf. Fusion 9, 1–25 (2019)
49. W. Sun, W. Lu, Q. Li, L. Chen, D. Mu, X. Yuan, WNN-LQE: wavelet-neural-network-based
link quality estimation for smart grid WSNs. IEEE Access 5, 12788–12797 (2017)
50. E.K. Lee, H. Viswanathan, D. Pompili, RescueNet: reinforcement-learning-based communi-
cation framework for emergency networking. Comput. Netw. 98, 14–28 (2016)
51. X. Chang, J. Huang, S. Liu, G. Xing, H. Zhang, J. Wang, L. Huang, Y. Zhuang, Accuracy-
aware interference modeling and measurement in wireless sensor networks. IEEE Trans. Mob.
Comput. 15(2), 278–291 (2016)
52. W. Kim, M.S. Stankovi, K.H. Johansson, H.J. Kim, A distributed support vector machine
learning over wireless sensor networks. IEEE Trans Cybern. 45(11), 2599–2611 (2015)
53. W. Elghazel, K. Medjaher, N. Zerhouni, J. Bahi, A. Farhat, C. Guyeux, M. Hakem, Random
forests for industrial device functioning diagnostics using wireless sensor networks, in
Aerospace Conference (IEEE, 2015), pp. 1–9
54. H. Chen, X. Li, F. Zhao, A reinforcement learning-based sleep scheduling algorithm for desired
area coverage in solar-powered wireless sensor networks. IEEE Sens. J. 16(8), 2763–2774
(2016)