PROTECTING INDIVIDUALS IN THE AGE OF AI
FEDERICO MARENGO
Order a digital copy at https://payhip.com/b/ocJgm
Privacy and AI. Protecting Individuals in the Age of AI.
Copyright © 2023 by Federico Marengo.
ISBN: 9798854287456 (printed edition)
All rights reserved.
First Edition: September 2023
Order a digital copy at https://payhip.com/b/ocJgm
Contents
PREFACE ......................................................................................... 9
ABOUT THE AUTHOR ................................................................... 12
INTRODUCTION............................................................................. 14
CONCEPTUALIZING AI ................................................................. 18
Introduction ................................................................................. 18
1.- The importance of data for the development of AI ................ 19
1.1.- The concept of data ....................................................... 19
1.2.- Personal data ................................................................. 22
2.- Artificial Intelligence ............................................................... 27
2.1.- Introduction .................................................................... 27
2.2.- Concept .......................................................................... 29
DATA PROTECTION PROVISIONS RELATED TO AI .................. 43
Introduction ................................................................................. 43
1.- Data protection as a fundamental right in Europe ................. 44
1.1.- Council of Europe legal framework ................................ 44
1.2.- European Union legal framework: primary legislation ... 45
2.- The General Data Protection Regulation .............................. 47
2.1.- Data protection principles .............................................. 48
Federico Marengo
2.2.- Lawful processing and lawful bases for processing ....... 64
ENSURING INDIVIDUAL RIGHTS IN AI ........................................ 76
Introduction ................................................................................. 76
1.- Rights related to automated decision-making including profiling
......................................................................................................... 77
1.1.- Introduction..................................................................... 77
1.2.- Content of the right ......................................................... 79
1.3.- Exceptions from the prohibition ...................................... 87
1.4.- Automatic decision-making based on special categories of
data .............................................................................................. 88
1.5.- Automated decisions and profiling of children ............... 90
1.6.- Safeguards ..................................................................... 91
2.- Rights derived from transparency obligations ....................... 97
2.1.- Information rights and right to access ............................ 97
2.2.- Information on automated decision making ................. 101
3.- Other data subject rights ..................................................... 107
3.1.- Right to rectification ...................................................... 107
3.2.- Right to erasure ............................................................ 109
3.3.- Right to restrict data processing................................... 112
3.4.- Right to object processing ............................................ 113
3.5.- Right to data portability ................................................ 114
4.- General GDPR accountability mechanisms ........................ 117
4.1.- Records of processing activities................................... 117
4.2.- Data Protection Officer ................................................. 119
4.3.- Data Protection by Design and by Default ................... 121
4.4.- Data Protection Impact Assessment ............................ 124
OVERCOMING THE LIMITATIONS OF THE GDPR ................... 131
Order a digital copy at https://payhip.com/b/ocJgm
Privacy and AI
Introduction ............................................................................... 131
1.- Addressing algorithmic transparency when processing personal
data using AI systems .................................................................... 132
1.1.- Outlining the problem of algorithmic transparency ...... 132
1.2.- Improving algorithmic transparency. Information to be
provided before the AI-powered decision is made .................... 135
1.3.- Information to be provided after the automated decision or
profiling ...................................................................................... 156
2.- Addressing fairness and non-discrimination when processing
personal data using AI systems ..................................................... 158
2.1.- Outlining the problem of algorithmic bias and fairness 158
2.2.- Addressing algorithmic biases and fairness ................ 162
MECHANISMS TO FURTHER MITIGATE THE RISKS POSED BY AI
SYSTEMS .......................................................................................... 188
Introduction ............................................................................... 188
1.- Register of AI systems or AI providers ................................ 189
2.- AI Ethical Officer to overcome limitations from DPOs ......... 192
3.- Standardisation of AI systems. Industry standards as a method
to fill legislative gaps ...................................................................... 195
4.- Certification of AI systems ................................................... 200
5.- Codes of Conduct for AI operators ...................................... 204
6.- Empowerment of Supervisory Authorities ........................... 207
6.1.- Supervisory authorities evaluating AI systems ............ 207
6.2.- Algorithmic disgorgement. Destroying AI systems that
used ill-gotten or tainted data for training .................................. 209
6.3.- Could algorithmic disgorgement be applied in the EU? 214
7.- Privacy by Design measures: reducing the identifiability of data
....................................................................................................... 216
Order a digital copy at https://payhip.com/b/ocJgm
Federico Marengo
7.1.- Anonymisation and pseudonymisation ........................ 216
7.2.- Encryption .................................................................... 220
7.3.- Synthetic data............................................................... 224
CONCLUSIONS ............................................................................ 229
ANNEX .......................................................................................... 246
REFERENCES .............................................................................. 260
Order a digital copy at https://payhip.com/b/ocJgm
INTRODUCTION
Artificial intelligence (hereinafter AI) has brought many societal
changes and is transforming the way to do business. AI systems are
pushing the boundaries of machine capabilities, cutting down the time
required to complete specific tasks, enabling the accomplishment of
complex operations that exceed human capacity, and easing repetitive
decision-making processes. In short, AI systems can provide a faster and
cheaper method to solve everyday problems in various areas, for
example, smart cities, fraud prevention, law enforcement, and
autonomous driving. The opportunities offered by these technologies are
countless. Yet, similar to what has happened with other innovations, AI
solutions can have detrimental consequences for individuals and society.
In particular, the processing of information using AI systems also entails
risks to the rights and freedoms of individuals, such as the lack of respect
for human autonomy, the production of material or moral harms,
discrimination, and lack of transparency in the decision-making process.
Whereas these features reveal the potential of artificial intelligence to
support most of the activities performed by individuals, it also triggers
many crucial questions, in particular, regarding the adequacy and
sufficiency of the EU legal framework on data protection to protect the
Order a digital copy at https://payhip.com/b/ocJgm
Privacy and AI
rights and freedoms of individuals when their personal information is
processed using AI systems.
This work attempts to discover the most critical challenges posed by
the processing of personal data supported by AI systems, how the
current European legal framework addresses these challenges, and it
also proposes alternative pathways to better protect the rights of
individuals without stifling innovation.
Since the entry into force of the General Data Protection Regulation
(hereinafter GDPR) in 2018, there has been massive interest from
researchers and industry stakeholders in data protection. The GDPR
harmonised the regulatory landscape of data protection and introduced
stringent requirements for the processing of personal data. Additionally,
it is a technology-neutral regulation, which means that it was conceived
to be flexible and adaptable to new technologies. It also included specific
provisions concerning automated decision-making to provide more
targeted protection against this particular way of processing.
When it comes to the interaction of AI and data protection,
researchers dedicated much of their efforts to particular fields: the
explainability of AI systems and the provisions concerning the right not
to be subject to automated decisions established in the GDPR and also
concerning the fairness of the decisions taken using AI systems.
Most reviewed materials provide highly detailed explanations about
the legal concepts involved, present new interpretations, and offer
recommendations about what should be done to improve individuals'
general situation at the legislative level. However, there is in general a
disconnection on how the legal concepts relate to operative aspects of
data protection. For the adequate protection of individuals, the analysis
cannot be only limited to legal texts and case law. It should also include
non-binding documents such as guidelines, standards, and best
practices that concretise and provide more detailed guidance on how the
15
Order a digital copy at https://payhip.com/b/ocJgm
Federico Marengo
high-level principles stipulated in the laws and regulations governing the
area should be translated into practical and operational requirements.
This work attempts to bridge, on the one hand, the gap between the
legislative and judicial interpretation and, on the other hand, the practical
and operative aspects concerning the protection of personal data.
This work is structured in 5 chapters as follows. The first chapter,
Conceptualizing AI, elaborates on the concept and importance of data,
in particular personal data, for the development of AI systems and the
conceptualization of artificial intelligence. The second chapter, Data
Protection Provisions Related to AI, explains how personal data is
protected in Europe. Starting with an overview of the fundamental legal
texts that govern the area, it then assesses more in detail the relevant
provisions of the GDPR that has a bearing with regard to the protection
of personal data where the processing is made using AI systems. In
particular, it elaborates on the impact of AI systems on the data
protection principles and the lawful basis to process personal data. The
third chapter, Ensuring Individual Rights in AI, provides an in-depth
appraisal of the protection of individual rights by the GDPR when
personal data is processed using AI systems. Whereas a large part of
this chapter is devoted to the rights related to automated decision-
making (together with its conceptualization, the exceptions and
safeguards), the remaining rights listed by the GDPR are also evaluated.
Particularly, rights derived from transparency obligations, the right to
rectification, erasure, restriction, objection and portability are reviewed in
the light of the challenges posed by the processing of personal data using
AI systems. Furthermore, this chapter introduces the most important
accountability mechanisms, which are also a crucial aspect regulated by
the GDPR and essential to mitigating the risks posed by the processing
of personal data using AI systems. The fourth chapter, Overcoming the
Limitations of the GDPR, provides more details about the limitations of
16
Order a digital copy at https://payhip.com/b/ocJgm
Privacy and AI
the current regime and explains how the weaknesses previously
identified could be overcome. It focuses on two of the most important
risks to the rights and freedoms of individuals: algorithmic transparency
and fairness and discrimination. Finally, the fifth chapter, Mechanisms to
Further Mitigate the Risks Posed by AI Systems, explores other
measures to further reduce the risks presented by the use of AI systems
to process personal data. In particular, it explores alternatives such as
the creation of registers for AI systems, the introduction of the role of the
AI ethical officer, the benefits of relying on standards on AI systems to fill
legislative gaps, certifications, and codes of conduct for AI system
operators, the provision of more powers to data protection authorities
and the reliance on privacy by design measures to reduce the risks of AI
systems.
17
Order a digital copy at https://payhip.com/b/ocJgm
CONCEPTUALIZING AI
Introduction
Artificial intelligence will definitively reshape the world in which we
live. Artificial Intelligence is the science of training machines to perform
human tasks. It uses concepts from statistics, computer science and
many other disciplines, to design algorithms that process data, make
predictions, and help make decisions. It establishes basic parameters
regarding the data and trains the machine to learn by itself by identifying
patterns using many layers of processing.
Though the term Artificial Intelligence was forged in 1956 by Minsky
and McCarthy, it was not until recently that it acquired its full significance.
Discussions about the potentialities and fears around AI are not new, but
technical features characterize and distinguish the current period. In
particular, the surge in artificial intelligence is mainly due to the enormous
increase in computational power and the access to huge amounts of data
to train machine learning models. Recent technological developments
have facilitated the transmission, processing and storage of huge
amounts of information. The borderless nature of the Internet along with
the vast volume of communications, create new regulatory issues for
states, mainly regarding national security and data protection. These
Order a digital copy at https://payhip.com/b/ocJgm
Privacy and AI
developments underpin the recent increase in machine learning
capabilities and justify the wide public attention on the matter.
This chapter proceeds as follows. Firstly, it accounts for the
importance of data and its free flow for the development of AI. It attempts
to disentangle what is meant by ‘data’ and it pictures the importance of
the free flow of information in the digital economy. Secondly, it assesses
the intricacies of artificial intelligence. As there is no universally agreed
definition of AI, it draws on the most common meaning of the term and
the proposal made by the European Commission in the AI Regulation
(hereinafter, AIA), as well as its foundations, capabilities and limitations.
Thirdly, it reviews some algorithmic models that are covered under the
umbrella term of AI. The purpose of this part is not to provide a complete
understanding of the mathematical underpinning of the models. Instead,
it leans on the assumption that there is a broad misunderstanding
regarding the current methods comprising AI and it attempts to explain
concisely the methods developers usually employ, leaving aside their
mathematical foundations. For this purpose, the methods are classified
according to their capacity to explain how they work and how they
produce the results since it provides the groundwork to evaluate which
methods could be more intelligible for users lacking specific technical
background.
1.- The importance of data for the development of AI
1.1.- The concept of data
Throughout history, humans have kept the information they produced
in diverse material means. Primitive societies painted walls to convey
messages, and later written text was recorded on papyrus or paper.
Technology allowed the digitalisation of information, which converted
analogue into digital information. The process of digitalisation brought
19
