Deep Learning based CCTV Footage

Super-resolu on for Human Subject

A PROJECT REPORT
SUBMITTED IN PARTIAL FULFILMENT OF THE
REQUIREMENTS FOR THE DEGREE OF

Master of Technology (Online)

IN

ELECTRONICS AND COMMUNICATION

BY

Maahi Rajpoot

Faculty of Engineering
Indian Ins tute of Science
Bangalore – 560 012 (INDIA)

November, 2023
 Declaration of Originality

I, Maahi Rajpoot, with SR No. 13-19-03-19-52-22-1-21866 hereby declare that the material
presented in the project report titled
Deep Learning based CCTV FootageSuper-resolution for Human Subject
represents original work carried out by me at Synopsys India pvt ltd as part of the project
credit requirements of the Master of Technology (Online) degree in ELECTRONICS AND
COMMUNICATION at the Indian Institute of Science, between August, 2022 to July, 2025.

With my signature, I certify that:

• I have not manipulated any of the data or results.

• I have not committed any plagiarism of intellectual property. I have clearly indicated and
referenced the contributions of others.

• I have explicitly acknowledged all collaborative research and discussions.

• I have understood that any false claim will result in severe disciplinary action.

• I have understood that the work may be screened for any form of academic misconduct.

Date: Student Signature

In our capacities as internal project guide and faculty mentor of the above-mentioned work, we
certify that the above statements are true to the best of our knowledge, and we have carried out
due diligence to ensure the originality of the report.

Internal Guide Name: Deep Shekhar Internal Guide Signature

Organisation: Analog Design, Staff Engineer

Faculty Mentor Name: Shalabh Bhatnagar Faculty Mentor Signature

Department: CSA
© Maahi Rajpoot November, 2024 All
rights reserved
 iv

ACKNOWLEDGEMENTS

I would like to express my heartfelt gratitude to everyone who has played a pivotal
role in the successful completion of my MTech project thesis on Deep
Learning-based CCTV Footage Super-resolution for Human Subject Recognition
at the Indian Institute of Science, Bangalore.

First and foremost, I extend my deepest thanks to Professor Shalabh Bhatnagar,

my thesis supervisor and faculty mentor, for his unwavering support, guidance,
and expertise throughout this research journey. His mentorship has been
invaluable, and I am truly fortunate to have had the opportunity to work under
his supervision. His insightful feedback and suggestions have greatly enhanced the
quality of this project and played a significant role in shaping its overall direction.

I would also like to extend my gratitude to Deep Shekhar, my internal guide, for
his invaluable advice and encouragement during the course of this project. His
technical expertise and practical insights have been instrumental in addressing key
challenges and achieving the project objectives.

A special thanks goes out to all those who have directly or indirectly contributed
to this endeavor, including my family and friends, for their constant support and
encouragement. Their unwavering belief in my abilities has been a source of
strength throughout this journey.

This project has been an incredibly rewarding experience, allowing me to delve

into cutting-edge concepts in deep learning and apply them to practical challenges.
The knowledge gained has not only contributed to my academic growth but has
also laid a strong foundation for future endeavors.
 v

Once again, I extend my sincere appreciation to everyone involved in this project.

Your contributions have been invaluable, and I am truly grateful for the
opportunity to work with such a dedicated and supportive team.
 Abstract

In recent years, the application of deep learning techniques to enhance CCTV

footage for human subject recognition has garnered significant attention, especially
for defense applications. Traditional image enhancement methods, such as
interpolation, often fall short in recovering fine details essential for accurate
recognition. This project focuses on improving the recognition accuracy of human
subjects in low-resolution CCTV footage.

We employ the FaceNet model for image recognition, comparing its

performance on original low-resolution images versus super-resolved images.
FaceNet is effective for recognition tasks as it maps facial images into a compact
Euclidean space, where distances correspond to face similarity.

Experiments are conducted on two datasets: the DroneSURF Dataset and the
CVBL-CCTV-Face Dataset. The DroneSURF Dataset includes images captured
from drones, presenting challenges such as varying altitudes, angles, and lighting
conditions. The CVBL-CCTV-Face Dataset comprises facial images collected
under various conditions, including different poses, expressions, and lighting
environments. These datasets provide a comprehensive benchmark for testing the
effectiveness of super-resolution techniques.

The CVBL-CCTV-Face dataset was meticulously crafted to support the

development and evaluation of facial recognition systems in low-resolution CCTV
environments. The dataset creation involved raw data collection, preprocessing to
remove irrelevant content, and manual verification and annotation to ensure
accurate face detection and matching to high-definition images.

The results are analyzed to determine the most effective super-resolution

method for enhancing recognition accuracy in defense-related applications. By
systematically comparing the performance of these super-resolution models, this
study aims to offer insights into their practical application for improving
 vii

surveillance system capabilities. This research contributes to the development of

more reliable and accurate human subject recognition systems, aiding
advancements in security and defense technologies.
Contents

Abstract vi

Contents viii

List of Figures x

List of Tables xi

Abbreviations xii

1 Introduction 1
1.1 History of Face Recognition . . . . . . . . . . . . . . . . . . . . . . . 5
1.1.1 Early Development . . . . . . . . . . . . . . . . . . . . . . . . 5
1.1.2 Linear Algebra Techniques . . . . . . . . . . . . . . . . . . . . 6
1.1.3 Statistical Approaches . . . . . . . . . . . . . . . . . . . . . . 7
1.1.4 Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.1.5 Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.3 Principal Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.4 Structure of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2 Related Work 15
2.1 Convolutional Neural Network Models . . . . . . . . . . . . . . . . . 15
2.2 Advanced CNN Models with Novel Loss Functions . . . . . . . . . . . 16
2.3 Machine Learning and Hybrid Models . . . . . . . . . . . . . . . . . . 16
2.4 Specialized Architectures . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.5 Our Contribution to the Field . . . . . . . . . . . . . . . . . . . . . . 18
2.5.1 Dataset Creation . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.5.2 Face Recognition Model Comparison . . . . . . . . . . . . . . 18
2.5.3 Practical Applications . . . . . . . . . . . . . . . . . . . . . . 19

3 Dataset 20
3.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

viii
 ix

3.2 DroneSURF Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

4 Methodology and Experimental Setup 23

4.1 Proposed Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.1.1 Detection by MTCNN . . . . . . . . . . . . . . . . . . . . . . 24
4.1.2 Super-Resolution . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.1.3 Recognition using FaceNet . . . . . . . . . . . . . . . . . . . . 24
4.2 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.2.1 Hardware Configuration . . . . . . . . . . . . . . . . . . . . . 25
4.2.2 Software Configuration . . . . . . . . . . . . . . . . . . . . . . 25

5 Results 27
5.1 Illustrative Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
5.2 Mathematical Result . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
5.2.1 DroneSURF Dataset . . . . . . . . . . . . . . . . . . . . . . . 28
5.3 Additional Face Recognition Models . . . . . . . . . . . . . . . . . . . 28
5.3.1 Dlib . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
5.3.2 ArcFace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
5.3.3 FaceNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

6 Conclusion and Future Work 31

6.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
6.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
6.2.1 Exploration of Advanced Super-Resolution Models . . . . . . 33
6.2.2 Real-World Testing and Applicability . . . . . . . . . . . . . . 34
6.2.3 Combination with Other Enhancement Methods . . . . . . . . 34
6.2.4 Hybrid Systems and Multi-Modal Recognition . . . . . . . . . 35
6.2.5 Scalability and Large-Scale Applications . . . . . . . . . . . . 36
6.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

Bibliography 37
List of Figures

4.1 Proposed architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

5.1 Illustrative result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

x
List of Tables

3.1 Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

5.1 Recognition accuracy on DroneSURF Dataset. . . . . . . . . . . . . . 28

5.2 Number of Embeddings Generated by Different Models on DroneSurf
Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

xi
Abbreviations

SR Super-Resolution
ESRGAN Enhanced Super-Resolution Generative Adversarial Network
VDSR Very Deep Super-Resolution
FSRCNN Feature Super-Resolution Convolutional Neural Network
SRCNN Super-Resolution Convolutional Neural Network
GAN Generative Adversarial Network
PSNR Peak Signal-to-Noise Ratio
SSIM Structural Similarity Index Measure
DIP Deep Image Prior
RCAN Residual Channel Attention Network

xii
Chapter 1

Introduction

The proliferation of CCTV systems has significantly enhanced surveillance

capabilities, enabling the monitoring of public and private spaces for security
purposes. These systems are increasingly used in urban areas, airports,
government facilities, commercial buildings, and private spaces to ensure safety,
prevent criminal activities, and support law enforcement. Their widespread
adoption has been driven by advances in camera technology, data storage, and
real-time video streaming capabilities. Today, these systems play a crucial role in
maintaining security in various settings, including monitoring public spaces for
safety, tracking movement in high-risk areas, and assisting in the investigation of
criminal activities.

However, despite their ubiquity, the effectiveness of CCTV systems is often

hindered by one significant limitation: the low resolution of the captured footage.
The quality of the images captured by CCTV cameras can be compromised by
several factors, such as the camera’s resolution, distance from the subject, lighting
conditions, and environmental variables such as fog or rain. This results in grainy,
pixelated images that obscure critical details, including the faces and identifying
features of individuals. In many cases, low-resolution footage makes it challenging,

1
 2

if not impossible, to accurately recognize or identify people, which is a serious

drawback in security and surveillance applications.

This limitation becomes particularly problematic in defense and security

applications, where precise identification is essential. For example, in national
security contexts, low-resolution footage captured in sensitive areas may be the
only available evidence for identifying suspects or verifying the movements of
individuals of interest. Similarly, in urban surveillance systems, authorities need to
reliably identify suspects, track criminal activities, and prevent potential threats in
real time. In such scenarios, the inability to identify individuals accurately due to
low-quality video is a significant issue that compromises the effectiveness of these
systems.

To address this challenge, the integration of advanced image enhancement

techniques, such as super-resolution (SR), has emerged as a promising solution.
Super-resolution is a technique used to improve the quality of low-resolution
images by reconstructing high-resolution details. By enhancing the clarity and
detail of the images before processing them through recognition algorithms,
super-resolution can significantly improve the accuracy of facial recognition
systems, even when the input images are of low quality.

The core objective of this project is to explore how super-resolution techniques can
be integrated with deep learning-based facial recognition models to improve human
subject identification in low-resolution CCTV footage. In particular, this study
focuses on using super-resolution techniques to enhance image quality and,
subsequently, improve the performance of the FaceNet model, a state-of-the-art
deep learning-based facial recognition system. FaceNet has gained widespread
popularity due to its ability to map facial images into a compact Euclidean space,
where the distances between points directly correspond to the similarity of faces.
This feature makes FaceNet particularly suitable for human identification tasks, as
it enables highly accurate recognition, even with subtle differences in facial
features.

IISc Bengaluru
 3

Super-resolution techniques can be broadly classified into two categories:

traditional methods and deep learning-based methods. Traditional
super-resolution techniques typically involve interpolating low-resolution images to
estimate the missing high-frequency details. These methods often struggle to
capture fine details, particularly in challenging conditions such as low-light
environments or images with significant noise. On the other hand, deep
learning-based super-resolution techniques, which utilize convolutional neural
networks (CNNs) and other advanced neural network architectures, have
demonstrated superior performance in enhancing image quality by learning
complex patterns in data. These methods are capable of generating high-resolution
images that are more detailed and accurate, making them ideal candidates for
improving the quality of low-resolution CCTV footage.

In this project, we focus on evaluating several deep learning-based super-resolution

models, including GFPGAN [1], CodeFormer [2], Unpaired SR [3], ESRGAN [4],
and Real-ESRGAN [5]. These models have been selected due to their impressive
performance in generating high-quality images from low-resolution inputs.
GFPGAN, for example, has shown exceptional results in face restoration tasks,
while ESRGAN and Real-ESRGAN are known for their ability to produce
high-quality images with minimal artifacts. CodeFormer and Unpaired SR are
notable for their ability to perform super-resolution without the need for paired
training data, making them well-suited for real-world surveillance scenarios where
paired low-resolution and high-resolution images are often unavailable.

The DroneSURF dataset [6] is used for evaluating the performance of these
super-resolution models. This dataset consists of images captured by drones,
offering a unique challenge due to the dynamic nature of the environment in which
the images are collected. The DroneSURF dataset includes various factors that
can impact the quality of the images, such as varying altitudes, angles, lighting
conditions, and poses of individuals. These variations make it an ideal testbed for
evaluating the robustness of super-resolution models in real-world surveillance

IISc Bengaluru
 4

scenarios. The dataset provides a diverse set of facial images, collected under
different conditions, including varying lighting environments, facial expressions,
and orientations. As such, it serves as a comprehensive benchmark for assessing
how well super-resolution techniques can enhance the performance of facial
recognition systems in complex, uncontrolled environments.

The significance of this research lies in its potential to address one of the most
persistent problems in surveillance and security systems—accurate human
identification from low-resolution footage. By combining advanced super-resolution
techniques with deep learning-based facial recognition models, this project aims to
develop a solution that can significantly improve recognition accuracy, even in
challenging conditions. The success of this approach could lead to the development
of more reliable and effective surveillance systems, particularly for defense
applications, where the need for precise identification is critical.

Furthermore, the findings of this research could have broader implications beyond
surveillance. Improved facial recognition accuracy in low-resolution footage could
benefit a wide range of applications, including law enforcement, border control,
crowd management, and even personal security systems. With the rapid expansion
of CCTV systems and the increasing reliance on video-based surveillance for public
safety, the ability to enhance image quality and improve recognition performance is
more important than ever. This research, therefore, not only addresses a critical
gap in current surveillance systems but also contributes to the broader field of
computer vision and machine learning, particularly in the context of real-world
applications where image quality can vary significantly.

In summary, this study seeks to demonstrate the potential of super-resolution

techniques in overcoming the limitations of low-resolution CCTV footage, thereby
improving the effectiveness of human subject recognition systems. By integrating
these techniques with advanced facial recognition models like FaceNet and testing
their performance on challenging datasets such as DroneSURF, we aim to provide
valuable insights into how these methods can enhance the robustness of

IISc Bengaluru
 5

surveillance systems in defense and security applications. The success of this

research could lead to the development of more accurate, reliable, and efficient
systems that can better serve the needs of public safety and national security.

This extended introduction now includes a more comprehensive background on the

importance of surveillance systems, the challenges posed by low-resolution CCTV
footage, the potential of super-resolution techniques, and the context in which
your research fits. The section also emphasizes the broader significance of the
study in the field of security and computer vision. Let me know if you’d like to
make further adjustments!

1.1 History of Face Recognition

Face recognition, as a field of study, has a rich history that spans several decades,
marked by advancements in technology and the evolution of scientific
understanding. The journey from manual methods to the sophisticated deep
learning algorithms used today showcases the rapid progress made in automating
human subject identification.

1.1.1 Early Development

The roots of face recognition can be traced back to the 1960s and 1970s when the
field was in its infancy. Early efforts at face recognition involved manually
extracting key facial features from photographs. Researchers would measure
distances between facial landmarks such as the eyes, nose, and mouth. These
measurements were then used for comparing different facial features and making
identifications. This manual process was time-consuming and often lacked the
precision necessary for reliable recognition, especially in large datasets.
Nevertheless, this period laid the foundation for more advanced computational

IISc Bengaluru
 6

approaches by demonstrating the feasibility of using facial features as a biometric

marker for identification.

One of the significant early milestones was the creation of the first computer-based
face recognition system in 1965 by Woodrow W. Bledsoe. Bledsoe’s approach
involved encoding facial features by manually plotting points on a face and storing
them as data, which was then compared to a database of known faces. Though
rudimentary by today’s standards, this work introduced the concept of using a
computer for facial analysis and set the stage for future developments.

1.1.2 Linear Algebra Techniques

The 1980s and 1990s marked the shift toward more mathematical and algorithmic
approaches to face recognition. Researchers began to explore the application of
linear algebra techniques to facial recognition. A key breakthrough in this era was
the development of the Eigenfaces method by Sirovich and Kirby in 1991, which
introduced the concept of dimensionality reduction using Principal Component
Analysis (PCA). The Eigenfaces method involved projecting high-dimensional
facial data into a lower-dimensional space, effectively capturing the most
significant variations in facial features while discarding noise. This allowed for
more efficient processing of facial data and improved the system’s ability to handle
large datasets.

Eigenfaces were a breakthrough because they transformed the face recognition

problem from a high-dimensional search to a more manageable one by reducing the
number of variables involved. Although this approach was groundbreaking, it still
had its limitations, particularly in dealing with variations in lighting, expression,
and orientation of faces. However, it demonstrated the power of mathematical
tools such as PCA in addressing the complexity of facial recognition tasks.

IISc Bengaluru
 7

1.1.3 Statistical Approaches

Building on the success of Eigenfaces, the next phase of development focused on

enhancing the accuracy and robustness of face recognition systems. Fisherfaces,
developed by Belhumeur, Hespanha, and Nayar in 1997, improved on the
Eigenfaces method by incorporating Linear Discriminant Analysis (LDA), a
statistical technique that maximizes the separation between different classes (i.e.,
individuals) while minimizing within-class variance. Fisherfaces offered better
performance in situations where faces varied due to lighting changes, facial
expressions, or occlusions, making it a more robust method than PCA alone.

These advancements were crucial because they addressed some of the fundamental
challenges in face recognition systems—variability in facial appearance and
background conditions. However, the reliance on handcrafted features and linear
models still limited the flexibility and scalability of these systems, especially when
faced with highly diverse datasets.

1.1.4 Machine Learning

The late 1990s and early 2000s saw the advent of machine learning algorithms that
pushed face recognition into a new era. Instead of relying solely on linear
transformations, machine learning approaches used algorithms that could learn
from data and improve over time. Support Vector Machines (SVMs), introduced
by Vapnik in 1995, became popular for face classification tasks due to their ability
to handle high-dimensional data efficiently. SVMs worked by finding the optimal
hyperplane that best separates different classes of data in a higher-dimensional
space, making them particularly well-suited for face recognition applications.

Alongside SVMs, neural networks began to gain traction in face recognition tasks.
The introduction of multi-layer perceptrons (MLPs) allowed researchers to train
deeper models capable of learning more complex relationships between facial

IISc Bengaluru
 8

features. Although these models were limited by computational power and data
availability, they set the stage for the next generation of face recognition systems
based on deep learning.

1.1.5 Deep Learning

The true revolution in face recognition came with the advent of deep learning
techniques, particularly Convolutional Neural Networks (CNNs), which have since
become the gold standard for many computer vision tasks, including face
recognition. CNNs, introduced by Yann LeCun and his colleagues in the late
1980s, became widely popular after the success of AlexNet in the 2012 ImageNet
competition. CNNs are able to automatically learn hierarchical feature
representations from raw image data, eliminating the need for manual feature
extraction, which was a major limitation in earlier methods.

This shift towards deep learning significantly improved the accuracy and
scalability of face recognition systems. Several landmark models in face recognition
emerged in the 2010s, including DeepFace (2014), developed by Facebook, which
utilized a deep CNN architecture to achieve human-level accuracy in face
recognition tasks. Another important model was FaceNet (2015), developed by
Google, which introduced a novel approach called triplet loss to train the network.
This method ensured that images of the same person were embedded closer
together in feature space, while images of different people were placed farther
apart. This led to a significant improvement in both the accuracy and
generalization of face recognition systems.

Other important deep learning models in this space include VGGFace, OpenFace,
and ArcFace, all of which have achieved state-of-the-art performance in face
recognition tasks. These models leverage vast amounts of labeled data and
advanced neural network architectures to achieve unprecedented accuracy in both
controlled and real-world settings.

IISc Bengaluru
 9

1.2 Motivation

The motivation behind this project stems from the growing need to enhance
surveillance and security systems in increasingly complex environments. As
urbanization increases and more CCTV cameras are deployed worldwide, the
reliance on video surveillance for public safety and law enforcement is stronger
than ever. However, one of the most persistent challenges faced by these systems is
the low quality of video footage, especially in large-scale surveillance systems that
capture footage from distant or obscured angles, in low-light conditions, or at high
altitudes. In such cases, the low resolution of the video footage makes it difficult to
accurately recognize and identify individuals, which is crucial for security purposes.

The inability to reliably identify individuals from low-resolution CCTV footage is

a significant challenge in areas such as urban surveillance, airport security, and
defense applications. In national security contexts, identifying individuals of
interest based on limited or poor-quality visual data can be a matter of critical
importance. In these situations, the need for highly accurate recognition systems
that can operate on low-resolution footage becomes paramount.

Traditional face recognition techniques, while effective under controlled conditions,

often struggle with real-world challenges such as low-resolution video, varying
lighting conditions, and occlusions. This is where super-resolution (SR) techniques
come into play. Super-resolution, a process of enhancing the resolution of images
or videos, has the potential to restore critical facial features lost in low-resolution
images. By increasing the quality of the input images, super-resolution techniques
provide a means to significantly improve the performance of facial recognition
systems, particularly in low-quality video footage.

The core motivation of this project is to explore how combining super-resolution

techniques with deep learning-based face recognition models can significantly
improve the accuracy of human subject recognition. The FaceNet model, a
state-of-the-art deep learning-based facial recognition system, has demonstrated

IISc Bengaluru
 10

remarkable performance in identifying individuals from high-resolution images.

However, its performance often deteriorates when faced with low-resolution
footage, as critical facial features become too distorted or blurred for accurate
recognition.

By integrating advanced super-resolution methods with FaceNet, this research

aims to overcome the limitations of low-resolution images. The goal is to improve
recognition accuracy even when the input footage is of poor quality, thereby
making face recognition systems more reliable and robust in real-world surveillance
applications. In doing so, this project seeks to contribute valuable insights and
solutions that can enhance the effectiveness of surveillance systems in defense, law
enforcement, and national security, ensuring better protection of public safety.

Furthermore, the findings from this research are expected to have broader
implications for surveillance systems worldwide, where the need for efficient,
high-accuracy recognition from low-resolution video footage is becoming
increasingly critical. Whether used for tracking suspects in criminal investigations,
managing crowds at public events, or improving security at critical infrastructure
sites, the ability to enhance the recognition accuracy of facial recognition systems
will play a crucial role in strengthening public safety and national security in the
years to come.

1.3 Principal Contributions

This research aims to tackle one of the most pressing issues in modern surveillance
systems: improving human subject recognition accuracy in low-resolution CCTV
footage. To this end, we present several key contributions to the field, each of
which adds significant value to the current state of research and provides practical
solutions for enhancing the performance of facial recognition systems in real-world
conditions.

IISc Bengaluru
 11

• Dataset Creation: One of the fundamental contributions of this research

is the development of the DroneSURF Dataset [6]. This dataset has been
specifically designed to address the challenges posed by low-resolution images
captured in dynamic and uncontrolled environments. It includes a diverse
collection of images captured from drones at varying altitudes, angles, and
lighting conditions, as well as images of individuals in different poses and
expressions. The DroneSURF dataset provides a comprehensive benchmark
for testing super-resolution and recognition models, enabling a rigorous eval-
uation of the effectiveness of advanced image enhancement techniques and
facial recognition algorithms. This dataset serves as a valuable resource for
researchers working in the field of facial recognition in surveillance applica-
tions, particularly in environments where high-quality images may not always
be available.

• Face Recognition Comparison: This research conducts a detailed and sys-

tematic comparison of several popular face recognition models, including Dlib
[7], ArcFace [8], and FaceNet [9]. Each of these models is evaluated based
on its performance in recognizing faces from low-resolution images, a critical
challenge in real-world surveillance scenarios. By comparing the performance
of these models across various recognition tasks, this study highlights their
respective strengths and limitations. In particular, we identify the most ef-
fective model for enhancing recognition accuracy under low-resolution condi-
tions. This comparison serves as an important reference for practitioners and
researchers seeking to select the most appropriate face recognition model for
deployment in surveillance systems.

• Super-Resolution Integration: Another major contribution is the integra-

tion of advanced super-resolution techniques with facial recognition systems to
improve the quality of low-resolution images. Super-resolution, a process that
enhances the resolution of images and restores fine-grained details, is explored
in depth through various methods, such as GFPGAN [1], CodeFormer [2], and

IISc Bengaluru
 12

Real-ESRGAN [5]. By applying these techniques to the DroneSURF dataset,

this research demonstrates how image enhancement can dramatically improve
the accuracy of face recognition systems. This integration of super-resolution
with face recognition models represents a significant step forward in address-
ing the challenges of low-quality surveillance footage.

• Fusion Model Development: Based on the findings from the previous ex-
periments, this study proposes a fusion model that combines the strengths
of super-resolution techniques with the best-performing face recognition mod-
els. The fusion approach optimizes the recognition accuracy by leveraging the
complementary strengths of each individual component. The experimental re-
sults indicate that the fusion model significantly outperforms the standalone
models, making it a promising approach for real-world surveillance systems.
This contribution provides a new avenue for future research and development
in facial recognition and image enhancement technologies.

• Evaluation and Benchmarking: To ensure the reliability and validity of the

proposed models, extensive evaluations and benchmarking are conducted. The
performance of various super-resolution and face recognition models is assessed
using rigorous evaluation metrics, including accuracy, precision, recall, and F1-
score. These metrics provide a clear picture of the strengths and weaknesses
of each approach, helping to guide future improvements and adaptations of
the models for deployment in different surveillance contexts.

1.4 Structure of the Thesis

The structure of this thesis is organized in a manner that systematically presents

the research, methodology, experimental setup, and results, ensuring a
comprehensive understanding of the work undertaken. The chapters are designed
to provide the reader with a clear view of the problem, solution approach,
experiments, and conclusions drawn from the study.

IISc Bengaluru
 13

• Chapter 1: Introduction – This chapter introduces the core problem of the

research, the challenges faced by low-resolution facial recognition systems in
surveillance environments, and the objectives of the study. It also outlines the
importance of super-resolution techniques and the FaceNet model in improving
recognition accuracy, specifically in defense and public safety applications.
The chapter concludes by highlighting the contribution of the research and its
potential impact on security systems.

• Chapter 2: Literature Review – In this chapter, a comprehensive review

of the existing literature on facial recognition, low-resolution image enhance-
ment, and super-resolution techniques is presented. Key milestones in the
development of face recognition models are discussed, along with the advance-
ments in super-resolution methods that have made it possible to improve the
quality of low-resolution images. The chapter provides a detailed analysis of
relevant studies, identifying gaps in the current research and setting the stage
for the contributions made in this thesis.

• Chapter 3: Dataset and Methodology – This chapter describes the

DroneSURF Dataset in detail, outlining its creation, structure, and the vari-
ous challenges it presents for facial recognition systems. The methodology for
the recognition task is explained, including the pre-processing steps, super-
resolution techniques, and the selection of facial recognition models. This
chapter also provides an overview of the experimental pipeline used to evalu-
ate the models.

• Chapter 4: Experimental Setup and Results – This chapter presents

the experimental setup, including the hardware and software used to train
and test the models. The evaluation metrics, such as accuracy, precision,
recall, and F1-score, are introduced, and the results of the experiments are
provided. Detailed comparisons between different facial recognition models,
both with and without super-resolution techniques, are discussed. The chapter

IISc Bengaluru
 14

also includes the performance of the proposed fusion model, demonstrating its
superiority over individual models in terms of recognition accuracy.

• Chapter 5: Conclusion and Future Work – The final chapter summa-

rizes the key findings of the research and discusses the implications of the
results. It also highlights the limitations of the current study and suggests
directions for future research. Potential areas for improvement include the ex-
ploration of more advanced super-resolution techniques, the use of additional
facial recognition models, and the application of the fusion model to other
types of datasets. This chapter concludes by reiterating the importance of
enhancing facial recognition systems for low-resolution CCTV footage and its
potential impact on security applications.

By following this structure, the thesis provides a comprehensive and logical

progression of the research, from problem definition to solution approach,
experimental validation, and conclusions. Each chapter builds on the previous one,
ensuring that the reader can follow the rationale behind the research and
understand the significance of the findings.

IISc Bengaluru
Chapter 2

Related Work

The development of face recognition models has seen significant advancements,

transitioning from early deep convolutional neural networks (CNNs) [10] to more
sophisticated models incorporating novel loss functions and transformer-based
architectures. This section reviews some of the key contributions in the field,
highlighting the progression and addressing the challenges tackled by successive
models.

2.1 Convolutional Neural Network Models

The earliest notable model, DeepFace [11], utilized a 9-layer deep CNN architecture
for face verification tasks. DeepFace [11] incorporated multi-task learning and
supervised pre-training on the Labeled Faces in the Wild (LFW) [12] dataset,
achieving near-human performance. However, it faced challenges in generalizing
across different demographics, poses, illumination conditions, and ages.

Following DeepFace [11], VGGFace [13] employed a deeper CNN [10] architecture
based on the VGG-16 model, trained on the VGGFace dataset [13] containing over
2.6 million images. VGGFace [13] improved accuracy but struggled with variations

15
 16

in facial expressions, occlusions, and aging. To address these limitations, the

VGGFace2 [14] dataset was introduced, offering more diverse images from over
9,000 identities to better generalize across various conditions. Despite
improvements, challenges remained in achieving robust performance in real-world
scenarios with complex variations.

2.2 Advanced CNN Models with Novel Loss

Functions

SphereFace [15] introduced the Angular Softmax (A-Softmax) loss function to

learn discriminative features by mapping face images onto a hypersphere manifold.
Although it achieved significant success, scaling to handle large-scale datasets and
improving robustness to variations posed ongoing challenges. ArcFace [8] and
InsightFace [16] both utilized the Additive Angular Margin (ArcFace) [8] loss
function to enhance discriminative power by optimizing angular differences
between feature embeddings. Both models continue to be refined for better
scalability and robustness in handling challenging scenarios.

2.3 Machine Learning and Hybrid Models

Dlib [7] Face Recognition combines machine learning algorithms, including a face
detection pipeline based on Histogram of Oriented Gradients (HOG) features and
a face recognition model using deep metric learning techniques. It can be trained
on various datasets, including LFW [12] and CIFAR-10 [17]. However, enhancing
Dlib’s capability to handle large-scale datasets with diverse facial variations and
improving its scalability for complex face recognition tasks remains a challenge.

DeepID3 [18] extends the DeepID [19] models by incorporating deeper layers and
attention mechanisms to enhance feature learning and discriminative power.

IISc Bengaluru
 17

Trained on datasets like LFW [12], YouTube Faces [20], and MegaFace [21],
DeepID3 [18] achieved high accuracy. Nevertheless, it faced challenges in handling
variations in pose, illumination, and expression, as well as scalability issues in
large-scale face recognition scenarios.

OpenFace [22] utilizes deep neural networks (DNNs) [23] for face detection, facial
landmark localization, and face verification, employing a multi-task learning
approach. Despite its versatility, optimizing OpenFace [22] for real-time
performance on resource-constrained devices and enhancing its accuracy in
handling diverse facial variations and environmental conditions remain areas for
improvement.

MobileFaceNet [24] is a compact CNN [10] architecture designed specifically for

face recognition tasks on mobile and embedded devices. It focuses on reducing
model size and computational complexity while maintaining high accuracy.
Trained on datasets such as CASIA-WebFace [25], MegaFace [21], and LFW [12],
MobileFaceNet [24] shows ongoing efforts to improve robustness in handling
variations in lighting, pose, and facial expressions, especially in real-world
scenarios.

2.4 Specialized Architectures

The Memory-Modulated Transformer Network (MMT) [26] integrates memory

modules with a transformer-based architecture to handle variations in facial
appearance due to factors like aging, pose, and expression. Evaluated on standard
face recognition datasets, MMT [26] aims to improve generalization across different
demographics and varying conditions.

In summary, the evolution of face recognition models has been marked by

continuous improvements in handling complex variations in facial images, from
early CNN [10] models to advanced models incorporating novel loss functions and

IISc Bengaluru
 18

transformer-based architectures. Each generation of models has built upon the

limitations of its predecessors, paving the way for more robust and accurate face
recognition systems in real-world applications.

2.5 Our Contribution to the Field

Our research makes significant contributions to the field of face recognition and
surveillance technology in several key areas:

2.5.1 Dataset Creation

We developed the DroneSURF [6] dataset, specifically designed to support the

development and evaluation of facial recognition systems in low-resolution CCTV
environments. This dataset includes a diverse range of images collected under
various conditions, such as different poses, expressions, and lighting environments.
The meticulous process of raw data collection, preprocessing, and manual
annotation ensures high-quality data that accurately represents real-world
surveillance scenarios. This dataset provides a comprehensive benchmark for
testing and improving face recognition models, contributing to more effective and
reliable surveillance systems.

2.5.2 Face Recognition Model Comparison

Our study conducts a detailed comparison of several prominent facial recognition

models, including Dlib [7], ArcFace [8], and FaceNet [9]. By evaluating these
models on both the original low-resolution images and the enhanced images from
our dataset, we provide valuable insights into the strengths and limitations of each
approach. This comparative analysis helps identify the most effective model for

IISc Bengaluru
 19

improving recognition accuracy in low-resolution surveillance footage, guiding

future research and practical applications in the field.

2.5.3 Practical Applications

The insights gained from our research have practical implications for enhancing
security and surveillance operations, particularly in defense and public safety
applications. By providing a detailed evaluation of face recognition models and a
high-quality dataset, we contribute to the development of more effective
surveillance technologies that can better protect public safety and national
security.

IISc Bengaluru
Chapter 3

Dataset

3.1 Dataset

Understanding the challenges faced in facial recognition from low-resolution CCTV

images is essential for improving the robustness of facial recognition systems.
Low-quality footage often introduces issues such as occlusion, motion blur, and
varying lighting conditions, all of which can significantly hinder accurate
identification. To address these challenges, advanced techniques such as image
enhancement, super-resolution, and more sophisticated recognition models are
crucial. These methods aim to enhance the quality of images, making it easier to
extract detailed facial features even from low-resolution footage.

By leveraging these techniques, it is possible to improve the performance of facial

recognition systems in real-world CCTV environments, leading to more accurate
and reliable security systems.

20
 21

3.2 DroneSURF Dataset

The DroneSURF [6] dataset is another critical resource utilized in this project,
specifically for testing the robustness of super-resolution techniques in facial
recognition. This dataset consists of images captured by drones, providing a unique
perspective on surveillance footage. The use of drones introduces several challenges
that traditional CCTV footage does not, such as varying altitudes, dynamic
angles, and unpredictable lighting conditions. These factors make the dataset
particularly useful for evaluating how well facial recognition systems can adapt to
real-world scenarios where environmental conditions are constantly changing.

DroneSURF features individuals captured in diverse outdoor settings, including

urban environments, open fields, and crowded areas. This variability helps
simulate the types of real-world conditions that surveillance systems face, where
factors such as motion blur, shadows, and distant subjects may degrade image
quality. Additionally, the dataset includes frames with varying facial expressions
and partial occlusions, further challenging the recognition models to handle such
complexities.

A key strength of the DroneSURF dataset is its wide range of video content,
capturing over 200 videos across 58 unique subjects. This variety ensures that the
models trained and tested on it can generalize well to different environments and
conditions. With a large number of annotated faces (over 786,000), the dataset
provides a comprehensive benchmark for evaluating the effectiveness of image
enhancement techniques, including super-resolution, in improving the accuracy of
facial recognition in low-resolution images.

The dataset’s diversity in terms of environmental factors and subject variety

makes it an excellent choice for testing the limits of facial recognition models. It
helps assess how well these models perform when exposed to challenging conditions
such as low altitudes, extreme angles, and changes in lighting—common
occurrences in real-world surveillance applications.

IISc Bengaluru
 22

Through the use of the DroneSURF dataset, we aim to push the boundaries of
facial recognition systems, ensuring that they can perform accurately and reliably
even in less-than-ideal conditions.
Table 3.1: Characteristics

Characteristic Active Passive Total

Videos 100 100 200
Subjects 58 58 58
Mean (secs.) 57 79 68
Min. (secs.) 18 34 18
Total Frames 172,263 239,188 411,451
Annotated
332,693 379,504 786,813
Faces

IISc Bengaluru
Chapter 4

Methodology and Experimental

Setup

4.1 Proposed Architecture

Figure 4.1: Proposed architecture

The architecture has three modules: the first module is for face detection and
cropping using MTCNN[27] (Figure 4.1(a)). The second module is
super-resolution for enhancing the image (Figure 4.1(b)). The third module is for
face recognition using the FaceNet[9] model (Figure 4.1(c)).

23
 24

4.1.1 Detection by MTCNN

The process begins with an input image containing multiple faces. Multi-task
Cascaded Convolutional Networks (MTCNN) [27] are employed to detect faces
within the input image due to their high accuracy and efficiency in face detection
tasks. MTCNN [27] works by running a series of neural networks to first identify
potential face regions and then refine these detections to pinpoint the exact
locations of the faces. The output of this stage is the detected faces, which are
cropped from the input image, resulting in several low-resolution (LR) facial
images ready for further processing.

4.1.2 Super-Resolution

The low-resolution facial images (LR) obtained from the MTCNN[27] detection
phase are then fed into a super-resolution model. This model can be one of the
advanced super- resolution techniques such as GFPGAN[1], CodeFormer[2], Un-
paired SR[3], ESRGAN[4], or Real-ESRGAN[5]. Each of these models applies deep
learning algorithms to enhance the resolution of the input images, producing
high-quality super-resolved (SR) images. The super-resolution process involves
upscaling the images and recovering high-frequency details that were lost in the
low-resolution version. The output of this stage is the super-resolved (SR) facial
images, which exhibit significantly improved clarity and detail compared to the
original low-resolution images.

4.1.3 Recognition using FaceNet

The final stage involves facial recognition using the FaceNet[9] deep neural
network. Both the super-resolved (SR) images and the high-resolution (HR)
reference images are passed through the FaceNet[9] network to perform facial
recognition. FaceNet[9] processes these images to generate their corresponding

IISc Bengaluru
 25

embeddings, which are fixed-size vectors that capture the essential features of the
faces. For the super-resolved images, the embeddings generated (SR Embedding)
represent the enhanced facial details, while the high-resolution images generate
their own set of embeddings (HR Embedding). These embeddings are then
compared to compute distance scores, which quantify the similarity between the
SR and HR embeddings. Lower distance scores indicate higher similarity,
facilitating accurate recognition of individuals. This process ensures that even
faces enhanced from low-resolution images can be accurately recognized,
demonstrating the effectiveness of combining super-resolution with advanced facial
recognition models.

4.2 Experimental Setup

The experiments were executed on a high-performance computing system with the

following specifications:

4.2.1 Hardware Configuration

GPUs: NVIDIA GPUs RTX 4090 with CUDA support for accelerated
computation.

CPU: High-performance multi-core CPU for handling preprocessing and data

management tasks.

Storage: High-capacity SSDs for fast data access and storage of large datasets
and model checkpoints.

4.2.2 Software Configuration

Operating System: Ubuntu 20.04 LTS

IISc Bengaluru
 26

Deep Learning Frameworks: TensorFlow and PyTorch for implementing and

training models.

CUDA and cuDNN: NVIDIA CUDA and cuDNN libraries for GPU
acceleration.

Python: Python 3.10 as the primary programming language, with various

libraries for data handling and preprocessing.

IISc Bengaluru
Chapter 5

Results

5.1 Illustrative Result

Figure 5.1: Illustrative result

The results of our thesis demonstrate the effectiveness of the proposed face
recognition enhancement process for CCTV footage. Initially, low-resolution video
frames with multiple individuals are used. The system detects and crops individual
faces, which are then enhanced to higher resolution. These enhanced faces are

27
 28

identified with unique IDs and matching scores, showcasing the improved accuracy
and reliability of face recognition. This process highlights the importance of
resolution enhancement in surveillance footage, significantly boosting recognition
performance from low-quality video inputs.

5.2 Mathematical Result

5.2.1 DroneSURF Dataset

The results of the experiments are presented in terms of the recognition accuracy of
the FaceNet [9] model on both regular and super-resolved images for each dataset.

Table 5.1: Recognition accuracy on DroneSURF Dataset.

S.No Model Name Pretrained Accuracy (%)

1 Without Super Resolution 42.81
2 CodeFormer [2] 58.92 60.72
3 Real-ESRGAN [5] 44.78 48.56

For the DroneSURF Dataset [6], GFPGAN [1] significantly outperformed other
models, achieving an accuracy of 69.16%. CodeFormer [2] also showed notable
improvement (59.56%), while ESRGAN [4] and Real-ESRGAN Real-ESRGAN [5]
showed relatively lower accuracy, indicating variability in performance across
different datasets.

5.3 Additional Face Recognition Models

Two other models were also evaluated for face recognition their performence is
given in the table below.

IISc Bengaluru
 29

Table 5.2: Number of Embeddings Generated by Different Models on DroneSurf

Dataset

Model No. of Embeddings on DroneSurf (Morning Category)

Dlib [7] 18,064
ArcFace [8] 27,657
FaceNet [9] 66,042
Total 66,633

5.3.1 Dlib

Dlib’s performance in generating embeddings was notably ineffective. The primary

challenge faced by Dlib’s face recognition module was its inability to manage the
variations and quality issues inherent in the dataset. Utilizing a ResNet-based
model for face recognition [7], Dlib struggled with the low-resolution and varying
conditions of the images, which resulted in poor overall performance. These
limitations highlighted the module’s inadequacy in handling datasets with
significant variations in image quality and resolution.

5.3.2 ArcFace

ArcFace demonstrated slightly better performance compared to Dlib. It was more

successful in generating embeddings, reflecting a higher robustness in dealing with
the dataset’s complexities. The key to ArcFace’s improved performance lies in its
use of an Additive Angular Margin Loss [8], which enhances the discriminative
power of the learned embeddings. By increasing the margin between classes,
ArcFace [8] effectively distinguishes between different individuals, thereby
achieving a higher success rate in embedding generation and recognition tasks.

5.3.3 FaceNet

FaceNet [9] outperformed both ArcFace [8] and Dlib [7], showing superior
performance in embedding generation with high accuracy and consistency. FaceNet

IISc Bengaluru
 30

[9]employs a deep convolutional network to produce embeddings and uses triplet

loss during training. This approach ensures that the distance between embeddings
of the same person is minimized while the distance between embeddings of
different persons is maximized. Such a method proved to be highly effective for the
given dataset, demonstrating FaceNet’s capability to recognize faces accurately
even under challenging conditions. This robust performance underscores FaceNet’s
suitability for tasks requiring high precision in face recognition.

IISc Bengaluru
Chapter 6

Conclusion and Future Work

6.1 Conclusion

This study aimed to address the challenge of recognizing individuals from

low-resolution CCTV footage, an issue that severely limits the effectiveness of
surveillance systems in security-critical applications such as national defense and
public safety. Through the integration of advanced super-resolution techniques
with the FaceNet facial recognition system, we aimed to significantly enhance
human subject recognition in such low-quality footage.

The research specifically evaluated five state-of-the-art super-resolution

models—GFPGAN [1], CodeFormer [2], Unpaired SR [3], ESRGAN [4], and
Real-ESRGAN [5]—to explore their impact on recognition accuracy. These models
were tested using the DroneSURF dataset [6], which provides a challenging
environment due to its dynamic and uncontrolled settings, such as varying
altitudes, lighting conditions, and angles.

Our results demonstrated that integrating super-resolution techniques with facial

recognition significantly improved accuracy. The GFPGAN model, in particular,
yielded the highest accuracy, reaching 69.16

31
 32

The importance of dataset-specific characteristics was also highlighted by our

findings. For example, certain models outperformed others depending on the type
of image quality and environmental variations within the dataset. This finding
emphasizes that no single super-resolution method is universally optimal across all
scenarios, and the choice of the model must be informed by the unique attributes
of the dataset and the problem at hand.

Our study contributes valuable insights to the growing field of surveillance

technology. By demonstrating that super-resolution can significantly enhance
recognition accuracy, this research paves the way for the development of more
reliable and effective facial recognition systems. The integration of advanced deep
learning techniques and super-resolution models can improve security and public
safety systems in applications such as urban surveillance, airport security, and law
enforcement. These systems are critical for monitoring large areas and detecting
potential threats or criminal activities, where real-time and accurate identification
is essential.

Moreover, the work demonstrates the increasing relevance of machine learning

techniques in enhancing the quality of CCTV footage. As surveillance technologies
evolve, the combination of super-resolution and facial recognition offers a pathway
to solving long-standing issues such as blurry, pixelated, or low-light images, which
are common in traditional CCTV systems. The findings suggest that future
security infrastructure can leverage these technologies to improve performance and
accuracy while maintaining efficiency in terms of computational resources.

In conclusion, this research has shown the potential of super-resolution techniques

in overcoming the limitations of low-resolution CCTV footage. The integration of
advanced models like GFPGAN with FaceNet has proven to enhance facial
recognition systems, offering a significant leap forward in the field of surveillance
and security technology. This work serves as a foundation for future research
aimed at further refining and optimizing these techniques, which can have broad

IISc Bengaluru
 33

implications for a variety of industries, including public safety, defense, and private
sector security.

6.2 Future Work

While this study demonstrates the potential of combining super-resolution

techniques with facial recognition for improving low-resolution CCTV footage,
there are several avenues for future research that could extend these findings and
address the remaining challenges in real-world applications. These include the
exploration of additional super-resolution models, improvements in real-time
applicability, integration with other enhancement methods, and the potential for
hybrid systems that combine multiple modalities of recognition.

6.2.1 Exploration of Advanced Super-Resolution Models

One area for future research lies in exploring additional state-of-the-art

super-resolution models that could potentially outperform the models tested in this
study. While models like GFPGAN and Unpaired SR showed promising results,
the field of super-resolution is rapidly advancing, with newer models frequently
emerging. These models often incorporate innovative loss functions, architectures,
or training strategies that may offer better performance on specific types of
images, such as those with extreme low-resolution, significant noise, or distortion.

Future studies could focus on integrating other cutting-edge techniques such as

Generative Adversarial Networks (GANs) or deep reinforcement learning (DRL)
approaches. GANs, in particular, have shown great promise in image generation
and enhancement tasks, offering the ability to generate high-quality images even
from extremely degraded inputs. Incorporating such models into the
super-resolution pipeline could lead to even higher-quality image reconstruction,
further enhancing the accuracy of facial recognition systems.

IISc Bengaluru
 34

Additionally, exploring unsupervised or self-supervised learning approaches for

super-resolution could be beneficial, as these techniques do not rely on large
amounts of labeled data, which is often difficult and expensive to obtain. This
could allow for the use of unannotated real-world surveillance footage to train
super-resolution models, making these methods more scalable and adaptable.

6.2.2 Real-World Testing and Applicability

While the results presented in this study were promising, they were based on
controlled datasets that, although challenging, do not fully replicate the
complexity of real-world surveillance environments. Future research should focus
on conducting field trials using real-world CCTV footage from various urban,
industrial, and defense environments. This will help assess the practical
applicability and performance of these techniques under more varied and
unpredictable conditions, such as changing weather, crowded scenes, or low-light
environments.

Furthermore, real-world testing will provide insights into the computational

efficiency of these models, which is critical for deployment in live surveillance
systems. It is essential that the super-resolution models do not introduce excessive
latency or require prohibitively high computational resources, especially for
real-time applications. Thus, future work should explore model optimization
techniques, such as pruning, quantization, or distillation, to make these models
more suitable for deployment in edge devices or resource-constrained environments.

6.2.3 Combination with Other Enhancement Methods

Another promising direction for future work involves the combination of

super-resolution with other image enhancement techniques. While super-resolution
focuses on improving the resolution of an image, other techniques, such as noise

IISc Bengaluru
 35

reduction, contrast adjustment, and sharpening, can further improve image

quality. Combining these methods could lead to more robust and high-quality
images that are better suited for accurate face recognition.

For example, noise reduction techniques, particularly those based on deep learning,
can be employed to remove artifacts introduced by low-light conditions or
poor-quality sensors. Contrast enhancement methods could help reveal subtle
facial features that are critical for recognition. These complementary techniques
could be used in a multi-step pipeline, where super-resolution is followed by
additional image enhancement steps to maximize recognition accuracy.

Moreover, incorporating pre-processing methods to handle environmental factors,

such as motion blur or occlusions (e.g., sunglasses or masks), could further improve
recognition in challenging surveillance environments. Integrating multi-modal
data, such as infrared or depth information, could also enhance recognition
accuracy, particularly in conditions where visible spectrum data is insufficient.

6.2.4 Hybrid Systems and Multi-Modal Recognition

Facial recognition systems could also benefit from integrating multiple recognition
modalities, such as combining facial recognition with other biometrics, like gait
analysis, voice recognition, or license plate recognition. A hybrid system that uses
different recognition techniques can provide more accurate and reliable results by
cross-validating information from multiple sources.

Furthermore, combining super-resolution with other machine learning techniques,

such as object detection or activity recognition, could enhance the overall
surveillance system’s ability to track and identify subjects in real-time.
Multi-modal recognition systems can offer greater robustness, especially in
complex scenarios where one modality (such as facial recognition) may not perform
well due to occlusions or low-quality images.

IISc Bengaluru
 36

6.2.5 Scalability and Large-Scale Applications

In addition to enhancing accuracy, future work should also focus on improving the
scalability of these techniques for large-scale deployments. Surveillance systems
often involve monitoring vast areas with numerous cameras, which results in a high
volume of video data. Processing this data in real-time or even offline for
recognition requires highly scalable solutions that can handle large amounts of
data efficiently.

Developing scalable algorithms that can process video frames in parallel or using
distributed computing resources would enable the use of super-resolution and
facial recognition technologies in large-scale urban and defense applications. These
solutions must strike a balance between accuracy, efficiency, and cost-effectiveness.

6.3 Summary

In conclusion, while this research has made significant strides in enhancing

low-resolution CCTV footage through super-resolution techniques, there remain
several exciting directions for future work. By exploring more advanced
super-resolution models, conducting real-world trials, integrating additional image
enhancement methods, and expanding to hybrid and multi-modal systems, future
studies can continue to improve the accuracy, efficiency, and applicability of facial
recognition technologies in surveillance and security systems. These advancements
will play a crucial role in ensuring that surveillance technologies remain effective
and reliable in increasingly complex and dynamic environments.

IISc Bengaluru
Bibliography

[1] X. Wang, Y. Li, H. Zhang, and Y. Shan, “Towards real-world blind face restora-
tion with generative facial prior,” in Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition (CVPR) Workshops, 2021.

[2] Q. Guo, X. Li, Y. Zhang, Y. Fu, and T. H. Li, “Codeformer: Towards robust
face restoration with codebook lookup transformer,” in Proceedings of the 30th
ACM International Conference on Multimedia (MM ’22), 2022.

[3] A. Bulat, J. Yang, and G. Tzimiropoulos, “To learn image super-resolution,

use a gan to learn how to do image degradation first,” in arXiv preprint
arXiv:2003.04047, 2021.

[4] X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, C. C. Loy, Y. Qiao, and
X. Tang, “Esrgan: Enhanced super-resolution generative adversarial networks,”
in Proceedings of the European Conference on Computer Vision Workshops
(ECCVW), 2018.

[5] X. Wang, L. Xie, C. Dong, and Y. Shan, “Real-esrgan: Training real-world

blind super-resolution with pure synthetic data,” in Proceedings of the IEEE
International Conference on Computer Vision (ICCV) Workshops, 2021.

[6] I. Kalra, M. Singh, S. Nagpal, R. Singh, M. Vatsa, and P. B. Sujit, “Dronesurf:

Benchmark dataset for drone-based face recognition,” in IEEE Xplore, 2021.

[7] D. E. King, “Dlib-ml: A machine learning toolkit,” Journal of Machine Learn-

ing Research, vol. 10, pp. 1755–1758, 2009.

37
Bibliography 38

[8] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin
loss for deep face recognition,” in Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR), 2019.

[9] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding

for face recognition and clustering,” in Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), 2015.

[10] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with

deep convolutional neural networks,” in Advances in neural information pro-
cessing systems, 2012, pp. 1097–1105.

[11] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “Deepface: Closing the gap
to human-level performance in face verification,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), 2014.

[12] G. B. Huang, M. Mattar, T. Berg, and E. Learned-Miller, “Labeled faces in the

wild: A database for studying face recognition in unconstrained environments,”
Technical Report 07-49, University of Massachusetts, Amherst, 2007.

[13] O. M. Parkhi, A. Vedaldi, and A. Zisserman, “Deep face recognition,” in British

Machine Vision Conference (BMVC), 2015.

[14] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman, “Vggface2: A

dataset for recognising faces across pose and age,” in International Conference
on Automatic Face and Gesture Recognition (FG), 2018.

[15] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song, “Sphereface: Deep hyper-
sphere embedding for face recognition,” in Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), 2017.

[16] J. Deng, J. Guo, X. Niannan, and S. Zafeiriou, “Arcface: Additive angular

margin loss for deep face recognition,” 2018.

[17] A. Krizhevsky, “Learning multiple layers of features from tiny images,” Tech-
nical Report, University of Toronto, 2009.

IISc Bengaluru
Bibliography 39

[18] Y. Sun, D. Liang, X. Wang, and X. Tang, “Deepid3: Face recognition with very
deep neural networks,” arXiv preprint arXiv:1502.00873, 2015.

[19] Y. Sun, X. Wang, and X. Tang, “Deep learning face representation by joint
identification-verification,” in Advances in neural information processing sys-
tems, 2014, pp. 1988–1996.

[20] L. Wolf, T. Hassner, and I. Maoz, “Face recognition in unconstrained videos

with matched background similarity,” in IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), 2011.

[21] I. Kemelmacher-Shlizerman, S. M. Seitz, D. Miller, and E. Brossard, “The

megaface benchmark: 1 million faces for recognition at scale,” in IEEE Con-
ference on Computer Vision and Pattern Recognition (CVPR), 2016.

[22] B. Amos, B. Ludwiczuk, and M. Satyanarayanan, “Openface: A general-

purpose face recognition library with mobile applications,” in CMU School of
Computer Science, 2016.

[23] J. Schmidhuber, “Deep learning in neural networks: An overview,” in Neural

Networks, vol. 61. Elsevier, 2015, pp. 85–117.

[24] Y. Chen, F. Wen, W. Zhu, Y. Zhang, and Z. Wang, “Mobilefacenets: Efficient

cnns for accurate real-time face verification on mobile devices,” in Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
2018.

[25] D. Yi, Z. Lei, S. Liao, and S. Z. Li, “Learning face representation from scratch,”
arXiv preprint arXiv:1411.7923, 2014.

[26] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,

Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in neural
information processing systems, 2017, pp. 5998–6008.

IISc Bengaluru
Bibliography 40

[27] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, “Joint face detection and alignment
using multi-task cascaded convolutional networks,” IEEE Signal Processing Let-
ters, vol. 23, no. 10, pp. 1499–1503, 2016.

IISc Bengaluru