PROJECT REPORT
KNOW-THEM
20CSTE401 - LIVE IN LAB-II
Submitted by
DEEPIKA LAKSHMI M.B - 412420104025
MINU DHAKSHINA. B.S - 412420104071
AANANDHINI S - 412420104002
in partial fulfillment for the award of
the degree
of
BACHELOR OF ENGINEERING
in
COMPUTER SCIENCE AND
ENGINEERING
SRI SAIRAM INSTITUTE OF TECHNOLOGY
(AUTONOMOUS),SAI LEO NAGAR, CHENNAI-44
ANNA UNIVERSITY: CHENNAI 600 025
2021-2022
ANNA UNIVERSITY: CHENNAI 600025
BONAFIDE CERTIFICATE
Certified that this project report “KNOW-THEM” is the bonafide work
of “DEEPIKA LAKSHMI M.B (412420104025),MINU DHAKSHINA
B.S (412420104071), AANANDHINI.S (412420104002)” who carried
out the 20CSTE401 - LIVE IN LAB-II Project Work under my
supervision.
SIGNATURE SIGNATURE SIGNATURE
Guide Lab in Charge HOD
Submitted for project Viva – Voce Examination held on _ _ _ _ _ _ _ _ _ _ _ _
INTERNAL EXAMINER EXTERNAL EXAMINER
ACKNOWLEDGEMENT
A successful man is one who can lay a firm foundation with the bricks
others have thrown at him.—David Brinkley
Such a successful personality is our beloved founder Chairman, Thiru.
MJF. Ln.LEO MUTHU. At first, we express our sincere gratitude to our
beloved chairman through prayers, who in the form of a guiding star has spread
his wings of external support with immortal blessings.
We express our gratitude to our CEO Mr. J. SAI PRAKASHLEOMUTHU
and our Trustee Mrs. J. SHARMILA RAJA for their constant encouragement
for completing this project.
We express our sincere thanks to our beloved principal,
Dr. K. PALANIKUMAR for having given us spontaneous and whole hearted
encouragement for completing this project.
We are indebted to our HEAD OF THE DEPARTMENT Dr. B.
SREEDEVI for her support during the entire course of this project work.
We express our gratitude and sincere thanks to our guide Mrs.S.SWATHI
for her valuable suggestions and constant encouragement for
successful completion of this project.
Our sincere thanks to our project coordinator for her kind
support in bringing out this project.
We thank all the teaching and non-teaching staff members of the
Department of Computer Science and Engineering and all others who
contributed
directly or indirectly for the successful completion of the project.
CHAPTER TITLE PAGE NO
NO
ABSTARCT 1
LIST OF FIGURES
1 INTRODUCTION 2
1.1 PROJECT AREA 2
1.2 AIM OF THE PROJECT 2
1.3 PURPOSE 2
2 LITERATURE SURVEY 3
3 SYSTEM DESIGN AND ARCHITECTURE
3.1 SYSTEM ARCHITECTURE 6
4 MODULE DESIGN 8
4.1 LIST OF MODULES
4.2 MODULE DESCRIPTION
4.2.1 FACE DETECTION
4.2.2 FACIAL EMOTION EXTRACTION
4.2.3 IMAGE CLASSIFICATION
4.2.4 TEXT TO VOICE CONVERSION
4.2.5 NAME IDENTIFICATION
5 SYSTEM IMPLEMENTATION 9
5.1 REQUIREMENT ANALYSIS
5.1.1 HARDWAE ENVIRONMNET
5.1.2 SOFTWARE ENVIRONMENT
5.2 LANGUAGE SPECIFICATION
5.2.1 NAME OF THE CODING LANGUAGE
5.3 IMPLEMENTATION TOOLS
5.3.1 OpenCV
5.3.2 Tensorflow
5.3.3 Keras
5.4.4 CNN
5.4.5 Visual Studio Code
5.4.6 Android Studio
6 RESULTS AND DISCUSSION
6.1 SCREENSHOTS 11
6.1.1 Screenshot for detecting the faces
6.1.2 Code used to train the model
6.1.3 Code used to test the features in the images
7 CONCLUSION AND FUTURE ENHANCEMENT 17
7.1 CONCLUSION
7.2 FUTURE ENHANCEMENT
REFERENCES
2
ABSTRACT
Humans are capable of producing thousands of facial actions during
communication that vary in complexity, intensity, and meaning. Emotions play a major role in
Human life. Visually challenged are one of the marginalized people who face day-to-day
problems without seeing the world. They find it difficult to predict the emotional state of a
person surrounding them, this project helps them to understand the emotions of others through
the facial expressions and thus interact well with people socially especially in public
places. The rapid advancement in technology has now helped visually challenged people to
explore the world through mobile phones. Few features that help them use these devices are
virtual assistants, text-to-voice converters, image recognizers etc. KnowThem is an android
application that uses Convolutional Neural Network (CNN) algorithms to identify the emotions
of the people that the visually challenged wanted to know. The goal of KNOW-THEM comes
under the Sustainable Development Goal (SDG) no 11-Sustainable cities and communities &
SDG goal no 3- Good Health and Wellbeing. Hence KNOW-THEM can enrich visually
challenged communities and enrich their day-to-day lives.
INTRODUCTION
1.1 PROJECT AREA
Understanding the emotions through the facial reactions of others while speaking is
important to understand people. Visually challenged people use all their senses effectively to
understand others’ emotions, but at times it becomes a challenging task to judge a new person
when exposed to numerous people, especially in public places.
1.2 AIM OF THE PROJECT
The main objective of this project is to help the visually challenged understand the public
people and make sure they are surrounded by safe people.
1.3 PURPOSE:
The goal is to develop a facial emotion recognition model which can understand human facial
expressions and detect the mood and state of a person who is opposite to the user. It uses the
concept of computer vision and machine learning to identify the emotions of the person based
on his/her facial expression. The purpose of this is to help the visually challenged. In general,we
have implemented the idea of bringing the vision level of the blind very close to the normal
ones. The project proposes a device which enhances participation of the visually impaired by
enabling them to be more effective in social interactions.
2
2. LITERATURE SURVEY
2.1 Project Title: Face Detection and Recognition System using Digital Image Processing
Link: https://ieeexplore.ieee.org/document/9074838
While recognizing any individual, the most important attribute is face. It serves as an individual
identity of everyone and therefore face recognition helps in authenticating any person's identity
using his personal characteristics. The whole procedure for authenticating any face data is
subdivided into two phases, in the first phase, the face detection is done quickly except for
those cases in which the object is placed quite far, followed by this the second phase is initiated
in which the face is recognized as an individual. Then the whole process is repeated thereby
helping in developing a face recognition model which is considered to be one of the most
extremely deliberated biometric technologies. Basically, there are two types of techniques that
are currently being followed in face recognition pattern, that is, the Eigenface method and the
Fisherface method. The Eigenface method basically makes use of the PCA (Principal
Component Analysis) to minimize the face dimensional space of the facial features. The area
of concern of this paper is using digital image processing to develop a face recognition system.
2.2 Project Title: Facial Expression Recognition Using Keras in Machine Learning
Link: https://ieeexplore.ieee.org/document/9725756
This paper aims at the extraction of expression features in facial expression recognition. Facial
expression identification has become increasingly significant in identifying the emotions of
those who are confronted with it, as well as in picture processing. In this paper dataset is used
namely the FER2013 dataset, it is a popularly used dataset for exploration and practices. The
FER2013 dataset is used to detect facial expressions efficiently. AFERS is applied in this case.
3
Facial detection, face feature extraction, and facial expression recognition are the three stages
of this method for detecting facial expressions. And some other methods are also used as Pre-
processing and Emotion classification. This paper outlines the efficiency of various widely
spread machine learning algorithms for recognizing facial expressions, namely Support Vector
Machine (SVM), Naïve Bayes.
2.3 Project Title: A Hybrid Approach of Facial Emotion Detection using Genetic
Algorithm along with Artificial Neural Network
Link: https://www.ijcaonline.org/archives/volume175/number4/sharan-2017-ijca-915494.pdf
Facial emotion recognition from facial expression is one of the most imperative intellectual
functions that our brain performs quite efficiently. For a single person, facial expressions may
be different at different instances and this is a great task to recognize the emotion from their
facial expressions. This work is an attempt to look at the task of emotion recognition using the
artificial intelligence which is cognitively very attractive and the same has been shown to
perform very well for emotion recognition. Facial emotion recognition is frequently used but
there is a problem occurring during the classification of emotion from the facial expressions
due to existing feature extraction techniques and their uniqueness. The major causes of the
problem in facial emotion recognition systems are the extraction of the best and most
appropriate feature sets from the faces according to the facial expressions. To minimize these
types of problems from facial emotion recognition systems, the SIFT descriptor along with a
genetic algorithm (GA) is the best solution according to the survey, and to achieve better
performance of proposed work, a novel objective function is being designed. In the proposed
work, an artificial Neural Network (ANN) is used as a classifier to train the facial emotion
recognition system and by using the public database Japanese Female Facial Expression
(JAFFE), the accuracy of facial emotion recognition is obtained at around 98% in MATLAB.
2.4 Project Title: Facial Recognition Based Emotion Analysis
Link: https://www.jetir.org/papers/JETIRFD06059.pdf
4
In this paper, human emotions are predicted using the Convolution Neural Network (CNN). In
this algorithm, the FER-2013 dataset has been applied for training. We recognized emotions
such as Angry, Disgust, Fear, Happy, Sad, Surprise, and Neutral. Keywords—Facial
Expression Recognition (FER), Facial Landmarks (FL), Facial Action Units (AU), Facial
Action Coding System (FACS).
2.5 Project Title: Human face recognition using facial feature detection techniques
Link: https://ieeexplore.ieee.org/document/7380598
This paper is a study and analysis of face recognition techniques using facial feature detection
techniques, taking into consideration the different variations, advantages, disadvantages, and
accuracy of the methods used. Recognizing the human face is a simple task for a human being
even seeing a person after several years. But doing the same task by a computer is not simple
because the computer will have problems recognizing the human face if there is a change in
the facial image like lighting conditions, complex background, pose and occlusion. Still
recognizing faces in images is an emerging trend of research in image processing streams.
5
3. SYSTEM DESIGN AND ARCHITECTURE
The workflow of this project consists of a series of steps detecting the face, feature extraction,
and then classifying the images. It involves pre-processed trained data to give the output.
The detailed description of the steps involved are represented using the below flow diagram.
A collection of data that involves various images are preprocessed, trained, and classified
which helps to classify the input images after feature extraction and feature matching.
6
Preprocessing - Image preprocessing is the step taken to format images before they are used
by model training and inference. Preprocessed images are used to reduce noise and blur
(filtering) – Identify structures within the image (segmentation) – Extract “useful” information
from the image (quantification) – Prepare the image for visualization (enhancement,
reconstruction).
Face Detection - Faces are detected from an image that includes multiple objects in it.
Facial recognition - A way of identifying or confirming an individual's identity using their
face.
Feature Extraction - Features from an image are extracted by matching the detected features
with similar images. Then with CNN, the whole dataset is trained.
Image Classification - Through the extracted features the test images can be classified with
the help of trained images. In this project, images are classified as Happy, Sad, Angry, Surprise,
Disgust and Neutral.
Fig 3.2
The image is converted to gray scale for extracting descriptors instead of operating on color
images directly is that grayscale simplifies the algorithm and reduces computational
requirements. It is a method used to detect the images .
7
4. MODULE DESIGN
MODULE:1 Face Detection:
The first module for the project is face detection. It is just detecting the presence of a face in
an image with multiple objects.
MODULE:2 Facial Emotion Extraction:
Features from the images are extracted and images get trained.
MODULE:3 Image Classification
According to the extracted features ,the images get classified using the Machine Learning
algorithms. The images are classified as Happy, Sad, Neutral, Surprise, Disgust and Angry.
MODULE:4 Text to Voice Conversion:
8
The output of how the image is classified is converted to voice format. As this project is being
focused for visually challenged the output should be in voice format and so text to voice
conversion is another module to convert the form of the output.
Module:5 Name Identification:
As the visually challenged be unaware of the person surrounding them all of a sudden, this
module works on keeping a record of people they already knew or whom they have come across
previously.
5. SYSTEM IMPLEMENTATION
To implement the system the following the components and specification are required.
5.1 REQUIREMENT ANALYSIS
5.1.1 HARDWARE ENVIRONMENT
To implement this system in user’s end, an Android smartphone or a electronic device like
laptop or a computer is required with a camera connected to the device. As well as a speaker
to receive the output needs to be integrated.
5.1.2 SOFTWARE ENVIRONMENT
The user device should have a smartphone with Android OS of version greater than 7.0.
5.2 LANGUAGE SPECIFICATION
Python is used for the system implementation
Python is a high-level, interpreted, general-purpose programming language.
Python has many in-built libraries. Impementing the machine learning algorithms is quite
simple using this language as well it has many modules for image processing which makes the
implementation to be effective.
5.3 IMPLEMENTATION TOOLS
9
5.3.1 OpenCV
OpenCV is the huge open-source library for the computer vision, machine learning, and
image processing. To Identify image pattern and its various features we use vector space and
perform mathematical operations on these features. It has C++, C, Python and Java interfaces
and supports Windows, Linux, Mac OS, iOS and Android. Image processing can be easily
implemented using OpenCV. “Image processing is the analysis and manipulation of a digitized
image, especially in order to improve its quality”. Digital-Image may be defined as a two-
dimensional function f(x, y), where x and y are spatial(plane) coordinates, and the amplitude
of fat any pair of coordinates (x, y) is called the intensity or grey level of the image at that
point. Image processing is basically signal processing in which input is an image and output is
image or characteristics according to the requirement associated with that image.
5.3.2 TensorFlow
TensorFlow is a free and open-source software library for machine learning and artificial
intelligence. It can be used across a range of tasks but has a particular focus on training and
inference of deep neural networks. It provides a collection of workflows to develop and train
models using Python or JavaScript, and to easily deploy in the cloud, on-prem, in the browser,
or on-device no matter what language is used.
5.3.3 Keras
Keras is an open-source software library that provides a Python interface for artificial neural
networks. Keras acts as an interface for the TensorFlow library. It mainly focuses on
developing and evaluating deep learning models.
5.3.4 CNN
In deep learning, a convolutional neural network is a class of artificial neural network often
used for image classification. By recognizing valuable features, CNN can identify different
objects on images, for finding patterns in images to recognize objects, faces, and scenes.
Applications that call for object recognition and computer vision — such as self-driving
vehicles and face-recognition applications — rely heavily on CNNs.
5.4.5 Visual Studio Code
Visual Studio Code also commonly referred to as VS Code, is a source-code editor made by
Microsoft for Windows, Linux and macOS is used to run the working of this system and
10
implement all the modules with support for development operations like debugging, task
running, and version control. It aims to provide just the tools a developer needs for a quick
code-build-debug cycle and leaves more complex workflows to fuller featured IDEs, such as
Visual Studio IDE. All the modules are implemented using Visual Studio Code.
5.4.6 Android Studio
The user-interface for this project is to be as an Android application. To implement the system
as an Android application, Android studio is used. Android Studio is the official integrated
development environment for Google's Android operating system, built on JetBrains' IntelliJ
IDEA software and designed specifically for Android development.
6. RESULTS AND DISCUSSION
Screenshots of the code
The sample code for detecting the faces in the captured image
11
Code used to train the model
Code used to test the features in the images
12
Code used to classify the images according to the detected emotions
13
Results of the implemented modules
Fig.6.0
The implemented program has detected various facial objects in the above image.
Fig.6.1
The facial objects in the image Fig 6.1 have been detected and then classified as happy.
Fig 6.2
14
The facial object has been recognized and classified as Sad.
Fig 6.3
In the image Fig 6.3, face has been detected and classified as a Surprise
Fig 6.4
The facial object has been detected in Fig 6.4 classified as Disgust
15
Fig 6.5
The facial object has been detected in Fig 6.5 and has been classified as Neutral
16
7.1 CONCLUSION:
To conclude, this report includes the implementation of KNOW-THEM which can be used
by visually challenged people to lead a comfortable and happy life without any struggles.
7.2 FUTURE ENHANCEMENT:
In the future, all the modules of the project will be completely implemented. Further new
features will be implemented to create a fruitful impact on the visually challenged.
REFERENCES
• https://www.researchgate.net/publication/326224279.pdf -Use of Mobile Applications
for Visually Impaired People
• https://ieeexplore.ieee.org/abstract/document/5626038 -Smartphone-based face
recognition tool for the blind
• https://ieeexplore.ieee.org/document/7392581 - Face detection and recognition
application for Android
• https://files.eric.ed.gov/fulltext/EJ1149519.pdf - Use of Mobile Applications for
People Who Are Visually Impaired
17
