Emotion Recognition in E-Learning

B.S. (SE) Final Year Project Proposal

Haleema Sadia
2018/COMP/BS(SE)/22974

Tahreem Nazim
2018/COMP/B S(SE)/23011

Uroobah Mumtaz
2018/COMP/B S(SE)/23015

Batch: 2018
Date: 04/03/2021

DEPARTMENT OF COMPUTER SCIENCE AND SOFTWARE ENGINEERING

JINNAH UNIVERSITY FOR WOMEN

JINNAH UNIVERSITY FOR WOMEN

1|Page
 Department Of Computer Science & Software Engineering
FINAL YEAR PROJECT APPROVAL FORM

The Chairperson Date: 04/03/2021

Computer Science & SE Department, Batch: 2018
Jinnah University for Women,
Karachi.

Subject: Bachelor of Science in Software Engineering Final Year Project

Madam,

We, the below listed students of Final Year BS 2018 class, desire to undertake work on the following project
Emotion Recognition in E-learning System under the supervision of Ms. Kanwal Zahoor & Ms. Tehreem
Qamar.
We request you to kindly grant approval for undertaking the work on the above-cited project. I abide by all
terms and conditions mentioned below.

1. I have selected this project on my own.

2. I am sure I can complete this project till December 2021.
3. I am eager to work under the supervision of advisor assigned to this project.
4. I understand that FYP committee can modify the scope of the project as and when required.
5. I know that if do not appear in regular project progress presentations my project will be disqualified.
6. I know that if I do not appear in mid project presentation, whenever it is scheduled, I will not be
eligible for final project viva
7. I fully understand that “cheating”* may lead to cancelation of my project.
8. I understand that the decision of the FYP evaluation committee, for all issues, would be final, and no
objections will be accepted.
9. I have no objection presenting my project to external or internal examiner assigned by the Head of the
Department.
10. I understand that it is my responsibility to update my advisor and FYP committee members with the
status of my project and submit reports on time.
11. I am choosing supervisor and group members on my own and that change request will not be allowed.
12. In case of dispute among students (group members), the project will be cancelled and all students will
be marked “fail”.
*-Copying code from any resources
-Using off the shelf components without prior permission
-Outsourcing your project

Yours sincerely,

S.No. Name Roll No. Signature

01 Haleema Sadia 2018/COMP/BS(SE)/22974

02 Tahreem Nazim 2018/COMP/B S(SE)/23011

03 Uroobah Mumtaz 2018/COMP/B S(SE)/23015

Table of Contents
PROJECT SUMMARY.......................................................................................................................................................1
OVERVIEW.......................................................................................................................................................................1
SCOPE..............................................................................................................................................................................1
In scope:......................................................................................................................................................................1
Out of scope:...............................................................................................................................................................1
Deliverables produced:...............................................................................................................................................2
OBJECTIVES.....................................................................................................................................................................2
PROJECT APPROACH......................................................................................................................................................2
TOOLS AND TECHNOLOGIES............................................................................................................................................3
ESTIMATED DURATION....................................................................................................................................................3
MARKET AND COMPETITION.........................................................................................................................................3
PROJECT APPROVALS....................................................................................................................................................4
 1

PROJECT SUMMARY
Emotion plays an important role in analyzing the student's interest in classroom lectures. Out of the various ways
to detect emotion, the quick way is to understand the emotional symptoms through facial expression. It is easy to
observe the students and their reaction on a particular topic which is being taught by the instructors. Most of the
time in this type of environment even an important lecture or course ends up with the boredom of the student.
All the effort of the instructor and their instructional aides become ineffective.

So we highlight this issue and making a system that will detect student emotion in online classes or physical
classes whether they are paying attention to the lecture or not, Observing each student's performance in an
intelligent environment, such as facial expressions, will help teachers dynamically adjust teaching methods by
receiving rapid feedback from this real-time interaction, and will be beneficial to the quality of education. Firstly
we will implement this in e-learning and later implement this in physical class (this will consider as a future
scope).

Our System will detect the emotional states of the learner and give feedback to an instructor about their instant
and weighted emotional states based on facial expressions. Though facial expressions do not necessarily convey
emotions, in the computer vision community.

The term facial expression recognition often refers to the classification of human facial features into one of the
six basic emotions: happiness, sadness, fear, disgust, surprise, and anger Herein, several classification algorithms
were applied to learn instant emotional state and the best accuracy rates. The Real-time facial emotion
recognition is divided into several phases including detecting general area of the human face, this process
includes tracking system which requires the hardware to monitor the general movement of facial layout, Facial
land marking, which pointing out more accurate facial point to be extracted.

Our focus is on understanding and interpolating the emotional state of the learner during a learning engagement.
Evaluating the emotion of a learner can progressively help in enhancing the learning experience physically and
emotionally. Firstly we will implement this in e-learning and later implement this in physical class (this will
consider as a future scope)

OVERVIEW
Online learners participate in various educational activities including reading, writing, watching video tutorials,
online exams, and online meetings. During the participation in these educational or different activities, they
show various engagement levels, such as boredom, frustration, delight, neutral, confusion, and learning gain. So,
it is important for online educators to detect their online learners’ engagement status precisely and efficiently.

SCOPE
The scope of our project is to develop a facial expression recognition system based on Python that will comprise
of a website and a mobile application with the aim to integrate it in E-Learning system that captures and detect
the student faces and classify their expressions to make E-Learning system interactive.

In scope:

· Provide a real-time feedback mechanism.

· Mobile Application
· Web Application

Out of scope:

· Emotion recognition in physical classes.

 Emotion recognition in industrial sectors to monitor the focus level of employee.

FYP Project Proposal

CS and SE Department, Jinnah University for Women
 2

Deliverables produced:
Our project will produce the following deliverables:

 Deliverable 1:
Mobile Application:
Emotion recognition system should be in real time and highly accurate and mobile device
has limited processing power, the algorithm in the emotion recognition system should be
implemented using less computation so, facial image is captured by an embedded camera
of a smart phone. Some representative frames are extracted from the video, and a face
detection module is applied to extract the face regions in the frames. After the face regions
are extracted then it will be capable of recognizing emotions of student.

 Deliverable 2:
Web Application:
Our site will be capable of recognizing different emotions (neutral, happy, sad, confused,
and surprised) of student. The emotions will change by time and these emotions change
the facial characteristics so the site will detect the emotions through camera.
Concerning the client-end of the system the functionality is simple enough. At predefined
interval a web camera captures frames from the users while they are visiting the website.
Each captured image is sent to the server for processing. As soon as the result comes back
from the server the interface is changed. On the server-end the learnt model is loaded from
the disk and the web application is ready to process the images.
The client-side of the website is developed using HTML for the context part of the page
and the Bootstrap framework for CSS styling. The most important component is the
Webcam JS JavaScript library (and open source MIT licensed library) which provides all
the necessary functions for an image to be captured and sent to the server. It is an AJAX
based communication and a callback function accepts the server response. The JQuery
library is used for accessing and altering the Document Object Model (DOM) of the page.
Customizations are i) the bootstrap theme is changed and b) a different images are shown
to the user (each with the recognized emotion). The server-side part is developed using the
Flask framework (BSD licensed) and python as programming language.

OBJECTIVES
Our project will have the following objectives:
 Identify emotions of a student.
 Automatically identify the type of face emotion.
 Detect Facial Expressions with Accuracy.
 Make E-Learning and physical classes Interactive.
 Enable Educator to be aware of the general emotional state of the virtual classroom.
 Provide a real-time feedback mechanism.

PROJECT APPROACH
We will be developing a facial expression recognition system based on Python that will comprise of a website
and a mobile application with the aim to integrate it in E-Learning system that captures and detect the student
faces and classify their expressions to make E-Learning system interactive.

The methodology which is applied to this project is split into three major parts:

The detection phase: This phase consists of the detection of faces, this phase follows two different steps: The
detection of emotions from still or video frames, and the pretreatment to eliminate the parasites caused by the
quality of the optical or electronic devices during the acquisition of the input image to extract the information
essential.

The extraction phase: It consists in extracting significant or discriminating information caused by facial
expressions, there are mainly two methods; local methods based on geometric characteristics and global methods

FYP Project Proposal

CS and SE Department, Jinnah University for Women
 3

based on appearance, which take the image of the face as a whole and use well-known statistical analysis
techniques.

The classification phase: After extracting all the features of the face region, the functionalities classification is
carried out at the last stage of an automatic facial expression analysis system. Precisely, this can be done by
classifying the characteristics according to the similarities between the characteristic data, using classifiers such
as artificial neural networks or linear classifiers.

TOOLS AND TECHNOLOGIES

 HTML
 CSS
 Deep Face
 Android Studio

ESTIMATED DURATION

Milestone Tentative Date Deliverable(s) completed

Software requirement and specification
Requirement Gathering January 1st, 2021
Document
Data modules like Flow diagram, ERD etc...
Designing February 15, 2021
Prototype for app and web

Hardware assembling March 30, 2021 Data gathering from hardware unit

App development May 1st , 2021 App with two modules

Remaining modules development and

App completion and integration June 15, 2021
integration
Website for Emotion Recognition In E-
Web development August 30, 2021
Learning System

Testing October 1st , 2021 System for acceptance testing

Deployment December 1st , 2021 Whole system integrates on site.

MARKET AND COMPETITION

1. What research have you conducted to understand your market, including your industry,
regions, customers, competitors?
• Focus groups
Here we will conduct a group interview involving a small number of
demographically similar people. Their reactions to specific researcher-posed
questions are studied.
• Personal experience
Here we want people to want you to reveal what you experienced while doing one or
another activity related to our project.

2. Similar Products Available

FYP Project Proposal
CS and SE Department, Jinnah University for Women
 4

Competitor 1:
Micro Expression
https://play.google.com/store/apps/details?id=com.Mazuzu.ExpressionTraining
Strengths:
 This app is designed to improve skills in recognizing concealed emotions.
 To the best of our knowledge the information contained within the app is correct.

Weaknesses:
 Information within this project is as yet unverified.
 Needs to improve accuracy and improve expression images.

Competitor 2:
Emoticon-Face Emotion Recognizer
https://play.google.com/store/apps/details?id=com.smartguyin98.Emotion_Recognizer
Strengths:
 This app detects complex and nuanced human Emotions and Cognitive states from
face.
 This App scientifically measures and reports the emotions and facial expressions
using sophisticated computer vision and machine learning techniques.
Weaknesses:
 Only One Face Will be detected.

Competitor 3:
Emotimeter
https://play.google.com/store/apps/details?
id=com.reaimagine.josem.emotimeter_facialemotionrecognizer
Strengths:
 Detect emotions in real-time from the images obtained by the camera.
 Take photos or record videos of these live detection sessions.
Weaknesses:
 Needs to improve accuracy and improve expression images..

3. What sets your product/project apart from your competitors?

Strengths:
Our Project aims
 To Detect Facial Expressions with Accuracy.
 To make E-Learning and physical classes Interactive.
 To provide a real-time feedback mechanism.
 This app mainly focus on 7 main emotions.
 Anger
 Disgust
 Surprise
 Sadness
 Happiness
 Confused
Weaknesses:
 The extracted information by the application may not be fully accurate.
 Requires Internet connection.

Current or future opportunities:

 In future we can Implement This Technology into physical classes or industrial
Sectors to monitor the focus level of student or employee.

4. How do you or will you promote your business?

FYP Project Proposal

CS and SE Department, Jinnah University for Women
 5

We will promote our business through:

a. Advertising (online)
b. Social media
c. Events and exhibitions

FYP Project Proposal

CS and SE Department, Jinnah University for Women