Proc.

of the International Conference on Electrical, Computer, Communications and Mechatronics Engineering (ICECCME 2023)
19-21 July 2023, Tenerife, Canary Islands, Spain
2023 3rd International Conference on Electrical, Computer, Communications and Mechatronics Engineering (ICECCME) | 979-8-3503-2297-2/23/$31.00 ©2023 IEEE | DOI: 10.1109/ICECCME57830.2023.10252617

Facial Emotion Recognition System for Mental Stress

Detection among University Students
Foo Jia Ming Shaik Shabana Anhum Shayla Islam Keoy Kay Hooi
Institute of Computer Science and Institute of Computer Science and Institute of Computer Science and Institute of Computer Science and
Digital Innovation Digital Innovation Digital Innovation Digital Innovation
UCSI University UCSI University UCSI University UCSI University
Kuala Lumpur, Malaysia Kuala Lumpur, Malaysia Kuala Lumpur, Malaysia Kuala Lumpur, Malaysia
[email protected] [email protected] [email protected] [email protected]

Abstract— Nowadays, mental stress has become a serious language, and paralanguage. Usually, nonverbal communication
concern, particularly among youngsters and university students. is expressed through facial expressions [4].
The age previously considered the most relaxed is now surrounded
by a lot of stress. Hence, this project aims to develop a Facial Human facial expressions are the most direct and effective
Emotion Recognition System for Mental Stress Detection in order way of expressing emotions and feelings. It is also able to
to bring advantages to university students and counseling identify human attention, including behavior, mental health,
departments of institutions in dealing with mental stress. Facial personality, criminal propensity, and lying [5]. In 1971, the
Emotion Recognition is a field of study that involves the use of research of psychologists Ekman and Friesen first proposed that
technology to identify and interpret human emotions based on human beings have six main emotions, and each emotion
facial expressions. It has been used in various applications such as reflects a unique psychological activity with a unique
healthcare, human-computer interaction, and security. Stress expression. These six main emotions are called basic emotions
detection is one of the most important applications of Facial and consist of anger, happiness, sadness, surprise, disgust, and
Emotion Recognition, which is beneficial in the early diagnosis and fear [6].
treatment of stress-related disorders. The system uses computer
vision and machine learning algorithms to analyses facial features II. INFORMATION GATHERING
and identify emotions such as happiness, sadness, anger, and fear.
The system is designed for mental stress detection, which means it A. Literature Reviews
can detect mental stress in individuals through analyzing the In the journal article of Anil Audumbar Pise at. El [7], the
user’s facial expressions. authors identified four phases in developing a prototypic Facial
Keywords—mental stress, facial recognition technology, Emotion Recognition (FER) system, which includes face
emotion, facial expression detection, face alignment, facial feature extraction, and emotion
I. INTRODUCTION classification. The authors claimed that the process of face
recognition begins with face detection, which is used to find and
In today's highly competitive world, mental stress has identify human faces in each input data. The second phase is face
become a serious concern, particularly for youngsters and alignment, which helps in identifying the geometric structure of
university students. The age group previously considered the human faces in digital images, allowing for more accurate face
most relaxed is now surrounded by a lot of stress. Stress is often localization. The third stage is facial feature extraction, which
used synonymously with negative life events that cause involves converting pixel data into a high-level representation of
physical, emotional, or psychological strain [1]. According to the face, including information about its color and texture. This
research by Rana et al. [2], increased stress among university phase is critical as it allows the system to identify unique
students causes several academic difficulties, including features of each individual's face, making it easier to distinguish
depression, lack of interest in attending class, struggles with between different people. The last stage is the emotion
comprehension, and suicidal ideation due to poor self-esteem. classification phase, which involves subjecting the feature data
The article also states that the emotional signs of high stress extracted from the facial images to further analysis.
levels include anger, impatience, frustration, or moodiness.
Therefore, this project aims to develop a Facial Emotion The accuracy of the recognition results is dependent on the
Recognition system for Mental Stress Detection to detect effectiveness of the feature extraction and the classification
indications of stress in university students based on their facial algorithms used. Therefore, it is essential to ensure that the
expressions. feature extraction and classification algorithms are optimized to
achieve the best results possible. There are various utilized
Facial expression is defined as a kind of nonverbal algorithms can be used to identify and classify the emotions
communication that uses one or more facial muscles movement based on the features extracted.
to express an individual’s feelings and emotional state [3].
Nonverbal communications refer to communication between According to the article of Mehta at. El. [8], There are three
humans through eye contact, facial expression, gesture, body popular Machine Learning algorithms were used for facial

979-8-3503-2297-2/23/$31.00 ©2023 IEEE

Authorized licensed use limited to: INDIRA GANDHI INSTITUTE OF TECHNOLOGY -Dhenkanal. Downloaded on December 05,2024 at 06:05:02 UTC from IEEE Xplore. Restrictions apply.
 emotion recognition, which are the Support Vector Machine and stress patterns in real-time. In this project, the dataset of
(SVM), Random Forest (RF), and k-Nearest Neighbor (kNN) FER2013 is used in order to perform on the facial expression
algorithms. These algorithms are used to identify emotions and recognition. FER2013 is a publicly available dataset that
intensity estimation of each recognized emotion. contains 35,887 grayscale facial expression images, which are
categorized into seven emotion categories including anger,
Other than the facial emotion recognition algorithm, feature disgust, fear, happiness, sadness, surprise, and neutral. This
extraction algorithms are also used in the development of the dataset was created for the Facial Expression Recognition
FER system in order to improve the accuracy of facial Challenge (FER2013) that was held at the IEEE International
classification. The algorithms include Convolutional Neural Conference on Computer Vision and Pattern Recognition
Networks (CNN), Constrained Local Model (CLM), and (CVPR) in 2013. The images were collected from the internet
Histogram of Oriented Gradient (HOG). and are of different sizes and qualities. FER2013 is a widely
used dataset in the field of facial expression recognition and has
Table I. Comparison of the current Facial Emotion Recognition
system [9]
been used for training and evaluating deep learning models for
emotion recognition.
Name Version Type Accuracy Feature Historical research method is also conducted to understand
extraction the cultural and social context around stress and how it is
Algorithm expressed through facial expressions. Besides that, this method
Affdex SDK v3.4.1 50.48% HOG and S can also provide important context for understanding stress and
VM classifiers how it has been studied in the past.
FaceVideo API v1.0 48.35% CNN Finally, in order to gain a more comprehensive
Cognitive API v1.0 52.61% N/A understanding of stress levels and their relationship with facial
Services: expressions, questionnaire methodology is often employed as a
Face means of gathering subjective data. This method involves the
FaceReader software v7.0 57.31% CNN use of carefully crafted questionnaires that are designed to elicit
FaceAnalysis SDK v1.0 55.07% CLM responses from participants about their experiences with stress
and how they perceive their own facial expressions in stressful
situations. By collecting and analysing this data, it can gain
The comparison results of current Facial Emotion Recognition valuable insights into the complex relationship between stress,
system facial expressions, and subjective experiences, which can help
• Most of the FER systems is using the Convolutional inform the development of the Facial Emotion Recognition
Neural Networks (CNN) algorithm for feature System for Mental Detection.
extraction.
III. ANALYSIS AND DISCUSSION
• The highest accuracy of the FER system is using the A questionnaire was conducted by online survey that
Convolutional Neural Networks (CNN) algorithm. consisted of a sample of 100 university students. The
For the mental stress detection, there have been many previous participants were randomly selected from a population of
works in this area. In the paper [10], the authors mentioned the students at any given university. The purpose of the
stress can be calculated using the classification algorithms such questionnaire was to gather data on this study topic, and the
as the Random Forest, Naïve Bayes, Support Vector Machine as results will used to analyse for draw conclusions and insights. A
well as the K-Nearest Neighbour. They found that the Support sample size of 100 students will provide a good representation
Vector Machine is performing well with the accuracy of of how students feel about the use of facial emotion recognition
85.71%. technology in mental health.
Compared to the paper [10], this project is more focus on the Through the online survey, the result shown that people may
facial expression and not much focus on the mental stress not always seek professional help or guidance to assess their
classification. Thus, the proposed mental stress detection mental well-being. Instead, they may rely on their own
algorithm in this project is based on the CNN algorithm used for perceptions and understanding of their emotions. Other than
facial expressions, where the frequency of stress-related facial that, it also proved that there are 30% of the respondents were
expressions is used to identify mental stress. not aware that they were experiencing symptoms of mental
stress.
B. Research Method
Additionally, the survey found that when students are faced
The FER system for stress detection is a complex system that with stressful situations, they tend to display “sadness” or “fear”
requires a multi-disciplinary approach. To develop such a on their emotions. On the other hand, when they are engaging in
system, this project combines quantitative methods, historical relaxing activities, they tend to have “happy” and “surprised”
research, and questionnaire methodology. emotions. This information can be useful in defining and
Quantitative research method can be used to collect and developing the mental stress detection algorithm.
analyse numerical data on facial expressions and physiological Lastly, this survey also highlights the need for accessible and
responses associated with stress. This data can then be used to user-friendly tools that can aid individuals in monitoring their
train machine learning algorithms that can recognize emotion mental health. Most of the respondents believe that the Facial

Authorized licensed use limited to: INDIRA GANDHI INSTITUTE OF TECHNOLOGY -Dhenkanal. Downloaded on December 05,2024 at 06:05:02 UTC from IEEE Xplore. Restrictions apply.
 Emotion Recognition Systems for Mental Stress Detection detection, the system initializes a window frame with a
could potentially provide more accurate and objective resolution of 1000x500 to serve as the user interface and loads
assessments of one’s emotional state and help individuals to the pictures that will be used in the user interface.
seek appropriate treatment or support when necessary.
Furthermore, the next step is to load the pre-trained facial
In a summary, the key finding of the survey showed that emotion recognition model "EmotionDetectionModel.h5" from
individuals may not always seek professional help to assess their Karansjc1 using "tf.keras". The authors trained and tested the
mental well-being, and 30% of respondents were not aware of model using the FER2013 dataset with a convolutional neural
experiencing mental stress symptoms. The survey also revealed network (CNN) that achieved 0.6662 accuracy. Additionally,
that individuals tend to display certain emotions in response to "tf.keras" is a high-level neural network API for TensorFlow
different situations. Finally, the survey highlighted most that provides an easy-to-use interface for building and training
respondents believing that facial emotion recognition systems deep learning models.
could provide more accurate assessments of emotional states
and aid in seeking appropriate treatment or support. After loading the model, the next step is to define a function
that calculates the mental stress score based on emotion
Most Frequent Emotions Most Frequent Emotions probabilities. According to the analysed result in this project,
Experienced by Students Experienced by Students when students are mentally stressed, they tend to display "Fear,"
During Stressful Situations During Relax Situations "Angry," and "Sad" emotions. Therefore, is to create an
algorithm to take in the predicted probabilities of different
• Sad • Happy emotions as input and returns a mental stress score as output. If
• Scared • Surprised users keep displaying "Fear," "Angry," and "Sad" emotions, the
mental stress score will increase. Once the mental stress score
reaches the limit, the system will notify and display a message.
B. Steps of the system algorithm

Fig. 1. Percentage of emotion when facing stressful situation

Fig. 3. System Architecture Diagram of FER System for Mental Stress

Detection

C. Stress Score
The limit for the stress score is defined based on the
Fig. 2. Percentage of emotion when relaxing activities probabilities of different emotions. The stress score is
calculated as the sum of the weighted emotion probabilities,
IV. SYNTHESIS where each stress emotion probability above 0.5 is given a
A. System Architecture Design weight of 0.25. Similarly, each non-stress emotion probability
above 0.5 is given a negative weight of 0.25, and neutral is not
The system begins by detecting the required libraries such as
dlib, opencv-python, and face_recognition, to ensure that these given any weight. Once the stress score reaches a weight of 1,
libraries are properly installed and working. Upon successful it is considered as stress.

Authorized licensed use limited to: INDIRA GANDHI INSTITUTE OF TECHNOLOGY -Dhenkanal. Downloaded on December 05,2024 at 06:05:02 UTC from IEEE Xplore. Restrictions apply.
 3) Face and Emotion Detected
Example algorithm for better understanding:
stress_score = 0.0
if emotion_probabilities[0] > 0.5: # Angry
stress_score += 0.25
if emotion_probabilities[3] > 0.5: # Happy
stress_score -= 0.25
D. User Interface Design and System Flow
1) Main Page

Fig. 6. Example of Face and Emotion Detection

Once the image data matched with the database, the system
will start performs emotion classification to generate the user's
emotion.

4) Message Box when Stress Detected

Fig. 4. Main Page of FER system for Mental Stress Detection

Firstly, press “Start System” to start the system. The

system’s main page displays two buttons: “Start System” and
“Check Record.” The “Start System” button activates the facial
emotion recognition and stress detection, while the "Check
Record" button displays the stress record in an additional
window.

2) Unknow Face Detected Fig. 7. Message Box when Stress

Using the data from emotion classification, the system

calculates the user's stress score. If the stress score exceeds the
limit set, the system classifies it as mental stress detected. Then,
a notification pops up on the user's screen, informing them to
take a break.

5) Check Record

Fig. 5. Example of Unknow Face Detected

When users show their face in front of the camera, the face
data will be capture and will be decoded. Then, it compares and
checks the image data in the database. If the image matches
with the database, the system performs emotion classification
to generate the user's emotion. Otherwise, the system requires
the person to press "r" to record a new face and fill in the name
and ID textboxes for information. Fig. 8. Mental Stress Record

The stress result, including name, ID, date, time, and

emotion, is saved in the database, and shown if the user presses
the “Check Record” button in the main page. If no stress is
detected, the system continues to generate the user's emotion
until the user presses "q" to terminate the system.

Authorized licensed use limited to: INDIRA GANDHI INSTITUTE OF TECHNOLOGY -Dhenkanal. Downloaded on December 05,2024 at 06:05:02 UTC from IEEE Xplore. Restrictions apply.
 V. EVALUATION instructions, and helpful feedback on emotion
The system has been tested by individuals of different ages, detection results.
races, and computer specifications to ensure its durability and • Address issues with response time by optimizing the
functionality. The participants were given a brief tutorial on performance and speed of the system.
how to use the system, and then asked to complete a series of • Consider incorporating additional features such as
tasks while thinking aloud and providing feedback on their audio recognition or multi-modal emotion detection to
satisfaction and experience. improve the accuracy and usefulness of the system.

A. Results and summary of the Usability Testing VI. CONCLUSION

After conducting usability testing on the Facial Emotion All in all, the development of the Facial Emotion
Recognition System for Stress Detection, the following Recognition system for Stress Detection with Python has been
limitations have been gathered: successfully developed and achieved the aims and objectives
• Accuracy of Emotion Detection stated at the initial planning phase. This system is developed
Participants noted that the accuracy of the system with the use of Python and its libraries such as cv2 library. In
varied depending on the lighting conditions, camera addition, the Rapid Application Development methodology is
angle, and facial expressions of the individuals in the also implemented in the development of the system. Moreover,
video clips. Some participants also reported that the this project has also concluded that 30% of students were not
aware that they were experiencing symptoms of mental stress.
system had difficulty recognizing more subtle or
When students faced with stressful situations, they tend to
nuanced emotions. Through the usability testing, we
display sad or fearful facial expressions. On the other hand,
can conclude that the accuracy of the emotion engaging in relaxing activities tends to show happy and
detection is 0.75. surprised emotions.
• Accuracy of Mental Stress Detection
However, the accuracy of mental stress detection is yet With the ability to detect mental stress and analyse facial
to be determined. This is due to the fact that stress expressions, this technology can provide valuable insights into
requires long-term testing and domain expertise for a user’s emotional state, allowing for early intervention and
evaluation. Therefore, further testing is necessary to treatment. Additionally, during user acceptance testing, the
system has successfully detected two individuals who showing
determine the accuracy of our mental stress detection
the signs of mental stress and suggested them to take a break or
system. We will be conducting these tests soon and
seek help. This is a remarkable achievement, as early detection
hope to determine the stress detection accuracy with of stress can prevent negative outcomes.
greater accuracy.
• User Interface Design In a nutshell, this system is particularly useful in high-stress
Several participants commented on the need for a environments such as schools, workplaces, and hospitals,
more attractive and fancier interface, with clearer where early detection of stress can prevent negative outcomes
instructions and more helpful feedback on the results such as burnout, poor performance, and health problems. Due
of emotion detection. to the reasons of Facial Emotion Recognition system for Stress
• Response Time Detection is still in the early stages of development, hence, the
Some participants noted that the system was slow to failure in recognition, inaccurate or mistake is still existed in
respond to their commands, which led to frustration this system. However, as technology continues to improve, the
and decreased efficiency. system is likely to become even more accurate and reliable,
making it an essential tool for stress management and
Overall, the system is able to perform its tasks, which is to prevention.
detect users, perform emotion classification and use the data A. Limitation
obtained from emotion classification to calculate the mental
stress score and send notification to users as well as save the The system has some limitations that need to be addressed.
record in the database. Users are generally satisfied with the 1) Limitation of the study
system and rate its overall usability as 4.3 out of 5.
• Cultural Differences
B. Recommendation One limitation of the proposed solution is that facial
Based on the results of the usability test, the following expressions and emotions may vary across different
recommendations are suggested to improve the usability and cultures. The FERS model may not accurately
user experience of the system. recognize emotions in individuals from different
• Improve the accuracy of emotion detection by refining cultural backgrounds, leading to inaccurate results.
the machine learning algorithms and computer vision • Sample Size
technology used by the system. Another limitation of the study is the sample size. In
• Enhance the user interface design by providing more this case, the sample size of the students who
intuitive and user-friendly controls, clearer participate in the study is only 100 respondents, which
can be considered as small and may not be
representative of the entire student population.

Authorized licensed use limited to: INDIRA GANDHI INSTITUTE OF TECHNOLOGY -Dhenkanal. Downloaded on December 05,2024 at 06:05:02 UTC from IEEE Xplore. Restrictions apply.
 Besides that, in educational settings, teachers and
2) Limitation of the proposed solution administrators can use these systems to identify stressed
• User Detection students and provide timely support, creating a more conducive
One of the main issues is that the proposed system learning environment and promoting overall well-being.
struggles to recognize more subtle or nuanced emotions Other than that, individuals can use facial emotion
and it is unable to detect user in the low-light condition, recognition systems to track their stress and make adjustments
which can be particularly problematic in accurate to their lifestyle, work-life balance, or seek professional help
contexts. when needed.
• Slow Respond Time Additionally, this system can contribute to ongoing research
The response time of the system is quite slow, which and development of stress and facial emotion recognition-
can be frustrating for users who are trying to interact related studies, furthering the understanding of the impact of
with it. stress on physical and mental health and improving
• Low Accuracy of the System interventions.
The accuracy of the system is not as high as it could be,
which may impact on its usefulness for its features. Overall, system has the potential to significantly improve the
well-being of individuals and society as a whole. With the
B. Future Development increasing need for early intervention and prevention of mental
The user's evaluation has provided a valuable insight into the health issues, the use of this systems can be an effective way to
system's performance. However, there is still a lot of room for promote better mental health outcomes.
further research and improvements to enhance the system's
REFERENCES
capabilities.
[1] S. Elizabeth, “How is stress affecting my health?,” Verywell Mind, Nov.
One potential area for further research is applying deep 07, 2022. https://www.verywellmind.com/stress-and-health-3145086.
learning methods in recognition with Artificial Intelligence (AI). (accessed Jan. 16, 2023)
The use of deep learning in recognition can improve accuracy [2] A. Rana, “Stress among students: An emerging issue,” Rana | Integrated
and reduce errors, making it a promising approach to explore. Journal of Social Sciences, Jul. 03, 2019.
https://pubs.iscience.in/journal/index.php/ijss/article/view/891
By incorporating AI into the system, it can learn from data and
[3] “APA Dictionary of Psychology.” https://dictionary.apa.org/facial-
provide more accurate results over time. expression
Another potential future development for this system is the [4] I. M. Revina and W. R. S. Emmanuel, “A Survey on Human Face
implementation of an online database or server for the dataset. Expression Recognition Techniques,” Journal of King Saud University -
Computer and Information Sciences, vol. 33, no. 6, pp. 619–628, Jul.
Currently, the dataset and record list are stored locally, which 2021, doi: 10.1016/j.jksuci.2018.09.002.
can make it inconvenient to control and access the data remotely.
[5] A. R. Khan, “Facial Emotion Recognition Using Conventional Machine
By storing the datasets and record list online, users would be Learning and Deep Learning Methods: Current Achievements, Analysis
able to use and access the record list from any location or device and Remaining Challenges,” Information, vol. 13, no. 6, p. 268, May
without limitations. 2022, doi: 10.3390/info13060268.
[6] B. Niwlikar, “Popular Theory of the 6 Basic Emotions by Paul Ekman,”
Lastly, integration with other technologies is also one of the Careershodh, Mar. 10, 2022. https://www.careershodh.com/popular-
potential future developments for this system. It could be theory-of-the-6-basic-emotions-by-paul-ekman/
integrated with other technologies, such as virtual reality or [7] A. A. Pise et al., “Methods for Facial Expression Recognition with
biofeedback devices, to provide more immersive and engaging Applications in Challenging Situations,” Computational Intelligence and
Neuroscience, vol. 2022, pp. 1–17, May 2022, doi:
stress management interventions. 10.1155/2022/9261438.
[8] D. Mehta, M. S. Siddiqui, and A. Y. Javaid, “Recognition of Emotion
C. Contribution to Society Intensities Using Machine Learning Algorithms: A Comparative Study,”
Sensors, vol. 19, no. 8, p. 1897, Apr. 2019, doi: 10.3390/s19081897.
Facial emotion recognition systems for stress detection can
have a significant positive impact on society. One of the key [9] G. Dimitrov, “A Study on the Impact of Academic Stress among College
Students in India,” Ideal Research An International Multidisciplinary,
contributions is the early intervention and prevention. By 2(4), pp. 21-35.
identifying signs of stress in individuals, the systems can [10] R. Ahuja,, & A. Banga, (2019). Mental stress detection in university
facilitate early intervention and treatment, potentially preventing students using machine learning algorithms. Procedia Computer Science,
the development of more serious mental health conditions such 152, 349-353.
as anxiety or depression.

Authorized licensed use limited to: INDIRA GANDHI INSTITUTE OF TECHNOLOGY -Dhenkanal. Downloaded on December 05,2024 at 06:05:02 UTC from IEEE Xplore. Restrictions apply.