See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/334481021

Detecting Student Engagement in e-learning Environment Based on Head

Pose Estimation and Eye Aspect Ratio

Conference Paper · April 2019

CITATION READS

1 616

2 authors:

Mustafa Uğur Uçar Ersin Özdemir

Havelsan İskenderun Teknik Üniversitesi
2 PUBLICATIONS 5 CITATIONS 23 PUBLICATIONS 117 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Mustafa Uğur Uçar on 16 July 2019.

The user has requested enhancement of the downloaded file.

 ICATCES 2019
Proceeding Book

International Conference on Advanced Technologies,

Computer Engineering and Science
26-28 Apr 2019 / Alanya, Turkey
Proceeding Book of the International Conference on Advanced Technologies,
Computer Engineering and Science (ICATCES 2019)

Editors
Assoc. Prof. Dr. Oguz FINDIK
Asst. Prof. Dr. Emrullah SONUÇ
Res. Asst. Yusuf Yargı BAYDİLLİ

Published, 2019.

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is
concerned. Nothing from this publication may be translated, reproduced, stored in a computerized system
or published in any form or in any manner.

http://icatces.org/
[email protected]

The individual contributions in this publication and any liabilities arising from them remain the responsi-
bility of the authors.

The publisher is not responsible for possible damages, which could be a result of content derived from
this publication.
International Conference on Advanced Technologies, Computer Engineering and Science (ICATCES 2019),
Apr 26-28, 2019 Alanya, Turkey

Honorary Committee
Prof. Dr. Refik Polat, Karabük University, Rector
Prof. Dr. Mehmet AKBABA, Karabük University

Chair
Assoc. Prof. Dr. İlker TÜRKER, Karabük University
Assoc. Prof. Dr. Oğuz FINDIK, Karabük University

Co-Chair
Asst. Prof. Dr. Emrullah SONUÇ, Karabük University

Organization Committee
Prof. Dr. İsmail Rakıp KARAŞ, Karabük University
Assoc. Prof. Dr. İlker TÜRKER, Karabük University
Assoc. Prof. Dr. Oğuz FINDIK, Karabük University
Asst. Prof. Dr. Caner ÖZCAN, Karabük University
Asst. Prof. Dr. Emrullah SONUÇ, Karabük University
Asst. Prof. Dr. Hakan KUTUCU, Karabük University
Res. Asst. Dr. Rafet DURGUT, Karabük University
Res. Asst. Sait DEMİR, Karabük University
Res. Asst. Yusuf Yargı BAYDİLLİ, Karabük University

Scientific Committee
Prof. Dr. Ali Karcı (İnönü University)
Prof. Dr. Basel Mahafzah (The University of Jordan)
Prof. Dr. Cemil ÖZ (Sakarya Üniversitesi)
Prof. Dr. Derviş KARABOĞA (Erciyes University)
Prof. Dr. Erkan ÜLKER (Konya Technical University)
Prof. Dr. Fatih Vehbi ÇELEBİ (Yıldırım Beyazıt University)
Prof. Dr. Ghulam Ali Mallah (Shah Abdul Latif University)
Prof. Dr. Haldun GÖKTAŞ (Yıldırım Beyazıt University)
Prof. Dr. Halil İbrahim BÜLBÜL (Gazi University)
Prof. Dr. Harun UĞUZ (Konya Technical University)
Prof. Dr. İhsan ULUER (Karabük University)
Prof. Dr. İsmail Rakıp KARAŞ (Karabük University)
Prof. Dr. Mario KOEPPEN (Kyushu Instutite Of Technology)
Prof. Dr. Mehmet AKBABA (Karabük University)
Prof. Dr. Mehmet ÖZALP (Karabük University)
Prof. Dr. Mykola S. Nikitchenko (Taras Shevchenko National University of Kyiv)
Prof. Dr. Nurhan KARABOĞA (Erciyes University)
Prof. Dr. Oleksandr I. Provotar (Taras Shevchenko National University of Kyiv)
Prof. Dr. Oleksandr O. Marchenko (Taras Shevchenko National University of Kyiv)
Prof. Dr. Raif BAYIR (Karabük University)
Prof. Dr. Sergiy D. Pogorilyy (Taras Shevchenko National University of Kyiv)
Prof. Dr. Serhii L. Kryvyi (Taras Shevchenko National University of Kyiv)
Prof. Dr. Vasyl M. Tereshchenko (Taras Shevchenko National University of Kyiv)
Prof. Dr. Valentina Emilia BALAS (University Aurel Vlaicu)
Prof. Dr. Yaşar BECERİKLİ (Kocaeli University)
Assoc. Prof. Dr. Abdrakhmanov RUSTAM (Ahmet Yesevi University)
Assoc. Prof. Dr. Adib HABBAL (Karabük University)
Assoc. Prof. Dr. Amirtayev KANAT (Ahmet Yesevi University)
Assoc. Prof. Dr. Bilal Alataş (Fırat University)
Assoc. Prof. Dr. Ergin YILMAZ (Bülent Ecevit University)
Assoc. Prof. Dr. Ivan Izonin (Lviv Polytechnic National University)
Assoc. Prof. Dr. İlhami Muharrem ORAK (Karabük University)
Assoc. Prof. Dr. İlker TÜRKER (Karabük University)

ii
International Conference on Advanced Technologies, Computer Engineering and Science (ICATCES 2019),
Apr 26-28, 2019 Alanya, Turkey

Assoc. Prof. Dr. İsmail BABAOĞLU (Konya Technical University)

Assoc. Prof. Dr. Kemal POLAT (Abant Izzet Baysal University)
Assoc. Prof. Dr. Mesut GÜNDÜZ (Konya Technical University)
Assoc. Prof. Dr. Mustafa Servet KIRAN (Konya Technical University)
Assoc. Prof. Dr. Necaattin Barışçı (Gazi University)
Assoc. Prof. Dr. Oleksii I. Chentsov (Taras Shevchenko National University of Kyiv)
Assoc. Prof. Dr. Rabie A. RAMADAN (Cairo University)
Assoc. Prof. Dr. Taras V. Panchenko (Taras Shevchenko National University of Kyiv)
Assoc. Prof. Dr. Tulep ABDIMUHAN (Ahmet Yesevi University)
Assoc. Prof. Dr. Yuliya Kozina (Odessa National Polytechnic University)
Asst. Prof. Dr. Ahmet BABALIK (Konya Technical University)
Asst. Prof. Dr. Berk Anbaroğlu (Hacettepe University)
Asst. Prof. Dr. Bilal Babayiğit (Erciyes University)
Asst. Prof. Dr. Burhan Konya Technical (Karabük University)
Asst. Prof. Dr. Caner ÖZCAN (Karabük University)
Asst. Prof. Dr. Ebubekir YAŞAR (Gaziosmanpaşa University)
Asst. Prof. Dr. Erkan DUMAN (Fırat University)
Asst. Prof. Dr. Ferhat ATASOY (Karabük University)
Asst. Prof. Dr. Hakkı SOY (Necmeddin Erbakan University)
Asst. Prof. Dr. Hannah INBARAN (Periyar University)
Asst. Prof. Dr. İlker YILDIZ (Abant İzzet Baysal University)
Asst. Prof. Dr. Kasım ÖZACAR (Karabük University)
Asst. Prof. Dr. Mehmet ŞİMŞEK (Duzce University)
Asst. Prof. Dr. Nesrin AYDIN ATASOY (Karabük University)
Asst. Prof. Dr. Nizar BANU (B S Abdur Rahman University)
Asst. Prof. Dr. Nursel YALÇIN(Gazi University)
Asst. Prof. Dr. Oktay AYTAR (Abant Izzet Baysal University)
Asst. Prof. Dr. Omar DAKKAK (Karabük University)
Asst. Prof. Dr. Ömer Kaan BAYKAN (Konya Technical University)
Asst. Prof. Dr. Ömer Muhammet Soysal (Louisiana State University)
Asst. Prof. Dr. Şafak BAYIR (Karabük University)
Asst. Prof. Dr. Şafak KAYIKÇI (Abant İzzet Baysal University)
Asst. Prof. Dr. Yasin ORTAKCI (Karabük University)
Asst. Prof. Dr. Yüksel ÇELİK (Karabük University)
Asst. Prof. Dr. Zafer ALBAYRAK (Karabük University)

Secretary
Res. Asst. Dr. Oğuzhan MENEMENCİOĞLU, Karabük University
Res. Asst. Ayşe Nur ALTINTAŞ, Karabük University
Res. Asst. Berna GÜNEŞ, Karabük University
Res. Asst. Elif KABULLAR, Karabük University
Res. Asst. Furkan SABAZ, Karabük University
Res. Asst. İdris KAHRAMAN, Karabük University
Res. Asst. Mehmet Zahid YILDIRIM, Karabük University
Res. Asst. Sait DEMİR, Karabük University
Res. Asst. Yasemin SANDAL, Karabük University

iii
International Conference on Advanced Technologies, Computer Engineering and Science (ICATCES 2019),
Apr 26-28, 2019 Alanya, Turkey

Welcome Address

It is a pleasure for us to offer you Abstracts Book for the 2nd International Conference on Advanced
Technologies, Computer Engineering and Science ICATCES 2019. Our goal was to bring together leading
academic scientists, researchers and research scholars to exchange and share their experiences and present
their latest research results, ideas, developments, and applications about all aspects of advanced technologies,
computer engineering and science. We decided to organize this event with the encouragement of our
colleagues in the hope of transforming the event into a symposium series. Our warmest thanks go to all
invited speakers, authors, and contributors of ICATCES 2019 for accepting our invitation. We hope that
you enjoy the symposium and look forward to meeting you again in one of the forthcoming ICATCES 2020
event.

iv
International Conference on Advanced Technologies, Computer Engineering and Science (ICATCES 2019),
Apr 26-28, 2019 Alanya, Turkey

Contents

Welcome Address iv

Proceedings viii
Improving classification performance on microscopic images using Generative Adversarial Net-
works (GAN) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Analyzing of the wound healing by implementing image processing algorithms on FPGA . . . . 7
Speckle reduction in images using non-local means filter and variational methods . . . . . . . . 11
Real time activity recognition using weighted dynamic time warping . . . . . . . . . . . . . . . 20
Remote controlled vehicle for surface and underground object detection . . . . . . . . . . . . . 25
Estimation relationship between electricity consumption and urban area from night satellite im-
agery: A case study for Istanbul . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Classification of flower species using convolutional neural networks . . . . . . . . . . . . . . . 33
Real-time diseases detection of grape and grape leaves using Faster R-CNN and SSD MobileNet
architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Improved network intrusion detection system using deep learning . . . . . . . . . . . . . . . . . 45
Sentiment analysis for hotel reviews with recurrent neural network architecture . . . . . . . . . . 50
Prediction of absenteeism at work with machine learning algorithms . . . . . . . . . . . . . . . 54
Automated diagnosis of tuberculosis using deep learning techniques . . . . . . . . . . . . . . . 58
A system based on image processing and deep CNN features for classification of defective fruits 63
A survey of joint routing and energy optimization techniques for underwater acoustic sensor
networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
An application of artificial bee colony algorithm to fatigue life estimation of magnesium alloy . . 77
An improved crow search algorithm for continuous optimization problems . . . . . . . . . . . . 81
Analysis of electronic countermeasure effects on air defense systems . . . . . . . . . . . . . . . 85
Performance evaluations of meta-heuristic algorithms on solving group elevator control problem 90
Performance comparison between genetic algorithm and ant colony optimization on traveling
salesman problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
On forecasting ability of a logistic model for prostate cancer under intermittent androgen suppres-
sion therapy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
Comparison of classification algorithms on heart disease data . . . . . . . . . . . . . . . . . . . 106
ECG signal classification with neural network ensemble . . . . . . . . . . . . . . . . . . . . . . 111
Detection of wart treatment method using machine learning algorithms . . . . . . . . . . . . . . 115
Detecting student engagement in e-learning environment based on head pose estimation and eye
aspect ratio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
Functional brain network analysis under cognitive task . . . . . . . . . . . . . . . . . . . . . . 124
A suggestion for electronic election system based on blockchain . . . . . . . . . . . . . . . . . 129
Assessing the architectural quality of software projects from an organizational perspective . . . . 135
An overview on data-driven prognostic methods of li-ion batteries . . . . . . . . . . . . . . . . 139
Game design for rehabilitation of children with disabilities by using depth sensor . . . . . . . . 142
Prioritizing manual black-box tests using topic modeling . . . . . . . . . . . . . . . . . . . . . 146
Increasing air traffic control efficiency with integrated human machine interface . . . . . . . . . 150
Comparison of classification algorithms in terms of model performance criteria in data mining
applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
Privacy scoring over professional OSNs: More central users are under higher risk . . . . . . . . 157

v
International Conference on Advanced Technologies, Computer Engineering and Science (ICATCES 2019),
Apr 26-28, 2019 Alanya, Turkey

Recent advances and future trends in industrial communication networks: M2M, I-IoT and inter-
operability issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
Raspberry Pi based personalized encoding ID system inspired by the quick response code software
and hardware design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
The usage statistics of new HTML5 semantic elements in the ClueWeb12 dataset . . . . . . . . 172
Internet of Things based remote monitoring system design for industrial plants . . . . . . . . . . 176
3D scene reconstruction using a swarm of drones . . . . . . . . . . . . . . . . . . . . . . . . . 180
Development of a smart helmet for digital data collection and applications on construction sites . 184
Android secure camera application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
A recommendation system for seattle public library using Naïve Bayes Classifier . . . . . . . . . 196
Server based indoor location and navigation using beacon devices . . . . . . . . . . . . . . . . 200
A conceptual design for managing internet of things devices in emergency situations . . . . . . 205
Can we fight social media with multimedia learning systems? . . . . . . . . . . . . . . . . . . . 211
Increasing the use of digital banking applications for the baby boomer generation by easy interface 218
Detection of P300 ERP waves by different classification methods . . . . . . . . . . . . . . . . . 222
Investigating personalised applications in MOOCs: the challenge of achieving transparency . . . 226
Unification of IT process models into a simpler framework . . . . . . . . . . . . . . . . . . . . 230
An IoT based mobile radar system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
Web and mobile based online joint working platform development for university students . . . . 245
Human computer interaction with one-shot learning . . . . . . . . . . . . . . . . . . . . . . . . 250
Local statistical features for multilingual artificial text detection from video images . . . . . . . 256
Real time distributed controller for delta robots . . . . . . . . . . . . . . . . . . . . . . . . . . 261
An overview of captcha systems and bypassing math captcha . . . . . . . . . . . . . . . . . . . 266
A review on Nvidia GauGan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
A simulation based harmony search algorithm for part routing optimization problem . . . . . . . 274
Multi class tag prediction on stack overflow dataset . . . . . . . . . . . . . . . . . . . . . . . . 278
A mobile indoor/outdoor augmented reality application for architecture . . . . . . . . . . . . . . 282
Reassembly of synthetically fractured objects . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
Classification of EEG patterns by using Katz Fractal Dimension . . . . . . . . . . . . . . . . . 290
Learning semi-supervised nonlinear embeddings for domain-adaptive pattern recognition . . . . 296
Tree-seed programming for symbollic regression . . . . . . . . . . . . . . . . . . . . . . . . . 301
An automated deep learning approach for bacterial image classification . . . . . . . . . . . . . . 305
Image processing on electrophoresis image with embedded system . . . . . . . . . . . . . . . . 310
Feature selection for text classification based on term frequency and inter-class standard deviation 312
Comparison of machine learning algorithms for franchise approval . . . . . . . . . . . . . . . . 316
Face detection using forensic software and deep learning methods on images and video files . . . 321
IP packet marking and forwarding based on content type with SDN . . . . . . . . . . . . . . . . 326
NetCar: A testbed for mobile sensor networks . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
Modeling and simulation of a reconfigurable microstrip antenna for wireless communication and
mobile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336
Node weighting method in centrality measure of complex networks . . . . . . . . . . . . . . . . 339
Deep learning based web application security . . . . . . . . . . . . . . . . . . . . . . . . . . . 346
Evaluation of student academics performance via machine learning algorithms . . . . . . . . . . 352
Criminological evaluation of cyber attacks on information and network security . . . . . . . . . 357
Digital forensics in social media; evaluation in the context of system, informatics, network and
cyber security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360
Gyroscope-accelerometer controlled smart disabled wheelchair . . . . . . . . . . . . . . . . . . 364
The optimization of the process of recognization fingerprint through the minutiae technique . . . 368
An automated GIS tool for property valuation . . . . . . . . . . . . . . . . . . . . . . . . . . . 373
Intelligent examination glove system design for use in medical education and user interface
application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
Brain tumor detection via active contours and scale invariant feature transform . . . . . . . . . . 381
Spatial preference system for roads maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . 385
Spatial preferences decision support tool for historical building restoration . . . . . . . . . . . . 387
Satellite images classification in geographic information systems . . . . . . . . . . . . . . . . . 389
Machine learning and satellite images for agricultural areas determination . . . . . . . . . . . . 391

vi
International Conference on Advanced Technologies, Computer Engineering and Science (ICATCES 2019),
Apr 26-28, 2019 Alanya, Turkey

TDMA scheduling for real-time flows in cluster-based wireless sensor networks . . . . . . . . . 393
Deep learning approaches for traffic flow predictions on signal-controlled intersections . . . . . 398
Securing TLS from MITM incursion using Diffie-Hellman . . . . . . . . . . . . . . . . . . . . 404

vii
 International Conference on Advanced Technologies, Computer Engineering and Science (ICATCES 2019),
Apr 26-28, 2019 Alanya, Turkey

Detecting Student Engagement in e-learning

Environment Based on Head Pose Estimation
and Eye Aspect Ratio
M. U. UÇAR1 and E. ÖZDEMİR2
1
Havelsan A.Ş., Konya/Turkey, [email protected]
2
Iskenderun Technical University, Hatay/Turkey, [email protected]

In traditional classrooms, teachers observe their students and

Abstract – In this paper, we explored the feasibility of detecting adjust the teaching activity according to the behavioral and
student engagement in e-learning environment based on Head emotional state of the students. However, it is not feasible for a
Pose Estimation and Eye Aspect Ratio of the student. 1000 video teacher to track each one of his/her students at the same time.
frames were extracted from UPNA Head Pose Database and used
as a dataset for training and testing. Euler angles (pitch, yaw and Therefore, providing a personalized learning environment for
roll) of the head were calculated based on Levenberg-Marquardt each student is not a realistic expectation.
optimization. We constructed our feature vector by utilizing Euler In order to automate personalization process for each student
angles and Eye Aspect Ratio for each frame. In addition to these and improve learning and teaching quality, computer vision and
behavioral features, each frame was annotated as “Engaged” or machine learning techniques are mostly used in recent days.
“Not Engaged” by five labelers. Engagement classifiers were In this study, we try to address engagement detection
trained with the labelled dataset. Experiments were made with
Random Forest, Decision Tree, SVM and KNN machine learning problem and uncover the contribution of Head Pose Estimation
algorithms to classify student engagement. SVM classifier and Eye Aspect Ratio to the student’s engagement. Our main
achieved 72.4% accuracy on the labelled dataset. Our study shows goal is to find out a way for automatically detecting engagement
that Head Pose and Eye Aspect Ratio are significant contributors by capturing and processing student’s video in real time with
to the student’s Visual Focus of Attention and engagement. the aid of a web camera, a computing device and advanced
computer vision and machine learning technologies.
Keywords – engagement detection, head pose estimation, eye
In the next section, we will review related studies in this field.
aspect ratio, computer vision, machine learning.
Then we will talk about our methodology and come up with a
cost-effective and simple engagement classification model.
I. INTRODUCTION Later, we will present the results of our experiments to reveal
the performance of the proposed solution. Finally, we will sum
Along with the technological developments, it has become up our findings and propose some improvements for future
feasible to process big data such as videos in real-time. work.
Software libraries that consists of the state-of-the-art
computer vision and machine learning algorithms such as II. RELATED WORK
OpenCV [1] and dlib [2] has accelerated the studies in image
processing field. Australian Council of Educational Research defined
The growing importance of Human-Computer Interaction student engagement as “students’ involvement with activities
and the introduction of smart devices in every aspect of our and conditions likely to generate high quality learning” [4].
lives have brought the necessity of gaining humanlike skills to According to Kuh (2008):
the smart devices. “Student engagement represents both the time and energy
Computer vision is one of these leading skills. One of the students invest in educationally purposeful activities and the
most important and common application of computer vision is effort institutions devote to using effective educational
face recognition. In contrast to face recognition, Head Pose practices.” [5].
Estimation has recently started to get more attention from Student engagement is a significant contributor to the student
researchers. success. Several researches made with students reveal the
Although the studies related with image processing in the strong relation between student engagement and academic
field of education are not very common [3], it is observed that achievement [6, 7]. Similarly, Trowler who has published a
they have increased in recent years. comprehensive literature review on student engagement states
Since engagement has a vital role for efficiency and success that a remarkable amount of literature has established robust
of a course regardless of whether it is a traditional classroom or correlations between student involvement and positive
e-learning course, learner engagement detection has been one outcomes of student success and development [8].
of the main research topics in this field. As cited in Fredricks et al., student engagement has three

119
 International Conference on Advanced Technologies, Computer Engineering and Science (ICATCES 2019),
Apr 26-28, 2019 Alanya, Turkey

dimensions: behavioral, cognitive and emotional engagement they showed the correlation between human and automatic
[9]. Behavioral engagement refers to student’s participation and engagement results [15].
involvement in learning activities. Cognitive engagement In addition to behavioral engagement studies, some
corresponds to student's psychological investment on the researchers focused on student’s emotional engagement. Ayvaz
learning process such as being thoughtful and focusing on and Gürüler developed an application to detect student's
achieving goals. On the other hand, emotional engagement can emotional state based on facial expressions. They created a
be understood as student's relationship with his/her teacher and dataset from 11680 emotional images of 12 student (5 female
friends; feelings about learning process such as being happy, and 7 male) and achieved 97.15% accuracy with SVM to
sad, bored, angry, interested and disappointed. classify student's emotional state [16].
In this study, we focus on behavioral engagement, since its Asteriadis et al. aimed a user-independent, non-intrusive
symptoms, such as drowsiness, looking at an irrelevant place engagement detection method, which is also resistant to
and focusing on the screen can be detected explicitly. different lighting conditions, and needs only a computing
Head pose, gaze and facial expressions are main contributors device, and a web-camera. They tracked head pose and eye gaze
to behavioral and emotional engagement. Automated to classify user's attention and frustration. They achieved above
engagement detection studies mostly based on these features. 80% accuracy with Neuro-Fuzzy system [17].
Vertegaal et al. presented an experiment with eye-trackers
attached to the subjects during a conversation and concluded III. METHODS
that gaze is an excellent predictor of conversational attention
A. Dataset and Labelling
[10].
Stiefelhagen and Zhu defined gaze as the direction where the In order to classify the student engagement, we employed
eyes are pointing in the space. They also point out that this several machine-learning algorithms that required labelled data
in training dataset for supervised learning.
direction has a close relation with the Visual Focus of Attention
(VFoA) and it is sum of head and eye orientation. In their study, We created student engagement dataset by using UPNA
they made an experiment to discover the contribution levels of Head Pose Database [18] that is suitable for our needs. This
head and eye orientation to the VFoA. As a result, they found database is publicly available for research purposes and created
that head pose contributes 68.9% to the overall gaze and 88.7% specifically for head pose estimation. There are totally 120
to VFoA estimation [11]. The results show us head pose is a videos in the database for 10 different persons (4 female and 6
good indicator for VFoA and student engagement detection. male) and 12 videos for each person. The videos are at 30 fps
Aslan et al. measured student engagement in 1:1 learning in MPEG-4 format with a resolution of 1280×720 pixels and
environment with 85% accuracy. They used multiple sensors: each video is 10 seconds long. In each video, subjects moved
(1) 2D camera for facial feature detection and analyzing their head combining translations and rotations along the three
spatial axes of head. Actual head pose angles and head position
learner's emotional state; (2) external eye tracking module to
data for each frame were measured with a sensor attached to the
detect student's gaze and region of interest on the screen; (3) 3D
subject’s head. These ground-truth measurements are shared as
camera (Intel's RealSense™ 3D camera and RealSense™ SDK)
a text file with the database.
to record depth data corresponding to each pixel of 2D camera.
Utilizing a 3D camera, they could extract more precise facial We extracted random 100 frames for each person from UPNA
landmark data resulting more accurate emotion recognition, Head Pose Database and created a dataset that consists of totally
head and body pose detection. They collected data from high 1000 images. Five human labelers annotated each image as (0)-
school students. Three human labelers annotated collected data “Not Engaged” or (1)-“Engaged”. We measured consistency
as “Engaged” or “NotEngaged”. They used labelled data to between the labelers based on Fleiss' kappa measure [19], which
train and test their Decision Tree, Random Forest and Naïve is a statistical approach for assessing the inter-rater reliability of
Bayes classifiers [12]. more than two raters. Percent agreement and Fleiss’ kappa
values of our five labelers are 0.95 and 0.85 respectively. These
Nezami et al. proposed a deep learning model to improve
values indicate there is strong agreement between labelers.
engagement detection for the face images in the Facial
Then, we labelled each image as majority decision of our
Expression Recognition-2013 (FER-2013) dataset. Their model labelers.
is trained with 4627 engaged and disengaged samples. On the
test set, proposed model achieves 72.38% classification Sample images in our dataset are shown below in Figure 1.
accuracy [13].
Bosch described results from several prior works that utilize
facial features to detect student engagement. He proposed a new
method based on Mind Wandering detection using face-based
techniques to improve engagement accuracies [14].
Whitehill et al. explored methods for automatic student
engagement recognition from facial expressions. They used
machine learning to automate process and found that
Figure 1. Sample images in the dataset.
engagement detection of 10-second videos can be reliably (First row: Engaged, Second row: Not Engaged)
predicted by using average label of the frames in video. Finally,

120
 International Conference on Advanced Technologies, Computer Engineering and Science (ICATCES 2019),
Apr 26-28, 2019 Alanya, Turkey

B. Face and Facial Landmark Detection detected by using EAR value. When student is sleepy or closed
We detected the face in the image by utilizing the Histogram his/her eye, EAR value is minimized. Facial landmarks used to
of Oriented Gradients algorithm of Dalal and Triggs [20], which calculate EAR value are shown below in Figure 4.
is already implemented in Dlib library.
After face detection, facial landmarks are extracted by using
Dlib's implementation of the Kazemi and Sullivan’s paper [21].
Detected 68 facial landmarks are illustrated in Figure 2.

Figure 4. Opened and closed eyes with related facial landmarks

We calculated EAR value based on Euclidean distance

formula by using facial landmark coordinates in the eye region
as shown in Equation 1 below:
ԡ’ʹ െ ’͸ԡ ൅ ԡ’͵ െ ’ͷԡ
 ൌ (1)
ʹ ൈ ԡ’ͳ െ ’Ͷԡ

E. Classification
Figure 2. Facial landmarks
We constructed our feature vector with four elements that
C. Head Pose Estimation consists of Pitch, Yaw, Roll and EAR values for each frame in
our previously labelled dataset. We used half of our dataset for
We used built-in solvePnP function in OpenCV library to
training purpose and the other part for testing. We trained SVM,
estimate head pose. This function solved Perspective-n-Point
KNN, Random Forest and Decision Tree classifiers with
problem by using Levenberg-Marquardt optimization [22]. As
training set. Thanks to these classifiers, we were able to classify
cited in Zheng et al., Perspective-n-Point is a problem of
a student image captured from camera as “Engaged” or “Not
estimating the pose of a calibrated camera given a set of n 3D
Engaged” in real-time. Our student engagement classification
(n≥3) point coordinates in the world and their corresponding 2D
software runs as a standalone application in student’s computer
projections in the image [23]. We used 2D facial landmark
during he/she takes an e-learning course. Screenshot of our
coordinates for 6 points (nose tip, chin, left corner of left eye,
application is shown in Figure 5.
right corner of right eye, mouth’s left corner and mouth’s right
corner) and their corresponding approximate 3D point
coordinates as Mallick suggested [24]. Then, we got
corresponding rotation and translation vector from solvePnP
function. We transformed the rotation vector to a rotation
matrix by using Rodrigues’ rotation formula [25]. Finally we
computed Euler angles (pitch, yaw and roll) from the rotation
matrix as described in Slabaugh’s paper [26]. Euler angles are
illustrated in Figure 3.

Figure 5. Screenshot of our application

IV. RESULTS
In order to analyze achievement of the student engagement
detection system, we performed two experiments.
In the first experiment, we aimed to measure the mean
absolute deviation of Euler angles. For this purpose, we
Figure 3. Euler angles processed 120 videos of 10 users in UPNA Head Pose
Database. Pitch, yaw and roll angles for each frame is calculated
D. Eye Aspect Ratio (EAR) and compared with the ground-truth pitch, yaw and roll angles,
In addition to Euler angles, another significant contributor to which are provided by UPNA Head Pose Database. The mean
the student engagement is EAR. Soukupová & Cech used EAR absolute deviation from ground-truth data in pitch, yaw and roll
value for real-time eye blink detection successfully [27]. angles are 1.3°, 4.9° and 4.3° respectively and details are
Similar to eye blink detection, student’s drowsiness can be provided in Table 1.

121
 International Conference on Advanced Technologies, Computer Engineering and Science (ICATCES 2019),
Apr 26-28, 2019 Alanya, Turkey

Table 1. Mean Absolute Deviations of Euler Angles VI. CONCLUSION

Pitch (°) Yaw (°) Roll (°) We have proposed a methodology to detect student
User_01 1.656645 2.685429 1.94371 engagement in e-learning environment based on Head Pose
User_02 1.109542 2.396021 1.527393 Estimation and EAR. We used UPNA Head Pose Database to
User_03 1.095765 5.770226 2.941756 produce our dataset and five human labelers annotated each
User_04 1.402484 3.083149 1.954714 image in the dataset with high inter-rater reliability (0.85 Fleis’
User_05 1.245349 6.498999 4.362588 kappa). We have illustrated that student engagement can be
User_06 1.768564 4.561221 4.521068 detected by machine learning models with 72.4% accuracy.
User_07 1.247303 4.029759 2.855128
Our application is based on computer vision techniques and
User_08 1.33414 9.503025 6.241764
User_09 0.968213 6.234931 5.708336
required a simple webcam. This method is more cost-effective,
User_10 1.590701 4.939992 11.44409 simple and applicable than sensor-based methods, which are
Deviation 1.341871 4.970275 4.350055 also expensive and intrusive. However, sensor-based methods
provide more accuracy. There is a natural trade-off between
high accuracy and non-intrusiveness.
In the second experiment, we aimed to measure the overall In future work, in order to increase student engagement
success of student engagement classification. SVM, KNN, accuracies, researchers might consider other contributors to the
Random Forest and Decision Tree student engagement
student engagement such as gaze, gestures and facial
classifiers, which we trained previously, are tested with the
expressions as well as Head Pose Estimation and EAR.
training part of our dataset. 72.4%, 71.6%, 70.6% and 70%
Additionally, utilizing more extensive dataset might boost
accuracies are achieved using SVM, KNN, Random Forest and
Decision Tree classifiers respectively as shown in Table 2. classification accuracies.

Table 2. Overall Success of the Classifiers REFERENCES

Accuracy (%) [1] G. Bradski, “The OpenCV Library,” Dr. Dobb’s Journal of Software
SVM 72.4 Tools, vol. 25, pp. 120-125, 2000. Available: https://opencv.org
KNN 71.6 [2] D. E. King, “Dlib-ml: A Machine Learning Toolkit,” Journal of Machine
Random Forest 70.6 Learning Research 10, pp. 1755-1758, 2009. Available: http://dlib.net
Decision Tree 70.0 [3] Ö. Sümer, P. Goldberg, K. Stürmer, T. Seidel, P. Gerjets, U. Trautwein,
and E. Kasneci, “Teachers' Perception in the Classroom,” CVPR
Additionally, we conducted second experiment with a new Workshops 2018, pp. 2315-2324.
dataset that is created by using ground-truth Euler angles, and [4] H. Coates, “Engaging Students for Success: 2008 Australasian Survey of
calculated EAR value. 78.6%, 77.8%, 77% and 76.4% Student Engagement,” Victoria, Australia, Australian Council for
accuracies are achieved using Random Forest, Decision Tree, Educational Research, 2008.
KNN and SVM classifiers respectively as shown in Table 3. [5] G. Kuh, T. Cruce, R. Shoup, J. Kinzie, and R. Gonyea, “Unmasking the
effects of student engagement on first year college grades and
Table 3. Overall Success of the Classifiers (With Ground-truth Data) persistence,” Journal of Higher Education, vol. 79 (5), pp. 540–563, 2008.
[6] S. Günüç, “The relationships between student engagement and their
Accuracy (%)
academic achievement,” International Journal on New Trends in
SVM 76.4
Education and Their Implications (IJONTE), vol. 5, pp. 216-231, 2014.
KNN 77.0
Random Forest 78.6 [7] M. J. Casuso-Holgado, A. I. Cuesta-Vargas, N. Moreno-Morales, M. T.
Decision Tree 77.8 Labajos-Manzanares, F. J. Barón-López, and M. Vega-Cuesta, “The
association between academic engagement and achievement in health
sciences students,” BMC medical education, 2013.
V. SYSTEM CONFIGURATION [8] V. Trowler, “Student Engagement Literature Review,” York, Higher
Our system configuration are shown in Table 4. Education Academy, 2010.
[9] J. A. Fredricks, P. C. Blumenfeld, and A. H. Paris, “School engagement:
Table 4. System Configuration potential of the concept, state of the evidence,” in Review of Educational
Research, vol. 74(1), pp. 59-109, 2004.
Processor Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz
[10] R. Vertegaal, R. Slagter, G. van der Veer, and A. Nijholt, "Eye Gaze
Graphics Card Gigabyte NVIDIA GeForce GTX 980 4 GB Patterns in Conversations: There is more to conversational agents than
meets the eyes," Proceedings of CHI 2001, Seattle, WA, vol. 3(1), pp.
RAM 16 GB
301-308, 2001.
Webcam Logitech HD Webcam C525 [11] R. Stiefelhagen, and J. Zhu, “Head Orientation and Gaze Direction in
OS Windows 10 Pro 64-bit Meetings,” CHI '02 Extended Abstracts on Human Factors in Computing
Systems (CHI EA '02), ACM, New York, NY, USA, pp. 858-859, 2002.
IDE Qt Creator 4.2.1
[12] S. Aslan, Z. Cataltepe, I. Diner, O. Dundar, A. A. Esme, R. Ferens, G.
Libraries OpenCV (version 3.3.1), dlib (version 19.8) Kamhi, E. Oktay, C. Soysal, and M. Yener, “Learner Engagement
Measurement and Classification in 1:1 Learning,” Proceedings - 2014

122
 International Conference on Advanced Technologies, Computer Engineering and Science (ICATCES 2019),
Apr 26-28, 2019 Alanya, Turkey

13th International Conference on Machine Learning and Applications,

ICMLA 2014, pp. 545-552, 2014.
[13] O. M. Nezami, L. Hamey, D. S. Richards, and M. Dras, “Engagement
Recognition using Deep Learning and Facial Expression,” eprint
arXiv:1808.02324, 08/2018.
[14] N. Bosch, “Detecting student engagement: Human versus machine,” In
Proceedings of the 2016 Conference on User Modeling Adaptation and
Personalization, UMAP ’16, pp. 317–320, New York, NY, USA, 2016.
[15] J. Whitehill, Z. Serpell, Y.C. Lin, A. Foster, and J. Movellan, “The faces
of engagement: Automatic recognition of student engagement from facial
expressions,” IEEE Transactions on Affective Computing vol. 5, no. 1,
pp. 86-98, 1 Jan.-March 2014.
[16] U. Ayvaz, and H. Gürüler, “Real-time detection of students' emotional
states in the classroom,” 25th Signal Processing and Communications
Applications Conference (SIU), 2017.
[17] S. Asteriadis, K. Karpouzis, and S. Kollias, “Feature extraction and
selection for inferring user engagement in an HCI environment,” in
Human-Computer Interaction, New Trends, Springer Berlin Heidelberg,
pp. 22-29, 2009.
[18] M. Ariz, J. J. Bengoechea, A. Villanueva, and R. Cabeza, “A novel 2D/3D
database with automatic face annotation for head tracking and pose
estimation,” Computer Vision and Image Understanding, vol. 148, pp.
201-210, ISSN 1077-3142, July 2016.
[19] K. L. Gwet, Handbook of Inter-Rater Reliability. 2nd ed., Gaithersburg:
Advanced Analytics, LLC ISBN 978-0-9708062-2-2, 2010.
[20] N. Dalal, and B. Triggs, “Histograms of Oriented Gradients for Human
Detection,” IEEE Computer Society Conference on Computer Vision and
Pattern Recognition (CVPR'05), 1, vol. 1, pp. 886-893, 2005.
[21] V. Kazemi, and J. Sullivan, “One Millisecond Face Alignment with an
Ensemble of Regression Trees,” 2014 IEEE Conference on Computer
Vision and Pattern Recognition pp. 1867-1874, 2014.
[22] J. J. Moré, “The Levenberg-Marquardt algorithm: Implementation and
theory,” In: A. G. Watson (eds) Numerical Analysis. Lecture Notes in
Mathematics, vol. 630. Springer, Berlin, Heidelberg, 1978.
[23] Y. Zheng, Y. Kuang, S. Sugimoto, K. Åström, and M. Okutomi,
“Revisiting the PnP Problem: A Fast, General and Optimal Solution,”
Proceedings of the IEEE International Conference on Computer Vision,
pp. 2344-2351, 2013.
[24] S. Mallick, “Head Pose Estimation using OpenCV and Dlib,” Learn
OpenCV, 26.09.2016. Available: http://www.learnopencv.com/head-
pose-estimation-using-opencv-and-dlib/
[25] L. Sorgi, “Two-view geometry estimation using the Rodrigues rotation
formula,” In 2011 18th IEEE International Conference on Image
Processing, pp. 1009-1012. IEEE, 2011.
[26] G. G. Slabaugh, “Computing Euler angles from a rotation matrix,” 1999.
Available: http://www.gregslabaugh.net/publications/euler.pdf
[27] T. Soukupová, and J. Cech, “Real-Time Eye Blink Detection using Facial
Landmarks,” 21st Computer Vision Winter Workshop, Rimske Toplice,
Slovenia, February 3–5, IEEE. 2016.

123

View publication stats