MIPRO 2017, May 22- 26, 2017, Opatija, Croatia

Design and Development of Contactless

Interaction with Computers Based on the Emotiv
EPOC+ Device

B. Šumak*, Matic Špindler* and M. Pušnik*

* Faculty of Electrical Engineering and Computer Science, University of Maribor, Slovenia
[Link]@[Link], [Link]@[Link], [Link]@[Link]

Abstract - Devices and methods for interaction with a Person’s quality of life, especially in case of persons
computer do not change very often. However, with with motor impairments, can be affected by different
technology advancements, solutions for new ways of human- factors such as physical health, psychological state, level
computer interaction are possible. Especially for physically of independence, social relationship, etc. [4]. Motor
disabled users, solutions for contactless interaction with a impaired users are mostly not able to use the standard
device promise new possibilities for independent computer computer interfaces (e.g. mouse, keyboard) due to their
use and management. In this study a solution for a health condition preventing them to use one or both hands.
contactless interaction with a computer was proposed based By being able to use a computer, motor impaired users can
on a brain-computer interface (BCI) device Emotiv EPOC+.
gain in their quality of life through the access to different
The EPOC+ device is able to detect brain signals, facial
expressions or levels of engagement (e.g. frustration,
online services, such as communication, education,
mediation, etc.), eye movement and head position through socialization, e-commerce, e-banking, etc. The ability to
the built-in gyroscope sensors. The contactless interaction control computers and other devices needed for accessing
solution proposed in this study was based on (1) mapping such services has become a necessary part of our lives in
facial expressions into user interface commands, (2) head- order to be able to live modern life [5].
movement detection for controlling the mouse cursor, and Latest advancements in different fields such as
(3) a virtual keyboard, which was developed to enable text
electronics, computers, wireless technologies, signal
input. The proposed solution was tested in a quasi-
processing and classifications have enabled development
experimental setting, where non-disabled users had to
complete several computer tasks. After completing the
of new non-invasive or user-friendly wireless devices,
tasks, perceived usefulness and perceived ease of use of the which can be used for implementing innovative solutions
proposed solution were assessed. At the end, limitations and for human-computer interaction (HCI), providing
future work are discussed. capabilities for computer controlling through speech
recognition, brain activity, eye-tracking, etc. Such
accessories and solutions for assistive technology (AT)
I. INTRODUCTION have enabled to design and implement new ways for
It is estimated that there is over a billion of people that controlling computers with hands-free solutions [5].
are living with disability [1]. Disability is a complex and People with motor impairments are confined within a
multidimensional human condition that almost everyone is home environment, unable to use devices like TV, lights,
going to experience at some point in life. Disability is a computers, kitchen accessories, etc. This study is focused
general term for different types of impairments. It is on designing an AT solution, which will enable motor
expected that almost everyone will get temporary or impaired users to independently control and use
permanently impaired and older people are going to computers based on a brain-computer interface (BCI)
experience increasing difficulties in functioning [1]. device. The BCI device provides users ability to control a
In this paper we are addressing people with motor computer by using brain activity only [6]. The BCI
disabilities especially. The Slovenian Paraplegic devices can be categorized based on their degree of
Association [2], founded in 1969, is a major national recording invasiveness of the electroencephalography
organization that represents and protects paraplegics and (EEG) method into one of the three categories [7]:
quadriplegics and enforces their rights and interests. The invasive (scalp EEG recording), partially or medium
most common causes of disability traffic are accidents, invasive (electrocorticography), and highly invasive
falls from heights, jumping into water, spinal cord (intracortical recording).
surgery, etc. In 2015 there were 1049 people registered in The BCI is one of the most growing fields of
the organization, which is approximately 0.051% of interdisciplinary research, involving disciplines such as
population living in Slovenia. Of these, more than half neuroscience, computer science, engineering and clinical
were people diagnosed with paraplegia and a third with rehabilitation [8]. In the past few years several
tetraplegia [3]. commercial BCI headsets were launched, which
accelerated the research and development of this
technology for various home environment settings. Recent

576
advancements in the field of EEG technology and BCI and it capabilities for providing information about facial
have offered acceptable quality-to-cost ratio and easy-to- expressions, head-movement and others.
use, out-of-the-box equipment with commercial BCI
headsets, which can be used for variety new applications III. EMOTIV EPOC+ DEVICE
[9].
Emotiv EPOC+ is a 14 channel wireless non-invasive
The rest of the paper is organized as follows. In next EEG and electromyography (EMG) device, which was
section, existing research based on the Emotiv EPOC+ designed for contextualized research and advanced BCI
device is summarized. Section that follows, describes the applications [24]. Electrodes of the EPOC+ headset are
Emotiv EPOC+ device and corresponding software located at AF3, AF4, F3, F4, F7, F8, FC5, FC6, P7, P8,
packages that were used in this study for developing the T7, T8, O1, O2 with two additional sensors that serve as
solution for the contactless interaction with the computer. CMS/DRL reference channels (one for the left and the
In section four, the proposed solution is presented. Results other for the right hemisphere of the head) [22].
of a pilot study, which aimed to test the proposed solution,
are discussed in section five. The last section concludes The EPOC+ headset is able to detect brain signals and
the paper together with directions and ideas for future facial expressions or levels of engagement (e.g.
research. frustration, mediation, etc.), eye movement and head
position through the built-in gyroscope sensors. The 9-
axis inertial motion sensor allows several
II. RELATED WORK motion/positional tracking and monitoring. The sampling
Results of several years of research are promising with rate of motion data is adjustable by the user (128Hz,
solutions with BCI applications such as BCI systems for 64Hz, 32Hz, or off). The device can be trained in order to
environmental control [10], speller systems [11,12], recognize cognitive neuro-activities. Emotiv provides
interfaces for robotic wheelchair steering [13–15], gaming several software solutions for receiving and analyzing
[16–18], etc. In existing literature, several studies were EEG data from the headset together with APIs and
published reporting results of evaluation of the Emotiv detection libraries:
EPOC+ device and its application. For example, the
EPOC+ device was used for designing and developing a - Performance Metrics & Emotional States suite
BCI-based system for generation of synthesized speech, provides capabilities for monitoring user’s
emotional states in real time and enables an extra
which works on eye-blinks detected from EEG signals
[19]. dimension in interaction by allowing the computer
to respond to user’s emotions. Using this suite
The EPOC+ device was also used for improving a user’s state of mind can be monitored and
speech interaction system by integrating the BCI device analyzed and information used for different
for the purpose of associating the brain signals with innovative solutions. For example, music and
commands for controlling wheel chairs for parallelized lighting can be adjusted according to user’s
people [20]. A study conducted by Vourvopoulos and experience in real time. Next, while in game, the
Liarokapis demonstrated, that that commercial BCIs like difficulty level of the game can be tailored and
EPOC+ can be used for effective and natural robot adjusted according to the user’s gaming
navigation both in the real and the virtual environments experience. This suite can be used in combination
[8]. with eye tracking devices and other solutions
which together provide rich information for real
Motor imagery is also an interesting topic in BCI time evaluation of the user’s experience while
research, which examines brain activity when imagining using the computer or mobile device. Information
motoric activities such as moving the left hand. EPOC+ provided by this suite can be used also for
device could be used for acquiring brainwaves, however a implementing adaptive user interfaces that can
recently published study showed that the device is not adjust according to user’s engagement,
recommended for implementing motor imagery excitement, and other states.
application [21].
- Facial expressions enables interpretation of the
In the field video games, the EPOC+ device can be signals measured by the EPOC+ headset into
used for assessing user experience and cognitive processes facial expressions in real time. This suite can be
caused by various stimulus modalities and gaming events used for example for implementing avatars that
while playing video games producing [22]. As react to user’s facial mimics in real time.
demonstrated in [23], the EPOC+ device can also be used
for analyzing how the frontal lobe of the brain (executive - Mental Commands is a suite that can read and
function) works in terms of prominent cognitive skills interpret user’s conscious thoughts and intent.
during games playing. With this suite solutions for manipulation of
virtual (e.g. an object in a computer game) or real
To the best of our knowledge, this study is a first objects (e.g. robotic arm) based on user’s thoughts
attempt to develop and evaluate a solution for contactless detection can be implemented.
interaction with a computer based on the EPOC+ device

577
 Figure 1. The Emotiv Xavier TestBench

Emotiv TestBench (see Fig. 1) is a software package Figure 2. Specifying the keystrokes based on facial epxressions using
that is used for displaying data from the Emotiv EPOC+ the Emotiv Xavier EmoKey tool
headset, including EEG signals, contact quality, FFT,
motion detection signals, wireless packet acquisition or All the above described features provided by the
loss display, marker events and headset battery level. The Emotiv EPOC+ devices can be used for implementing
software is able to record and replay files in a binary innovative human-computer interaction solutions that can
EEGLAB format. The binary format can also be converted replace standard computer devices such as the mouse and
into a .csv format for analyzing using open source EEG the keyboard completely.
data analysis software such as OpenViBE, BCI2000,
Matlab libraries, etc. IV. DESIGN OF THE SOLUTION FOR CONTACTLESS
In EEG systems it is difficult to separate the muscle COMPUTER INTERACTION BASED ON EPOC+
signals from the brain signals, therefore most medical For designing the solution for hands-free HCI,
EEG systems require that patients don’t move and stay together with the Emotiv EPOC+ headset following
completely still while recording. Due to the sensitivity of software was required:
these EEG recording systems the recordings can be biased
through eye blinks and swallowing. However, Emotiv - Emotiv Xavier EmoKey (see Fig. 2) enables to
solutions are able to filter these noises and actually use specify the translation map between specific facial
these noise information as their advantage when compared expressions and computer commands. It can be
with other EEG solutions. Eight of the fourteen EEG used for simulating the use of mouse buttons,
sensors of the EPOC+ device are placed in the frontal keyboard buttons or combinations of keys.
head lobe, which can read signals from the facial muscles - Emotiv Xavier ControlPanel (see Fig. 3), which is
and eyes. Most EEG systems treat these signals as noise the basic software package for managing the
and these are being filtered and eliminated from further EPOC+ device. This software is necessary when
analysis. Emotiv has adjusted the detection of brain first placing the headset on the head and
signals and based on their classification algorithms they establishing the communication link between the
are able to detect which muscles have caused the noise. device and the computer. It enables to check the
Facial expressions such as left eye wink, right eye wink, quality and strength of the signals, the ability for
raising eyebrows, smile, etc. can be detected through the facial expression detection, the cursor movement
analysis of noises caused by different muscles. Detection ability through the gyroscope sensors, etc. The
of the eye movement is also possible because eye is ControlPanel software can also be used for
electrically polarized and its movement causes electrical training purposes in operating the computer
signals, which in turn can be detected. through mental commands.
- Emotiv EPOC Brain Activity Map – is a solution
that shows brain activity on individual sensors and
provides additional graphical information about it.
Solutions for hands-free interaction with the computer
usually also need virtual keyboard similar to the
keyboards on mobile devices with touchscreens. There are
several solutions for virtual keyboard available such as
OptiKey. There are also solutions that enable writing
based on the speech recognition. However, due to the
compatibility issues with Emotiv software and because
speech recognition didn’t work for the Slovenian
Figure 3. Facial expression detection in Emotiv Xavier Controlpanel
language, we developed a custom virtual keyboard. The
developed keyboard allowed inserting special characters

578
 a specific time frame, and (6) completing an online
purchase in an online shop. Prior executing the tasks,
TABLE II. THE TRANSLATION MAP users were logged in to online services and they didn’t
have to register and/or login. During the execution of the
Facial Expression Computer command tasks users were not allowed to use the mouse and the
Left eye wink Click of the left mouse button computer keyboard.
Right eye wink Click of the right mouse button In the pilot study, which objective was to evaluate the
proposed solution, ten non-disabled users participated. At
Raised eyebrows Pressing the button “Enter” the beginning of the testing session we explained them the
Clenching teeth Pressing the combination of keys “Ctrl+V” purpose of the experiment and explained how the Emotiv
Raised righ corner of EPOC+ device functions. Then, all sensors of the device
Pressing the button “Backspace” were moistened and the headset was put on the head of the
the mouth
Head movement Moving the mouse cursor participants. After the quality of the signals was checked,
the participants entered the training phase of the
experiment (see Fig. 4). The training phase is important
such as ‘š’, ‘č’, etc. Some extra buttons/functionalities for users as well as for the Emotiv software and its
were added as well such as “copy all”, “delete all”, “clear machine learning algorithms. For training purposes the
clipboard”, etc. Emotiv Xavier Control Panel was used. After users felt
comfortable with using the headset and the sensitivity of
To be able to use the computer for tasks such as
writing an e-mail, browsing on the web, watching videos,
etc., the solution for hands-free computer use has to TABLE I. DECRIPTIVE STATISTICS OF PU AND PEOU
enable triggering actions equal to traditional input
interfaces. The EmoKey software enables to emulate Item median 1 2 3 4 5
computer keys and mouse buttons, which are triggered
Perceived Usefulness (PU) (adapted from [25])
based on the specified facial expression. In this study,
several facial expressions were translated into computer PU1 - Using the EPOC+ based solution 1.5 5 3 1 0 1
commands using the EmoKey software (see Table I for in my job would enable me to
definitions). With combination of the virtual keyboard, the accomplish tasks more quickly.
proposed solution enables to opening applications, PU2 - Using the EPOC+ based solution 1 6 3 0 1 0
positioning the mouse cursor on the specific button or would improve my job performance.
application area, etc.
PU3 - Using the EPOC+ based solution 1 6 2 1 1 0
in my job would increase my
V. EVALUATION OF THE PROPOSED SOLUTION: A productivity
PILOT STUDY PU4 - Using the EPOC+ based solution 1 6 1 2 1 0
The proposed solution was evaluated at the faculty in a would enhance my effectiveness on the
controlled lab environment. For testing purposes, six job.
computer tasks were specified, which users had to PU5 - Using the EPOC+ based solution 1 6 3 0 1 0
complete using the proposed solution only. The specified would make it easier to do my job.
tasks were: (1) writing a short “Hello”-type e-mail using a PU6 - I would find the EPOC+ based 2 4 4 1 0 1
web browser and an Gmail account, (2) finding a photo in solution useful in my job.
the photo gallery on the computer using the Windows
explorer, (3) downloading an e-book (PDF) from a
Perceived Ease of Use (PEOU) (adapted from [25])
specific web address, opening the e-book and finding a
specific chapter in the book, (4) making a short Skype PEOU1 - Learning to operate the EPOC+ 3.5 1 0 4 5 0
call, (5) finding a YouTube video and viewing the video at based solution would be easy for me.
PEOU2 - I would find it easy to get the 2.5 1 4 4 1 0
EPOC+ based solution to do what I want
it to do.
PEOU3 - My interaction with the 4 0 0 1 6 3
EPOC+ based solution would be clear
and understandable.
PEOU4 - I would find the EPOC+ based 3.5 2 1 2 4 1
solution to be flexible to interact with.
PEOU5 - It would be easy for me to 4 0 3 1 5 1
become skillful at using the EPOC+
based solution.
PEOU6 - I would find the EPOC+ based 3 1 0 5 4 0
solution easy to use.

Figure 4. The evaluation of the solution

579
the facial expressions detections were adjusted according as self-efficacy, compatibility, user experience quality,
to individuals’ facial expression capabilities, participants etc., which need to be considered and assessed. In order to
got instructions for completing the six tasks. fully understand the potential of the proposed solution,
future research is needed that will include real disabled
The main purpose of this study was to evaluate users’ users. This way, we will be able to understand what are
perceptions about usefulness (PU) and ease of use the positive and/or negative effects of the proposed
(PEOU) of the proposed solution. Times needed for solution and how can we improve it. Future research is
completing tasks were not measured. After users also needed in order to understand whether the proposed
completed the tasks, they answered a questionnaire solution is in some way better if compared to existing AT
consisting of items that were adapted from the solutions that disabled users are using in their daily
Technology Acceptance Model [25]. PU and PEOU items computer use activities. We also plan to compare the
were measured using a 5-point Likert type scale with outcomes between the non-disabled and disabled users.
values from 1 (“Strongly Disagree”) to 5 (“Strongly We assume that disabled-users will perceive the solution
Agree”). Descriptive statistics about the users’ perceptions more useful than non-disabled users, since they need such
about the usefulness and ease of use of the proposed AT solutions more.
solution are presented in Table II. It was expected that
users won’t perceive the proposed solution very much
useful and the scores for all PU items were very low. ACKNOWLEDGMENT
However, assessments of users’ perceptions in easiness of The authors acknowledge the financial support from
use of the proposed solution showed that with exception the Slovenian Research Agency (research core funding
of PEOU2 users didn’t experience difficulties in learning No. P2-0057).
and using the proposed solution. This outcome is
promising, because the solution is intended for disabled
REFERENCES
users that are not able to use one or both hands. For such
users, the proposed solution can be an alternative to their [1] The Lancet. World Report on Disability. Lancet
existing AT solution which they already use. 2011;377:1977. doi:10.1016/S0140-6736(11)60844-1.

Next step in evaluating the proposed solution will be [2] Slovenian Paraplegic Association n.d. [Link]
testing the solution using real disabled users and to [Link]/eng/.
measure their efficiency and perceptions in using the [3] Kastelic D, Ermenc H. Letno poročilo za leto 2015 2016:1–
proposed solution. We expect that disabled users will have 34.
higher perceptions about the usefulness of the proposed
[4] Siegel C, Dorner TE. Information technologies for active and
solution.
assisted living—Influences to the quality of life of an ageing
society. Int J Med Inform 2017;100:32–45.
VI. CONLUSION
doi:10.1016/[Link].2017.01.012.
Motor impaired users based on their condition usually [5] Kumar DK, Arjunan SP. Human–Computer Interface
have difficulties in using standard computer interfaces and
Technologies for the Motor Impaired. 2016.
input devices such as the keyboard, mouse, etc. because of
doi:10.1201/b19274.
the lack of controlling hands. Therefore, different AT
solutions were designed and developed that enable [6] Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G,
alternative interactions with a computer. Such AT require Vaughan TM. Brain-computer interfaces for communication
design and development of innovative hardware and and control. Clin Neurophysiol 2002;113:767–91.
software solutions. This paper reports results of a pilot [7] Pires G, Nunes U, Castelo-Branco M. Evaluation of Brain-
study, in which a solution for contactless interaction with
computer Interfaces in Accessing Computer and other
computer was designed and developed using the Emotiv
Devices by People with Severe Motor Impairments. Procedia
EPOC+ device. The EPOC+ device is an affordable non-
invasive EEG device that can be used for innovative Comput Sci 2012;14:283–92.
solutions based on brain activity analysis and facial doi:10.1016/[Link].2012.10.032.
expressions detection. It also provides head motion [8] Vourvopoulos A, Liarokapis F. Evaluation of commercial
sensors, which together can fully replace standard input brain–computer interfaces in real and virtual world
devices such as the computer mouse and keyboard. environment: A pilot study. Comput Electr Eng 2014;40:714–
Emotiv software for the EPOC+ device and custom 29. doi:10.1016/[Link].2013.10.009.
software packages (e.g. virtual keyboard, speech
[9] Göhring D, Latotzky D, Wang M, Rojas R. Semi-autonomous
recognition, etc.) represent a basis for developing new and
innovative AT for users, who are not able to use standard Car Control Using Brain Computer Interfaces, 2013, p. 393–
computer devices and interfaces. 408. doi:10.1007/978-3-642-33932-5_37.
[10] Cincotti F, Mattia D, Aloise F, Bufalari S, Schalk G, Oriolo
In addition to the development of different AT
solutions it is very important to fully understand how such G, et al. Non-invasive brain–computer interface system:
solutions impact real users and their experience in using Towards its application as assistive technology. Brain Res
computers. The main limitation of this study is that Bull 2008;75:796–803.
involved only non-disabled users. Next, only users’ doi:10.1016/[Link].2008.01.007.
perceptions about usefulness and ease of use of the [11] Pires G, Nunes U, Castelo-Branco M. Comparison of a row-
solution were assessed. There are also other factors such column speller vs. a novel lateral single-character speller:

580
 Assessment of BCI for severe motor disabled patients. Clin [18] Holz EM, Höhne J, Staiger-Sälzer P, Tangermann M, Kübler
Neurophysiol 2012;123:1168–81. A. Brain–computer interface controlled gaming: Evaluation of
doi:10.1016/[Link].2011.10.040. usability by severely motor restricted end-users. Artif Intell
[12] Pires G, Nunes U, Castelo-Branco M. Statistical spatial Med 2013;59:111–20. doi:10.1016/[Link].2013.08.001.
filtering for a P300-based BCI: Tests in able-bodied, and [19] Soman S, Murthy BK. Using Brain Computer Interface for
patients with cerebral palsy and amyotrophic lateral sclerosis. Synthesized Speech Communication for the Physically
J Neurosci Methods 2011;195:270–81. Disabled. Procedia Comput Sci 2015;46:292–8.
doi:10.1016/[Link].2010.11.016. doi:10.1016/[Link].2015.02.023.
[13] Millan J d. R, Galan F, Vanhooydonck D, Lew E, Philips J, [20] Al-Hudhud G. Affective command-based control system
Nuttin M. Asynchronous non-invasive brain-actuated control integrating brain signals in commands control systems.
of an intelligent wheelchair. 2009 Annu. Int. Conf. IEEE Eng. Comput Human Behav 2014;30:535–41.
Med. Biol. Soc., IEEE; 2009, p. 3361–4. doi:10.1016/[Link].2013.06.038.
doi:10.1109/IEMBS.2009.5332828. [21] Fakhruzzaman MN, Riksakomara E, Suryotrisongko H. EEG
[14] Lopes AC, Pires G, Vaz L, Nunes U. Wheelchair navigation Wave Identification in Human Brain with Emotiv EPOC for
assisted by human-machine shared-control and a P300-based Motor Imagery. Procedia Comput Sci 2015;72:269–76.
Brain Computer Interface. 2011 IEEE/RSJ Int. Conf. Intell. doi:10.1016/[Link].2015.12.140.
Robot. Syst., IEEE; 2011, p. 2438–44. [22] McMahan T, Parberry I, Parsons TD. Modality specific
doi:10.1109/IROS.2011.6048355. assessment of video game player’s experience using the
[15] Pires G, Castelo-Branco M, Nunes U. Visual P300-based BCI Emotiv. Entertain Comput 2015;7:1–6.
to steer a wheelchair: A Bayesian approach. 2008 30th Annu. doi:10.1016/[Link].2015.03.001.
Int. Conf. IEEE Eng. Med. Biol. Soc., IEEE; 2008, p. 658–61. [23] Mondéjar T, Hervás R, Johnson E, Gutierrez C, Latorre JM.
doi:10.1109/IEMBS.2008.4649238. Correlation between videogame mechanics and executive
[16] Pires G, Torres M, Casaleiro N, Nunes U, Castelo-Branco M. functions through EEG analysis. J Biomed Inform
Playing Tetris with non-invasive BCI. 2011 IEEE 1st Int. 2016;63:131–40. doi:10.1016/[Link].2016.08.006.
Conf. Serious Games Appl. Heal., IEEE; 2011, p. 1–6. [24] Inc E. Emotiv EPOC+ 2017. [Link]
doi:10.1109/SeGAH.2011.6165454.
[25] Davis FD. Perceived Usefulness, Perceived Ease of Use, and
[17] Wang Q, Sourina O, Nguyen MK. EEG-Based “Serious” User Acceptance of Information Technology. MIS Q
Games Design for Medical Applications. 2010 Int. Conf. 1989;13:319. doi:10.2307/249008.
Cyberworlds, IEEE; 2010, p. 270–6.
doi:10.1109/CW.2010.56.

581