Facial Expression Recognition using MATLAB

Bhakthi Rai#1, Chaya Shetty#2

#
Dept. Of Electronics & Communication
NMAMIT, Nitte
1
[email protected]
[email protected]

Abstract— Automatic facial expression recognition has become and powerful verbal languages, facial expression role in interactions
an interesting and challenging area for the computer vision field remain essential, and sometimes even critical.
and its application areas are not limited to mental state
identification, security, automatic counselling systems, face The facial expression detection [2] scheme is based on an
expression synthesis, lie detection, music for mood, automatic information theory approach that decomposes face images into a
tutoring systems, operator fatigue detection etc. One of the non small set of characteristic feature images called “eigenfaces,” which
verbal communication method by which one understands the may be thought of as the principal components of the initial training
mood/mental state of a person is the expression of face for e.g. set of face images. Recognition is performed by projecting a new
happy, sad, fear, disgust, surprise and anger. Improving the image into the subspace spanned by the eigenfaces and then
skills of reading expressions is an important step towards classifying the face by comparing its position in face space with the
successful relations. Facial expression recognition has increasing positions of known individuals [3].
applications and requires more accurate and reliable facial
expression recognition system. The main aim of this paper is to This paper deals with the facial expression recognition of a dynamic
increase the accuracy rate of predicting the expression and to images or a sequence of images or a video. The faces detected from
have application based on dynamic images/sequence of the input image/video, are cropped and stored in a folder. The
images/videos and also to handle occlusion. cropped faces are then processed and compared with the database
images consisting all the different expressions to be recognised. The
Keywords-- Viola-Jones Algorithm, Eigen Face Recognition, appropriate match is found based on the concept of eigenface and the
Information theory approach, Principal component analysis, result is stored in the text format.
Facial Expression Recognition.

II. FACIAL DETECTION APPROACH

I. INTRODUCTION The Viola-Jones algorithm [1] uses digital image features, that is, a
The face is a primary focus of attention in social intercourse, playing scalar product between the image and some digital image templates.
a major role in conveying identity and emotion [1]. Many times, More precisely, let I and P denote an image and a pattern, both of the
body languages and especially facial expressions, tells more than same size N X N . The feature associated with pattern P of image I is
words about one’s state of mind. Computational models of face defined by
recognition, in particular, are interesting because they can contribute   I i , j 1 p ( i , j ) iswhite    I i , j 1 p ( i , j ) isblack
not only to theoretical insights but also to practical applications. 1 i  N 1 j  N 1 i  N 1 j  N (1)
The face detection using Viola-Jones algorithm [1] invovles
A face detector has to tell whether an image of arbitrary size contains following calculations:
a human face and if so, where it is. One natural framework for The defined exponential loss is
considering this problem is that of binary classification, in which a x, y   R d   1,1, L y , f x   exp yf x , (2)
classifier is constructed to minimize the misclassification risk. A new
face detector for color images can be developed where the objective where the classifier f : Rd  R takes the form of a linear
is to develop very efficient algorithm in terms of low computational combination of weak classifiers
complexity, with maximum number of face detections and the T
minimum number of false alarms. f     t ht ,
t 1 (3)
This task can be achieved using Viola-Jones algorithm [1] which is With T  N , min1t T  t  0 and t , ht    1,1. Naturally, the
an operational algorithm that works with a reasonable computational overall Objective is
budget. Viola-Jones algorithm [1] is highly efficient, which is fed n T
 
with a real time image sequence generated from a standard webcam, min  wi 1L yi ,   t ht xi ,
it performs well on a standard PC. ht , t 0 i 1  t 1  (4)
with some initial probabilistic weight wi(1).
A facial expression is a gesture executed with the facial muscles,
which convey the emotional state of the subject to observers. An An approach is deployed to deduce the optimal classifiers ht and
expression sends a message about a person’s internal feeling. Facial weights at one after another, although there is no guarantee that the
expressions are a primary means of conveying non verbal
objective is minimized. Given  s , h s 1 s t , let Z t 1 be the weighted
information among humans, though many animal species display
facial expressions too. Although human developed a very wide range exponential loss and seek to minimize it through ht , t .
 n To extract the relevant information in a face image, encode it as
Z t 1  min  wi 1e yi s 1 s hsxi 
t
efficiently as possible, and compare one face encoding with a
ht, t 0 i 1 database of models encoded similarly. A simple approach to
n n extracting the information contained in an image of a face is to
 min  Di t  e t 1y ht xi 1   Di t  e t 1y ht xi 1 somehow capture the variation in a collection of face images,
i i
ht , t 0 i 1 i 1
independent of any judgment of features, and use this information to
n
 n
 min e t  Di t   e t  e t  Di t 1y ht xi 1
i
encode and compare individual face images.
ht t 0 i 1 i 1
Mathematically, to find the principal components of the distribution

ht t 0
 n D t 

i 1 Z t i

 Z t min e  t  e t  e  t  i 1y ht xi 1
 
(5) of faces, or the eigenvectors of the co-variance matrix of the set of
face images, treating an image as a point (or vector) in a very high
dimensional space. The eigenvectors are ordered, each one
Therefore the optimization of Zt+1 can be carried out in two stages: accounting for a different amount of the variation among the face
first, because of  t ' s assumed positivity, the weighted error using a images.These eigenvectors can be thought of as a set of features that
together characterize the variation between face images.
base learning algorithm, a decision stump for instance is minimized
n
t  min Z t1  Di t 1y i h xi 1,
h i 1 (6)
n
ht  arg min Z t1  Di t 1y i h xi 1,
h i 1 (7)
In case of multiple minimizers, take ht to be any of them. Next
choose

t 
1 1  t
2
ln  arg min e  e  e t .  
t  0 (8)
Hence t < 0.5 is necessary, which imposes a minimal condition on
the training set and the base learning algorithm. Also obtained is
Z t 1  2 Z t t 1  t   Z t ,
(9)
a recursive relation asserting the decreasing behavior of the
exponential risk. Weight change thus depends on whether an Fig.1. Detected face output of the preprocessed input image
observation is misclassified
Di t  1 Each image location contributes more or less to each eigenvector, so
wi t  1 
Z t 1 that the eigenvector as a sort of ghostly face which is called an
 y  t h t x i  eigenface [3] is displayed.
D t e i
 i
2 Z t t 1  t  This paper considers five basic expressions to recognize, i.e., happy,
sad, disgust, anger and neutral as shown in the Fig.3. which includes
wi t   1 1 following features [4]:
  1h t x i  y i  1h t x i  y 
2  1  t i
t  (10)
1. Happy: The expression usually involves a smile with both corner
The final boosted classifier is thus a weighted committee
of the mouth rising, the eyes are squinting and wrinkles appear at
T  eyes corners.
f   sign   t ht .
t 1  (11) 2. Sad: The expression involves a slight pulling down of lip corners
and inner side of eyebrows is rising.
The Fig.1. depicts the face detection approach. Post processing
procedure eliminates a number of false positives and beautifies the 3. Anger: The expression involves three main features i.e., teeth
detected result using a 31 layer cascade. revealing, eyebrows down and inner side tightening, squinting eyes.
Each face detected is cropped using the concept of computer vision
and the result is displayed in Fig.2. 4. Disgust: The expression involves wrinkled nose and mouth.
Sometimes even involves tongue coming out.

III. FACIAL EXPRESSION DETECTION APPROACH 5. Neutral: The expression is characterized by neutral positioning of
The information theory approach of coding and decoding face the facial features, implying a lack of strong emotion.
images may give insight into the information content of face images,
emphasizing the significant local and global “features." Such features
may or may not be directly related to the intuitive notion of face A. CALCULATING EIGEN FACES
features such as the eyes, nose, lips, and hair. The simplest and the efficient way of facial expression recognition
is the eigenface approach because the computational complexity of
eigenfaces is much lesser compared to any other method of the facial 1, ifl  k
expression recognition . U Tl U k   lk   (13)
0, otherwise
Let a face image l x, y  be a two-dimensional N by N array of The vectors u k , and scalar k are the eigenvectors and eigenvalues,
intensity values. An image may also be considered as a vector of respectively, of the covariance matrix
dimension N 2 ,so that a typical image of size 256 by 256 becomes a C
1 M
  n Tn (14)
vector of dimension 65,536, or, equivalently, a point in 65.536 M n1
dimensional space.
 AAT
An ensemble of images, then maps to a collection of points in this where the matrix A  1 2 .... M . The matrix C, however is
huge space. Images of faces, being similar in overall configuration, 2 2 2
N by N and determining the N eigenvectors and eigenvalues
will not be randomly distributed in this huge image space and thus is an intractable task for typical image sizes.
can be described by a relatively low dimensional subspace.
If the number of data points in the image space is less than the
The main idea of the principal component analysis [3] is to find the
vectors that best account for the distribution of face images within dimension of the space (M < N 2 ). there will be only M-1, rather
the entire image space. These vectors define the subspace of face than N 2 meaningful eigenvectors. Fortunately it can be solved for
2
images, which is called “facespace." Each vector is of length N , the N 2 dimensional eigenvectors in this case by first solving for the
describes an N by N image, and is a linear combination of the eigenvectors of an M by M matrix--e.g, solving a 16 X 16 matrix
original face images. Because these vectors are the eigenvectors of rather than a 16,584 X 16,384 matrix and then taking appropriate
the co-variance matrix corresponding to the original face images and linear combinations of the face images  i .
because they are face like in appearance, these are refered as
“eigenfaces.” An example of eigen faces is shown in fig.5. Consider the eigenvectors Vi, of AAT such that
AAT Vi   i Vi (15)
Premultiplying both sides by A.
AAT AVi   i AVi (16)
from which its seen that Avi, are the eigenvectors of C = AAT .

Following this analysis, the M by M matrix is constructed, L  AAT ,

where Lmn  Tm  n , and find the M eigenvectors, Vl, of L. These
vectors determine linear combinations of the M training set face
images to form the eigenfaces Ul.
M
U l   V lk  k , l  1,...., M (17)
k 1
With this analysis the calculations are greatly reduced, from the order
of the number of pixels in the images ( N 2 ) to the order of the
number of images in the training set (M). In practice, the training set
Fig.2.Cropped faces from the pre processed image of face images will be relatively small (M < N 2 ), and the
calculations become quite manageable. The associated eigenvalues
Let the training set of face images be 1, 2, 3, .........M . The allow to rank the eigenvectors according to their usefulness in
characterizing the variation among the images. Fig.5. shows
1 M eigenface derived from the input images of Fig.4.
average face of the set is defined by    n .Each face
M n 1
differs from the average by the vector  i  i  . An example
training set is shown in Fig.4(a), with the average face  shown in B. CLASSIFYING A FACE IMAGE
Fig.4(b). The eigenface images calculated from the eigenvectors of L, span a
basis set, with which to describe face images. In practice, a smaller
This set of very large vectors is then subject to principal component M' is sufficient for identification, since accurate reconstruction of the
analysis [2], which seeks a set of orthonormal vectors, u n , which image is not a requirement, In this framework, identification becomes
a pattern recognition task. The eigenfaces span an M’-dimensional
best describes the distribution of the data.
subspace of the original N 2 image space. The M’ significant
The k th vector, u k , is chosen such that
eigenvectors of the L matrix are chosen as those with the largest
k 
1 M T

 U n
M n1 k
2 (12)
associated eigenvalues.

is a maximum, subject to A new face image (  ) is transformed into its eigenface components
by a simple operation,
wk  U Tk     (18) (4) distant from face space and not near a known face class.
for k = 1, . . . .M’. This describes a set of point by point image In the first case, an individual is recognized and identified. In the
multiplications and summations, operations performed at second case, an unknown individual is present. The last two cases
approximately frame rate on current image processing hardware. indicate that the image is not a face image. Case three typically
Fig.6. shows an image and its projection into the face space. shows up as a false positive in most recognition systems, in this
framework, however, the false recognition may be detected because
of the significant distance between the image and the subspace of
expected face images [3].

Happy Sad Anger

Disgust Neutral
Fig.3. Different facial expressions

The weights form a vector T  1, 2 ....M '  that describes the (a)

contribution of each eigenface in representing the input face image

treating the eigenfaces as a basis set for face images. The vector may
then be used in a standard pattern recognition algorithm to find which
of a number of predefined face classes, if any, best describes the face.
The simplest method for determining which face class provides the
best description of an input face image is to find the face class k that
minimizes the Euclidian distance.
2k     k 
2
(19)
Where  k is a vector describing the k th face class. The face
classes  i are calculated by averaging the results of the eigenface
(b)
representation over a small number of face images of each individual.
A face is classified as belonging to class k when the minimum k is Fig.4(a).Face images used in database and
(b)Average face 
below some chosen threshold . Otherwise the face is classified as
“unknown,“ and optionally used to create a new face class.

Because creating the vector of weights is equivalent to projecting the

original face image onto the low dimensional face space, many
images will project onto a given pattern vector. This is not a problem
for the system, however, since the distance  between the image
and the face space is simply the squared distance between the mean
adjusted input image      and  f  iM1i ui , its projection
onto face space:
2
2     f (20)
Thus there are four possibilities for an input image and its pattern
vector:
(1) near face space and near a face class,
Fig.5. Calculated eigenface after lighting compensation and greyscale
(2) near face space but not near a known face class,
conversion
(3) distant from face space and near a face class, and
 IV. METHODOLOGY compared with the database images based on the features of
eigenfaces and the result is displayed in the text file. The expressions
The captured image is taken as input. The faces in the input image
corresponding database images are included in a text file with the
detected using Viola-Jones Algorithm [1] are cropped and stored in same label as the database images.
the folder. The cropped faces in the folder is considered to be the test
image. The folder of database images consisting the expressions to This paper is able to analyze each of the five facial expressions and
be recognized is loaded, followed by the test images. The various the success percentage of each expression is plotted in the Fig.8. The
expressions are included in the text file and is loaded along with the happy and sad faces are effectively recognized with 70% accuracy,
database images. The respective calculations are performed and the the angry faces are detected with 85% accuracy and the disgust and
image features are extracted using the concept of eigenfaces [3]. The neutral expressions are detected with 87.5% accuracy. The success
test images are compared with the database images and the facial percentage can be increased by including more images in the database.
expression is detected. The result is then stored in the text file More the images in the database higher is the accuracy of the facial
displaying the image label and the corresponding expression. The expression recognition.
flowchart in Fig.7 shows the execution flow.

Fig.6. Original face image and its projection onto the face space defined by
the eigenfaces of Fig.5

Fig.8. The success percentage of each of the five basic facial expression based
on eigen face approach

VI. CONCLUSIONS
In this paper, a simple approach to Facial Expression Recognition is
presented. The proposed system is able to perform human face
detection, feature point extraction and facial expression recognition
from the images. The eigenface approach is a practical solution to the
problem of face recognition. This algorithm is the better option based
on the less computational and time consuming complexity. It enables
to extract the precise facial features and is appropriate to real time
facial expression and emotion characterization task. It is fast,
relatively simple and has been shown to work well in a constrained
environment. This paper shows that the eigenface technique can be
made to perform at higher accuracy, although with a substantial
“unknown" rejection rate, and thus is potentially well suited to
various applications.

REFERENCES
Fig.7. Flowchart showing the execution flow [1] Yi-QingWang, “An Analysis of the Viola-Jones Face Detection
Algorithm.” Image Processing On Line, CMLA, ENS
Cachan, France , 2014.
V. SIMULATION RESULTS [2] Mu-Chun Su, Yi-Jwu Hsieh and De-Yuan Haung”A Simple
The Facial Expression Recognition [2] tests were performed using Approach to Facial Expression.” International Conference on
sequence of images in a database. A total of 40 images are included in Computer Engineering and Applications, Gold Coast, Australia,
the database. The subjects were instructed by an experimenter to Pg.456-461, 2007.
perform a series of facial display. The face images are resized into a [3] Matthew Turk and Alex Pentland, “Eigen Faces For
common size and the image sequences begin with happy face and Recognition.”Journal of cognitive Neuroscience, Volume
were digitized with grey scale. The faces in the input image, which are 3, number1, 2013.
cropped and stored in a folder, is taken as the test image and is [4] NitzanCooper, Ben-Gurion University of Negev “Facial
processed in a similar way as that of the database images. It is then Expression Recognition.”Project Report, 2013.