2nd INTERNATIONAL CONFERENCE ON ADVANCED COMPUTING AND SOFTWARE ENGINEERING (ICACSE-2019)

Sentiment Analysis by using Recurrent

Neural Network
Alpna Patel1 and Arvind Kumar Tiwari2

Abstract—Sentiment analysis is the process of emotion extraction and opinion mining from given text. This research paper gives the
detailed overview of different feature selection methods, sentiment classification techniques and deep learning approaches for
sentiment analysis. The feature selection methods include n-grams, stop words and negation handling. This paper also discusses about
various sentiment classification techniques named as machine learning based approach and lexicon based approach. There is various
classification algorithms such as SVM, Maximum Entropy and Naïve Bayes used for sentiment classification. In this paper we also
discuss about deep learning models such as RNN, CNN and LSTM which is used for sentiment analysis. There are various application
of sentiment analysis in decision making, prediction and business application.
Keywords: Sentiment Analysis, Deep Learning, Sentiment Classification, Machine Learning


I. INTRODUCTION a. Positive: I love natural beauty.

Sentiment analysis refers as opinion mining or subjectivity b. Negative: This is disgusting.

analysis for a given task [1]. Sentiment analysis is the c. Neutral: I usually go outside in the evening.
combination of emotions, opinion and sentiments towards
the given topic. It is the procedure of extracting opinion 4. Sentiment level: There are various level of sentiment
from particular task. Sentiment analysis is the powerful analysis such as document level, sentence level and
tool for specific event such as movies reviews and general phrase level.
elections [2]. It shows the user’s view for different II. METHODOLOGY
commercial sites such as Flipkart, Amazon, Myntra etc.
Sentiment analysis focuses on user’s likes and dislikes to We focus on opinion mining for classification task. The
the prediction of particular products review, box office main objective of classification is related to sentiment
performance and general elections. Sentiment is polarity analysis such as box office performance, analysis of blogs,
based analysis. Polarity can be further divided into three web texts and general elections. We extract most relevant
parts positive, negative and neutral. feature from sentences for sentiment analysis. We trained
The aim of sentiment analysis is naturally determine the different models by using different classification
public view whether it is positive, negative or neutral [3]. techniques and different set of features.
We can also apply sentiment analysis on commercial Feature Selection is a primary task to extract the feature
websites, audios, videos, social networking sites and from sentences for sentiment classification. In order to
images [4]. select the feature, feature vector can be divided into two
Now a day’s opinion extraction has become a important categories such binary feature and numerical feature
field of research because generally people are usually which represents the frequency occurrences. Here the
show their opinion on social networking sites and following texts feature:
commercial sites. There are some important notations  N- grams: It can be defined as n term in texts. If
related to sentiment analysis have given below: one can take one (unigram) or more than one word
1. Subjective: Gold is awesome movie.1 (bigram) at a time accordingly.
2. Objective: Reema Kagti is the director of Gold.  Stop words: In article ‘a’ and ‘the’, in preposition
‘in’, ‘near’, ‘beside’ and in pronouns ‘he’, ‘she’,
3. Polarity: Polarity can be divided into three parts ‘it’ are stop words.
on the basis of opinion.
 POS tagging: It is related to parts of speech such as
noun, pronoun, adjective, adverb etc. In a sentence
1,2Department
adverb and objective holds the maximum
of Computer Science & Engineering sentiments.
Kamla Nehru Institute of Technology
Sultanpur, India  Stemming: It is defined as the procedure of wrap
E-mail: [email protected] out the suffixes and prefixes from text. For a
[email protected] example: ‘writing’, ‘written’ can be ‘write’.

Electronic copy available at: https://ssrn.com/abstract=3349572

Sentiment Analysis by using Recurrent Neural Network 109

 Discourse Feature: A sentence may contain B. Machine Learning Based Approach

multiple sentiments which can be represented using
discourse based connectors. There are generally two type of approach such as
supervised learning method and unsupervised learning
 Negation: It is related to negation word ‘not’. For approach used to train and test datasets. Supervised
an example: Dosa is not good. In this sentence learning method uses well labeled data to train the
‘good’ is the positive word but ‘not’ inverts the machine. Each class contains different kind of features
meaning of whole sentence. which labeled with it. Supervised learning method further
divided into four classification methods named as
III. TECHNIQUES OF SENTIMENT CLASSIFICATION
Probabilistic classifier, Decision tree classifier, Linear
Sentiment Analysis classifier, Rule based classifier. There are different kinds
of algorithms such as such as SVM [11], Neural Network
[10], Maximum Entropy [9], and Naïve bayes [8] are used
for classification.
Machine Learning Approach Lexicon Based
Approach
IV. DEEP LEARNING BASED MODELS
A. Recurrent Neural Network
Supervised Unsupervised Corpous Based Dictionary
Learning Learning Based RNN is one of the deep learning approaches which are
used for sentiment analysis. It produces the output on the
Probabilistic Bayesian basis of previous computation by using sequential
Classifier Network Statistical Semantic
information. Previously, traditional neural network uses
independent inputs which are unfit for some task in
Rule Based Naïve Bayes Natural Language Processing. For an example: word
Classifier
prediction in a given sentence. RNN is efficient model for
Maximum
sentiment analysis. RNN uses memory cell that capable to
Decision Tree Entropy capture information about long sequences, shown in fig. 2.
Classifier

SVM

Linear
Classifier
Neural
Network
Figure. 1: Techniques of sentiment classification Figure. 2: RNN framework
The given equation shows the basic formula for RNN:
A. Lexicon Based Approach = (ℎ , ) (1)
The objective of lexicon based approach is to find the Where represents the output from previous node,
opinion lexicon from given text. It is used for calculating activation function f is tanh function and denotes the
sentiments. It focuses on counting the number of positive input sequences ( , , , … … , ). Recursive neural
words and negative words in a given sentence. If the networks represent the deep tree structure which is able to
sentence gives more positive opinion then the sentence grab the semantic of texts. This process is time consuming
assigned to a positive score and if the sentence contains task that is the disadvantage of Recursive neural network
more negative words then it assigned to a negative score. [22]. Recurrent neural network has improved the time
If the sentence represents the equal number of positive complexity. In RNN model, analysis is done word by
words and negative words then it is assigned to a neutral word and it is time taking process [12].
score. Opinion lexicon is a tool for it [5]. There are two It is able to learn recent words compare to earlier words
type of lexicon based approach that leads the vanishing gradient problem. RNN uses
1. Dictionary based approach: It is the collection of LSTM to solve the problem of vanishing gradient.
opinion words which are collected manually.
Opinion words help to form seed list. When we B. LSTM
found new words (synonyms & antonyms) during LSTM (Long Short Term Memory) is the part of RNN
sentiment classification, then it is added to seed which is used to learn long-range dependencies for text
list.
sequences. LSTM contains memory blocks which also
2. Corpus based approach: It is the collection of known as gates to control the text flow. The memory
large amount of words based on specific topic. It blocks contain three gates named as input gate; forget gate
helps to expand seed list [6]. It uses two basic and output gate to control the flow of information [13],
approach named as statistical and semantic [7]. which is shown in fig. 3.

Electronic copy available at: https://ssrn.com/abstract=3349572

110 2nd International Conference on Advanced Computing and Software Engineering (ICACSE-2019)

Neural Network. RNN model is used to evaluate the

performance.
To achieve the sentiment analysis, split the dataset
into two label; label 0 for negative review and label 1 for
positive review. Deep learning methods are used to train
data. After training the data we able to identify valid data
for test classifier. The shape of train data is 15000, valid
data shape 10000 and test data shape 25000.
In order to find unique word, NLTK uses different
algorithm to remove punctuation, stop words and spaces
from texts. The Sklearn uses different techniques such
text, tfidfvectorizor for feature extraction by using data
preprocessing techniques. The NLTK use porter stemmer
and snowball stemmer algorithms for stemming the words.
Figure 3: Long short term memory
The word ‘disappointed’ stemmed as disappoint.
In order to get unique word, the NLTK used different
C. Convolutional Neural Network feature selection methods such as stop word removing,
negation handling, POS tagging and stemming.
CNN is one of the deep learning models that were used in
The given graph shows the model training evaluation
classification of sentence as opinion extraction and text
on dataset:
tokenization [14]. CNN perform very well for Natural
Language Processing. It preserves 2-D spatial orientation
in texts, computer vision and images. CNN uses three
layers such as convolutional layer, pooling layer and fully
connected layer to perform the task.
D. Word Embedding
Deep learning models need word embedding as a input
variable in NLP [21]. Word embedding is the process
language modeling that converts texts into real numbers
(e.g., word “fat”-(…., 0.18…0.26….0.35...)).In word Figure 4: Model training evolution
embedding technique, each word is represented using
A. Performance Measure
dimension. Word2Vec is the prediction model which
computes word embeddings from sentences. It includes There are several parameters that is used to measure the
CBOW (Continuous Bag of words) model and Skip- Gram performance. It is measured by using confusion matrix.
model for feature learning. Confusion matrix also known as error matrix. It includes
TP (true positive), FN (false negative), TN (true negative),
V. DATASET DESCRIPTION and FP (false positive) on test data.
To evaluate the accuracy and error analysis of our model,
Table 1: Confusion matrix
we used IMDB movie reviews dataset for sentiment
Label 1 Label 2
analysis [23]. This dataset contains movie reviews along (Predicted) (Predicted)
with their associated binary sentiment polarity labels. It is Label 1 TN FP
intended to serve as a benchmark for sentiment (Actual)
classification. The core dataset contains 50,000 reviews Label 2 FN TP
split evenly into 25k train and 25k test sets. The overall (Actual)
distribution of labels is balanced (25k pos and 25k neg). With the help of these parameters, we calculate
We also include an additional 50,000 unlabeled accuracy, recall, specificity etc.
documents for unsupervised learning. We used 20% of  Accuracy: It is used for evaluating the performance
labeled training sets as validation set. of models by using binary classification. It is
calculated by confusion matrix.
VI. RESULT AND DISCUSSION
Accuracy=
In this paper to perform sentiment analysis, we used
 Recall: It is also known as sensitivity and it is
IMDB movie review dataset. Anaconda open source tool
calculated by using formula:
is used for python language to perform machine learning
task. This paper used jupyter notebook for deep learning Recall =
and Keras sequential model to implement Recurrent

Electronic copy available at: https://ssrn.com/abstract=3349572

Sentiment Analysis by using Recurrent Neural Network 111

 Specificity: It is calculated by using confusion [2] B. Heredia, T. M. Khoshgoftaar, J. Prusa, and M. Crawford, Cross-
matrix. Domain Sentiment Analysis: An Empirical Investigation, 2016
IEEE 17th Int. Conf. Inf. Reuse Integr., pp. 160165, 2016.
[3] F. Luo, C. Li, and Z. Cao, Affective-feature-based sentiment
Specificity= analysis using SVM classifier, 2016 IEEE 20th Int. Conf. Comput.
 Precision: It is also known as positive predictive Support. Coop. Work Des., pp. 276281, 2016.
values. It is calculated by using formula: [4] Medhat, Walaa, Ahmed Hassan, and Hoda Korashy. “Sentiment
analysis algorithms and applications: A survey” Ain Shams
Engineering Journal 5.4:1093-1113, 2014.
Precision= [5] Kang Hanhoon, Yoo Seong Joon, Han Dongil., “Senti-lexicon and
 F-measure: It gives the measure on test’s accuracy improved Naı¨ve Bayes algorithms for sentiment analysis of
that is used both precision and recall. F-measure or restaurant reviews”, Expert Syst Appl ,39:6000–10, 2012.
[6] Keshtkar Fazel, Inkpen Diana., “A bootstraping method for
F score calculated by using formula: extracting paraphrases of emotion expressions from texts” Comput
∗ ∗ Intell;vol. 0, 2012.
F-measure= [7] Medhat, Walaa, Ahmed Hassan, and Hoda Korashy. “Sentiment
analysis algorithms and applications: A survey” Ain Shams
Engineering Journal 5.4 :1093-1113, 2014.
 MCC: MCC stands for mathew correlation [8] Kang Hanhoon, Yoo Seong Joon, Han Dongil., “Senti-lexicon and
coefficient that is used for binary classification. It improved Naı¨ve Bayes algorithms for sentiment analysis of
varies between ranges from -1 to +1. It can be restaurant reviews”, Expert Syst Appl ,39:6000–10, 2012.
calculated by using given formula: [9] Duric Adnan, Song Fei., “Feature selection for sentiment analysis
based on content and syntax models”, Decis Support Syst,53:704–
∗ ∗ 11, 2012.
MCC=[( )∗( )∗( )∗( )] / [10] Moraes Rodrigo, Valiati Joa˜o Francisco, Gavia˜o Neto Wilson
P.,“Document-level sentiment classification: an empirical
The result achieved from test data is given in table II: comparison between SVM and ANN”, Expert Syst Appl ,40:621–
33, 2013.
Table II: Evaluation of classifiers
[11] Rui Huaxia, Liu Yizao, Whinston Andrew., “Whose and what
chatter matters? The effect of tweets on movie sales”, Decis
F-measure
Classifiers

Specificity

Support Syst 2013.

Accuracy

Precision

[12] Socher, R., et al. Parsing natural scenes and natural language with
Recall

recursive neural networks. in Proceedings of the 28th international

MCC

conference on machine learning (ICML-Il). 2011.

[13] Senior, A. (n.d.). Long Short-Term Memory Recurrent Neural
RNN 0.8742 0.8717 0.8765 0.8753 0.8734 0.7484 Network Architectures for Large Scale Acoustic Modeling Has.
http://doi.org/arXiv:1402.1128
In this paper, RNN model gives the accuracy 87.42% [14] Senior, A. (n.d.). Long Short-Term Memory Recurrent Neural
on IMDB movie review dataset. In order to evaluate the Network Architectures for Large Scale Acoustic Modeling Has.
http://doi.org/arXiv:1402.1128
performance, we achieved; recall 87.17%, specificity [15] Kaur, H., Mangat, V., & Nidhi. (2017). A survey of sentiment
87.65%, precision 87.53%, f- measure 87.34% and MCC analysis techniques. Proceedings of the International Conference on
74.14% value. IoT in Social, Mobile, Analytics and Cloud, I-SMAC 2017, 921–
925. https://doi.org/10.1109/I-SMAC.2017.8058315
VII. CONCLUSION [16] Yelena, M. (2009). Sentiment Analysis: An Overview
Comprehensive Exam Paper. Computer Science Department,
Sentiment analysis is the process of extracting sentiments, (May).
attitude, emotions and opinion in given sentence. This [17] Boiy, E., & Moens, M. F. (2009). A machine learning approach to
sentiment analysis in multilingual web texts. Information Retrieval,
research paper covered different sentiment classification 12(5), 526–558. https://doi.org/10.1007/s10791-008-9070-z
techniques. Sentiment classification uses two approaches [18] Hassan, A., & Mahmood, A. (2017). Deep Learning approach for
such as machine learning based approach and lexicon sentiment analysis of short texts. 2017 3rd International Conference
based approach for classification. Machine learning based on Control, Automation and Robotics, ICCAR 2017, 705–710.
https://doi.org/10.1109/ICCAR.2017.7942788
approach is more suitable than lexicon based approach. [19] Yin, W., Kann, K., Yu, M., & Schütze, H. (2017). Comparative
This survey paper concludes various deep learning models Study of CNN and RNN for Natural Language Processing, (July).
for sentiment analysis. CNN and RNN are most popular https://doi.org/10.14569/IJACSA.2017.080657
deep learning algorithms for opinion mining. Recurrent [20] Zvarevashe, K., & Olugbara, O. O. (2018). A framework for
sentiment analysis with opinion mining of hotel reviews. 2018
Neural Network model produces the output on the basis of Conference on Information Communications Technology and
previous computation by using sequential information. Society, ICTAS 2018 - Proceedings, 1–4.
This paper concludes that RNN gives the accuracy 87.42% https://doi.org/10.1109/ICTAS.2018.8368746
on movie review dataset. Presently, sentiment analysis is [21] Collobert R, Weston J, Bottou L, Karlen M, Kavukcuoglu K, and
Kuksa P. Natural language processing (almost) from scratch.
the popular field of research. Journal of Machine Learning Research, 2011.
[22] Socher, R., et al. Parsing natural scenes and natural language with
REFERENCES recursive neural networks. in Proceedings of the 28th international
[1] Pang B, Lee L. , “Opinion mining and sentiment analysis” conference on machine learning (ICML-Il). 2011.
FoundTrends Inform Retriev:1–135, 2008. [23] https://www.kaggle.com/desiredewaele/sentiment-analysis-on-
imdb-reviews/data

Electronic copy available at: https://ssrn.com/abstract=3349572