Deep learning for nlp
4 likes2,045 views
This document provides an overview of deep learning techniques for natural language processing (NLP). It discusses some of the challenges in language understanding like ambiguity and productivity. It then covers traditional ML approaches to NLP problems and how deep learning improves on these approaches. Some key deep learning techniques discussed include word embeddings, recursive neural networks, and language models. Word embeddings allow words with similar meanings to have similar vector representations, improving tasks like sentiment analysis. Recursive neural networks can model hierarchical structures like sentences. Language models assign probabilities to word sequences.
![Deep Learning: Why for NLP ?
• Beat state of the art
– Language Modeling (Mikolov et al. 2011) [WSJ AR task]
– Speech Recognition (Dahl et al. 2012, Seide et al 2011;
following Mohammed et al. 2011)
– Sentiment Classification (Socher et al. 2011)
– MNIST hand-written digit recognition (Ciresan et al.
2010)
– Image Recognition (Krizhevsky et al. 2012) [ImageNet]
5](https://image.slidesharecdn.com/deeplearningfornlp-151008043922-lva1-app6892/85/Deep-learning-for-nlp-5-320.jpg)
![Language semantics
• What is the meaning of a word?
(Lexical semantics)
• What is the meaning of a sentence?
([Compositional] semantics)
• What is the meaning of a longer piece of
text?
(Discourse semantics)
6](https://image.slidesharecdn.com/deeplearningfornlp-151008043922-lva1-app6892/85/Deep-learning-for-nlp-6-320.jpg)
![One-hot encoding
• The one-hot vector of an ID is a vector filled with 0s, except
for a 1at the position associated with the ID
– for vocabulary size D=10, the one-hot vector of word ID w=4 is e(w)
= [ 0 0 0 1 0 0 0 0 0 0 ]
• A one-hot encoding makes no assumption about word
similarity
• All words are equally different from each other
8](https://image.slidesharecdn.com/deeplearningfornlp-151008043922-lva1-app6892/85/Deep-learning-for-nlp-8-320.jpg)
![N-gram models
• An n-gram is a sequence of n words
– unigrams(n=1):’‘is’’,‘‘a’’,‘‘sequence’’,etc.
– bigrams(n=2): [‘‘is’’,‘‘a’’], [‘’a’’,‘‘sequence’’],etc.
– trigrams(n=3): [‘’is’’,‘‘a’’,‘‘sequence’’],
[‘‘a’’,‘‘sequence’’,‘‘of’’],etc.
• n-gram models estimate the conditional from n-
grams counts
• The counts are obtained from a training corpus (a
dataset of word text)
63](https://image.slidesharecdn.com/deeplearningfornlp-151008043922-lva1-app6892/85/Deep-learning-for-nlp-63-320.jpg)
More Related Content
Similar to Deep learning for nlp (20)
Deep learning for nlp
- 1. Deep learning for Natural language processing Viet-Trung Tran 1
- 2. Some of the challenges in Language Understanding • Language is ambiguous: – Every sentence has many possible interpretations. • Language is productive: – We will always encounter new words or new constructions • Language is culturally specific 2 fruit flies like a banana NN NN VB DT NN NN VB P DT NN NN NN P DT NN NN VB VB DT NN
- 3. ML: Traditional Approach • For each new problem/question – Gather as much LABELED data as you can get – Throw some algorithms at it (mainly put in an SVM and keep it at that) – If you actually have tried more algos: Pick the best – Spend hours hand engineering some features / feature selection / dimensionality reduction (PCA, SVD, etc) – Repeat… 3
- 4. Deep learning vs the rest 4
- 5. Deep Learning: Why for NLP ? • Beat state of the art – Language Modeling (Mikolov et al. 2011) [WSJ AR task] – Speech Recognition (Dahl et al. 2012, Seide et al 2011; following Mohammed et al. 2011) – Sentiment Classification (Socher et al. 2011) – MNIST hand-written digit recognition (Ciresan et al. 2010) – Image Recognition (Krizhevsky et al. 2012) [ImageNet] 5
- 6. Language semantics • What is the meaning of a word? (Lexical semantics) • What is the meaning of a sentence? ([Compositional] semantics) • What is the meaning of a longer piece of text? (Discourse semantics) 6
- 7. One-hot encoding • Form vocabulary of words that maps lemmatized words to a unique ID (position of word in vocabulary) • Typical vocabulary sizes will vary between 10 000 and 250 000 7
- 8. One-hot encoding • The one-hot vector of an ID is a vector filled with 0s, except for a 1at the position associated with the ID – for vocabulary size D=10, the one-hot vector of word ID w=4 is e(w) = [ 0 0 0 1 0 0 0 0 0 0 ] • A one-hot encoding makes no assumption about word similarity • All words are equally different from each other 8
- 9. Word representation • Standard – Bag of Words – A one-hot encoding – 20k to 50k dimensions – Can be improved by factoring in document frequency • Word embedding – Neural Word embeddings – Uses a vector space that attempts to predict a word given a context window – 200-400 dimensions Word embeddings make seman0c similarity and synonyms possible 9
- 10. Distributional representations • “You shall know a word by the company it keeps” (J. R. Firth 1957) • One of the most successful ideas of modern • statistical NLP! 10
- 11. • Word Embeddings (Bengio et al, 2001; Bengio et al, 2003) based on idea of distributed representations for symbols (Hinton 1986) • Neural Word embeddings (Mnih and Hinton 2007, Collobert & Weston 2008, Turian et al 2010; Collobert et al. 2011, Mikolov et al.2011) 11
- 12. Neural distributional representations • Neural word embeddings • Combine vector space semantics with the prediction of probabilistic models • Words are represented as a dense vector Human = 12
- 14. Word embeddings Turian, J., Ra0nov, L., Bengio, Y. (2010). Word representa0ons: A simple and general method for semi-‐supervised learning 14
- 15. 15
- 16. • What words have embeddings closest to a given word? From Collobert et al. (2011) 16
- 17. Word Embeddings for MT: Mikolov (2013) 17
- 18. Word Embeddings • one of the most exciting area of research in deep learning • introduced by Bengio, et al. 2013 • W:words→Rn is a paramaterized function mapping words in some language to high-dimensional vectors (200 to 500). – W(‘‘cat")=(0.2, -0.4, 0.7, ...) – W(‘‘mat")=(0.0, 0.6, -0.1, ...) • Typically, the function is a lookup table, parameterized by a matrix, θ, with a row for each word: Wθ(wn)=θn • W is initialized as random vectors for each word. • Word embedding learns to have meaningful vectors to perform some task. 18
- 19. Learning word vectors (Collobert et al. JMLR 2011) • Idea: A word and its context is a positive training example, a random word in the same context give a negative training example 19
- 20. Example • Train a network for is predicting whether a 5- gram (sequence of five words) is ‘valid.’ • Source – any text corpus (wikipedia) • Break half number of 5-grams to get negative training examples – Make 5-gram nonsensical – "cat sat song the mat” 20
- 21. Neural network to determine if a 5-gram is 'valid' (Bottou 2011) • Look up each word in the 5-gram through W • Feed those into R network • R tries to predict if the 5-gram is 'valid' or 'invalid' – R(W(‘‘cat"), W(‘‘sat"), W(‘‘on"), W(‘‘the"), W(‘‘mat"))= 1 – R(W(‘‘cat"), W(‘‘sat"), W(‘‘song"), W(‘‘the"), W(‘‘mat"))=0 • The network needs to learn good parameters for both W and R. 21
- 22. 22
- 23. Idea • “a few people sing well” → “a couple people sing well” • the validity of the sentence doesn’t change • if W maps synonyms (like “few” and “couple”) close together – R’s perspective little changes. 23
- 24. Bingo • The number of possible 5-grams is massive • But, small number of data points to learn from • Similar class of words – “the wall is blue” → “the wall is red” • Multiple words – “the wall is blue” → “the ceiling is red” • Shifting “red” closer to “blue” makes the network R perform better. 24
- 25. Word embedding property • Analogies between words encoded in the difference vectors between words. – W(‘‘woman")−W(‘‘man") ≃ W(‘‘aunt")−W(‘‘uncle") – W(‘‘woman")−W(‘‘man") ≃ W(‘‘queen")−W(‘‘king") 25
- 26. Linguis0c Regulari0es: Mikolov (2013) 26
- 27. Word embedding property: Shared representations • The use of word representations… has become a key “secret sauce” for the success of many NLP systems in recent years, across tasks including named entity recognition, part-of-speech tagging, parsing, and semantic role labeling. (Luong et al. (2013)) 27
- 28. • W and F learn to perform task A. Later, G can learn to perform B based on W 28
- 30. English – Chinese word mapping 30
- 31. Embed images and words in a single representation 31
- 32. Feedforward neural net language model (NNLM) Belgio et. al., 2003 • Long training time 32
- 33. Recurrent neural network based language model (Mikolov et. al., 2010) • Elman Network 33
- 34. Simple RNN training • Input vector: 1-of-N encoding (one hot) • Repeated epoch – S(0): vector of small value (0,1) – Hidden layer: 30 – 500 units – All training data from corpus are sequentially presented – Init learning rate: 0.1 – Error function – Standard backpropagation with stochastic gradient descent • Conversion achieved after 10 – 20 epochs 34
- 35. Word2vec (Mikolov et. al., 2013) • Log-linear model • Previous models: non-linear hidden layer -> complexity • Continuous word vectors are learned using simple model 35
- 36. Continuous BoW (CBOW) Model • Similar to the feed-forward NNLM, but – Non-linear hidden layer removed • Called CBOW (continuous BoW) because the order of the words is lost
- 37. CBOW Model
- 38. Continuous Skip-gram Model • Similar to CBOW, but – Tries to maximize classification of a word based on another word in the same sentence • Predicts words within a certain window • Observations – Larger window size => better quality of the resulting word vectors, higher training time – More distant words are usually less related to the current word than those close to it – Give less weight to the distant words by sampling less from those words in the training examples
- 41. Modular Network that learns word embeddings • Fixed number of inputs 41
- 42. Recursive neural networks • Output of a module go into a module of the same type • tree-structured neural networks • No fixed number of inputs 42
- 43. Building on Word Vector Space Models • But how can we represent the meaning of longer phrases? • By mapping them into the same vector space! 43
- 44. How should we map phrases into a vector space? 44
- 45. Sentence Parsing: What we want 45
- 46. Learn Structure and Representation 46
- 47. Recursive Neural Networks for Structure Prediction 47
- 48. Recursive Neural Network Definition 48
- 49. Recursive Application of Relational Operators 49
- 50. Parsing a sentence with an RNN 50
- 54. Labeling in Recursive Neural Networks 54
- 56. Recursive neural tensor network 56
- 57. Socher et al. 2013: Sentence sentiment analysis 57
- 59. Reversible sentence representation BoNou 2011 • Bilingual sentence representation 59
- 60. Cho et al. (2014) 60
- 61. Credits • Richard Socher, Christopher – Stanford University – nlp.stanford.edu/courses/NAACL2013/ • Roelof Pieters, PhD candidate KTH/CSC • http://colah.github.io/ • Bengio GSS 2012 61
- 62. Language Modeling • A language model is a probabilistic model that assigns probabilities to any sequence of words p(w1, ... ,wT) • Language modeling is the task of learning a language model that assigns high probabilities to well formed sentences • Plays a crucial role in speech recognition and machine translation systems 62
- 63. N-gram models • An n-gram is a sequence of n words – unigrams(n=1):’‘is’’,‘‘a’’,‘‘sequence’’,etc. – bigrams(n=2): [‘‘is’’,‘‘a’’], [‘’a’’,‘‘sequence’’],etc. – trigrams(n=3): [‘’is’’,‘‘a’’,‘‘sequence’’], [‘‘a’’,‘‘sequence’’,‘‘of’’],etc. • n-gram models estimate the conditional from n- grams counts • The counts are obtained from a training corpus (a dataset of word text) 63