Neural Network Language
Models and word2vec
Tambet Matiisen
8.10.2014
Sources
• Yoshua Bengio. Neural net language models.
• Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.
Efficient Estimation of Word Representations in Vector Space.
• Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean.
Distributed Representations of Words and Phrases and their
Compositionality.
• Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig.
Linguistic Regularities in Continuous Space Word Representations.
• Tomas Mikolov, Quoc V. Le and Ilya Sutskever.
Exploiting Similarities among Languages for Machine Translation.
Language models
• A language model captures the statistical
characteristics of sequences of words in a
natural language, typically allowing one to
make probabilistic predictions of the next
word given preceding ones.
• E.g. the standard “trigram” method:
count ( wt 2 wt 1wt )
P( wt | wt 2 , wt 1 )
count ( wt 2 wt 1 )
Neural network language models
• A neural network language model is a language
model based on neural networks, exploiting their
ability to learn distributed representations.
• A distributed representation of a word is a vector
of activations of neurons (real values) which
characterizes the meaning of the word.
• A distributed representation is opposed to a local
representation, in which only one neuron (or very
few) is active at each time.
NNLM architecture
Softmax output layer
V nodes
(one unit per next word)
HxV weights
Hidden layer to predict output from features of the input words H nodes
HxD weights HxD weights
Learned distributed Learned distributed
D nodes D nodes
representation of word t-2 representation of word t-1
VxD weights (shared)
Sparse representation of Sparse representation of
V nodes V nodes
word t-2 word t-1
word2vec
• An efficient implementation of the continuous
bag-of-words and skip-gram architectures for
computing vector representations of words.
• The word vectors can be used to significantly
improve and simplify many NLP applications.
CBOW architecture
sparse representation
weights = distributed representation
NB! Shared!
softmax
Predicts current word given the context.
Skip-gram architecture
softmax
output weights
softmax
sparse representation
softmax
weights = distributed representation softmax
Predicts the surrounding words given the current word
Linguistic regularities
The word vector space implicitly encodes many regularities
among words, i.e. vector(KINGS) – vector(KING) +
vector(QUEEN) is close to vector(QUEENS)
Semantic-Syntactic Word Relationship
test set
Accuracy
days
minutes
hours
From words to phrases
• Find words that appear frequently together and
infrequently in other contexts.
count ( wi w j )
score( wi , w j )
count ( wi ) count ( w j )
• The bigrams with score above the chosen
threshold are then used as phrases.
• The δ is used as a discounting coefficient and
prevents too many phrases consisting of very
infrequent words to be formed.
Examples - analogy
Examples – distance (rare words)
Examples – addition
Parameters
• Architecture: skip-gram (slower, better for infrequent
words) vs CBOW (fast)
• The training algorithm: hierarchical softmax (better for
infrequent words) vs negative sampling (better for
frequent words, better with low dimensional vectors)
• Sub-sampling of frequent words: can improve both
accuracy and speed for large data sets (useful values
are in range 1e-3 to 1e-5)
• Dimensionality of the word vectors: usually more is
better, but not always
• Context (window) size: for skip-gram usually around
10, for CBOW around 5
Machine translation
using distributed representations
1. Build monolingual models of languages using
large amounts of text.
2. Use a small bilingual dictionary to learn a
linear projection between the languages.
3. Translate a word by projecting its vector
representation from the source language
space to the target language space.
4. Output the most similar word vector from
target language space as the translation.
English vs Spanish
Translation accuracy
English Spanish English Vietnamese
How is this related to neuroscience?
How to calculate similarity matrix
import sys
import gensim
if len([Link]) < 2:
print "Usage: [Link] <vectorfile> <wordfile>"
[Link](1)
model = [Link].Word2Vec.load_word2vec_format([Link][1],
binary=True)
with open([Link][2]) as f:
words = [Link]().splitlines()
for w1 in words:
s = ""
for w2 in words:
if s != "": s += ","
s += str([Link](w1, w2))
print s
Discovery of structural form - animals
Discovery of structural form - cities
