The Poisson Distribution Mixture Models Expectation-Maximization Wrap-up

Poisson Mixture Models

Brandon Malone
Much of this material is adapted from Bilmes 1998 and Tomasi 2004.
Many of the images were taken from the Internet

February 20, 2014

Brandon Malone Poisson Mixture Models

The Poisson Distribution Mixture Models Expectation-Maximization Wrap-up

Poisson Mixture Models

Suppose we have a dataset D which consists of DNA sequences observed

from a mixture of k bacteria. We do not know which sequence belongs
to which species.

Sequence Species Count

CAGAGGAT ? 5
TCAGTGTC ? 13
CTCTGTGA ? 2
AACTGTCG ? 7
CGCGTGGA ? 15
GGATGAGA ? 1

Which DNA sequences belong to the same species?

Brandon Malone Poisson Mixture Models

The Poisson Distribution Mixture Models Expectation-Maximization Wrap-up

Poisson Mixture Models

Suppose we have a dataset D which consists of DNA sequences observed

from a mixture of k bacteria. We do not know which sequence belongs
to which species.

Sequence Species Count

CAGAGGAT ? 5
λk P
TCAGTGTC ? 13 K
CTCTGTGA ? 2
AACTGTCG ? 7
⇒
CGCGTGGA ? 15
GGATGAGA ? 1 Dl zl
M
Which DNA sequences belong to the same species?

This can be described by a Poisson mixture model.

Brandon Malone Poisson Mixture Models

The Poisson Distribution Mixture Models Expectation-Maximization Wrap-up

1 The Poisson Distribution

2 Mixture Models

3 Expectation-Maximization

4 Wrap-up

Brandon Malone Poisson Mixture Models

The Poisson Distribution Mixture Models Expectation-Maximization Wrap-up

Multiple Bernoulli trials

Suppose we have a Bernoulli-distributed variable (a weighted coin

flip with parameter θ).

If we flip two coins, what is our probability of seeing exactly one H?

Brandon Malone Poisson Mixture Models

The Poisson Distribution Mixture Models Expectation-Maximization Wrap-up

Multiple Bernoulli trials

Suppose we have a Bernoulli-distributed variable (a weighted coin

flip with parameter θ).

If we flip two coins, what is our probability of seeing exactly one H?

C1 C2 P(C1 , C2 )
H H θ·θ
H T θ · (1 − θ)
T H (1 − θ) · θ
T T (1 − θ) · (1 − θ)

So, P(exactly one H) = 2 · θ · (1 − θ).

Brandon Malone Poisson Mixture Models

The Poisson Distribution Mixture Models Expectation-Maximization Wrap-up

Multiple Bernoulli trials

Suppose we have a Bernoulli-distributed variable (a weighted coin

flip with parameter θ).

If we flip two coins, what is our probability of seeing exactly one H?

C1 C2 P(C1 , C2 )
H H θ·θ
H T θ · (1 − θ)
T H (1 − θ) · θ
T T (1 − θ) · (1 − θ)

So, P(exactly one H) = 2 · θ · (1 − θ).

n


In general, P(exactly m successes in n trials) = m · θm · (1 − θ)n−m .

Brandon Malone Poisson Mixture Models

The Poisson Distribution Mixture Models Expectation-Maximization Wrap-up

Take it, to the limit, one more time

What if we have an infinite number of trials and expect to see λ

successes?
λm
lim P(exactly m successes in n trials) = exp {−λ}
n→∞ m!
This is called the Poisson distribution.

We will write g (m : λ) to mean P(exactly m successes given λ).

(See the videos for a detailed derivation.)

Brandon Malone Poisson Mixture Models

The Poisson Distribution Mixture Models Expectation-Maximization Wrap-up

Mixtures of distributions
Suppose we have K Poisson distributions (components) with
parameters λ1 . . . λK mixed together with proportions p1 . . . pK .
We often write P = {p1 . . . pK } and θ = {λ1 . . . λK , P}.

procedure GenerateDataset(Poisson parameters λ1 . . . λk , mixing proportions

)
p1 . . . pk , samples N

D←∅
for l = 1 to N do
component zl ←sample(Mult(p1 . . . pK ))
observation Dl ←sample(Poisson(λzl ))
D ← D ∪ Dl
end for
return D
end procedure
Brandon Malone Poisson Mixture Models
The Poisson Distribution Mixture Models Expectation-Maximization Wrap-up

Mixtures of distributions
Suppose we have K Poisson distributions (components) with
parameters λ1 . . . λK mixed together with proportions p1 . . . pK .
We often write P = {p1 . . . pK } and θ = {λ1 . . . λK , P}.

λk P
K

Dl zl
M

Figure: Generative model for a Poisson mixture model (PMM)

Brandon Malone Poisson Mixture Models

The Poisson Distribution Mixture Models Expectation-Maximization Wrap-up

Likelihood of data
We can write the (log) probability of any mixture model as follows.

K
X
P(D : θ) = pk g (D : λk )
k
N X
Y K
P(D : θ) = pk g (Dl : λk )
l k
N X
Y K
`(D : θ) = log pk g (Dl : λk )
l k
N
X K
X
`(D : θ) = log pk g (Dl : λk )
l k

The learning problem can be formulated as follows.

θ∗ = arg max `(D : θ)

θ

Brandon Malone Poisson Mixture Models

The Poisson Distribution Mixture Models Expectation-Maximization Wrap-up

Membership probabilities

Notation

q(k, l) ..= pk g (Dl : λk ) joint probability of Dl and component k

P(k|l) ..= P(zl = k|Dl ) conditional probability of component k given Dl

The probability that Dl came from comonent k is expressed as

follows.
q(k, l)
P(k|l) = PK
m q(m, l)

Also, we know each observation came from some component.

X
P(k|l) = 1
k

Brandon Malone Poisson Mixture Models

The Poisson Distribution Mixture Models Expectation-Maximization Wrap-up

Jensen’s Inequality
Recall the likelihood of the mixture model.
N
X K
X
`(D : θ) = log q(k, l)
l k

Jensen’s inequality shows the following.

K
X K
X
log πk αk ≥ πk log αk when π is a distribution
k k

We can make this work for any values.

K K K K
X X πk X ck X ck
log ck = log ck = log πk ≥ πk log
πk πk πk
k k k k

Brandon Malone Poisson Mixture Models

The Poisson Distribution Mixture Models Expectation-Maximization Wrap-up

Expectation-Maximization (EM)

Our learning problem is formulated as follows.

θ∗ = arg max `(D : θ)

θ

EM begins with a (bad) set of estimates for θ.

1 Use Jensen’s inequality to estimate a bound b on `
called the expectation of `
2 Find values of θ which maximize b

EM is guaranteed to find θs which do not decrease b.

Brandon Malone Poisson Mixture Models

The Poisson Distribution Mixture Models Expectation-Maximization Wrap-up

Expectation and the Q function

Recall the definition of ` and Jensen’s inequality.
N
X K
X
`(D : θ) = log q(k, l)
l k
N K
XX q(k, l)
≥ P(k|l) log
P(k|l)
l k

This gives the expectation of ` with our current parameters θ.

Based on this equation, we define Q(θ) which we want to maximize.

N X
X K
Q(θ) = P(k|n) log q(k, l)
l k

(See the handout for a detailed derivation of Q.)

Brandon Malone Poisson Mixture Models
The Poisson Distribution Mixture Models Expectation-Maximization Wrap-up

Maximization and the Q function

We use the following process to maximize Q for a particular

parameter θi .
1 Differentiate Q w.r.t θi
2 Set the derivative equal to 0
3 Solve for θi
(See the handout for detailed derivations.)

PN
P(k|l)Dl
l
λk =
Z (k)
Z (k)
pk =
N

Brandon Malone Poisson Mixture Models

The Poisson Distribution Mixture Models Expectation-Maximization Wrap-up

The EM algorithm for PMMs

procedure pmmEM(data D, inital p1 . . . pK , λ1 . . . λK , convergence criteria C)

while C has not been met do
. Update the expectations
q(k, l) ← pk · g (Dl , λk )
P(k|l) ← PKq(k,l)
m q(m,l)
PN
. Maximize the parameters
P(k|l)Dl
λk ← l
Z (k)
Z (k)
pk ← N
end while
end procedure

Brandon Malone Poisson Mixture Models

The Poisson Distribution Mixture Models Expectation-Maximization Wrap-up

Grouping the DNA sequences into clusters

After running EM, we have several useful pieces of information

about our metagenomics sample.
P(k|l). The distribution over species for each sequence.
pk . The relative genome sizes of the species.
λk . The abundance of the species.

Other questions...
Do we really know how many species there are?
Can we differentiate species with similar abundances?
How do we pick “good” initial parameters?
When have we converged?

Brandon Malone Poisson Mixture Models

The Poisson Distribution Mixture Models Expectation-Maximization Wrap-up

More on EM

EM is a general framework that is useful whenever data is missing.

If used to estimate class probabilities in naive Bayes models, it
is called Bayesian clustering
If used in HMMs, it is called the Baum-Welch algorithm
Can be used in general Bayesian networks to calculate
parameters when some data is missing
If used with structure learning algorthms, it is called
Structural EM
Many, many others...
We maximize likelihood with EM. What if we want MAP
parameters?

Brandon Malone Poisson Mixture Models

The Poisson Distribution Mixture Models Expectation-Maximization Wrap-up

Recap

During this part of the course, we have discussed:

Mixture models as a probabilistic clustering method
Expectation-maximization as a framework for estimating
parameters when variables are hidden

Brandon Malone Poisson Mixture Models

The Poisson Distribution Mixture Models Expectation-Maximization Wrap-up

Next in probabilistic models

We will see a Bayesian version of EM.

Estimating parameters in topic models

Brandon Malone Poisson Mixture Models