Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Expectation-Maximization algorithm can be used for the latent variables (variables that are not directly observable
and are actually inferred from the values of the other observed variables) too in order to predict their values with the
condition that the general form of probability distribution governing those latent variables is known to us. This
algorithm is actually used at the base of many unsupervised clustering algorithms in the field of machine learning. It
is used to find the local maximum likelihood parameters of a statistical model in the cases where latent variables
are involved and the data is missing or incomplete.
In statistics, an expectation–maximization (EM) algorithm is an iterative method to find maximum likelihood or
maximum a posteriori (MAP) estimates of parameters in statistical models, where the model depends on unobserved
latent variables.
Algorithm:
1. Given a set of incomplete data, consider a set of starting parameters.
2. Expectation step (E – step): Using the observed available data of the dataset, estimate (guess) the values of the
missing data.
3. Maximization step (M – step): Complete data generated after the expectation (E) step is used in order to update
the parameters.
4. Repeat step 2 and step 3 until convergence.
The essence of Expectation-Maximization algorithm is to use the available observed data of the dataset to estimate
the missing data and then using that data to update the values of the parameters. Let us understand the EM algorithm
in detail.
Initially, a set of initial values of the parameters are considered. A set of incomplete observed data is given to the
system with the assumption that the observed data comes from a specific model.
The next step is known as “Expectation” – step or E-step. In this step, we use the observed data in order to
estimate or guess the values of the missing or incomplete data. It is basically used to update the variables.
The next step is known as “Maximization”-step or M-step. In this step, we use the complete data generated in the
preceding “Expectation” – step in order to update the values of the parameters. It is basically used to update the
hypothesis.
Now, in the fourth step, it is checked whether the values are converging or not, if yes, then stop otherwise
repeat step-2 and step-3 i.e. “Expectation” – step and “Maximization” – step until the convergence occurs.
Usage of EM algorithm –
It can be used to fill the missing data in a sample.
It can be used as the basis of unsupervised learning of clusters.
It can be used for the purpose of estimating the parameters of Hidden Markov Model (HMM).
It can be used for discovering the values of latent variables.
Advantages of EM algorithm –
It is always guaranteed that likelihood will increase with each iteration.
The E-step and M-step are often pretty easy for many problems in terms of implementation.
Solutions to the M-steps often exist in the closed form.
Disadvantages of EM algorithm –
It has slow convergence.
It makes convergence to the local optima only.
It requires both the probabilities, forward and backward (numerical optimization requires only forward
probability).
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
Scanned by CamScanner
