Conjugate prior

In Bayesian probability theory, if, given a likelihood function , the posterior distribution is in the same probability distribution family as the
prior probability distribution , the prior and posterior are then called conjugate distributions with respect to that likelihood function and the prior is called
a conjugate prior for the likelihood function .

A conjugate prior is an algebraic convenience, giving a closed-form expression for the posterior; otherwise, numerical integration may be necessary. Further,
conjugate priors may give intuition by more transparently showing how a likelihood function updates a prior distribution.

The concept, as well as the term "conjugate prior", were introduced by Howard Raiffa and Robert Schlaifer in their work on Bayesian decision theory.[1] A
similar concept had been discovered independently by George Alfred Barnard.[2]

Example
The form of the conjugate prior can generally be determined by inspection of the probability density or probability mass function of a distribution. For example,
consider a random variable which consists of the number of successes in Bernoulli trials with unknown probability of success in [0,1]. This random
variable will follow the binomial distribution, with a probability mass function of the form

The usual conjugate prior is the beta distribution with parameters ( , ):

where and are chosen to reflect any existing belief or information ( and would give a uniform distribution) and is the Beta function
acting as a normalising constant.

In this context, and are called hyperparameters (parameters of the prior), to distinguish them from parameters of the underlying model (here ). A typical
characteristic of conjugate priors is that the dimensionality of the hyperparameters is one greater than that of the parameters of the original distribution. If all
parameters are scalar values, then there will be one more hyperparameter than parameter; but this also applies to vector-valued and matrix-valued parameters.
(See the general article on the exponential family, and also consider the Wishart distribution, conjugate prior of the covariance matrix of a multivariate normal
distribution, for an example where a large dimensionality is involved.)

If we sample this random variable and get successes and failures, then we have

which is another Beta distribution with parameters . This posterior distribution could then be used as the prior for more samples, with the
hyperparameters simply adding each extra piece of information as it comes.

Interpretations

Pseudo-observations
It is often useful to think of the hyperparameters of a conjugate prior distribution corresponding to having observed a certain number of pseudo-observations
with properties specified by the parameters. For example, the values and of a beta distribution can be thought of as corresponding to successes and
failures if the posterior mode is used to choose an optimal parameter setting, or successes and failures if the posterior mean is used to choose an
optimal parameter setting. In general, for nearly all conjugate prior distributions, the hyperparameters can be interpreted in terms of pseudo-observations. This
can help provide intuition behind the often messy update equations and help choose reasonable hyperparameters for a prior.

Dynamical system
One can think of conditioning on conjugate priors as defining a kind of (discrete time) dynamical system: from a given set of hyperparameters, incoming data
updates these hyperparameters, so one can see the change in hyperparameters as a kind of "time evolution" of the system, corresponding to "learning". Starting
at different points yields different flows over time. This is again analogous with the dynamical system defined by a linear operator, but note that since different
samples lead to different inferences, this is not simply dependent on time but rather on data over time. For related approaches, see Recursive Bayesian
estimation and Data assimilation.

Practical example
Suppose a rental car service operates in your city. Drivers can drop off and pick up cars anywhere inside the city limits. You can find and rent cars using an app.

Suppose you wish to find the probability that you can find a rental car within a short distance of your home address at any time of day.

Over three days you look at the app and find the following number of cars within a short distance of your home address:

Suppose we assume the data comes from a Poisson distribution. In that case, we can compute the maximum likelihood estimate of the parameters of the model,
which is Using this maximum likelihood estimate, we can compute the probability that there will be at least one car available on a given
day:

This is the Poisson distribution that is the most likely to have generated the observed data . But the data could also have come from another Poisson
distribution, e.g., one with , or , etc. In fact, there is an infinite number of Poisson distributions that could have generated the observed data. With
relatively few data points, we should be quite uncertain about which exact Poisson distribution generated this data. Intuitively we should instead take a weighted
average of the probability of for each of those Poisson distributions, weighted by how likely they each are, given the data we've observed .

Generally, this quantity is known as the posterior predictive distribution where is a new data point, is the observed data and

are the parameters of the model. Using Bayes' theorem we can expand therefore Generally, this

integral is hard to compute. However, if you choose a conjugate prior distribution , a closed-form expression can be derived. This is the posterior predictive
column in the tables below.

Returning to our example, if we pick the Gamma distribution as our prior distribution over the rate of the Poisson distributions, then the posterior predictive is
the negative binomial distribution, as can be seen from the table below. The Gamma distribution is parameterized by two hyperparameters , which we have
to choose. By looking at plots of the gamma distribution, we pick , which seems to be a reasonable prior for the average number of cars. The choice
of prior hyperparameters is inherently subjective and based on prior knowledge.

Given the prior hyperparameters and we can compute the posterior hyperparameters and

Given the posterior hyperparameters, we can finally compute the posterior predictive of

This much more conservative estimate reflects the uncertainty in the model parameters, which the posterior predictive takes into account.

Table of conjugate distributions

Let n denote the number of observations. In all cases below, the data is assumed to consist of n points (which will be random vectors in the
multivariate cases).
If the likelihood function belongs to the exponential family, then a conjugate prior exists, often also in the exponential family; see Exponential family:
Conjugate distributions.

When the likelihood function is a discrete distribution

Model Conjugate prior (and Prior Posterior

Likelihood Posterior pred
parameters posterior) distribution hyperparameters hyperparameters[note 1] Interpretation of hyperparameters

p
Bernoulli
(probability)
Beta successes, failures[note 3]
(Bernoulli)

Binomial
with known p
Beta successes, failures[note 3]
number of (probability) (beta-binomial)
trials, m

Negative total successes, failures[note 3]

binomial
with known
p
Beta (beta-negativ
(probability) (i.e., experiments, assuming
failure number, binomial)
r stays fixed)

total occurrences in intervals

(negative binom
Poisson λ (rate) Gamma

[note 4] total occurrences in intervals

(negative binom

p
(probability
vector), k where is
Categorical (number of Dirichlet [note 3]
the number of observations in occurrences of category
categories; category i
i.e., size of
(categorical)
p)

p
(probability
vector), k
Multinomial (number of Dirichlet [note 3]
occurrences of category
categories; (Dirichlet-multin
i.e., size of
p)
Hypergeometric
M (number
with known
total population
of target Beta-binomial[3] successes, failures[note 3]
members)
size, N

p0
Geometric Beta experiments, total failures[note 3]
(probability)

When likelihood function is a continuous distribution

Model Conjugate prior (and Prior

Likelihood
parameters posterior) distribution hyperparameters Posterior hyperparameters[note 1] Interpretation of hyperparameters

mean was estimated from

Normal
observations with total precision
with known μ (mean) Normal (sum of all individual precisions)
variance σ2 and with sample mean

mean was estimated from

Normal observations with total precision
with known μ (mean) Normal (sum of all individual precisions)
precision τ
and with sample mean

variance was estimated from

Normal observations with sample variance
with known σ2 (variance) Inverse gamma [note 6] (i.e. with sum of squared
mean μ deviations , where deviations are
from known mean )

Normal variance was estimated from

with known σ2 (variance) Scaled inverse chi-squared observations with sample variance
mean μ

precision was estimated from

Normal observations with sample variance
with known τ (precision) Gamma [note 4] (i.e. with sum of squared
mean μ deviations , where deviations are
from known mean )
 mean was estimated from
μ and σ2 observations with sample mean ;
Normal[note 7] Assuming Normal-inverse gamma variance was estimated from
exchangeability observations with sample mean
and sum of squared deviations
is the sample mean

mean was estimated from

observations with sample mean ,
μ and τ
and precision was estimated from
Normal Assuming Normal-gamma
observations with sample mean
exchangeability
and sum of squared deviations
is the sample mean

Multivariate mean was estimated from

normal with observations with total precision
μ (mean
known Multivariate normal (sum of all individual precisions)
vector)
covariance
is the sample mean and with sample mean
matrix Σ

Multivariate mean was estimated from

normal with observations with total precision
μ (mean
known Multivariate normal (sum of all individual precisions)
vector)
precision is the sample mean
and with sample mean
matrix Λ

Multivariate
covariance matrix was estimated
normal with Σ (covariance
Inverse-Wishart from observations with sum of
known mean matrix)
pairwise deviation products
μ
Multivariate
covariance matrix was estimated
normal with Λ (precision
Wishart from observations with sum of
known mean matrix)
μ pairwise deviation products

mean was estimated from

μ (mean observations with sample mean ;
Multivariate vector) and Σ covariance matrix was estimated
normal-inverse-Wishart from observations with sample
normal (covariance
is the sample mean
matrix) mean and with sum of pairwise
deviation products

mean was estimated from

μ (mean observations with sample mean ;
Multivariate vector) and Λ covariance matrix was estimated
normal-Wishart from observations with sample
normal (precision
matrix) is the sample mean mean and with sum of pairwise
deviation products

observations with maximum value

Uniform Pareto

observations with sum of the

Pareto order of magnitude of each
with known k (shape) Gamma observation (i.e. the logarithm of the
minimum xm ratio of each observation to the
minimum )

Weibull
observations with sum of the
with known θ (scale) Inverse gamma[3] β'th power of each observation
shape β

Log-normal Same as for the normal distribution after applying the natural logarithm to the data for the posterior hyperparameters. Please refer to Fink (1997, pp. 21–22) to

Exponential λ (rate) Gamma [note 4] [5]

observations that sum to

Gamma
with known β (rate) Gamma observations with sum
shape α

Inverse
Gamma β (inverse observations with sum
Gamma
with known scale)
shape α

Gamma
or observations ( for estimating
with known α (shape)
, for estimating ) with product
rate β
was estimated from
α (shape), β observations with product ; was
Gamma[3] (inverse scale) estimated from observations with
sum

and were estimated from

Beta α, β observations with product and
product of the complements
See also
Beta-binomial distribution

Notes
1. Denoted by the same symbols as the prior hyperparameters with primes added ('). For instance is denoted
2. This is the posterior predictive distribution of a new data point given the observed data points, with the parameters marginalized out.
Variables with primes indicate the posterior values of the parameters.
3. The exact interpretation of the parameters of a beta distribution in terms of number of successes and failures depends on what function is
used to extract a point estimate from the distribution. The mean of a beta distribution is which corresponds to successes and

failures, while the mode is which corresponds to successes and failures. Bayesians generally prefer to use the
posterior mean rather than the posterior mode as a point estimate, justified by a quadratic loss function, and the use of and is more
convenient mathematically, while the use of and has the advantage that a uniform prior corresponds to 0 successes
and 0 failures. The same issues apply to the Dirichlet distribution.
4. β is rate or inverse scale. In parameterization of gamma distribution,θ = 1/β and k = α.
5. This is the posterior predictive distribution of a new data point given the observed data points, with the parameters marginalized out.
Variables with primes indicate the posterior values of the parameters. and refer to the normal distribution and Student's t-distribution,
respectively, or to the multivariate normal distribution and multivariate t-distribution in the multivariate cases.
6. In terms of the inverse gamma, is a scale parameter
7. A different conjugate prior for unknown mean and variance, but with a fixed, linear relationship between them, is found in the normal variance-
mean mixture, with the generalized inverse Gaussian as conjugate mixing distribution.
8. is a compound gamma distribution; here is a generalized beta prime distribution.

References
1. Howard Raiffa and Robert Schlaifer. Applied Statistical Decision Theory. Division of Research, Graduate School of Business Administration,
Harvard University, 1961.
2. Jeff Miller et al. Earliest Known Uses of Some of the Words of Mathematics ([Link] "conjugate prior
distributions" ([Link] Electronic document, revision of November 13, 2005, retrieved December 2, 2005.
3. Fink, Daniel (1997). "A Compendium of Conjugate Priors" ([Link]
s/df36/CONJINTRnew%[Link]) (PDF). CiteSeerX [Link].5540 ([Link]
0). Archived from the original ([Link] (PDF) on May
29, 2009.
4. Murphy, Kevin P. (2007), Conjugate Bayesian analysis of the Gaussian distribution ([Link]
(PDF)
5. Liu, Han; Wasserman, Larry (2014). Statistical Machine Learning ([Link] (PDF). p. 314.

Retrieved from "[Link]