Rejection Sampling in Artificial Intelligence

Rejection sampling generates random samples from a complex target distribution. It is useful when direct sampling from the target distribution is difficult or computationally expensive. The method relies on sampling from a simpler, more tractable proposal distribution and then applying a rejection criterion to ensure that the samples are consistent with the target distribution.

This article provides an overview of rejection sampling, its theoretical foundations, practical applications, and limitations.

Foundation of Rejection Sampling

Rejection sampling is based on a simple yet elegant idea: instead of sampling directly from a target distribution p(x), we sample from a proposal distribution q(x) that is easier to work with. The proposal distribution must satisfy the condition that there exists a constant M such that:

M⋅q(x)≥p(x) \text{ for all } x

This ensures that the proposal distribution q(x) "covers" the target distribution p(x) when scaled by M.

The rejection sampling algorithm proceeds as follows:

1. Sample a candidate point x from the proposal distribution q(x).
2. Compute the acceptance probability α(x):
   α(x)= \frac{p(x)}{M⋅q(x)}
3. Generate a uniform random number u from the interval  [0,1].
4. Accept the sample x if u≤α(x); otherwise, reject it.
5. Repeat the process until the desired number of samples is obtained.

The accepted samples are guaranteed to follow the target distribution p(x).

Key Considerations in Rejection Sampling

Choice of Proposal Distribution

The efficiency of rejection sampling depends heavily on the choice of the proposal distribution q(x). A good proposal distribution should:

• Be easy to sample from.
• Closely resemble the target distribution p(x).
• Minimize the constant M to reduce the rejection rate.

If q(x) is poorly chosen, the rejection rate may be high, leading to inefficient sampling.

Acceptance Rate

The acceptance rate is the proportion of candidate samples that are accepted. It is given by:

\text{Acceptance Rate} = \frac{1}{M}

A smaller M leads to a higher acceptance rate, making the algorithm more efficient. However, finding an optimal M and q(x) can be challenging, especially for high-dimensional or complex distributions.

Practical Applications of Rejection Sampling

Rejection sampling has a wide range of applications in various fields, including:

1. Monte Carlo Simulations: Rejection sampling is often used in Monte Carlo methods to generate samples from complex distributions that arise in physics, finance, and engineering.
2. Bayesian Inference: In Bayesian statistics, rejection sampling can be used to draw samples from posterior distributions when direct sampling is infeasible.
3. Generative Models: Rejection sampling is employed in generative modeling to create synthetic data that follows a specific distribution.
4. Random Variate Generation: The technique is used to generate random numbers from non-standard distributions, such as truncated or mixed distributions.

Limitations of Rejection Sampling

While rejection sampling is a versatile and conceptually simple method, it has several limitations:

1. Inefficiency in High Dimensions: As the dimensionality of the target distribution increases, finding a suitable proposal distribution q(x) with a small M becomes increasingly difficult. This leads to a high rejection rate and poor computational efficiency.
2. Dependence on Proposal Distribution: The performance of rejection sampling is highly sensitive to the choice of q(x). A poor choice can result in excessive computational overhead.
3. Difficulty with Complex Distributions: For distributions with sharp peaks, heavy tails, or multiple modes, designing an effective proposal distribution can be challenging.

Extensions and Variants

To address some of the limitations of rejection sampling, several extensions and variants have been developed:

1. Adaptive Rejection Sampling: This method adaptively refines the proposal distribution based on the samples generated, improving efficiency over time.
2. Importance Sampling: Instead of rejecting samples, importance sampling assigns weights to the samples based on their likelihood under the target distribution.
3. Markov Chain Monte Carlo (MCMC): MCMC methods, such as the Metropolis-Hastings algorithm, provide an alternative approach to sampling from complex distributions by constructing a Markov chain that converges to the target distribution.