COMPSCI 614: Randomized Algorithms with
Applications to Data Science
Prof. Cameron Musco
University of Massachusetts Amherst. Spring 2024.
Lecture 19
1
Logistics
• I will send responses to project progress reports soon.
2
Summary
Last Week: Start on Markov Chains.
• Start on Markov chains and their analysis
• Markov chain based algorithms for satisfiability: ≈ n2 time for
2-SAT, and ≈ (4/3)n for 3-SAT.
Today: Markov Chains Continued
• The gambler’s ruin problem.
• Aperiodicity and stationary distribution of a Markov chain.
• The fundamental theorem of Markov chains.
3
Markov Chain Review
• A discrete time stochastic process is a Markov chain if is it
memoryless:
Pr(Xt = at |Xt−1 = at−1 , . . . , X0 = a0 ) = Pr(Xt = at |Xt−1 = at−1 )
• If each Xt can take m possible values, the Markov chain is
specified by the transition matrix P ∈ [0, 1]m×m with
Pi,j = Pr(Xt+1 = j|Xt = i).
• Let qt ∈ [0, 1]1×m be the distribution of Xi . Then qt+1 = qt P.
4
Markov Chain Review
Often viewed as an underlying state transition graph. Nodes
correspond to possible values that each Xt can take.
The Markov chain is irreducible if the underlying graph consists of
single strongly connected component.
5
Gambler’s Ruin
5
Gambler’s Ruin
• You and ‘a friend’ repeatedly toss a fair coin. If it hits heads, you
give your friend $1. If it hits tails, they give you $1.
• You start with $ℓ1 and your friend starts with $ℓ2 . When either of
you runs out of money the game terminates.
• What is the probability that you win $ℓ2 ?
6
Gambler’s Ruin Markov Chain
Let X0 , X1 , . . . be the Markov chain where Xt is your profit at
step t. X0 = 0 and:
P−ℓ1 ,−ℓ1 = Pℓ2 ,ℓ2 = 1
Pi,i+1 = Pi,i−1 = 1/2 for − ℓ1 < i < ℓ2
• ℓ1 and ℓ2 are absorbing states.
• All i with −ℓ1 < i < ℓ2 are transient states. I.e.,
Pr[Xt′ = i for some t′ > t | Xt = i] < 1.
Observe that this Markov chain is also a Martingale since
E[Xt+1 |Xt ] = Xt .
7
Gambler’s Ruin Analysis
Let X0 , X1 , . . . be the Markov chain where Xt is your profit at step t.
X0 = 0 and:
P−ℓ1 ,−ℓ1 = Pℓ2 ,ℓ2 = 1
Pi,i+1 = Pi,i−1 = 1/2 for − ℓ1 < i < ℓ2
We want to compute q = limt→∞ Pr[Xt = ℓ2 ].
By linearity of expectation, for any i, E[Xi ] = 0. Further, for
q = limt→∞ Pr[Xt = ℓ2 ], since −ℓ1 , ℓ2 are the only non-transient states,
lim E[Xt ] = ℓ2 q + −ℓ1 (1 − q) = 0.
t→∞
Solving for q, we have q = ℓ1 +ℓ2 .
ℓ1
8
Gambler’s Ruin Thought Exercise
What if you always walk away as soon as you win just $1. Then
what is your probability of winning, and what are your
expected winnings?
9
Stationary Distributions
9
Stationary Distribution
A stationary distribution of a Markov chain with transition matrix
P ∈ [0, 1]m×m is a distribution π ∈ [0, 1]m such that π = πP.
I.e. if Xt ∼ π, then Xt+1 ∼ πP = π.
Think-pair-share: Do all Markov chains have a stationary
distribution?
10
Claim (Existence of Stationary Distribution)
Any Markov chain with a finite state space, and transition
matrix P ∈ [0, 1]m×m has a stationary distribution π ∈ [0, 1]m
with π = πP.
Follows from the Brouwer fixed point theorem: for any
continuous function f : S → S, where S is a compact convex
set, there is some x such that f(x) = x.
11
Periodicity
The periodicity of a state i is defined as:
T = gcd{t > 0 : Pr(Xt = i | X0 = i) > 0}.
The state is aperiodic if it has periodicity T = 1.
A Markov chain is aperiodic if all states are aperiodic.
12
Periodicity
Claim
If a Markov chain is irreducible, and has at least one
self-loop, then it is aperiodic.
13
Fundamental Theorem
Theorem (The Fundamental Theorem of Markov Chains)
Let X0 , X1 , . . . be a Markov chain with a finite state space and
transition matrix P ∈ [0, 1]m×m . If the chain is both irreducible and
aperiodic,
1. There exists a unique stationary distribution π ∈ [0, 1]m with
π = πP.
2. For any states i, j, limt→∞ Pr[Xt = i|X0 = j] = π(i). I.e., for any
initial distribution q0 , limt→∞ qt = limt→∞ q0 Pt = π.
3. π(i) = E[min(t:X1t =i)|X0 =i] . I.e., π(i) is the inverse of the average
expected return time from state i back to i.
In the limit, the probability of being at any state i is independent of
the starting state.
14
Stationary Distribution Example 1
Shuffling Markov Chain: Given a pack of c cards. At each step draw
two random cards, swap them and repeat.
• What is the state space of this chain?
• What is the transition probability Pi,j ? How does it compare to
Pj,i ?
• This Markov chain is symmetric and thus its stationary
distribution is uniform, π(i) = c!1 .
Letting m = c! denote the size of the state space,
∑ ∑ 1 ∑ 1
πP:,i = π(j)Pj,i = π(j)Pi,j = Pi,j = = π(i).
m m
j j j
Once we have exhibited a stationary distribution, we know that it is
unique and that the chain converges to it in the limit!
15
Stationary Distribution Example 2
Random Walk on an Undirected Graph: Consider a random
walk on an undirected graph. If it is at node i at step t, then it
moves to any of i’s neighbors at step t + 1 with probability d1 .
i
• What is the state space of this chain?
• What is the transition probability Pi,j ?
• Is this chain aperiodic?
• If the graph is not bipartite, then there is at least one odd
cycle, making the chain aperiodic.
16
Stationary Distribution Example 2
Random Walk on an Undirected Graph: Consider a random
walk on an undirected graph. If it is at node i at step t, then it
moves to any of i’s neighbors at step t + 1 with probability d1 .
i
Claim: When the graph is not bipartite, the unique stationary
di
distribution of this Markov chain is given by π(i) = 2|E| .
∑ ∑ dj 1 ∑ 1 d
πP:,i = π(j)Pj,i = · = = i = π(i).
2|E| dj 2|E| 2|E|
j j j
I.e., the probability of being at a given node i is dependent
only on the node’s degree, not on the structure of the graph in
any other way.
17
