CMPSC 448: Machine Learning

Lecture 18. Dynamic Programming for Markov Decision Processes

Rui Zhang
Fall 2024

1
Outline of RL
● Introduction to Reinforcement learning
● Multi-armed Bandits
● Markov Decision Processes (MDP)
○ Dynamic Programming when we know the world

● Learning in MDP: When we don't know the world

○ Monte Carlo Methods

○ Temporal-Difference Learning (TD): SARSA and Q-Learning

Note: All of the lectures are tabular methods; we will only briefly discuss the
motivation function approximation methods (e.g., DQN, policy gradient, deep
reinforcement learning)
2
Two Problems in MDP
Input: a perfect model of RL as a finite MDP

Two problems:
1. evaluation (prediction): given a policy , what is value function ?
2. control: find the optimal policy or optimal value functions, i.e., or

In fact, in order to solve problem 2, we must first know how to solve problem 1.

3
Solution 1: Write out Bellman Equations and Solve Them
Solve systems of equations
● Write Bellman Equations or Bellman Optimality Equations for all state and
state-action pairs
● Solve systems of linear Equations for Evaluation (i.e., compute and )
● Solve systems of nonlinear Equations for Control (i.e., compute and )

We discussed this in our previous lecture.

4
Solution 2: Dynamic Programming (DP) for MDP
Idea: Use Dynamic Programming on Bellman equations for value functions to
organize and structure the search.

Dynamic Programming in context of MDP/RL, refers to collection of algorithms to

compute optimal policies given a perfect model of the environment as a Markov
Decision Process (MDP). Note that we know all the information about MDP
including states, rewards, transition probabilities, etc.

This is the focus of this lecture.

5
Outline of DP for MDP
We introduce two DP methods to find an optimal policy for a given MDP:

Policy Iteration
● Policy Evaluation
● Policy Improvement

Value Iteration
● One-sweep Policy Evaluation + One-step Policy Improvement

Both methods rely on Bellman condition on the optimality of a policy (from the
previou lecture): The value of a state under an optimal policy must equal the
expected return for the best action from that state.
6
Policy Iteration

Policy Evaluation: Estimate (iterative policy evaluation)

Policy Improvement: Generate (Greedy policy improvement)

7
Policy Evaluation
Policy Evaluation: for a given arbitrary policy compute the state-value function

8
Solution 1: Solving A System of Linear Equations
Recall: state-value function for policy

Recall: Bellman equation for

A system of simultaneous equations

Note: environment dynamics are completely known

9
 Solution 2: Iterative Policy Evaluation using Dynamic Programming

Start with a random guess of for all states, then iteratively update them using
Bellman Equation.

10
Iterative Policy Evaluation by Bellman Expectation Backup Operator

A sweep consists of applying a backup operation

to each state.

11
Bellman Expectation Backup Operator
Recall Bellman Equation for

From this, let's define Bellman Expectation Backup Operator on

12
Example: a small GridWorld

An un-discounted episodic MDP with

Actions = {up, down, right, left}
Nonterminal states: {1, 2, . . ., 14}
One terminal state (shown twice as shaded squares)
Actions that would take agent off the grid leave state unchanged
Reward is –1 until the terminal state is reached
13
Iterative Policy Evaluation for the small GridWorld

14
Iterative Policy Evaluation for the small GridWorld

15
Iterative Policy Evaluation for the small GridWorld

16
Iterative Policy Evaluation for the small GridWorld

The final estimate is in fact , which is this case gives, for each state, the
negative of the expected number of steps from that state until termination
17
Iterative Policy Evaluation

18
Policy Improvement
Suppose we have computed for a deterministic policy

For a given state , tells us how good it is to follow

For a given state , would it be better to do an action ?

19
Policy Improvement
Suppose we have computed for a deterministic policy

For a given state , tells us how good it is to follow

For a given state , would it be better to do an action ?

Let’s take action and thereafter follow the policy and see what happens to
the agent’s reward, this is just

20
Policy Improvement Theorem

The theorem can be easily generalized to stochastic policies (actions are selected
by different probabilities at every states under policy, which is more realistic)

21
Example of Greedification

22
Policy Improvement Theorem - Proof

23
Policy improvement with greedification
Do this for all states to get a new policy that is greedy with respect to

24
Policy improvement with greedification
Do this for all states to get a new policy that is greedy with respect to

Note that then from policy

improvement theorem, we have

25
Policy improvement with greedification
Do this for all states to get a new policy that is greedy with respect to

Note that then from policy

improvement theorem, we have

What if the policy is unchanged by this? then the policy satisfies Bellman
Optimality Equation, and it must be the optimal policy! 26
Policy Iteration: Iterate between Evaluation and Improvement

27
 Policy Iteration for the Small GridWorld

An un-discounted episodic MDP with

Actions = {up, down, right, left}
Nonterminal states: {1, 2, . . ., 14}
One terminal state (shown twice as shaded squares)
Actions that would take agent off the grid leave state
unchanged
Reward is –1 until the terminal state is reached
28
 Policy Improvement in the Middle

An un-discounted episodic MDP with

Actions = {up, down, right, left}
Nonterminal states: {1, 2, . . ., 14}
One terminal state (shown twice as shaded squares)
Actions that would take agent off the grid leave state
unchanged
Reward is –1 until the terminal state is reached
29
Policy Improvement in the Middle

Do we need to do policy
evaluation until convergence
before greedification or it could
be truncated somehow?

30
 From Policy Iteration to Value Iteration

An un-discounted episodic MDP with

Actions = {up, down, right, left}
Nonterminal states: {1, 2, . . ., 14}
One terminal state (shown twice as shaded squares)
Actions that would take agent off the grid leave state
unchanged
Reward is –1 until the terminal state is reached
31
From Policy Iteration to Value Iteration
Recall Policy Iteration alternates between the following two steps:

1. Policy Evaluation: Multiple Sweeps of Bellman Expectation Backup Operation until Convergence

2. Policy Improvement: One step of greedification

32
From Policy Iteration to Value Iteration
But, we don't need to do policy evaluation until convergence.
Instead, in Value Iteration:
1. Just One Sweep of Bellman Expectation Backup Operation

2. One step of greedification

33
An example MDP

The dynamics of MDP is given: 34

An example MDP

Our goal is to learn the

optimal policy such that if
we follow this policy, we
maximize the cumulative
reward!

We can find the optimal

policy in many ways.

Let's use value iteration.

35
value iteration: initialization

36
value iteration: one sweep evaluation

37
value iteration: one sweep evaluation

38
value iteration: one sweep evaluation

39
value iteration: one sweep evaluation

0.25
0 40
value iteration: one sweep greedification

0.25
41
value iteration: one sweep greedification

0.25
42
value iteration: one sweep greedification

0.25
43
value iteration: one sweep greedification

0.25) = 10.2 0.25

44
value iteration: one sweep greedification

0.25) = 3.08 0.25

45
Value Iteration: Combine two steps in a single update
Let’s interleave the evaluation and greedification
ONE sweep of evaluation is followed by ONE step of greedification
Combine these two together gives one update of value iteration as following

We call this Bellman Optimality Backup Operator

In this way, we don't need to explicitly maintain a policy

46
Bellman Optimality Backup Operator
Bellman Optimality Equation for

Bellman Optimality Backup Operator on

47
Value Iteration

48
Convergence of Policy Iteration and Value Iteration
Both Policy Iteration and Value Iteration are guaranteed to converge to the optimal
policy and the optimal value functions!

49
Summary
Policy Evaluation: Bellman expectation backup operators (without a max)
Policy Improvement: form a greedy policy, if only locally
Policy Iteration: alternate the above two processes
Value Iteration: Bellman optimality backup operators (with a max)

DP is used when we know how the world works. Biggest limitation of DP is that it
requires a probability model (as opposed to a generative or simulation model).
DP uses Full Backups (to be contrasted later with sample backups)
Next Lecture: MC and TD when we don't know how the world works

50