Artificial

Intelligence
Lecture 9 – Markov Decision Processes II
Dr. Shivanjali Khare
[email protected]
 Example: Grid World

 A maze-like problem
 The agent lives in a grid
 Walls block the agent’s path
 Noisy movement: actions do not always go as planned
 80% of the time, the action North takes the agent North
 10% of the time, North takes the agent West; 10% East
 If there is a wall in the direction the agent would have
been taken, the agent stays put
 The agent receives rewards each time step
 Small “living” reward each step (can be negative)
 Big rewards come at the end (good or bad)
 Goal: maximize sum of (discounted) rewards
Recap: MDPs
• Markov decision processes:
• States S
• Actions A
• Transitions P(s’|s,a) (or T(s,a,s’)) s
• Rewards R(s,a,s’) (and discount )
a
• Start state s0
s, a

s,a,s’
• Quantities: ’s
• Policy = map of states to actions
• Utility = sum of discounted rewards
• Values = expected future utility from a state (max node)
• Q-Values = expected future utility from a q-state (chance node)
Optimal Quantities

 The value (utility) of a state s:

V*(s) = expected utility starting in s and s s is a
acting optimally state
a
 The value (utility) of a q-state (s,a): (s, a) is a
s, a q-state
Q*(s,a) = expected utility starting out
having taken action a from state s and s,a,s’ (s,a,s’) is a
(thereafter) acting optimally s transition
’
 The optimal policy:
*(s) = optimal action from state s

[Demo: gridworld values (L9D1)]

Gridworld Values V*
Gridworld: Q*
The Bellman Equations

How to be optimal:
Step 1: Take correct first action
Step 2: Keep being optimal
The Bellman Equations

• Definition of “optimal utility” via expectimax recurrence gives a simple one- s

step lookahead relationship amongst optimal utility values
a
s, a

s,a,s’
’s

• These are the Bellman equations, and they characterize optimal values in a
way we’ll use over and over
Value Iteration

• Bellman equations characterize the optimal values: V(s)

a
s, a

s,a,s’
V(s’)
• Value iteration computes them:

• Value iteration is just a fixed point solution method

• … though the Vk vectors are also interpretable as time-limited values
Example: Value Iteration

3.5 2.5 0

2 1 0
Assume no discount!

0 0 0
Policy Extraction
Computing Actions from
Values
• Let’s imagine we have the optimal values V*(s)

• How should we act?

• It’s not obvious!

• We need to do a mini-expectimax (one step)

• This is called policy extraction, since it gets the policy implied by the values
Computing Actions from
Q-Values
• Let’s imagine we have the optimal q-values:

• How should we act?

• Completely trivial to decide!

• Important lesson: actions are easier to select from q-values than values!
Policy Evaluation
Fixed Policies

Do the optimal action Do what  says to do

s s

a (s)
s, a s, (s)

s,a,s’ s, (s),s’
’s ’s

• Expectimax trees max over all actions to compute the optimal values

• If we fixed some policy (s), then the tree would be simpler – only one action per state
• … though the tree’s value would depend on which policy we fixed
Utilities for a Fixed Policy

• Another basic operation: compute the utility of a state s

under a fixed (generally non-optimal) policy s

(s)
• Define the utility of a state s, under a fixed policy : s, (s)
V(s) = expected total discounted rewards starting in s and following 
s, (s),s’
’s
• Recursive relation (one-step look-ahead / Bellman equation):
Example: Policy Evaluation

Always Go Right Always Go Forward

Example: Policy Evaluation

Always Go Right Always Go Forward

Policy Evaluation
• How do we calculate the V’s for a fixed policy ?

• Idea 1: Turn recursive Bellman equations into updates s

(like value iteration) (s)
s, (s)

s, (s),s’
’s

• Efficiency: O(S2) per iteration

• Idea 2: Without the maxes, the Bellman equations are just a linear system
• Solve with Matlab (or your favorite linear system solver)
Policy Methods
Problems with Value Iteration
• Value iteration repeats the Bellman updates:

• Problem 1: It’s slow – O(S2A) per iteration s, a

s,a,s’
• Problem 2: The “max” at each state rarely changes ’s

• Problem 3: The policy often converges long before the values

[Demo: value iteration (L9D2)]

k=0

Noise = 0.2
Discount = 0.9
Living reward = 0
k=1

Noise = 0.2
Discount = 0.9
Living reward = 0
k=2

Noise = 0.2
Discount = 0.9
Living reward = 0
k=3

Noise = 0.2
Discount = 0.9
Living reward = 0
k=4

Noise = 0.2
Discount = 0.9
Living reward = 0
k=5

Noise = 0.2
Discount = 0.9
Living reward = 0
k=6

Noise = 0.2
Discount = 0.9
Living reward = 0
k=7

Noise = 0.2
Discount = 0.9
Living reward = 0
k=8

Noise = 0.2
Discount = 0.9
Living reward = 0
k=9

Noise = 0.2
Discount = 0.9
Living reward = 0
k=10

Noise = 0.2
Discount = 0.9
Living reward = 0
k=11

Noise = 0.2
Discount = 0.9
Living reward = 0
k=12

Noise = 0.2
Discount = 0.9
Living reward = 0
k=100

Noise = 0.2
Discount = 0.9
Living reward = 0
Policy Iteration
Policy Iteration
• Alternative approach for optimal values:
• Step 1: Policy evaluation: calculate utilities for some fixed policy (not optimal
utilities!) until convergence
• Step 2: Policy improvement: update policy using one-step look-ahead with
resulting converged (but not optimal!) utilities as future values
• Repeat steps until policy converges

• This is policy iteration

• It’s still optimal!
• Can converge (much) faster under some conditions
Policy Iteration
• Evaluation: For fixed current policy , find values with policy evaluation:
• Iterate until values converge:

• Improvement: For fixed values, get a better policy using policy extraction
• One-step look-ahead:
Comparison

• Both value iteration and policy iteration compute the same thing (all optimal values)

• In value iteration:
• Every iteration updates both the values and (implicitly) the policy
• We don’t track the policy, but taking the max over actions implicitly recomputes it

• In policy iteration:
• We do several passes that update utilities with fixed policy (each pass is fast because we
consider only one action, not all of them)
• After the policy is evaluated, a new policy is chosen (slow like a value iteration pass)
• The new policy will be better (or we’re done)

• Both are dynamic programs for solving MDPs

Summary: MDP
Algorithms
• So you want to….
• Compute optimal values: use value iteration or policy iteration
• Compute values for a particular policy: use policy evaluation
• Turn your values into a policy: use policy extraction (one-step lookahead)

• These all look the same!

• They basically are – they are all variations of Bellman updates
• They all use one-step lookahead expectimax fragments
• They differ only in whether we plug in a fixed policy or max over actions
 Questions: Policy Iteration
Consider the gridworld where Left and Right actions are successful 100% of the
time. Specifically, the available actions in each state are to move to the
neighboring grid squares.
From state a, there is also an exit action available, which results in going to the
terminal state and collecting a reward of 10. Similarly, in state e, the reward for the
exit action is 1. Exit actions are successful 100% of the time.
• The discount factor ( 1) is 0.9.

We will execute one round of policy iteration.

42
 Policy evaluation

43
 Policy improvement

44
Next Time: Reinforcement Learning!