CS 188: Artificial Intelligence
Markov Decision Processes
Instructors: Sergey Levine and Stuart Russell
University of California, Berkeley
[slides adapted from Dan Klein and Pieter Abbeel http://ai.berkeley.edu.]
Non-Deterministic Search
max
chance
10 10
4 5
9 100
7
Example: Grid World
A maze-like problem
The agent lives in a grid
Walls block the agent’s path
Noisy movement: actions do not always go as planned
80% of the time, the action North takes the agent North
(if there is no wall there)
10% of the time, North takes the agent West; 10% East
If there is a wall in the direction the agent would have
been taken, the agent stays put
The agent receives rewards each time step
Small “living” reward each step (can be negative)
Big rewards come at the end (good or bad)
Goal: maximize sum of rewards
Grid World Actions
Deterministic Grid World Stochastic Grid World
Markov Decision Processes
An MDP is defined by:
A set of states s S
A set of actions a A
A transition function T(s, a, s’)
Probability that a from s leads to s’, i.e., P(s’| s, a)
Also called the model or the dynamics
A reward function R(s, a, s’)
Sometimes just R(s) or R(s’)
A start state
Maybe a terminal state
MDPs are non-deterministic search problems
One way to solve them is with expectimax search
We’ll have a new tool soon
[Demo – gridworld manual intro (L8D1)]
Video of Demo Gridworld Manual Intro
What is Markov about MDPs?
“Markov” generally means that given the present state, the
future and the past are independent
For Markov decision processes, “Markov” means action
outcomes depend only on the current state
Andrey Markov
(1856-1922)
This is just like search, where the successor function could only
depend on the current state (not the history)
Policies
In deterministic single-agent search problems,
we wanted an optimal plan, or sequence of
actions, from start to a goal
For MDPs, we want an optimal policy *: S → A
A policy gives an action for each state
An optimal policy is one that maximizes
expected utility if followed
An explicit policy defines a reflex agent
Expectimax didn’t compute entire policies
It computed the action for a single state only
Optimal Policies
R(s) = -0.01 R(s) = -0.03
R(s) = -0.4 R(s) = -2.0
Example: Racing
Example: Racing
A robot car wants to travel far, quickly
Three states: Cool, Warm, Overheated
Two actions: Slow, Fast
0.5 +1
Going faster gets double reward
1.0
Fast
Slow -10
+1
0.5
Warm
Slow
Fast 0.5 +2
Cool 0.5
+1 Overheated
1.0
+2
Racing Search Tree
MDP Search Trees
Each MDP state projects an expectimax-like search tree
s s is a state
(s, a) is a
s, a
q-state
(s,a,s’) called a transition
s,a,s’ T(s,a,s’) = P(s’|s,a)
R(s,a,s’)
s’
Utilities of Sequences
Utilities of Sequences
What preferences should an agent have over reward sequences?
More or less? [1, 2, 2] or [2, 3, 4]
Now or later? [0, 0, 1] or [1, 0, 0]
Discounting
It’s reasonable to maximize the sum of rewards
It’s also reasonable to prefer rewards now to rewards later
One solution: values of rewards decay exponentially
Worth Now Worth Next Step Worth In Two Steps
Discounting
How to discount?
Each time we descend a level, we
multiply in the discount once
Why discount?
Sooner rewards probably do have
higher utility than later rewards
Also helps our algorithms converge
Example: discount of 0.5
U([1,2,3]) = 1*1 + 0.5*2 + 0.25*3
U([1,2,3]) < U([3,2,1])
Stationary Preferences
Theorem: if we assume stationary preferences:
Then: there are only two ways to define utilities
Additive utility:
Discounted utility:
Quiz: Discounting
Given:
Actions: East, West, and Exit (only available in exit states a, e)
Transitions: deterministic
Quiz 1: For = 1, what is the optimal policy?
Quiz 2: For = 0.1, what is the optimal policy?
Quiz 3: For which are West and East equally good when in state d?
Infinite Utilities?!
Problem: What if the game lasts forever? Do we get infinite rewards?
Solutions:
Finite horizon: (similar to depth-limited search)
Terminate episodes after a fixed T steps (e.g. life)
Gives nonstationary policies ( depends on time left)
Discounting: use 0 < < 1
Smaller means smaller “horizon” – shorter term focus
Absorbing state: guarantee that for every policy, a terminal state will eventually
be reached (like “overheated” for racing)
Recap: Defining MDPs
Markov decision processes: s
Set of states S
Start state s0 a
Set of actions A s, a
Transitions P(s’|s,a) (or T(s,a,s’))
Rewards R(s,a,s’) (and discount ) s,a,s’
s’
MDP quantities so far:
Policy = Choice of action for each state
Utility = sum of (discounted) rewards
Solving MDPs
Optimal Quantities
The value (utility) of a state s:
V*(s) = expected utility starting in s and s s is a
acting optimally state
a
(s, a) is a
The value (utility) of a q-state (s,a): s, a q-state
Q*(s,a) = expected utility starting out
s,a,s’ (s,a,s’) is a
having taken action a from state s and
transition
(thereafter) acting optimally s’
The optimal policy:
*(s) = optimal action from state s
[Demo – gridworld values (L8D4)]
Snapshot of Demo – Gridworld V Values
Noise = 0.2
Discount = 0.9
Living reward = 0
Snapshot of Demo – Gridworld Q Values
Noise = 0.2
Discount = 0.9
Living reward = 0
Values of States
Fundamental operation: compute the (expectimax) value of a state
Expected utility under optimal action
s
Average sum of (discounted) rewards
This is just what expectimax computed! a
s, a
Recursive definition of value:
s,a,s’
s’
Racing Search Tree
Racing Search Tree
Racing Search Tree
We’re doing way too much
work with expectimax!
Problem: States are repeated
Idea: Only compute needed
quantities once
Problem: Tree goes on forever
Idea: Do a depth-limited
computation, but with increasing
depths until change is small
Note: deep parts of the tree
eventually don’t matter if γ < 1
Time-Limited Values
Key idea: time-limited values
Define Vk(s) to be the optimal value of s if the game ends
in k more time steps
Equivalently, it’s what a depth-k expectimax would give from s
[Demo – time-limited values (L8D6)]
k=0
Noise = 0.2
Discount = 0.9
Living reward = 0
k=1
Noise = 0.2
Discount = 0.9
Living reward = 0
k=2
Noise = 0.2
Discount = 0.9
Living reward = 0
k=3
Noise = 0.2
Discount = 0.9
Living reward = 0
k=4
Noise = 0.2
Discount = 0.9
Living reward = 0
k=5
Noise = 0.2
Discount = 0.9
Living reward = 0
k=6
Noise = 0.2
Discount = 0.9
Living reward = 0
k=7
Noise = 0.2
Discount = 0.9
Living reward = 0
k=8
Noise = 0.2
Discount = 0.9
Living reward = 0
k=9
Noise = 0.2
Discount = 0.9
Living reward = 0
k=10
Noise = 0.2
Discount = 0.9
Living reward = 0
k=11
Noise = 0.2
Discount = 0.9
Living reward = 0
k=12
Noise = 0.2
Discount = 0.9
Living reward = 0
k=100
Noise = 0.2
Discount = 0.9
Living reward = 0
