CS 188: Artificial Intelligence
Reinforcement Learning
Instructors: Dan Klein and Pieter Abbeel
University of California, Berkeley
[These slides were created by Dan Klein and Pieter Abbeel for CS188 Intro to AI at UC Berkeley. All CS188 materials are available at
Reinforcement Learning
Reinforcement Learning
Agent
State: s
Reward: r
Actions: a
Environm
ent
Basic idea:
Receive feedback in the form of rewards
Agents utility is defined by the reward function
Must (learn to) act so as to maximize expected
rewards
All learning is based on observed samples of
Example: Learning to Walk
Initial
Kohl and Stone, ICRA 2004]
A Learning Trial
After Learning [1K
Trials]
Example: Learning to Walk
Kohl and Stone, ICRA 2004]
Initial
[Video: AIBO WALK initial]
Example: Learning to Walk
Kohl and Stone, ICRA 2004]
Training
[Video: AIBO WALK training
Example: Learning to Walk
Kohl and Stone, ICRA 2004]
Finished
[Video: AIBO WALK finishe
Example: Toddler Robot
edrake, Zhang and Seung, 2005]
[Video: TODDLER 40s
The Crawler!
[Demo: Crawler Bot (L10D1)] [You, in Projec
Video of Demo Crawler Bot
Reinforcement Learning
Still assume a Markov decision process (MDP):
A
A
A
A
set of states s S
set of actions (per state) A
model T(s,a,s)
reward function R(s,a,s)
Still looking for a policy (s)
New twist: dont know T or R
I.e. we dont know which states are good or what the actions do
Must actually try actions and states out to learn
Offline (MDPs) vs. Online (RL)
Offline
Solution
Online
Learning
Model-Based Learning
Model-Based Learning
Model-Based Idea:
Learn an approximate model based on experiences
Solve for values as if the learned model were correct
Step 1: Learn empirical MDP model
Count outcomes s for each s, a
Normalize to give an estimate of
Discover each
when we experience (s, a, s)
Step 2: Solve the learned MDP
For example, use value iteration, as before
Example: Model-Based Learning
Input
Policy
Observed Episodes
(Training)
Episode
Episode
1 C,
2 C,
B, east,
B, east,
E
Assume: = 1
-1
C, east, D,
-1
D,
exit, x,
Episode
+10
3 C, -1
E, north,
C, east, D, -1
D, exit, x,
+10
-1
C, east, D,
-1
D,
exit, x,
Episode
+10
4 C, -1
E, north,
C, east, A, -1
A, exit, x,
-10
Learned
Model
T(s,a,s).
T(B, east, C) =
1.00
T(C, east, D) =
0.75
T(C, east, A) =
0.25
R(s,a,s).
R(B, east, C) =
-1
R(C, east, D) =
-1
Example: Expected Age
Goal: Compute expected age of cs188
students
Known
P(A)
Without P(A), instead collect samples [a1, a2, aN]
Why does this
work?
Because
eventually
you learn the
right model.
Unknown P(A): Model
Based
Unknown P(A): Model
Free
Why does this
work?
Because
samples
appear with
the right
frequencies.
Model-Free Learning
Passive Reinforcement Learning
Passive Reinforcement Learning
Simplified task: policy evaluation
Input: a fixed policy (s)
You dont know the transitions T(s,a,s)
You dont know the rewards R(s,a,s)
Goal: learn the state values
In this case:
Learner is along for the ride
No choice about what actions to take
Just execute the policy and learn from experience
This is NOT offline planning! You actually take actions in the world.
Direct Evaluation
Goal: Compute values for each state
under
Idea: Average together observed
sample values
Act according to
Every time you visit a state, write down
what the sum of discounted rewards
turned out to be
Average those samples
This is called direct evaluation
Example: Direct Evaluation
Input Policy
E
Assume: = 1
Observed Episodes
(Training)
Episode
Episode
1 C,
2 C,
B, east,
B, east,
-1
C, east, D,
-1
D,
exit, x,
Episode
+10
3 C, -1
E, north,
C, east, D, -1
D, exit, x,
+10
-1
C, east, D,
-1
D,
exit, x,
Episode
+10
4 C, -1
E, north,
C, east, A, -1
A, exit, x,
-10
Output Values
-10
A
+8 +4 +1
B C D0
-2
Problems with Direct Evaluation
Whats good about direct evaluation?
Its easy to understand
It doesnt require any knowledge of T, R
It eventually computes the correct
average values, using just sample
transitions
What bad about it?
It wastes information about state
connections
Each state must be learned separately
So, it takes a long time to learn
Output Values
-10
A
+8 +4 +1
B C D0
-2
If B and E both go to
C under this policy,
how can their values
be different?
Why Not Use Policy Evaluation?
Simplified Bellman updates calculate V for a fixed policy: s
Each round, replace V with a one-step-look-ahead layer over V (s)
s, (s)
s,
(s),s
This approach fully exploited the connections between the states
Unfortunately, we need T and R to do it!
Sample-Based Policy Evaluation?
We want to improve our estimate of V by computing these
averages:
Idea: Take samples of outcomes s (by doing the action!) and
s
average
(s)
s, (s)
s, (s),s
's2
's 1
's3
Almost! But we cant
rewind time to get
sample after sample
from state s.
Temporal Difference Learning
Big idea: learn from every experience!
Update V(s) each time we experience a transition (s, a, s, r)
Likely outcomes s will contribute updates more often
Temporal difference learning of values
Policy still fixed, still doing evaluation!
Move values toward value of whatever successor occurs:
running average
Sample of V(s):
Update to V(s):
Same update:
s
(s)
s,
(s)
s
Exponential Moving Average
Exponential moving average
The running interpolation update:
Makes recent samples more important:
Forgets about the past (distant past values were wrong anyway)
Decreasing learning rate (alpha) can give converging
averages
Example: Temporal Difference Learning
States
Observed Transitions
B, east, C,
-2
E
Assume: = 1,
= 1/2
0
0
C, east, D,
-2
0
-1
0
0
0
8
-1
3
0
Problems with TD Value Learning
TD value leaning is a model-free way to do policy
evaluation, mimicking Bellman updates with running
sample averages
However, if we want to turn values into a (new) policy,
s
were sunk:
a
s, a
s,a,s
Idea: learn Q-values, not values
Makes action selection model-free too!
Active Reinforcement Learning
Active Reinforcement Learning
Full reinforcement learning: optimal policies (like
value iteration)
You dont know the transitions T(s,a,s)
You dont know the rewards R(s,a,s)
You choose the actions now
Goal: learn the optimal policy / values
In this case:
Learner makes choices!
Fundamental tradeoff: exploration vs. exploitation
This is NOT offline planning! You actually take actions in the
world and find out what happens
Detour: Q-Value Iteration
Value iteration: find successive (depth-limited) values
Start with V0(s) = 0, which we know is right
Given Vk, calculate the depth k+1 values for all states:
But Q-values are more useful, so compute them instead
Start with Q0(s,a) = 0, which we know is right
Given Qk, calculate the depth k+1 q-values for all q-states:
Q-Learning
Q-Learning: sample-based Q-value
iteration
Learn Q(s,a) values as you go
Receive a sample (s,a,s,r)
Consider your old estimate:
Consider your new sample estimate:
Incorporate the new estimate into a running
average:
[Demo: Q-learning gridworld
(L10D2)]
Video of Demo Q-Learning -- Gridworld
Video of Demo Q-Learning -- Crawler
Q-Learning Properties
Amazing result: Q-learning converges to optimal
policy -- even if youre acting suboptimally!
This is called off-policy learning
Caveats:
You have to explore enough
You have to eventually make the learning rate
small enough
but not decrease it too quickly
Basically, in the limit, it doesnt matter how you select
actions (!)
