Artificial

Intelligence
Lecture 11 – Reinforcement Learning II
Dr. Shivanjali Khare
[email protected]
Reinforcement Learning
• We still assume an MDP:
• A set of states s  S
• A set of actions (per state) A
• A model T(s,a,s’)
• A reward function R(s,a,s’)
• Still looking for a policy (s)

• New twist: don’t know T or R, so must try out actions

• Big idea: Compute all averages over T using sample outcomes

The Story So Far: MDPs and RL

Known MDP: Offline Solution

Goal Technique
Compute V*, Q*, * Value / policy iteration

Evaluate a fixed policy  Policy evaluation

Unknown MDP: Model-Based Unknown MDP: Model-Free

Goal Technique Goal Technique
Compute V*, Q*, * VI/PI on approx. MDP Compute V*, Q*, * Q-learning

Evaluate a fixed policy  PE on approx. MDP Evaluate a fixed policy  Value Learning
 Analogy: Expected Age

Goal: Compute expected age of cs188 students

Known P(A)

Without P(A), instead collect samples [a1, a2, … aN]

Unknown P(A): “Model Based” Unknown P(A): “Model Free”

Why does this Why does this

work? Because work? Because
eventually you samples appear
learn the right with the right
model. frequencies.
Sample-Based Policy
Evaluation?
• We want to improve our estimate of V by computing these averages:

• Idea: Take samples of outcomes s’ (by doing the action!) and average
s
(s)
s,
(s)
s, (s),s’
's2 's 1 's3
'
Almost! But we
can’t rewind time to
get sample after
sample from state s.
Model-Free Learning

s
• Model-free (temporal difference) a
learning s, a
• Experience world through episodes r
’s
a’
• Update estimates each transition s’, a’

’’s
• Over time, updates will mimic Bellman
updates
Temporal Difference
Learning

s
• Temporal difference learning of values (s)
• Policy still fixed, still doing evaluation!
• Move values toward value of whatever successor occurs: running
s,
average (s)
’s

Sample of V(s):

Update to V(s):
Example: Temporal Difference
Learning

States Observed Transitions

B, east, C, - C, east, D, -
2 2

A 0 0 0

B C D 0 0 8 -1 0 8 -1 3 8

E 0 0 0

Assume:  = 1, α =
1/2
Q-Learning

• Q-Learning: sample-based Q-value iteration

• Learn Q(s,a) values as you go

• Receive a sample (s,a,s’,r)
• Consider your old estimate:
• Consider your new sample estimate:
no longer
policy
• Incorporate the new estimate into a running average:
evaluation!

[Demo: Q-learning – gridworld (L10D2)]

[Demo: Q-learning – crawler (L10D3)]
Video of Demo Q-Learning -- Gridworld
Video of Demo Q-Learning -- Crawler
 Q-Learning Properties

• Amazing result: Q-learning converges to optimal policy -- even if

you’re acting suboptimally!

• This is called off-policy learning

• Caveats:
• You have to explore enough
• You have to eventually make the learning rate
small enough
• … but not decrease it too quickly
• Basically, in the limit, it doesn’t matter how you select actions (!)

[Demo: Q-learning – auto – cliff grid (L11D1)]

Active Reinforcement
Learning
 Model-Free Learning

• act according to current optimal (based on Q-Values)

• but also explore…
Model-Based Learning

Input Policy 

A
act according to current optimal
B C D also explore!

E
Exploration vs.
Exploitation
Video of Demo Q-learning – Manual
Exploration – Bridge Grid
How to Explore?

• Several schemes for forcing exploration

• Simplest: random actions (-greedy)
• Every time step, flip a coin
• With (small) probability , act randomly
• With (large) probability 1-, act on current policy

• Problems with random actions?

• You do eventually explore the space, but keep
thrashing around once learning is done
• One solution: lower  over time
• Another solution: exploration functions
[Demo: Q-learning – manual exploration – bridge grid (L11D2)]
[Demo: Q-learning – epsilon-greedy -- crawler (L11D3)]
Video of Demo Q-learning – Epsilon-
Greedy – Crawler
Exploration Functions

• When to explore?
• Random actions: explore a fixed amount
• Better idea: explore areas whose badness is not
(yet) established, eventually stop exploring

• Exploration function
• Takes a value estimate u and a visit count n, and
returns an optimistic utility, e.g.
Regular Q-Update:
Modified Q-Update:

• Note: this propagates the “bonus” back to states that lead to unknown states as well!

[Demo: exploration – Q-learning – crawler – exploration function (L11D4)]

Video of Demo Q-learning – Exploration
Function – Crawler
Regret

• Even if you learn the optimal policy,

you still make mistakes along the way!
• Regret is a measure of your total
mistake cost: the difference between
your (expected) rewards, including
youthful suboptimality, and optimal
(expected) rewards
• Minimizing regret goes beyond
learning to be optimal – it requires
optimally learning to be optimal
• Example: random exploration and
exploration functions both end up
optimal, but random exploration has
higher regret
Approximate Q-Learning
Generalizing Across
States

• Basic Q-Learning keeps a table of all q-values

• In realistic situations, we cannot possibly learn

about every single state!
• Too many states to visit them all in training
• Too many states to hold the q-tables in memory

• Instead, we want to generalize:

• Learn about some small number of training states from
experience
• Generalize that experience to new, similar situations
• This is a fundamental idea in machine learning, and
we’ll see it over and over again

[demo – RL pacman]
Example: Pacman

Let’s say we discover In naïve q-learning, Or even this one!

through experience we know nothing
that this state is bad: about this state:

[Demo: Q-learning – pacman – tiny – watch all (L11D5)]

[Demo: Q-learning – pacman – tiny – silent train (L11D6)]
[Demo: Q-learning – pacman – tricky – watch all (L11D7)]
Video of Demo Q-Learning Pacman –
Tiny – Watch All
Video of Demo Q-Learning Pacman –
Tiny – Silent Train
Video of Demo Q-Learning Pacman –
Tricky – Watch All
Feature-Based
Representations

• Solution: describe a state using a vector of

features (properties)
• Features are functions from states to real numbers
(often 0/1) that capture important properties of the
state
• Example features:
• Distance to closest ghost
• Distance to closest dot
• Number of ghosts
• 1 / (dist to dot)2
• Is Pacman in a tunnel? (0/1)
• …… etc.
• Is it the exact state on this slide?
• Can also describe a q-state (s, a) with features (e.g.
action moves closer to food)
Linear Value Functions

• Using a feature representation, we can write a q function (or value function) for any
state using a few weights:

• Advantage: our experience is summed up in a few powerful numbers

• Disadvantage: states may share features but actually be very different in value!
Approximate Q-Learning

• Q-learning with linear Q-functions:

Exact Q’s

Approximate Q’s

• Intuitive interpretation:
• Adjust weights of active features
• E.g., if something unexpectedly bad happens, blame the features that were on: disprefer all
states with that state’s features

• Formal justification: online least squares

Example: Q-Pacman

[Demo: approximate Q-
learning pacman
(L11D10)]
Video of Demo Approximate Q-
Learning -- Pacman
 DeepMind Atari () approximate Q-
learning with neural nets

36
Q-Learning and Least
Squares
Linear Approximation:
Regression

40

26

24
20
22

20

30
40
0 20
30
0 20
10 20
10
0 0

Prediction: Prediction:
Optimization: Least
Squares

Error or “residual”
Observation

Prediction

0
0 20
Minimizing Error

Imagine we had only one point x, with features f(x), target value y, and weights w:

Approximate q update explained:

“target” “prediction”
Overfitting: Why Limiting Capacity
Can Help

30

25

20
Degree 15 polynomial
15

10

-5

-10

-15
0 2 4 6 8 10 12 14 16 18 20
Policy Search
Policy Search
• Problem: often the feature-based policies that work well (win games, maximize
utilities) aren’t the ones that approximate V / Q best
• E.g. your value functions from project 2 were probably horrible estimates of future rewards, but
they still produced good decisions
• Q-learning’s priority: get Q-values close (modeling)
• Action selection priority: get ordering of Q-values right (prediction)
• We’ll see this distinction between modeling and prediction again later in the course

• Solution: learn policies that maximize rewards, not the values that predict them

• Policy search: start with an ok solution (e.g. Q-learning) then fine-tune by hill climbing
on feature weights
Policy Search
• Simplest policy search:
• Start with an initial linear value function or Q-function
• Nudge each feature weight up and down and see if your policy is better than
before

• Problems:
• How do we tell the policy got better?
• Need to run many sample episodes!
• If there are a lot of features, this can be impractical

• Better methods exploit lookahead structure, sample wisely, change

multiple parameters…
The Story So Far: MDPs and RL

Known MDP: Offline Solution

Goal Technique
Compute V*, Q*, * Value / policy iteration

Evaluate a fixed policy  Policy evaluation

Unknown MDP: Model-Based Unknown MDP: Model-Free

*use features *use features
Goal to generalize Technique Goal to generalize Technique
Compute V*, Q*, * VI/PI on approx. MDP Compute V*, Q*, * Q-learning

Evaluate a fixed policy  PE on approx. MDP Evaluate a fixed policy  Value Learning
 Discussion: Model-Based vs Model-
Free RL

47
 RL: Helicopter Flight

[Andrew Ng] [Video: HELICOPTER]

 RL: Learning Locomotion

[Schulman, Moritz, Levine, Jordan, Abbeel, ICLR 2016] [Video: GAE]

 RL: Learning Soccer

[Bansal et al, 2017]

 RL: Learning
Manipulation

[Levine*, Finn*, Darrell, Abbeel, JMLR 2016]

 RL: NASA SUPERball

Pieter Abbeel -- UC Berkeley | Gradescope |

[Geng*, Zhang*, Bruce*, Caluwaerts, Vespignani, Sunspiral, Abbeel, Levine, ICRA 2017] Covariant.AI
RL: In-Hand
Manipulation

Pieter Abbeel -- UC Berkeley | Gradescope |

Covariant.AI
 OpenAI: Dactyl

Trained with domain randomization

[OpenAI]
Conclusion

• We’re done with Part I: Search and Planning!

• We’ve seen how AI methods can solve

problems in:
• Search
• Constraint Satisfaction Problems
• Games
• Markov Decision Problems
• Reinforcement Learning

• Next up: Part II: Uncertainty and Learning!