Generalization Bounds and Stability

9.520 Class 14, 03 April 2006

Sasha Rakhlin

 Plan

• Generalization Bounds
• Stability
• Generalization Bounds Using Stability
 Algorithms

We define an algorithm A to be a mapping from a training

set S = {z1, . . . , zn} to a function fS . Here, zi ≡ (xi, yi).

Throughout the next several lectures, we assume that A is

deterministic, and that A does not depend on the ordering
of the points in the training set.

How can we measure quality of fS ?

 Risks

Recall that in Lecture 2 we’ve defined the true (expected)

risk:
�
I[fS ] = IE(x,y) [V (fS (x), y)] = V (fS (x), y)dµ(x, y)
and the empirical risk:
n
1 �
IS [fS ] = V (fS (xi), yi).
n i=1

Note: the true and empirical risks are denoted in Bous-

ˆ
quet & Elisseeff as R(A, S) and R(A, S), respectively, to
emphasize the algorithm that produced fS .

Note: we will denote the loss function as V (f, z) or as

V (f (x), y), where z = (x, y).
 Generalization Bounds

Our goal is to choose an algorithm A so that I[fS ] will be

small. This is difficult because we can’t measure I[fS ].

We can, however, measure IS [fS ]. A generalization bound

is a (probabilistic) bound on how big the defect

D[fS ] = I[fS ] − IS [fS ]

can be. If we can bound the defect and we can observe
that IS [fS ] is small, then I[fS ] must be small.
 Properties of Generalization Bounds, I

What will a generalization bound depend on? A gener­

alization bound is a way of saying that the performance
of a function on the training set has to be similar to its
performance on future examples. For this reason, gener­
alization bounds are always probabilistic: they hold with
some (high) probability, to take into account the (low)
chance that you’ll see a very unrepresentative training set.
 Properties of Generalization Bounds, II

Generalization bounds depend on some measure of the size

of the hypothesis space we allow ourselves to choose from.
As the hypothesis space gets smaller, the generalization
bound will get tighter (but the empirical performance will
often go down). As the hypothesis space gets bigger, the
generalization bound will get looser.

The bound will depend on the number of samples we have.

In general, we would like the bounds to get tighter at least
as fast as √1n .
 Properties of Generalization Bounds, III

A good generalization bound will not depend on the prob­

ability distribution P from which the examples are drawn.
If it did, we couldn’t measure it, since P is unknown.
 Generalization Bounds By Bounding the

Hypothesis Space

In 9.520, we discuss two different ways to obtain general­

ization bounds:

One way is to explicitly bound the size of the hypothesis

space H. For example, functions in an RKHS with ||f ||2
K ≤
M form a bounded hypothesis space whose “size” can be
measured and used to obtain generalization bounds (recall
uGC classes of functions).

� �
IPS sup |IS [f ] − I[f ]| > � < δ
f ∈H

This approach will be discussed in future lectures.

 Generalization Bounds By Stability

The other approach is to use stability of algorithms. Here,

the basic idea is that we bound how much the function
produced by an algorithm can change when we modify
the training set slightly. In this class and the next class,
we will explain and develop this approach to generalization
bounds, and show that Tikhonov reguarization in an RKHS
exhibits the necessary stability.

Note that in this approach we are not concerned with

“good performance” of all functions, but only the one
produced by our algorithm:

IPS (|IS [fS ] − I[fS ]| > �) < δ

 Uniform Stability

Given a training set S, we define S i,z to be the new training

set obtained when point i of S is replaced by the new point
z ∈ Z . Given this definition, we say that an algorithm A

has uniform stability β (is β-stable) if

∀(S, z) ∈ Z n+1, ∀i, sup |V (fS , u) − V (fS i,z , u)|

≤ β.
u

An algorithm is β-stable if, for any possible training set, we

can replace an arbitrary training point with any other pos­
sible training point, and the loss at any point will change
by no more than β.
 Uniform Stability Cont’d

Uniform stability is a strong requirement, because it ig­

nores the fact that the points are drawn from a probability
distribution. For uniform stability, the function still has
to change very little even when a very unlikely (“bad”)
training set is drawn.

In general, the stability β is a function of n, and should

perhaps be written βn.
 Stability and Concentration Inequalities

Question: Given that an algorithm A has stability β, how

can we get bounds on its performance?

Answer: Concentration Inequalities. In particular, we will

use McDiarmid’s Inequality.

Concentration Inequalities show how a variable is concen­

trated around its mean.

Michel Talagrand:

A random variable that depends (in a “smooth” way) on

the influence of many independent variables (but not too
much on any of them) is essentially constant.
 McDiarmid’s Inequality

Given random variables v1, . . . , vn, and a function F : v n →

IR satisfying

sup |F (v1, . . . , vn) − F (v1, . . . , vi−1, vi� , vi+1, . . . , vn)| ≤ ci,

v1 ,...,vn ,vi�

the following statement holds:

� �
2�2
IP (|F (v1, . . . , vn) − IES (F (v1, . . . , vn))| > �) ≤ 2 exp − �n 2
.
i=1 ci

This is an example of the law of large numbers.

 Example: Hoeffding’s Inequality

Suppose each vi ∈ [a, b], and we define F (v1, . . . , vn) =

1 �n v , the average of the v . Then, c = 1 (b − a).
n i=1 i i i n
Applying McDiarmid’s Inequality, we have that
�
�

2�2

IP (|F (v) − IE(F (v))| > �) ≤ 2 exp − �n 2

⎛ i=1 ci ⎞
2�2
= 2 exp
⎝
− �n 1 (b − a))2
⎠

(
i=1 n
�
�
2n�2

= 2 exp − .
(b − a)2

We have easily recovered the famous “Hoeffding’s Inequal­

ity”. (Of course, we did not prove McDiarmid’s Inequality.)

 Generalization Bounds via McDiarmid’s

Inequality

We will use β-stability to apply McDiarmid’s inequality to

the defect D[fS ] = I[fS ] − IS [fS ]. To do this, we will need
two things:

1. the expectation of the defect (we can’t measure it, but

we can bound its expectation) and

2. a bound on how much the defect can change when we

replace a point.

In order to bound the deviation (the second quantity), we

require that there exist an upper bound M on the loss.
 Bounding The Expectation of The Defect

IES D[fS ]
=
IES [IS [fS ] − I[fS ]]
⎡ ⎤
n
�
1 ⎣
= IES,z V (fS (xi), yi) − V (fS (x), y)⎦
n i=1
⎡ ⎤
n
1
�

= IES,z V (fS i,z (x), y) − V (fS (x), y)

⎦

n i=1
≤ β

The second equality follows by exploiting the “symmetry”

of expectation: The expected value of a training set on
a training point doesn’t change when we “rename” the
points.
 Bounding The Deviation of The Defect

|D[fS ] − D[fS i,z ]| = |IS [fS ] − I[fS ] − IS i,z [fS i,z ] + I[fS i,z ]|
≤ |I[fS ] − I[fS i,z ]| + |IS [fS ] − IS i,z [fS i,z ]|
1
≤ β + |V (fS (xi), yi) − V (fS i,z (x), y)|
n
1
�
+ |V (fS (xj ), yj ) − V (fS i,z (xj ), yj )|
n j=i
�
M
≤ β+ + β

n
M
= 2β +
n
 Applying McDiarmid’s Inequality

By McDiarmid’s Inequality, for any �,

⎛ ⎞
2�2

IP (|D[fS ] − IED[fS ]| > �) ≤ 2 exp ⎝

− �n ⎠=
i=1 (2(β +M
n )) 2
⎛ ⎞ �
�

�2 n�2
= 2 exp ⎝
− M )2
⎠
= 2 exp
−
2n(β + n
2(nβ + M )2

Note that
IP(D[fS ] > β + �) = IP(D[fS ] − IED[fS ] > �)
≤ IP(|D[fS ] − IED[fS ]| > �)

Hence,
� �

n�2

IP(IS [fS ] − I[fS ] > β + �) ≤ 2 exp −

2(nβ + M )2
 A Different Form Of The Bound

If we define
� �
n�2
δ ≡ 2 exp − .

2(nβ + M )2

Solving for � in terms of δ, we find that

�
2 ln(2/δ)
� = (nβ + M ) .

By varying δ (and �), we can say that for any δ ∈ (0, 1),
with probability 1 − δ,
�
2 ln(2/δ)
I[fS ] ≤ IS [fS ] + β + (nβ + M ) .
n
 Fast Convergence

k for some k, we can restate our bounds

Note that if β = n
as
� � � �
k n�2
P |I[fS ] − IS [fS ]| ≥ + � ≤ 2 exp − 2
,
n 2(k + M )
and with probability 1 − δ,
�
k 2 ln(2/δ)
I[fS ] ≤ IS [fS ] + + (2k + M ) .
n n
 Fast Convergence, Cont’d

For the uniform stability approach we’ve described, β = n k

(for some constant k) is “good enough”. Obviously, the

best possible stability would be β = 0 — the function
can’t change at all when you change the training set. An
algorithm that always picks the same function, regardless
of its training set, is maximally stable and has β = 0. Using
β = 0 in the last bound, with probability 1 − δ,
�
2 ln(2/δ)
I[fS ] ≤ IS [fS ] + M .
n
� �
The convergence is still O √1n . So once β = O( n 1 ), further

increases in stability don’t change the rate of convergence.

 Other kinds of stabilities

Notation: ∀δ means “for all except a set of measure δ.

An algorithm A : Z n → F is

uniformly β hypothesis stable:

∀i, (S, u) ∈ Z n+1, sup{|V (fS , z) − V (fS i,u , z)|} ≤ β.

z∈Z
(β, δ) leave-one-out stable:
� �
δ � �
∀ S, ∀i, �
V (fS , zi) − V (fS i , zi)�
≤ β.

(β, δ) error stable:

� �
δ � �
∀ (S, u), ∀i, �
I[fS ] − I[fS i,u ]�
≤ β.
(β, δ) cross-validation stable:
�
�

� �
∀
δ S ∈ Z n, ∀i, u ∈ Z, �V (fS , u) − V (fS i,u , u)� ≤ β.
 Thoughts on stability and open questions

Stability is a new research area – still things to be done.

Good generalization bounds can be proved for specific al­

gorithms if certain types of stabilities can be shown.

There might be a way to � �apply other concentration in­

1 convergence.
equalities to get faster O n
 Summary

We used McDiarmid’s inequality to prove a generalization

bound for a uniformly β-stable algorithm. Note that this
bound cannot tell us that the expected error will be low
a priori, it can only tell us that with high probability, the
expected error will be close to the empirical error. We have
to actually observe a low empirical error to conclude that
we have a low expected error.
� �
1 seems to be a strong require-
Uniform stability of O n
ment. Next time, we will show that Tikhonov regulariza­
tion possesses this property.