Convex Optimization Theory

A SUMMARY

BY

DIMITRI P. BERTSEKAS

We provide a summary of theoretical concepts and results relating to convex

analysis, convex optimization, and duality theory. In particular, we list the
relevant definitions and propositions (without proofs) of the author’s book
“Convex Optimization Theory,” Athena Scientific, 2009. For ease of use,
the chapter, section, definition, and proposition numbers of the latter book
are identical to the ones of this appendix.

CHAPTER 1: Basic Concepts of Convex Analysis

Section 1.1. Convex Sets and Functions

Definition 1.1.1: A subset C of ℜn is called convex if

αx + (1 − α)y ∈ C, ∀ x, y ∈ C, ∀ α ∈ [0, 1].

Proposition 1.1.1:
(a) The intersection ∩i∈I Ci of any collection {Ci | i ∈ I} of convex
sets is convex.
(b) The vector sum C1 + C2 of two convex sets C1 and C2 is convex.
(c) The set λC is convex for any convex set C and scalar λ. Fur-
thermore, if C is a convex set and λ1 , λ2 are positive scalars,

467
468 Chap. 1

(λ1 + λ2 )C = λ1 C + λ2 C.

(d) The closure and the interior of a convex set are convex.
(e) The image and the inverse image of a convex set under an affine
function are convex.

A hyperplane is a set of the form {x | a′ x = b}, where a is a nonzero

vector and b is a scalar. A halfspace is a set specified by a single linear
inequality, i.e., a set of the form {x | a′ x ≤ b}, where a is a nonzero vector
and b is a scalar. A set is said to be polyhedral if it is nonempty and it has
the form {x | a′j x ≤ bj , j = 1, . . . , r}, where a1 , . . . , ar and b1 , . . . , br are
some vectors in ℜn and scalars, respectively. A set C is said to be a cone
if for all x ∈ C and λ > 0, we have λx ∈ C.

Definition 1.1.2: Let C be a convex subset of ℜn . We say that a

function f : C 7→ ℜ is convex if


f αx + (1 − α)y ≤ αf (x) + (1 − α)f (y), ∀ x, y ∈ C, ∀ α ∈ [0, 1].

A convex function f : C 7→ ℜ is called strictly convex if

f αx + (1 − α)y < αf (x) + (1 − α)f (y)
for all x, y ∈ C with x 6= y, and all α ∈ (0, 1). A function f : C 7→ ℜ, where
C is a convex set, is called concave if the function (−f ) is convex.
The epigraph of a function f : X 7→ [−∞, ∞], where X ⊂ ℜn , is
defined to be the subset of ℜn+1 given by

epi(f ) = (x, w) | x ∈ X, w ∈ ℜ, f (x) ≤ w .
The effective domain of f is defined to be the set

dom(f ) = x ∈ X | f (x) < ∞ .
We say that f is proper if f (x) < ∞ for at least one x ∈ X and f (x) > −∞
for all x ∈ X, and we say that f improper if it is not proper. Thus f is
proper if and only if epi(f ) is nonempty and does not contain a vertical
line.

Definition 1.1.3: Let C be a convex subset of ℜn . We say that an

extended real-valued function f : C 7→ [−∞, ∞] is convex if epi(f ) is
a convex subset of ℜn+1 .
Sec. B.1.1 Convex Sets and Functions 469

Definition 1.1.4: Let C and X be subsets of ℜn such that C is

nonempty and convex, and C ⊂ X. We say that an extended real-
valued function f : X 7→ [−∞, ∞] is convex over C if f becomes
convex when the domain of f is restricted to C, i.e., if the function
f˜ : C 7→ [−∞, ∞], defined by f˜(x) = f (x) for all x ∈ C, is convex.

We say that a function f : X 7→ [−∞, ∞] is closed if epi(f ) is a

closed set. We say that f is lower semicontinuous at a vector x ∈ X if
f (x) ≤ lim inf k→∞ f (xk ) for every sequence {xk } ⊂ X with xk → x. We
say that f is lower semicontinuous if it is lower semicontinuous at each
point x in its domain X. We say that f is upper semicontinuous if −f is
lower semicontinuous.

Proposition 1.1.2: For a function f : ℜn 7→ [−∞, ∞], the following

are equivalent:

(i) The level set Vγ = x | f (x) ≤ γ is closed for every scalar γ.
(ii) f is lower semicontinuous.
(iii) epi(f ) is closed.

Proposition 1.1.3: Let f : X 7→ [−∞, ∞] be a function. If dom(f )

is closed and f is lower semicontinuous at each x ∈ dom(f ), then f is
closed.

Proposition 1.1.4: Let f : ℜm 7→ (−∞, ∞] be a given function, let

A be an m × n matrix, and let F : ℜn 7→ (−∞, ∞] be the function

F (x) = f (Ax), x ∈ ℜn .

If f is convex, then F is also convex, while if f is closed, then F is

also closed.

Proposition 1.1.5: Let fi : ℜn 7→ (−∞, ∞], i = 1, . . . , m, be given

functions, let γ1 , . . . , γm be positive scalars, and let F : ℜn 7→ (−∞, ∞]
470 Chap. 1

be the function

F (x) = γ1 f1 (x) + · · · + γm fm (x), x ∈ ℜn .

If f1 , . . . , fm are convex, then F is also convex, while if f1 , . . . , fm are

closed, then F is also closed.

Proposition 1.1.6: Let fi : ℜn 7→ (−∞, ∞] be given functions for

i ∈ I, where I is an arbitrary index set, and let f : ℜn 7→ (−∞, ∞] be
the function given by
f (x) = sup fi (x).
i∈I

If fi , i ∈ I, are convex, then f is also convex, while if fi , i ∈ I, are

closed, then f is also closed.

Proposition 1.1.7: Let C be a nonempty convex subset of ℜn and

let f : ℜn 7→ ℜ be differentiable over an open set that contains C.
(a) f is convex over C if and only if

f (z) ≥ f (x) + ∇f (x)′ (z − x), ∀ x, z ∈ C.

(b) f is strictly convex over C if and only if the above inequality is

strict whenever x 6= z.

Proposition 1.1.8: Let C be a nonempty convex subset of ℜn and

let f : ℜn 7→ ℜ be convex and differentiable over an open set that
contains C. Then a vector x∗ ∈ C minimizes f over C if and only if

∇f (x∗ )′ (z − x∗ ) ≥ 0, ∀ z ∈ C.

When f is not convex but is differentiable over an open set that

contains C, the condition of the above proposition is necessary but not
sufficient for optimality of x∗ (see e.g., [Ber99], Section 2.1).
Sec. B.1.1 Convex Sets and Functions 471

Proposition 1.1.9: (Projection Theorem) Let C be a nonempty

closed convex subset of ℜn , and let z be a vector in ℜn . There exists a
unique vector that minimizes kz − xk over x ∈ C, called the projection
of z on C. Furthermore, a vector x∗ is the projection of z on C if and
only if
(z − x∗ )′ (x − x∗ ) ≤ 0, ∀ x ∈ C.

Proposition 1.1.10: Let C be a nonempty convex subset of ℜn and

let f : ℜn 7→ ℜ be twice continuously differentiable over an open set
that contains C.
(a) If ∇2 f (x) is positive semidefinite for all x ∈ C, then f is convex
over C.
(b) If ∇2 f (x) is positive definite for all x ∈ C, then f is strictly
convex over C.
(c) If C is open and f is convex over C, then ∇2 f (x) is positive
semidefinite for all x ∈ C.

Strong Convexity

If f : ℜn 7→ ℜ is a function that is continuous over a closed convex set

C ⊂ ℜn , and σ is a positive scalar, we say that f is strongly convex over
C with coefficient σ if for all x, y ∈ C and all α ∈ [0, 1], we have

σ
f αx + (1 − α)y + α(1 − α)kx − yk2 ≤ αf (x) + (1 − α)f (y).
2

Then f is strictly convex over C. Furthermore, there exists a unique x∗ ∈ C

that minimizes f over C, and by applying the definition with y = x∗ and
letting α ↓ 0, it can be seen that
σ
f (x) ≥ f (x∗ ) + kx − x∗ k2 , ∀ x ∈ C.
2

If int(C), the interior of C, is nonempty, and f is continuously differentiable

over int(C), the following are equivalent:
(i) f is strongly convex with coefficient σ over C.

′
(ii) ∇f (x) − ∇f (y) (x − y) ≥ σkx − yk2 , ∀ x, y ∈ int(C).
(iii) f (y) ≥ f (x) + ∇f (x)′ (y − x) + σ2 kx − yk2 , ∀ x, y ∈ int(C).
472 Chap. 1

Furthermore, if f is twice continuously differentiable over int(C), the above

three properties are equivalent to:
(iv) The matrix ∇2 f (x) − σI is positive semidefinite for every x ∈ int(C),
where I is the identity matrix.
A proof may be found in the on-line exercises of Chapter 1 of [Ber09].

Section 1.2. Convex and Affine Hulls

The convex hull of a set X, denoted conv(X), is the intersection of all

convex sets containing
PmX. A convex combination of elements of X is a
vector of the form i=1 αi xi , where m is a positive integer, x1 , . . . , xm
belong to X, and α1 , . . . , αm are scalars such that
Xm
αi ≥ 0, i = 1, . . . , m, αi = 1.
i=1
The convex hull conv(X) is equal to the set of all convex combinations of
elements of X. Also, for any set S and linear transformation A, we have
conv(AS) = A conv(S). From this it follows that for any sets S1 , . . . , Sm ,
we have conv(S1 + · · · + Sm ) = conv(S1 ) + · · · + conv(Sm ).
If X is a subset of ℜn , the affine hull of X, denoted aff(X), is the
intersection of all affine sets containing X. Note that aff(X) is itself an
affine set and that it contains conv(X). The dimension of aff(X) is defined
to be the dimension of
the subspace
parallel to aff(X). It can be shown that
aff(X) = aff conv(X) = aff cl(X) . For a convex set C, the dimension of
C is defined to be the dimension of aff(C).
Given a nonempty subset X of ℜP n , a nonnegative combination of
m
elements of X is a vector of the form i=1 αi xi , where m is a positive
integer, x1 , . . . , xm belong to P
X, and α1 , . . . , αm are nonnegative scalars. If
m
the scalars αi are all positive, i=1 αi xi is said to be a positive combination.
The cone generated by X, denoted cone(X), is the set of all nonnegative
combinations of elements of X.

Proposition 1.2.1: (Caratheodory’s Theorem) Let X be a non-

empty subset of ℜn .
(a) Every nonzero vector from cone(X) can be represented as a pos-
itive combination of linearly independent vectors from X.
(b) Every vector from conv(X) can be represented as a convex com-
bination of no more than n + 1 vectors from X.

Proposition 1.2.2: The convex hull of a compact set is compact.

Sec. B.1.3 Relative Interior and Closure 473

Section 1.3. Relative Interior and Closure

Let C be a nonempty convex set. We say that x is a relative interior point

of C if x ∈ C and there exists an open sphere S centered at x such that

S ∩ aff(C) ⊂ C,

i.e., x is an interior point of C relative to the affine hull of C. The set

of relative interior points of C is called the relative interior of C, and is
denoted by ri(C). The set C is said to be relatively open if ri(C) = C. The
vectors in cl(C) that are not relative interior points are said to be relative
boundary points of C, and their collection is called the relative boundary of
C.

Proposition 1.3.1: (Line Segment Principle) Let C be a nonempty

convex set. If x ∈ ri(C) and x ∈ cl(C), then all points on the line seg-
ment connecting x and x, except possibly x, belong to ri(C).

Proposition 1.3.2: (Nonemptiness of Relative Interior) Let C

be a nonempty convex set. Then:
(a) ri(C) is a nonempty convex set, and has the same affine hull as
C.
(b) If m is the dimension of aff(C) and m > 0, there exist vectors
x0 , x1 , . . . , xm ∈ ri(C) such that x1 − x0 , . . . , xm − x0 span the
subspace parallel to aff(C).

Proposition 1.3.3: (Prolongation Lemma) Let C be a nonempty

convex set. A vector x is a relative interior point of C if and only if
every line segment in C having x as one endpoint can be prolonged
beyond x without leaving C [i.e., for every x ∈ C, there exists a γ > 0
such that x + γ(x − x) ∈ C].

Proposition 1.3.4: Let X be a nonempty convex subset of ℜn , let

f : X 7→ ℜ be a concave function, and let X ∗ be the set of vectors
where f attains a minimum over X, i.e.,
474 Chap. 1

 
X∗ = x∗ ∈ X f (x∗ ) = inf f (x) .
x∈X

If X ∗ contains a relative interior point of X, then f must be constant

over X, i.e., X ∗ = X.

Proposition 1.3.5: Let C be a nonempty convex set.

(a) We have cl(C) = cl ri(C) .


(b) We have ri(C) = ri cl(C) .
(c) Let C be another nonempty convex set. Then the following three
conditions are equivalent:
(i) C and C have the same relative interior.
(ii) C and C have the same closure.
(iii) ri(C) ⊂ C ⊂ cl(C).

Proposition 1.3.6: Let C be a nonempty convex subset of ℜn and

let A be an m × n matrix.
(a) We have A · ri(C) = ri(A · C).
(b) We have A · cl(C) ⊂ cl(A · C). Furthermore, if C is bounded,
then A · cl(C) = cl(A · C).

Proposition 1.3.7: Let C1 and C2 be nonempty convex sets. We

have

ri(C1 + C2 ) = ri(C1 ) + ri(C2 ), cl(C1 ) + cl(C2 ) ⊂ cl(C1 + C2 ).

Furthermore, if at least one of the sets C1 and C2 is bounded, then

cl(C1 ) + cl(C2 ) = cl(C1 + C2 ).

Sec. B.1.3 Relative Interior and Closure 475

Proposition 1.3.8: Let C1 and C2 be nonempty convex sets. We

have

ri(C1 ) ∩ ri(C2 ) ⊂ ri(C1 ∩ C2 ), cl(C1 ∩ C2 ) ⊂ cl(C1 ) ∩ cl(C2 ).

Furthermore, if the sets ri(C1 ) and ri(C2 ) have a nonempty intersec-

tion, then

ri(C1 ∩ C2 ) = ri(C1 ) ∩ ri(C2 ), cl(C1 ∩ C2 ) = cl(C1 ) ∩ cl(C2 ).

Proposition 1.3.9: Let C be a nonempty convex subset of ℜm , and

let A be an m × n matrix. If A−1 · ri(C) is nonempty, then

ri(A−1 · C) = A−1 · ri(C), cl(A−1 · C) = A−1 · cl(C),

where A−1 denotes inverse image of the corresponding set under A.

Proposition 1.3.10: Let C be a convex subset of ℜn+m . For x ∈ ℜn ,

denote
Cx = {y | (x, y) ∈ C},
and let
D = {x | Cx 6= Ø}.
Then 
ri(C) = (x, y) | x ∈ ri(D), y ∈ ri(Cx ) .

Continuity of Convex Functions

Proposition 1.3.11: If f : ℜn 7→ ℜ is convex, then it is continuous.

More generally, if f : ℜn 7→ (−∞, ∞] is a proper convex function,
then f , restricted to dom(f ), is continuous over the relative interior of
dom(f ).
476 Chap. 1

Proposition 1.3.12: If C is a closed interval of the real line, and

f : C 7→ ℜ is closed and convex, then f is continuous over C.

Closures of Functions
The closure of the epigraph of a function f : X 7→ [−∞, ∞] can be seen
to be a legitimate epigraph of another function. This function, called the
closure of f and denoted cl f : ℜn 7→ [−∞, ∞], is given by



(cl f )(x) = inf w | (x, w) ∈ cl epi(f ) , x ∈ ℜn .

The closure of the convex hull of the epigraph of f is the epigraph of some
function, denoted cl ˇ f called the convex closure of f . It can be seen that
ˇ
cl f is the closure of the function F : ℜn 7→ [−∞, ∞] given by



F (x) = inf w | (x, w) ∈ conv epi(f ) , x ∈ ℜn . (B.1)

It is easily shown that F is convex, but it need not be closed and its
ˇ f ) (it can be seen though that
domain may be strictly contained in dom(cl
ˇ f coincide).
the closures of the domains of F and cl

Proposition 1.3.13: Let f : X 7→ [−∞, ∞] be a function. Then

ˇ f )(x),
inf f (x) = inf (cl f )(x) = infn (cl f )(x) = infn F (x) = infn (cl
x∈X x∈X x∈ℜ x∈ℜ x∈ℜ

where F is given by Eq. (B.1). Furthermore, any vector that attains

ˇ f.
the infimum of f over X also attains the infimum of cl f , F , and cl

Proposition 1.3.14: Let f : ℜn 7→ [−∞, ∞] be a function.

(a) cl f is the greatest closed function majorized by f , i.e., if g :
ℜn 7→ [−∞, ∞] is closed and satisfies g(x) ≤ f (x) for all x ∈ ℜn ,
then g(x) ≤ (cl f )(x) for all x ∈ ℜn .
ˇ f is the greatest closed and convex function majorized by f ,
(b) cl
i.e., if g : ℜn 7→ [−∞, ∞] is closed and convex, and satisfies
ˇ f )(x) for all x ∈ ℜn .
g(x) ≤ f (x) for all x ∈ ℜn , then g(x) ≤ (cl
Sec. B.1.3 Relative Interior and Closure 477

Proposition 1.3.15: Let f : ℜn 7→ [−∞, ∞] be a convex function.

Then:
(a) We have





cl dom(f ) = cl dom(cl f ) , ri dom(f ) = ri dom(cl f ) ,


(cl f )(x) = f (x), ∀ x ∈ ri dom(f ) .
Furthermore, cl f is proper if and only if f is proper.


(b) If x ∈ ri dom(f ) , we have


(cl f )(y) = lim f y + α(x − y) , ∀ y ∈ ℜn .
α↓0

Proposition 1.3.16: Let f : ℜm 7→ [−∞, ∞] be a convex function

and A be an
m × n matrix such that the range of A contains a point
in ri dom(f ) . The function F defined by

F (x) = f (Ax),

is convex and

(cl F )(x) = (cl f )(Ax), ∀ x ∈ ℜn .

Proposition 1.3.17: Let fi : ℜn 7→ [−∞, ∞], i = 1, . . . , m, be con-

vex functions such that


∩m
i=1 ri dom(fi ) 6= Ø.

The function F defined by

F (x) = f1 (x) + · · · + fm (x),

478 Chap. 1

is convex and

(cl F )(x) = (cl f1 )(x) + · · · + (cl fm )(x), ∀ x ∈ ℜn .

Section 1.4. Recession Cones

Given a nonempty convex set C, we say that a vector d is a direction of

recession of C if x + αd ∈ C for all x ∈ C and α ≥ 0. The set of all
directions of recession is a cone containing the origin, called the recession
cone of C, and denoted by RC .

Proposition 1.4.1: (Recession Cone Theorem) Let C be a nonem-

pty closed convex set.
(a) The recession cone RC is closed and convex.
(b) A vector d belongs to RC if and only if there exists a vector
x ∈ C such that x + αd ∈ C for all α ≥ 0.

Proposition 1.4.2: (Properties of Recession Cones) Let C be

a nonempty closed convex set.
(a) RC contains a nonzero direction if and only if C is unbounded.
(b) RC = Rri(C) .
(c) For any collection of closed convex sets Ci , i ∈ I, where I is an
arbitrary index set and ∩i∈I Ci 6= Ø, we have

R∩i∈I Ci = ∩i∈I RCi .

(d) Let W be a compact and convex subset of ℜm , and let A be an

m × n matrix. The recession cone of the set

V = {x ∈ C | Ax ∈ W }

(assuming this set is nonempty) is RC ∩ N (A), where N (A) is

the nullspace of A.
Sec. B.1.4 Recession Cones 479

Given a convex set C the lineality space of C, denoted by LC , is the

set of directions of recession d whose opposite, −d, are also directions of
recession:
LC = RC ∩ (−RC ).

Proposition 1.4.3: (Properties of Lineality Space) Let C be a

nonempty closed convex subset of ℜn .
(a) LC is a subspace of ℜn .
(b) LC = Lri(C) .
(c) For any collection of closed convex sets Ci , i ∈ I, where I is an
arbitrary index set and ∩i∈I Ci 6= Ø, we have

L∩i∈I Ci = ∩i∈I LCi .

(d) Let W be a compact and convex subset of ℜm , and let A be an

m × n matrix. The lineality space of the set

V = {x ∈ C | Ax ∈ W }

(assuming it is nonempty) is LC ∩ N (A), where N (A) is the

nullspace of A.

Proposition 1.4.4: (Decomposition of a Convex Set) Let C be

a nonempty convex subset of ℜn . Then, for every subspace S that is
contained in the lineality space LC , we have

C = S + (C ∩ S ⊥ ).

The notion of direction of recession of a convex function f can be

described in terms of its epigraph via the following proposition.

Proposition 1.4.5: Let f : ℜn 7→ (−∞, ∞] be a closed proper convex

function and consider the level sets

Vγ = x | f (x) ≤ γ , γ ∈ ℜ.
480 Chap. 1

Then:
(a) All the nonempty level sets Vγ have the same recession cone,
denoted Rf , and given by

Rf = d | (d, 0) ∈ Repi(f ) ,

where Repi(f ) is the recession cone of the epigraph of f .

(b) If one nonempty level set Vγ is compact, then all of these level
sets are compact.

For a closed proper convex function f : ℜn 7→ (−∞, ∞], the (com-

mon) recession cone Rf of the nonempty level sets is called the recession
cone of f . A vector d ∈ Rf is called a direction of recession of f . The
recession function of f , denoted rf , is the closed proper convex function
whose epigraph is Rf .
The lineality space of the recession cone Rf of a closed proper convex
function f is denoted by Lf , and is the subspace of all d ∈ ℜn such that
both d and −d are directions of recession of f , i.e.,
Lf = Rf ∩ (−Rf ).
We have that d ∈ Lf if and only if
f (x + αd) = f (x), ∀ x ∈ dom(f ), ∀ α ∈ ℜ.
Consequently, any d ∈ Lf is called a direction in which f is constant , and
Lf is called the constancy space of f .

Proposition 1.4.6: Let f : ℜn 7→ (−∞, ∞] be a closed proper convex

function. Then the recession cone and constancy space of f are given
in terms of its recession function by
 
Rf = d | rf (d) ≤ 0 , Lf = d | rf (d) = rf (−d) = 0 .

Proposition 1.4.7: Let f : ℜn 7→ (−∞, ∞] be a closed proper convex

function. Then, for all x ∈ dom(f ) and d ∈ ℜn ,

f (x + αd) − f (x) f (x + αd) − f (x)

rf (d) = sup = lim .
α>0 α α→∞ α
Sec. B.1.4 Recession Cones 481

Proposition 1.4.8: (Recession Function of a Sum) Let fi :

ℜn 7→ (−∞, ∞], i = 1, . . . , m, be closed proper convex functions such
that the function f = f1 + · · · + fm is proper. Then

rf (d) = rf1 (d) + · · · + rfm (d), ∀ d ∈ ℜn .

Nonemptiness of Set Intersections

Let {Ck } be a sequence of nonempty closed sets in ℜn with Ck+1 ⊂ Ck for

all k (such a sequence is said to be nested ). We are concerned with the
question whether ∩∞k=0 Ck is nonempty.

Definition 1.4.1: Let {Ck } be a nested sequence of nonempty closed

convex sets. We say that {xk } is an asymptotic sequence of {Ck } if
xk 6= 0, xk ∈ Ck for all k, and

xk d
kxk k → ∞, → ,
kxk k kdk

where d is some nonzero common direction of recession of the sets Ck ,

d 6= 0, d ∈ ∩∞
k=0 RCk .

A special case is when all the sets Ck are equal. In particular, for a
nonempty closed convex set C, and a sequence {xk } ⊂ C, we say that {xk }
is an asymptotic sequence of C if {xk } is asymptotic (as per the preceding
definition) for the sequence {Ck }, where Ck ≡ C.
Given any unbounded sequence {xk } such that xk ∈ Ck for each k,
there exists a subsequence {xk }k∈K that is asymptotic
 for the corresponding
subsequence {Ck }k∈K . In fact, any limit point of xk /kxk k is a common
direction of recession of the sets Ck .

Definition 1.4.2: Let {Ck } be a nested sequence of nonempty closed

convex sets. We say that an asymptotic sequence {xk } is retractive if
for the direction d corresponding to {xk } as per Definition 1.4.1, there
exists an index k such that
482 Chap. 1

xk − d ∈ Ck , ∀ k ≥ k.
We say that the sequence {Ck } is retractive if all its asymptotic se-
quences are retractive. In the special case Ck ≡ C, we say that the set
C is retractive if all its asymptotic sequences are retractive.

A closed halfspace is retractive. Intersections and Cartesian products,

involving a finite number of sets, preserve retractiveness. In particular, if
{Ck1 }, . . . , {Ckr } are retractive nested sequences of nonempty closed convex
sets, the sequences {Nk } and {Tk } are retractive, where

Nk = Ck1 ∩ Ck2 ∩ · · · ∩ Ckr , Tk = Ck1 × Ck2 × · · · × Ckr , ∀ k,

and we assume that all the sets Nk are nonempty. A simple consequence
is that a polyhedral set is retractive, since it is the nonempty intersection
of a finite number of closed halfspaces.

Proposition 1.4.9: A polyhedral set is retractive.

The importance of retractive sequences is motivated by the following

proposition.

Proposition 1.4.10: A retractive nested sequence of nonempty closed

convex sets has nonempty intersection.

Proposition 1.4.11: Let {Ck } be a nested sequence of nonempty

closed convex sets. Denote

R = ∩∞
k=0 RCk , L = ∩∞
k=0 LCk .

(a) If R = L, then {Ck } is retractive, and ∩∞

k=0 Ck is nonempty.
Furthermore,
∩∞k=0 Ck = L + C̃,

where C̃ is some nonempty and compact set.

(b) Let X be a retractive closed convex set. Assume that all the sets
C k = X ∩ Ck are nonempty, and that
Sec. B.1.4 Recession Cones 483

RX ∩ R ⊂ L.
Then, {C k } is retractive, and ∩∞
k=0 C k is nonempty.

Proposition 1.4.12: (Existence of Solutions of Convex Quad-

ratic Programs) Let Q be a symmetric positive semidefinite n × n
matrix, let c and a1 , . . . , ar be vectors in ℜn , and let b1 , . . . , br be
scalars. Assume that the optimal value of the problem

minimize x′ Qx + c′ x
subject to a′j x ≤ bj , j = 1, . . . , r,

is finite. Then the problem has at least one optimal solution.

Closedness under Linear Transformation and Vector Sum

The conditions of Prop. 1.4.11 can be translated to conditions guaranteeing

the closedness of the image, A C, of a closed convex set C under a linear
transformation A.

Proposition 1.4.13: Let X and C be nonempty closed convex sets

in ℜn , and let A be an m × n matrix with nullspace denoted by N (A).
If X is a retractive closed convex set and

RX ∩ RC ∩ N (A) ⊂ LC ,

then A(X ∩ C) is a closed set.

A special case relates to vector sums.

Proposition 1.4.14: Let C1 , . . . , Cm be nonempty closed convex sub-

sets of ℜn such that the equality d1 + · · · + dm = 0 for some vectors
di ∈ RCi implies that di ∈ LCi for all i = 1, . . . , m. Then C1 +· · ·+Cm
is a closed set.

When specialized to just two sets, the above proposition implies that
if C1 and −C2 are closed convex sets, then C1 − C2 is closed if there is no
484 Chap. 1

common nonzero direction of recession of C1 and C2 , i.e.

RC1 ∩ RC2 = {0}.

This is true in particular if either C1 or C2 is bounded, in which case either

RC1 = {0} or RC2 = {0}, respectively. For an example of two unbounded
closed convex sets in the plane whose vector sum is not closed, let
 
C1 = (x1 , x2 ) | x1 x2 ≥ 0 , C2 = (x1 , x2 ) | x1 = 0 .

Some other conditions asserting the closedness of vector sums can be

derived from Prop. 1.4.13. For example, we can show that the vector sum
of a finite number of polyhedral sets is closed, since it can be viewed as the
image of their Cartesian product (clearly a polyhedral set) under a linear
transformation. Another useful result is that if X is a polyhedral set, and
C is a closed convex set, then X + C is closed if every direction of recession
of X whose opposite is a direction of recession of C lies also in the lineality
space of C. In particular, X + C is closed if X is polyhedral, and C is
closed.

Section 1.5. Hyperplanes

A hyperplane in ℜn is a set of the form

{x | a′ x = b},

where a is nonzero vector in ℜn (called the normal of the hyperplane), and

b is a scalar. The sets

{x | a′ x ≥ b}, {x | a′ x ≤ b},

are called the closed halfspaces associated with the hyperplane (also referred
to as the positive and negative halfspaces, respectively). The sets

{x | a′ x > b}, {x | a′ x < b},

are called the open halfspaces associated with the hyperplane.

Proposition 1.5.1: (Supporting Hyperplane Theorem) Let C

be a nonempty convex subset of ℜn and let x be a vector in ℜn . If
x is not an interior point of C, there exists a hyperplane that passes
through x and contains C in one of its closed halfspaces, i.e., there
exists a vector a 6= 0 such that

a′ x ≤ a′ x, ∀ x ∈ C.
Sec. B.1.5 Hyperplanes 485

Proposition 1.5.2: (Separating Hyperplane Theorem) Let C1

and C2 be two nonempty convex subsets of ℜn . If C1 and C2 are
disjoint, there exists a hyperplane that separates C1 and C2 , i.e., there
exists a vector a 6= 0 such that

a′ x1 ≤ a′ x2 , ∀ x1 ∈ C1 , ∀ x2 ∈ C2 .

Proposition 1.5.3: (Strict Separation Theorem) Let C1 and

C2 be two disjoint nonempty convex sets. Then under any one of
the following five conditions, there exists a hyperplane that strictly
separates C1 and C2 , i.e., a vector a 6= 0 and a scalar b such that

a′ x1 < b < a′ x2 , ∀ x1 ∈ C1 , ∀ x2 ∈ C2 .

(1) C2 − C1 is closed.
(2) C1 is closed and C2 is compact.
(3) C1 and C2 are polyhedral.
(4) C1 and C2 are closed, and

RC1 ∩ RC2 = LC1 ∩ LC2 ,

where RCi and LCi denote the recession cone and the lineality
space of Ci , i = 1, 2.
(5) C1 is closed, C2 is polyhedral, and RC1 ∩ RC2 ⊂ LC1 .

Proposition 1.5.4: The closure of the convex hull of a set C is the

intersection of the closed halfspaces that contain C. In particular,
a closed convex set is the intersection of the closed halfspaces that
contain it.

Let C1 and C2 be two subsets of ℜn . We say that a hyperplane

properly separates C1 and C2 if it separates C1 and C2 , and does not fully
contain both C1 and C2 . If C is a subset of ℜn and x is a vector in ℜn , we
say that a hyperplane properly separates C and x if it properly separates
486 Chap. 1

C and the singleton set {x}.

Proposition 1.5.5: (Proper Separation Theorem) Let C be a

nonempty convex subset of ℜn and let x be a vector in ℜn . There
exists a hyperplane that properly separates C and x if and only if
x∈/ ri(C).

Proposition 1.5.6: (Proper Separation of Two Convex Sets)

Let C1 and C2 be two nonempty convex subsets of ℜn . There exists a
hyperplane that properly separates C1 and C2 if and only if

ri(C1 ) ∩ ri(C2 ) = Ø.

Proposition 1.5.7: (Polyhedral Proper Separation Theorem)

Let C and P be two nonempty convex subsets of ℜn such that P is
polyhedral. There exists a hyperplane that separates C and P , and
does not contain C if and only if

ri(C) ∩ P = Ø.

Consider a hyperplane in ℜn+1 with a normal of the form (µ, β),

where µ ∈ ℜn and β ∈ ℜ. We say that such a hyperplane is vertical if
β = 0, and nonvertical if β 6= 0.

Proposition 1.5.8: (Nonvertical Hyperplane Theorem) Let C

be a nonempty convex subset of ℜn+1 that contains no vertical lines.
Let the vectors in ℜn+1 be denoted by (u, w), where u ∈ ℜn and
w ∈ ℜ. Then:
(a) C is contained in a closed halfspace corresponding to a nonverti-
cal hyperplane, i.e., there exist a vector µ ∈ ℜn , a scalar β 6= 0,
and a scalar γ such that
Sec. B.1.6 Conjugate Functions 487

µ′ u + βw ≥ γ, ∀ (u, w) ∈ C.

(b) If (u, w) does not belong to cl(C), there exists a nonvertical hy-
perplane strictly separating (u, w) and C.

Section 1.6. Conjugate Functions

Consider an extended real-valued function f : ℜn 7→ [−∞, ∞]. The conju-

gate function of f is the function f ⋆ : ℜn 7→ [−∞, ∞] defined by

f ⋆ (y) = sup x′ y − f (x) , y ∈ ℜn . (B.2)
x∈ℜn

Proposition 1.6.1: (Conjugacy Theorem) Let f : ℜn 7→ [−∞, ∞]

be a function, let f ⋆ be its conjugate, and consider the double conju-
gate f ⋆⋆ = (f ⋆ )⋆ . Then:
(a) We have
f (x) ≥ f ⋆⋆ (x), ∀ x ∈ ℜn .

(b) If f is convex, then properness of any one of the functions f , f ⋆ ,

and f ⋆⋆ implies properness of the other two.
(c) If f is closed proper convex, then

f (x) = f ⋆⋆ (x), ∀ x ∈ ℜn .

ˇ f are equal. Fur-

(d) The conjugates of f and its convex closure cl
ˇ
thermore, if cl f is proper, then

ˇ f )(x) = f ⋆⋆ (x),
(cl ∀ x ∈ ℜn .

Positively Homogeneous Functions and Support Functions

Given a nonempty set X, consider the indicator function of X, defined by

0 if x ∈ X,
n
δX (x) =
∞ if x ∈
/ X.
The conjugate of δX is given by
σX (y) = sup y ′ x
x∈X
488 Chap. 1

and is called the support function of X.

Let C be a convex cone. The conjugate of its indicator function δC
is its support function,
σC (y) = sup y ′ x.
x∈C

The support/conjugate function σC is the indicator function δC ∗ of the

cone
C ∗ = {y | y ′ x ≤ 0, ∀ x ∈ C},
called the polar cone of C. By the Conjugacy Theorem [Prop. 1.6.1(d)],
the polar cone of C ∗ is cl(C). In particular, if C is closed, the polar of
its polar is equal to the original. This is a special case of the Polar Cone
Theorem, given in Section 2.2.
A function f : ℜn 7→ [−∞, ∞] is called positively homogeneous if its
epigraph is a cone in ℜn+1 . Equivalently, f is positively homogeneous if
and only if
f (γx) = γ f (x), ∀ γ > 0, ∀ x ∈ ℜn .
Positively homogeneous functions are closely connected with support
functions. Clearly, the support function σX of a set X is closed convex
and positively homogeneous. Moreover, if σ : ℜn 7→ (−∞, ∞] is a proper
convex positively homogeneous function, then we claim that the conjugate
of σ is the indicator function of the closed convex set

X = x | y ′ x ≤ σ(y), ∀ y ∈ ℜn ,

and that cl σ is the support function of X. For a proof, let δ be the

conjugate of σ: 
δ(x) = sup y ′ x − σ(y) .
y∈ℜn

Since σ is positively homogeneous, we have for any γ > 0,

 
γ δ(x) = sup γy ′ x − γ σ(y) = sup (γy)′ x − σ(γy) .
y∈ℜn y∈ℜn

The right-hand sides of the preceding two relations are equal, so we obtain

δ(x) = γ δ(x), ∀ γ > 0,

which implies that δ takes only the values 0 and ∞ (since σ and hence also
its conjugate δ is proper). Thus, δ is the indicator function of a set, call it
X, and we have

X = x | δ(x) ≤ 0
( )

′
= x sup y x − σ(y) ≤ 0
y∈ℜn

= x| y′x ≤ σ(y), ∀ y ∈ ℜn .
Sec. B.2.1 Extreme Points 489

Finally, since δ is the conjugate of σ, we see that cl σ is the conjugate of

δ; cf. the Conjugacy Theorem [Prop. 1.6.1(c)]. Since δ is the indicator
function of X, it follows that cl σ is the support function of X.
We now discuss a characterization of the support function of the 0-
level set of a closed proper convex function f : ℜn 7→ (−∞, ∞]. The
closure of the cone generated by epi(f ), is the epigraph of a closed convex
positively homogeneous function, called the closed function generated by
f , and denoted by gen f . The epigraph of gen f is the intersection of all
the closed cones that contain epi(f ). Moreover, if gen f is proper, then
epi(gen f ) is the intersection of all the halfspaces that contain epi(f ) and
contain 0 in their boundary.
Consider the conjugate f ⋆ of a closed proper  convex function f :
n
ℜ 7→ (−∞, ∞]. We claim that if the level set y | f ⋆ (y) ≤ 0 [or the

level set x | f (x) ≤ 0 ] is nonempty, its support
 function is gen f (or
respectively gen f ⋆ ). Indeed, if the level set y | f ⋆ (y) ≤ 0 is nonempty,
any y such that f ⋆ (y) ≤ 0, or equivalently y ′ x ≤ f (x) for all x, defines a
nonvertical hyperplane that separates the origin from epi(f ), implying that
the epigraph of gen f does not contain a line, so gen f is proper. Since gen f
is also closed, convex, and positively homogeneous, by our earlier analysis
it follows that gen f is the support function of the set

Y = y | y ′ x ≤ (gen f )(x), ∀ x ∈ ℜn .
Since epi(gen f ) is the intersection of all the halfspaces that contain epi(f )
and contain 0 in their boundary, the set Y can be written as
 
 
′ n ′
Y = y | y x ≤ f (x), ∀ x ∈ ℜ = y sup y x − f (x) ≤ 0 .
x∈ℜn

We thus obtain that gen f is the support function of the set


Y = y | f ⋆ (y) ≤ 0 ,
assuming this set is nonempty.
Note that the method used to characterize the 0-level sets of f and
f ⋆ can be applied to any level set. In particular, a nonempty level set
Lγ = x | f (x) ≤ γ is the 0-level set of the function fγ defined by
fγ (x) = f (x) − γ, and its support function is the closed function generated
by fγ⋆ , the conjugate of fγ , which is given by fγ⋆ (y) = f ⋆ (y) + γ.

CHAPTER 2: Basic Concepts of Polyhedral Convexity

Section 2.1. Extreme Points

In this chapter, we discuss polyhedral sets, i.e., nonempty sets specified by

systems of a finite number of affine inequalities
a′j x ≤ bj , j = 1, . . . , r,
490 Chap. 2

where a1 , . . . , ar are vectors in ℜn , and b1 , . . . , br are scalars.

Given a nonempty convex set C, a vector x ∈ C is said to be an
extreme point of C if it does not lie strictly between the endpoints of any
line segment contained in the set, i.e., if there do not exist vectors y ∈ C
and z ∈ C, with y 6= x and z 6= x, and a scalar α ∈ (0, 1) such that
x = αy + (1 − α)z.

Proposition 2.1.1: Let C be a convex subset of ℜn , and let H be a

hyperplane that contains C in one of its closed halfspaces. Then the
extreme points of C ∩ H are precisely the extreme points of C that
belong to H.

Proposition 2.1.2: A nonempty closed convex subset of ℜn has at

least one extreme point if and only if it does not contain a line, i.e.,
a set of the form {x + αd | α ∈ ℜ}, where x and d are vectors in ℜn
with d 6= 0.

Proposition 2.1.3: Let C be a nonempty closed convex subset of ℜn .

Assume that for some m × n matrix A of rank n and some b ∈ ℜm , we
have
Ax ≥ b, ∀ x ∈ C.
Then C has at least one extreme point.

Proposition 2.1.4: Let P be a polyhedral subset of ℜn .

(a) If P has the form

P = x | a′j x ≤ bj , j = 1, . . . , r ,

where aj ∈ ℜn , bj ∈ ℜ, j = 1, . . . , r, then a vector v ∈ P is an

extreme point of P if and only if the set

Av = aj | a′j v = bj , j ∈ {1, . . . , r}

contains n linearly independent vectors.

Sec. B.2.2 Polar Cones 491

(b) If P has the form

P = {x | Ax = b, x ≥ 0},

where A is an m × n matrix and b is a vector in ℜm , then a

vector v ∈ P is an extreme point of P if and only if the columns
of A corresponding to the nonzero coordinates of v are linearly
independent.
(c) If P has the form

P = {x | Ax = b, c ≤ x ≤ d},

where A is an m × n matrix, b is a vector in ℜm , and c, d are

vectors in ℜn , then a vector v ∈ P is an extreme point of P if
and only if the columns of A corresponding to the coordinates
of v that lie strictly between the corresponding coordinates of c
and d are linearly independent.

Proposition 2.1.5: A polyhedral set in ℜn of the form


x | a′j x ≤ bj , j = 1, . . . , r

has an extreme point if and only if the set {aj | j = 1, . . . , r} contains

n linearly independent vectors.

Section 2.2. Polar Cones

We return to the notion of polar cone of nonempty set C, denoted by C ∗ ,

and given by C ∗ = {y | y ′ x ≤ 0, ∀ x ∈ C}.

Proposition 2.2.1:
(a) For any nonempty set C, we have

∗
∗
∗
C ∗ = cl(C) = conv(C) = cone(C) .
492 Chap. 2

(b) (Polar Cone Theorem) For any nonempty cone C, we have

(C ∗ )∗ = cl conv(C) .

In particular, if C is closed and convex, we have (C ∗ )∗ = C.

Section 2.3. Polyhedral Sets and Functions

We recall that a polyhedral cone C ⊂ ℜn is a polyhedral set of the form

C = {x | a′j x ≤ 0, j = 1, . . . , r},

where a1 , . . . , ar are some vectors in ℜn , and r is a positive integer. We

say that a cone C ⊂ ℜn is finitely generated , if it is generated by a finite
set of vectors, i.e., if it has the form

 
 r
X 


C = cone {a1 , . . . , ar } = x x= µj aj , µj ≥ 0, j = 1, . . . , r ,
 
j=1

where a1 , . . . , ar are some vectors in ℜn , and r is a positive integer.

Proposition 2.3.1: (Farkas’ Lemma) Let a1 , . . . , ar be vectors
in

ℜn . Then, {x | a′j x ≤ 0, j = 1, . . . , r} and cone {a1 , . . . , ar } are
closed cones that are polar to each other.

Proposition 2.3.2: (Minkowski-Weyl Theorem) A cone is poly-

hedral if and only if it is finitely generated.

Proposition 2.3.3: (Minkowski-Weyl Representation) A set P

is polyhedral if and only if there is a nonempty finite set {v1 , . . .
, vm }
and a finitely generated cone C such that P = conv {v1 , . . . , vm } +C,
i.e.,
Sec. B.2.3 Polyhedral Sets and Functions 493

 
 m
X m
X 
P = x x= µj vj + y, µj = 1, µj ≥ 0, j = 1, . . . , m, y ∈ C .
 
j=1 j=1

Proposition 2.3.4: (Algebraic Operations on Polyhedral Sets)

(a) The intersection of polyhedral sets is polyhedral, if it is nonempty.
(b) The Cartesian product of polyhedral sets is polyhedral.
(c) The image of a polyhedral set under a linear transformation is a
polyhedral set.
(d) The vector sum of two polyhedral sets is polyhedral.
(e) The inverse image of a polyhedral set under a linear transforma-
tion is polyhedral.

We say that a function f : ℜn 7→ (−∞, ∞] is polyhedral if its epigraph

is a polyhedral set in ℜn+1 . Note that a polyhedral function f is, by
definition, closed, convex, and also proper [since f cannot take the value
−∞, and epi(f ) is closed, convex, and nonempty (based on our convention
that only nonempty sets can be polyhedral)].

Proposition 2.3.5: Let f : ℜn 7→ (−∞, ∞] be a convex function.

Then f is polyhedral if and only if dom(f ) is a polyhedral set and

f (x) = max {a′j x + bj }, ∀ x ∈ dom(f ),

j=1,...,m

where aj are vectors in ℜn , bj are scalars, and m is a positive integer.

Some common operations on polyhedral functions, such as sum and

linear composition preserve their polyhedral character as shown by the
following two propositions.

Proposition 2.3.6: The sum of two polyhedral functions f1 and f2 ,

such that dom(f1 ) ∩ dom(f2 ) 6= Ø, is a polyhedral function.
494 Chap. 3

Proposition 2.3.7: If A is a matrix and g is a polyhedral function

such that dom(g) contains a point in the range of A, the function f
given by f (x) = g(Ax) is polyhedral.

Section 2.4. Polyhedral Aspects of Optimization

Polyhedral convexity plays a very important role in optimization. The

following are two basic results related to linear programming, the mini-
mization of a linear function over a polyhedral set.

Proposition 2.4.1: Let C be a closed convex subset of ℜn that has

at least one extreme point. A concave function f : C 7→ ℜ that attains
a minimum over C attains the minimum at some extreme point of C.

Proposition 2.4.2: (Fundamental Theorem of Linear Pro-

gramming) Let P be a polyhedral set that has at least one extreme
point. A linear function that is bounded below over P attains a mini-
mum at some extreme point of P .

CHAPTER 3: Basic Concepts of Convex Optimization

Section 3.1. Constrained Optimization

Let us consider the problem

minimize f (x)
subject to x ∈ X,
where f : ℜn 7→ (−∞, ∞] is a function and X is a nonempty subset of
ℜn . Any vector x ∈ X ∩ dom(f ) is said to be a feasible solution of the
problem (we also use the terms feasible vector or feasible point ). If there
is at least one feasible solution, i.e., X ∩ dom(f ) 6= Ø, we say that the
problem is feasible; otherwise we say that the problem is infeasible. Thus,
when f is extended real-valued, we view only the points in X ∩ dom(f ) as
candidates for optimality, and we view dom(f ) as an implicit constraint set.
Furthermore, feasibility of the problem is equivalent to inf x∈X f (x) < ∞.
We say that a vector x∗ is a minimum of f over X if
x∗ ∈ X ∩ dom(f ), and f (x∗ ) = inf f (x).
x∈X
Sec. B.3.2 Constrained Optimization 495

We also call x∗ a minimizing point or minimizer or global minimum of f

over X. Alternatively, we say that f attains a minimum over X at x∗ , and
we indicate this by writing

x∗ ∈ arg min f (x).

x∈X

If x∗ is known to be the unique minimizer of f over X, with slight abuse

of notation, we also occasionally write

x∗ = arg min f (x).

x∈X

We use similar terminology for maxima.

Given a subset X of ℜn and a function f : ℜn 7→ (−∞, ∞], we say
that a vector x∗ is a local minimum of f over X if x∗ ∈ X ∩ dom(f ) and
there exists some ǫ > 0 such that

f (x∗ ) ≤ f (x), ∀ x ∈ X with kx − x∗ k < ǫ.

A local minimum x∗ is said to be strict if there is no other local mini-

mum within some open sphere centered at x∗ . Local maxima are defined
similarly.

Proposition 3.1.1: If X is a convex subset of ℜn and f : ℜn 7→

(−∞, ∞] is a convex function, then a local minimum of f over X is
also a global minimum. If in addition f is strictly convex, then there
exists at most one global minimum of f over X.

Section 3.2. Existence of Optimal Solutions

Proposition 3.2.1: (Weierstrass’ Theorem) Consider a closed

proper function f : ℜn 7→ (−∞, ∞], and assume that any one of the
following three conditions holds:
(1) dom(f ) is bounded.
(2) There exists a scalar γ such that the level set

x | f (x) ≤ γ

is nonempty and bounded.

496 Chap. 3

(3) f is coercive, i.e., if for every sequence {xk } such that kxk k → ∞,
we have limk→∞ f (xk ) = ∞.
Then the set of minima of f over ℜn is nonempty and compact.

Proposition 3.2.2: Let X be a closed convex subset of ℜn , and let

f : ℜn 7→ (−∞, ∞] be a closed convex function with X ∩ dom(f ) 6= Ø.
The set of minima of f over X is nonempty and compact if and only
if X and f have no common nonzero direction of recession.

Proposition 3.2.3: (Existence of Solution, Sum of Functions)

Let fi : ℜn 7→ (−∞, ∞], i = 1, . . . , m, be closed proper convex func-
tions such that the function f = f1 + · · · + fm is proper. Assume that
the recession function of a single function fi satisfies rfi (d) = ∞ for
all d 6= 0. Then the set of minima of f is nonempty and compact.

Section 3.3. Partial Minimization of Convex Functions

Functions obtained by minimizing other functions partially, i.e., with re-

spect to some of their variables, arise prominently in the treatment of dual-
ity and minimax theory. It is then useful to be able to deduce properties of
the function obtained, such as convexity and closedness, from correspond-
ing properties of the original.

Proposition 3.3.1: Consider a function F : ℜn+m 7→ (−∞, ∞] and

the function f : ℜn 7→ [−∞, ∞] defined by

f (x) = infm F (x, z).

z∈ℜ

Then:
(a) If F is convex, then f is also convex.
(b) We have 



P epi(F ) ⊂ epi(f ) ⊂ cl P epi(F ) ,
Sec. B.3.3 Partial Minimization of Convex Functions 497

where P (·) denotes projectionon the space of (x, w), i.e., for any
subset S of ℜn+m+1 , P (S) = (x, w) | (x, z, w) ∈ S .

Proposition 3.3.2: Let F : ℜn+m 7→ (−∞, ∞] be a closed proper

convex function, and consider the function f given by

f (x) = infm F (x, z), x ∈ ℜn .

z∈ℜ

Assume that for some x ∈ ℜn and γ ∈ ℜ the set


z | F (x, z) ≤ γ

is nonempty and compact. Then f is closed proper convex. Further-

more, for each x ∈ dom(f ), the set of minima in the definition of f (x)
is nonempty and compact.

Proposition 3.3.3: Let X and Z be nonempty convex sets of ℜn and

ℜm , respectively, let F : X × Z 7→ ℜ be a closed convex function, and
assume that Z is compact. Then the function f given by

f (x) = inf F (x, z), x ∈ X,

z∈Z

is a real-valued convex function over X.

Proposition 3.3.4: Let F : ℜn+m 7→ (−∞, ∞] be a closed proper

convex function, and consider the function f given by

f (x) = infm F (x, z), x ∈ ℜn .

z∈ℜ

Assume that for some x ∈ ℜn and γ ∈ ℜ the set


z | F (x, z) ≤ γ

is nonempty and its recession cone is equal to its lineality space. Then
498 Chap. 4

f is closed proper convex. Furthermore, for each x ∈ dom(f ), the set

of minima in the definition of f (x) is nonempty.

Section 3.4. Saddle Point and Minimax Theory

Let us consider a function φ : X × Z 7→ ℜ, where X and Z are nonempty

subsets of ℜn and ℜm , respectively. An issue of interest is to derive condi-
tions guaranteeing that
sup inf φ(x, z) = inf sup φ(x, z), (B.3)
z∈Z x∈X x∈X z∈Z

and that the infima and the suprema above are attained.

Definition 3.4.1: A pair of vectors x∗ ∈ X and z ∗ ∈ Z is called a

saddle point of φ if

φ(x∗ , z) ≤ φ(x∗ , z ∗ ) ≤ φ(x, z ∗ ), ∀ x ∈ X, ∀ z ∈ Z.

Proposition 3.4.1: A pair (x∗ , z ∗ ) is a saddle point of φ if and only

if the minimax equality (B.3) holds, and x∗ is an optimal solution of
the problem
minimize sup φ(x, z)
z∈Z
subject to x ∈ X,
while z ∗ is an optimal solution of the problem

maximize inf φ(x, z)

x∈X

subject to z ∈ Z.

CHAPTER 4: Geometric Duality Framework

Section 4.1. Min Common/Max Crossing Duality

We introduce a geometric framework for duality analysis, which aims to

capture the most essential characteristics of duality in two simple geomet-
rical problems, defined by a nonempty subset M of ℜn+1 .
Sec. B.4.1 Min Common/Max Crossing Duality 499

(a) Min Common Point Problem: Consider all vectors that are common
to M and the (n + 1)st axis. We want to find one whose (n + 1)st
component is minimum.
(b) Max Crossing Point Problem: Consider nonvertical hyperplanes that
contain M in their corresponding “upper” closed halfspace, i.e., the
closed
 halfspace whose recession cone contains the vertical halfline
(0, w) | w ≥ 0 . We want to find the maximum crossing point of
the (n + 1)st axis with such a hyperplane.
We refer to the two problems as the min common/max crossing (MC/MC)
framework , and we will show that it can be used to develop much of the
core theory of convex optimization in a unified way.
Mathematically, the min common problem is

minimize w
subject to (0, w) ∈ M.

We also refer to this as the primal problem, and we denote by w∗ its optimal
value,
w∗ = inf w.
(0,w)∈M

The max crossing problem is to maximize over all µ ∈ ℜn the maxi-

mum crossing level corresponding to µ, i.e.,

maximize inf {w + µ′ u}
(u,w)∈M
(B.4)
subject to µ ∈ ℜn .

We also refer to this as the dual problem, we denote by q ∗ its optimal value,

q ∗ = sup q(µ),
µ∈ℜn

and we refer to q(µ) as the crossing or dual function.

Proposition 4.1.1: The dual function q is concave and upper semi-

continuous.

The following proposition states that we always have q ∗ ≤ w∗ ; we

refer to this as weak duality. When q ∗ = w∗ , we say that strong duality
holds or that there is no duality gap.

Proposition 4.1.2: (Weak Duality Theorem) We have q ∗ ≤ w∗ .

500 Chap. 4

The feasible solutions of the max crossing problem are restricted by

the horizontal directions of recession of M . This is the essence of the
following proposition.

Proposition 4.1.3: Assume that the set


M = M + (0, w) | w ≥ 0

is convex.
 Then the set of feasible solutions of the max crossing prob-
lem, µ | q(µ) > −∞ , is contained in the cone

µ | µ′ d ≥ 0 for all d with (d, 0) ∈ RM ,

where RM is the recession cone of M .

Section 4.2. Some Special Cases

There are several interesting special cases where the set M is the epigraph of
some function. For example, consider the problem of minimizing a function
f : ℜn 7→ [−∞, ∞]. We introduce a function F : ℜn+r 7→ [−∞, ∞] of the
pair (x, u), which satisfies
f (x) = F (x, 0), ∀ x ∈ ℜn . (B.5)
Let the function p : ℜr 7→ [−∞, ∞] be defined by
p(u) = infn F (x, u), (B.6)
x∈ℜ

and consider the MC/MC framework with

M = epi(p).
The min common value w∗ is the minimal value of f , since
w∗ = p(0) = infn F (x, 0) = infn f (x).
x∈ℜ x∈ℜ

The max crossing problem (B.4) can be written as

maximize q(µ)
subject to µ ∈ ℜr ,
where the dual function is
 
q(µ) = inf {w + µ′ u} = inf p(u) + µ′ u = inf F (x, u) + µ′ u .
(u,w)∈M u∈ℜr (x,u)∈ℜn+r
(B.7)
Sec. B.4.2 Some Special Cases 501

Note that from Eq. (B.7), an alternative expression for q is


q(µ) = − sup −µ′ u − F (x, u) = −F ⋆ (0, −µ),
(x,u)∈ℜn+r

where F ⋆ is the conjugate of F , viewed as a function of (x, u). Since

q ∗ = sup q(µ) = − inf r F ⋆ (0, −µ) = − inf r F ⋆ (0, µ),

µ∈ℜr µ∈ℜ µ∈ℜ

the strong duality relation w∗ = q ∗ can be written as

inf F (x, 0) = − inf r F ⋆ (0, µ).

x∈ℜn µ∈ℜ

Different choices of function F , as in Eqs. (B.5) and (B.6), yield

corresponding MC/MC frameworks and dual problems. An example of
this type is minimization with inequality constraints:

minimize f (x)
(B.8)
subject to x ∈ X, g(x) ≤ 0,

where X is a nonempty subset
of ℜn , f : X 7→ ℜ is a given function, and

g(x) = g1 (x), . . . , gr (x) with gj : X 7→ ℜ being given functions. We
introduce a “perturbed constraint set” of the form

Cu = x ∈ X | g(x) ≤ u , u ∈ ℜr , (B.9)

and the function 

f (x) if x ∈ Cu ,
F (x, u) =
∞ otherwise,
which satisfies the condition F (x, 0) = f (x) for all x ∈ C0 [cf. Eq. (B.5)].
The function p of Eq. (B.6) is given by

p(u) = infn F (x, u) = inf f (x), (B.10)

x∈ℜ x∈X, g(x)≤u

and is known as the primal function or perturbation function. It captures

the essential structure of the constrained minimization problem, relating to
duality and other properties, such as sensitivity. Consider now the MC/MC
framework corresponding to M = epi(p). From Eq. (B.7), we obtain with
some calculation
( 
inf x∈X f (x) + µ′ g(x) if µ ≥ 0,
q(µ) =
−∞ otherwise.

The following proposition derives the primal and dual functions in

the minimax framework. In this proposition, for a given x, we denote by
502 Chap. 4

ˆ φ)(x, ·) the concave closure of φ(x, ·) [the smallest concave and upper
(cl
semicontinuous function that majorizes φ(x, ·)].

Proposition 4.2.1: Let X and Z be nonempty subsets of ℜn and

ℜm , respectively, and let φ : X × Z 7→ ℜ be a function. Assume
ˆ φ)(x, ·) is proper for all x ∈ X, and consider the MC/MC
that (−cl
framework corresponding to M = epi(p), where p is given by

p(u) = inf sup φ(x, z) − u′ z , u ∈ ℜm .
x∈X z∈Z

Then the dual function is given by

ˆ φ)(x, µ),
q(µ) = inf (cl ∀ µ ∈ ℜm .
x∈X

Section 4.3. Strong Duality Theorem

The following propositions give general results for strong duality.

Proposition 4.3.1: (MC/MC Strong Duality) Consider the min

common and max crossing problems, and assume the following:
(1) Either w∗ < ∞, or else w∗ = ∞ and M contains no vertical lines.
(2) The set 
M = M + (0, w) | w ≥ 0
is convex.

Then, we have q ∗ = w∗ if and only if for every sequence (uk , wk ) ⊂
M with uk → 0, there holds w∗ ≤ lim inf k→∞ wk .

Proposition 4.3.2: Consider the MC/MC framework, assuming that

w∗ < ∞.
(a) Let M be closed and convex. Then q ∗ = w∗ . Furthermore, the
function
Sec. B.4.4 Existence of Dual Optimal Solutions 503


p(u) = inf w | (u, w) ∈ M , u ∈ ℜn ,
is convex and its epigraph is the set

M = M + (0, w) | w ≥ 0 .

If in addition −∞ < w∗ , then p is closed and proper.

(b) q ∗ is equal to the optimal value

of the min common problem
corresponding to cl conv(M ) .
(c) If M is of the form

M = M̃ + (u, 0) | u ∈ C ,

where M̃ is a compact set and C is a closed convex set, then

q ∗ is equal to the optimal value of the min common problem
corresponding to conv(M ).

Section 4.4. Existence of Dual Optimal Solutions

The following propositions give general results for strong duality, as well
existence of dual optimal solutions.

Proposition 4.4.1: (MC/MC Existence of Max Crossing So-

lutions) Consider the MC/MC framework and assume the following:
(1) −∞ < w∗ .
(2) The set 
M = M + (0, w) | w ≥ 0
is convex.
(3) The origin is a relative interior point of the set

D = u | there exists w ∈ ℜ with (u, w) ∈ M }.

Then q ∗ = w∗ , and there exists at least one optimal solution of

the max crossing problem.
504 Chap. 4

Proposition 4.4.2: Let the assumptions of Prop. 4.4.1 hold. Then

Q∗ , the set of optimal solutions of the max crossing problem, has the
form
⊥
Q∗ = aff(D) + Q̃,

where Q̃ is a nonempty, convex, and compact set. In particular, Q∗ is

compact if and only if the origin is an interior point of D.

Section 4.5. Duality and Polyhedral Convexity

The following propositions address special cases where the set M has par-
tially polyhedral structure.

Proposition 4.5.1: Consider the MC/MC framework, and assume

the following:
(1) −∞ < w∗ .
(2) The set M has the form

M = M̃ − (u, 0) | u ∈ P ,

where M̃ and P are convex sets.

(3) Either ri(D̃) ∩ ri(P ) 6= Ø, or P is polyhedral and ri(D̃) ∩ P 6= Ø,
where D̃ is the set given by

D̃ = u | there exists w ∈ ℜ with (u, w) ∈ M̃ }.

Then q ∗ = w∗ , and Q∗ , the set of optimal solutions of the max crossing

problem, is a nonempty subset of RP∗ , the polar cone of the recession
cone of P . Furthermore, Q∗ is compact if int(D̃) ∩ P 6= Ø.

Proposition 4.5.2: Consider the MC/MC framework, and assume

that:
(1) −∞ < w∗ .
(2) The set M is defined in terms of a polyhedral set P , an r × n
matrix A, a vector b ∈ ℜr , and a convex function f : ℜn 7→
(−∞, ∞] as follows:
Sec. B.5.1 Nonlinear Farkas’ Lemma 505


M = (u, w) | Ax − b − u ∈ P for some (x, w) ∈ epi(f ) .


(3) There is a vector x ∈ ri dom(f ) such that Ax − b ∈ P .
Then q ∗ = w∗ and Q∗ , the set of optimal solutions of the max crossing
problem, is a nonempty subset of RP∗ , the polar cone of the recession
cone of P . Furthermore, Q∗ is compact

if the matrix A has rank r and
there is a vector x ∈ int dom(f ) such that Ax − b ∈ P .

CHAPTER 5: Duality and Optimization

Section 5.1. Nonlinear Farkas’ Lemma

A nonlinear version of Farkas’ Lemma captures the essence of convex pro-

gramming duality. The lemma involves a nonempty convex set X ⊂ ℜn ,
and functions f : X 7→ ℜ and gj : X 7→ ℜ, j = 1, . . . , r. We denote

′
g(x) = g1 (x), . . . , gr (x) , and use the following assumption.

Assumption 5.1: The functions f and gj , j = 1, . . . , r, are convex,

and
f (x) ≥ 0, ∀ x ∈ X with g(x) ≤ 0.

Proposition 5.1.1: (Nonlinear Farkas’ Lemma) Let Assumption

5.1 hold and let Q∗ be the subset of ℜr given by

Q∗ = µ | µ ≥ 0, f (x) + µ′ g(x) ≥ 0, ∀ x ∈ X .

Assume that one of the following two conditions holds:

(1) There exists x ∈ X such that gj (x) < 0 for all j = 1, . . . , r.
(2) The functions gj , j = 1, . . . , r, are affine, and there exists x ∈
ri(X) such that g(x) ≤ 0.
Then Q∗ is nonempty, and under condition (1) it is also compact.

The interior point condition (1) in the above proposition, and other
propositions that follow, is known as the Slater condition. By selecting f
and gj to be linear, and X to be the entire space in the above proposition,
506 Chap. 5

we obtain a version of Farkas’ Lemma (cf. Section 2.3) as a special case.

Proposition 5.1.2: (Linear Farkas’ Lemma) Let A be an m × n

matrix and c be a vector in ℜm .
(a) The system Ay = c, y ≥ 0 has a solution if and only if

A′ x ≤ 0 ⇒ c′ x ≤ 0.

(b) The system Ay ≥ c has a solution if and only if

A′ x = 0, x ≥ 0 ⇒ c′ x ≤ 0.

Section 5.2. Linear Programming Duality

One of the most important results in optimization is the linear program-

ming duality theorem. Consider the problem
minimize c′ x
subject to a′j x ≥ bj , j = 1, . . . , r,
where c ∈ ℜn , aj ∈ ℜn , and bj ∈ ℜ, j = 1, . . . , r. In the following
proposition, we refer to this as the primal problem. We consider the dual
problem
maximize b′ µ
X r
subject to aj µj = c, µ ≥ 0,
j=1

which can be derived from the MC/MC duality framework in Section 4.2.
We denote the primal and dual optimal values by f ∗ and q ∗ , respectively.

Proposition 5.2.1: (Linear Programming Duality Theorem)

(a) If either f ∗ or q ∗ is finite, then f ∗ = q ∗ and both the primal and
the dual problem have optimal solutions.
(b) If f ∗ = −∞, then q ∗ = −∞.
(c) If q ∗ = ∞, then f ∗ = ∞.

Note that the theorem allows the possibility f ∗ = ∞ and q ∗ = −∞.

Another related result is the following necessary and sufficient condition
for primal and dual optimality.
Sec. B.5.3 Convex Programming Duality 507

Proposition 5.2.2: (Linear Programming Optimality Condi-

tions) A pair of vectors (x∗ , µ∗ ) form a primal and dual optimal so-
lution pair if and only if x∗ is primal-feasible, µ∗ is dual-feasible, and

µ∗j (bj − a′j x∗ ) = 0, ∀ j = 1, . . . , r.

Section 5.3. Convex Programming Duality

We first focus on the problem

minimize f (x)
(B.11)
subject to x ∈ X, g(x) ≤ 0,

′
where X is a convex set in ℜn , g(x) = g1 (x), . . . , gr (x) , f : X 7→ ℜ and
gj : X 7→ ℜ, j = 1, . . . , r, are convex functions. The dual problem is
maximize inf L(x, µ)
x∈X

subject to µ ≥ 0,
where L is the Lagrangian function
L(x, µ) = f (x) + µ′ g(x), x ∈ X, µ ∈ ℜr .
For this and other similar problems, we denote the primal and dual opti-
mal values by f ∗ and q ∗ , respectively. We always have the weak duality
relation q ∗ ≤ f ∗ ; cf. Prop. 4.1.2. When strong duality holds, dual optimal
solutions are also referred to as Lagrange multipliers. The following eight
propositions are the main results relating to strong duality in a variety of
contexts. They provide conditions (often called constraint qualifications),
which guarantee that q ∗ = f ∗ .

Proposition 5.3.1: (Convex Programming Duality – Exis-

tence of Dual Optimal Solutions) Consider problem (B.11). As-
sume that f ∗ is finite, and that one of the following two conditions
holds:
(1) There exists x ∈ X such that gj (x) < 0 for all j = 1, . . . , r.
(2) The functions gj , j = 1, . . . , r, are affine, and there exists x ∈
ri(X) such that g(x) ≤ 0.
Then q ∗ = f ∗ and the set of optimal solutions of the dual problem is
nonempty. Under condition (1) this set is also compact.
508 Chap. 5

Proposition 5.3.2: (Optimality Conditions) Consider problem

(B.11). There holds q ∗ = f ∗ , and (x∗ , µ∗ ) are a primal and dual
optimal solution pair if and only if x∗ is feasible, µ∗ ≥ 0, and

x∗ ∈ arg min L(x, µ∗ ), µ∗j gj (x∗ ) = 0, j = 1, . . . , r.

x∈X

The condition µ∗j gj (x∗ ) = 0 is known as complementary slackness,

and generalizes the corresponding condition for linear programming, given
in Prop. 5.2.2. The preceding proposition actually can be proved without
the convexity assumptions of X, f , and g, although this fact will not be
useful to us.
The analysis for problem (B.11) can be refined by making more spe-
cific assumptions regarding available polyhedral structure in the constraint
functions and the abstract constraint set X. Here is an extension of prob-
lem (B.11) with additional linear equality constraints:

minimize f (x)
(B.12)
subject to x ∈ X, g(x) ≤ 0, Ax = b,

′
where X is a convex set, g(x) = g1 (x), . . . , gr (x) , f : X 7→ ℜ and
gj : X 7→ ℜ, j = 1, . . . , r, are convex functions, A is an m × n matrix, and
b ∈ ℜm . The corresponding Lagrangian function is

L(x, µ, λ) = f (x) + µ′ g(x) + λ′ (Ax − b),

and the dual problem is

maximize inf L(x, µ, λ)
x∈X
subject to µ ≥ 0, λ ∈ ℜm .
In the special case of a problem with just linear equality constraints:
minimize f (x)
(B.13)
subject to x ∈ X, Ax = b,
the Lagrangian function is

L(x, λ) = f (x) + λ′ (Ax − b),

and the dual problem is

maximize inf L(x, λ)
x∈X
subject to λ ∈ ℜm .
Sec. B.5.3 Convex Programming Duality 509

Proposition 5.3.3: (Convex Programming – Linear Equality

Constraints) Consider problem (B.13).
(a) Assume that f ∗ is finite and that there exists x ∈ ri(X) such
that Ax = b. Then f ∗ = q ∗ and there exists at least one dual
optimal solution.
(b) There holds f ∗ = q ∗ , and (x∗ , λ∗ ) are a primal and dual optimal
solution pair if and only if x∗ is feasible and

x∗ ∈ arg min L(x, λ∗ ).

x∈X

Proposition 5.3.4: (Convex Programming – Linear Equality

and Inequality Constraints) Consider problem (B.12).
(a) Assume that f ∗ is finite, that the functions gj are linear, and
that there exists x ∈ ri(X) such that Ax = b and g(x) ≤ 0. Then
q ∗ = f ∗ and there exists at least one dual optimal solution.
(b) There holds f ∗ = q ∗ , and (x∗ , µ∗ , λ∗ ) are a primal and dual
optimal solution pair if and only if x∗ is feasible, µ∗ ≥ 0, and

x∗ ∈ arg min L(x, µ∗ , λ∗ ), µ∗j gj (x∗ ) = 0, j = 1, . . . , r.

x∈X

Proposition 5.3.5: (Convex Programming – Linear Equality

and Nonlinear Inequality Constraints) Consider problem (B.12).
Assume that f ∗ is finite, that there exists x ∈ X such that Ax = b
and g(x) < 0, and that there exists x̃ ∈ ri(X) such that Ax̃ = b. Then
q ∗ = f ∗ and there exists at least one dual optimal solution.

Proposition 5.3.6: (Convex Programming – Mixed Polyhe-

dral and Nonpolyhedral Constraints) Consider problem (B.12),
where X is the intersection of a polyhedral set P and a convex set C,
510 Chap. 5

X = P ∩ C,

′
g(x) = g1 (x), . . . , gr (x) , the functions f : ℜn 7→ ℜ and gj : ℜn 7→ ℜ,
j = 1, . . . , r, are defined over ℜn , A is an m × n matrix, and b ∈ ℜm .
Assume that f ∗ is finite and that for some r with 1 ≤ r ≤ r, the
functions gj , j = 1, . . . , r, are polyhedral, and the functions f and gj ,
j = r + 1, . . . , r, are convex over C. Assume further that:
(1) There exists a vector x̃ ∈ ri(C) in the set

P̃ = P ∩ x | Ax = b, gj (x) ≤ 0, j = 1, . . . , r .

(2) There exists x ∈ P̃ ∩C such that gj (x) < 0 for all j = r +1, . . . , r.
Then q ∗ = f ∗ and there exists at least one dual optimal solution.

We will now give a different type of result, which under some com-
pactness assumptions, guarantees strong duality and that there exists an
optimal primal solution (even if there may be no dual optimal solution).

Proposition 5.3.7: (Convex Programming Duality – Exis-

tence of Primal Optimal Solutions) Assume that problem (B.11)
is feasible, that the convex functions f and gj are closed, and that the
function 
f (x) if g(x) ≤ 0, x ∈ X,
F (x, 0) =
∞ otherwise,
has compact level sets. Then f ∗ = q ∗ and the set of optimal solutions
of the primal problem is nonempty and compact.

We now consider another important optimization framework, the

problem
minimize f1 (x) + f2 (Ax)
(B.14)
subject to x ∈ ℜn ,

where A is an m × n matrix, f1 : ℜn 7→ (−∞, ∞] and f2 : ℜm 7→ (−∞, ∞]

are closed proper convex functions. We assume that there exists a feasible
solution.

Proposition 5.3.8: (Fenchel Duality)

(a) If f ∗ is finite and A · ri dom(f1 ) ∩ ri dom(f2 ) 6= Ø, then
f ∗ = q ∗ and there exists at least one dual optimal solution.
Sec. B.5.4 Subgradients and Optimality Conditions 511

(b) There holds f ∗ = q ∗ , and (x∗ , λ∗ ) is a primal and dual optimal

solution pair if and only if
 
x∗ ∈ arg minn f1 (x)− x′ A′ λ∗ and Ax∗ ∈ arg minm f2 (z)+ z ′ λ∗ .
x∈ℜ z∈ℜ
(B.15)

An important special case of Fenchel duality involves the problem

minimize f (x)
(B.16)
subject to x ∈ C,

where f : ℜn 7→ (−∞, ∞] is a closed proper convex function and C is a

closed convex cone in ℜn . This is known as a conic program, and some of its
special cases (semidefinite programming, second order cone programming)
have many practical applications.

Proposition 5.3.9: (Conic Duality Theorem) Assume that the

optimal value

of the primal conic problem (B.16) is finite, and that
ri dom(f ) ∩ ri(C) 6= Ø. Consider the dual problem

minimize f ⋆ (λ)
subject to λ ∈ Ĉ,

where f ⋆ is the conjugate of f and Ĉ is the dual cone,

Ĉ = −C ∗ = {λ | λ′ x ≥ 0, ∀ x ∈ C}.

Then there is no duality gap and the dual problem has an optimal
solution.

Section 5.4. Subgradients and Optimality Conditions

Let f : ℜn 7→ (−∞, ∞] be a proper convex function. We say that a vector

g ∈ ℜn is a subgradient of f at a point x ∈ dom(f ) if

f (z) ≥ f (x) + g ′ (z − x), ∀ z ∈ ℜn . (B.17)

The set of all subgradients of f at x is called the subdifferential of f at

x and is denoted by ∂f (x). By convention, ∂f (x) is considered empty
for all x ∈
/ dom(f ). Generally, ∂f (x) is closed and convex, since based
on the subgradient inequality (B.17), it is the intersection of a collection
512 Chap. 5

of closed halfspaces. Note that we restrict attention to proper functions

(subgradients are not useful and make no sense for improper functions).

Proposition 5.4.1: Let f : ℜ
n 7→ (−∞, ∞] be a proper convex func-

tion. For every x ∈ ri dom(f ) ,

∂f (x) = S ⊥ + G,

where S is the subspace that is parallel to the affine hull of dom(f ),

and G is a
nonempty convex and compact set. In particular, if x ∈
int dom(f ) , then ∂f (x) is nonempty and compact.

It follows from the preceding proposition that if f is real-valued, then

∂f (x) is nonempty and compact for all x ∈ ℜn . An
important property is
that if f is differentiable at some x ∈ int dom(f ) , its gradient ∇f (x) is
the unique subgradient at x. We give a proof of these facts, together with
the following proposition, in Section 3.1.

Proposition 5.4.2: (Subdifferential Boundedness and Lips-

chitz Continuity) Let f : ℜn 7→ ℜ be a real-valued convex function,
and let X be a nonempty bounded subset of ℜn .
(a) The set ∪x∈X ∂f (x) is nonempty and bounded.
(b) The function f is Lipschitz continuous over X, i.e., there exists
a scalar L such that

f (x) − f (z) ≤ L kx − zk, ∀ x, z ∈ X.

Section 5.4.1. Subgradients of Conjugate Functions

We will now derive an important relation between the subdifferentials of a

proper convex function f : ℜn 7→ (−∞, ∞] and its conjugate f ⋆ . Using the
definition of conjugacy, we have

x′ y ≤ f (x) + f ⋆ (y), ∀ x ∈ ℜn , y ∈ ℜn .

This is known as the Fenchel inequality. A pair (x, y) satisfies this inequal-
ity as an equation if and only if x attains the supremum in the definition

f ⋆ (y) = sup y ′ z − f (z) .
z∈ℜn
Sec. B.5.4 Subdifferential Calculus 513

Pairs of this type are connected with the subdifferentials of f and f ⋆ , as

shown in the following proposition.

Proposition 5.4.3: (Conjugate Subgradient Theorem) Let f :

ℜn 7→ (−∞, ∞] be a proper convex function and let f ⋆ be its conju-
gate. The following two relations are equivalent for a pair of vectors
(x, y):
(i) x′ y = f (x) + f ⋆ (y).
(ii) y ∈ ∂f (x).
If in addition f is closed, the relations (i) and (ii) are equivalent to
(iii) x ∈ ∂f ⋆ (y).

For an application of the Conjugate Subgradient Theorem, note that

the necessary and sufficient optimality condition (B.15) in the Fenchel Du-
ality Theorem can be equivalently written as

A′ λ∗ ∈ ∂f1 (x∗ ), λ∗ ∈ −∂f2 (Ax∗ ).

The following proposition gives some useful corollaries of the Conjugate

Subgradient Theorem:

Proposition 5.4.4: Let f : ℜn 7→ (−∞, ∞] be a closed proper convex

function and let f ⋆ be its conjugate.


(a) f ⋆ is differentiable at a vector y ∈ int dom(f ⋆ ) if and only if
the supremum of x′ y − f (x) over x ∈ ℜn is uniquely attained.
(b) The set of minima of f is given by

arg minn f (x) = ∂f ⋆ (0).

x∈ℜ

Section 5.4.2. Subdifferential Calculus

We now generalize some of the basic theorems of ordinary differentiation

(Section 3.1 gives proofs for the case of real-valued functions).

Proposition 5.4.5: (Chain Rule) Let f : ℜm 7→ (−∞, ∞] be a

convex function, let A be an m × n matrix, and assume that the
function F given by
514 Chap. 5

F (x) = f (Ax)
is proper. Then

∂F (x) ⊃ A′ ∂f (Ax), ∀ x ∈ ℜn .

Furthermore, if either f is polyhedral or else the range of A contains

a point in the relative interior of dom(f ), we have

∂F (x) = A′ ∂f (Ax), ∀ x ∈ ℜn .

We also have the following proposition, which is a special case of the

preceding one [cf. the proof of Prop. 3.1.3(b)].

Proposition 5.4.6: (Subdifferential of Sum of Functions) Let

fi : ℜn 7→ (−∞, ∞], i = 1, . . . , m, be convex functions, and assume
that the function F = f1 + · · · + fm is proper. Then

∂F (x) ⊃ ∂f1 (x) + · · · + ∂fm (x), ∀ x ∈ ℜn .

Furthermore, if ∩m
i=1 ri dom(fi ) 6= Ø, we have

∂F (x) = ∂f1 (x) + · · · + ∂fm (x), ∀ x ∈ ℜn .

More generally, the same is true if for some m with 1 ≤ m ≤ m, the

functions fi , i = 1, . . . , m, are polyhedral and
  

∩m m
i=1 dom(fi ) ∩ ∩i=m+1 ri dom(fi ) 6= Ø.

Section 5.4.3. Optimality Conditions

It can be seen from the definition of subgradient that a vector x∗ minimizes
f over ℜn if and only if 0 ∈ ∂f (x∗ ). We give the following generalization
of this condition to constrained problems.

Proposition 5.4.7: Let f : ℜn 7→ (−∞, ∞] be a proper convex func-

tion, let X be a nonempty convex subset of ℜn , and assume that one
of the following four conditions holds:
Sec. B.5.4 Directional Derivatives 515

(1) ri dom(f ) ∩ ri(X) 6= Ø.
(2) f is polyhedral and dom(f ) ∩ ri(X) 6= Ø.


(3) X is polyhedral and ri dom(f ) ∩ X 6= Ø.
(4) f and X are polyhedral, and dom(f ) ∩ X 6= Ø.
Then, a vector x∗ minimizes f over X if and only if there exists g ∈
∂f (x∗ ) such that

g ′ (x − x∗ ) ≥ 0, ∀ x ∈ X. (B.18)

The relative interior condition (1) of the preceding proposition is au-

tomatically satisfied when f is real-valued [we have dom(f ) = ℜn ]; Section
3.1 gives a proof of the proposition for this case. If in addition, f is differ-
entiable, the optimality condition (B.18) reduces to the one of Prop. 1.1.8
of this appendix:

∇f (x∗ )′ (x − x∗ ) ≥ 0, ∀ x ∈ X.

Section 5.4.4. Directional Derivatives

For a proper convex function f : ℜn 7→ (−∞, ∞], the directional derivative

at any x ∈ dom(f ) in a direction d ∈ ℜn , is defined by

f (x + αd) − f (x)
f ′ (x; d) = lim . (B.19)
α↓0 α

An important fact here is that the ratio in Eq. (B.19) is monotonically

nonincreasing as α ↓ 0, so that the limit above is well-defined. To verify
this, note that for any α > 0, the convexity of f implies that for all α ∈
(0, α),
α  α α

f (x + αd) ≤ f (x + αd) + 1 − f (x) = f (x) + f (x + αd) − f (x) ,
α α α
so that
f (x + αd) − f (x) f (x + αd) − f (x)
≤ , ∀ α ∈ (0, α). (B.20)
α α
Thus the limit in Eq. (B.19) is well-defined (as a real number, or ∞, or
−∞) and an alternative definition of f ′ (x; d) is

f (x + αd) − f (x)
f ′ (x; d) = inf , d ∈ ℜn . (B.21)
α>0 α
516 Chap. 5

The directional derivative is related to the support function of the

subdifferential ∂f (x), as indicated in the following proposition.

Proposition 5.4.8: (Support Function of the Subdifferential)

Let f : ℜn 7→ (−∞, ∞] be a proper convex function, and let (cl f ′ )(x; ·)
be the closure of the directional derivative f ′ (x; ·).
(a) For all x ∈ dom(f ) such that ∂f (x) is nonempty, (cl f ′ )(x; ·) is
the support function of ∂f (x).


(b) For all x ∈ ri dom(f ) , f ′ (x; ·) is closed and it is the support
function of ∂f (x).

Directional Derivative of an Expected Value

A useful subdifferential formula relates to the subgradients of an expected

value function 
f (x) = E F (x, ω) ,
where ω is a random variable taking values in a set Ω, and F (·, ω) : ℜn 7→ ℜ
is a real-valued convex function such that f is real-valued (note that f is
easily verified to be convex). If ω takes a finite number of values with
probabilities p(ω), then the formulas
 
f ′ (x; d) = E F ′ (x, ω; d) , ∂f (x) = E ∂F (x, ω) , (B.22)
hold because they can be written in terms of finite sums as
X X
f ′ (x; d) = p(ω)F ′ (x, ω; d), ∂f (x) = p(ω)∂F (x, ω),
ω∈Ω ω∈Ω

so Prop. 5.4.6 applies. However, the formulas (B.22) hold even in the case
where Ω is uncountably infinite, with appropriate
 mathematical interpre-
tation of the integral of set-valued functions E ∂F (x, ω) as the set of
integrals Z
g(x, ω) dP (ω), (B.23)
ω∈Ω

where g(x, ω) ∈ ∂F (x, ω), ω ∈ Ω (measurability issues must be addressed

in this context). For a formal proof and analysis, see the author’s papers
[Ber72], [Ber73], which also provide a necessary and sufficient condition for
f to be differentiable, even when F (·, ω) is not. In this connection, it is
important to note that the integration over ω in Eq. (B.23) may smooth out
the nondifferentiabilities of F (·, ω) if ω is a “continuous” random variable.
This property can be used in turn in algorithms, including schemes that
bring to bear the methodology of differentiable optimization.
Sec. B.5.4 Directional Derivatives 517

Danskin’s Theorem

The following proposition derives its origin from a theorem by Danskin

[Dan67] that provides a formula for the directional derivative of the max-
imum of a (not necessarily convex) directionally differentiable function.
When adapted to a convex function f , this formula yields the expression
for ∂f (x) given in the proposition.

Proposition 5.4.9: (Danskin’s Theorem) Let Z ⊂ ℜm be a

compact set, and let φ : ℜn × Z 7→ ℜ be continuous and such that
φ(·, z) : ℜn 7→ ℜ is convex for each z ∈ Z.
(a) The function f : ℜn 7→ ℜ given by

f (x) = max φ(x, z) (B.24)

z∈Z

is convex and has directional derivative given by

f ′ (x; y) = max φ′ (x, z; y),

z∈Z(x)

where φ′ (x, z; y) is the directional derivative of the function φ(·, z)

at x in the direction y, and Z(x) is the set of maximizing points
in Eq. (B.24)
 
Z(x) = z φ(x, z) = max φ(x, z) .
z∈Z

In particular, if Z(x) consists of a unique point z and φ(·, z) is

differentiable at x, then f is differentiable at x, and ∇f (x) =
∇x φ(x, z), where ∇x φ(x, z) is the vector with coordinates

∂φ(x, z)
, i = 1, . . . , n.
∂xi

(b) If φ(·, z) is differentiable for all z ∈ Z and ∇x φ(x, ·) is continuous

on Z for each x, then

∂f (x) = conv ∇x φ(x, z) | z ∈ Z(x) , ∀ x ∈ ℜn . (B.25)

In particular, if φ is linear in x for all z ∈ Z, i.e.,

φ(x, z) = a′z x + bz , ∀ z ∈ Z,
518 Chap. 5

then 
∂f (x) = conv az | z ∈ Z(x) .

Proof: See Prop. 4.5.1 of the book by Bertsekas, Nedic, and Ozdaglar
[BNO03] or the author’s Convex Optimization Algorithms book [Ber15]
(with solution included). Q.E.D.

There is an extension of Danskin’s Theorem, which provides a more

general formula for the subdifferential ∂f (x) of the function

f (x) = sup φ(x, z), (B.26)

z∈Z

where Z is a compact set. This version of the theorem does not require
that φ(·, z) is differentiable. Instead it assumes that φ(·, z) is an extended

real-valued closed proper convex function
 for each z ∈ Z, that int dom(f )
[the interior of the set dom(f ) = x |
f (x) < ∞ ] is nonempty, and that

φ is continuous on the set int dom(f ) × Z. Then for all x ∈ int dom(f ) ,
we have 
∂f (x) = conv ∂φ(x, z) | z ∈ Z(x) ,
where ∂φ(x, z) is the subdifferential of φ(·, z) at x for any z ∈ Z, and Z(x)
is the set of maximizing points in Eq. (B.26); for a formal statement and
proof of this result, see Prop. A.22 of the author’s Ph.D. thesis, which may
be found on-line [Ber71].

Section 5.5. Minimax Theory

We will now provide theorems regarding the validity of the minimax equal-
ity and the existence of saddle points. These theorems are obtained by
specializing the MC/MC theorems of Chapter 4. We will assume through-
out this section the following:
(a) X and Z are nonempty convex subsets of ℜn and ℜm , respectively.
(b) φ : X × Z 7→ ℜ is a function such that φ(·, z) : X 7→ ℜ is convex and
closed for each z ∈ Z, and −φ(x, ·) : Z 7→ ℜ is convex and closed for
each x ∈ X.

Proposition 5.5.1: Assume that the function p given by


p(u) = inf sup φ(x, z) − u′ z , u ∈ ℜm ,
x∈X z∈Z
Sec. B.5.5 Minimax Theory 519

satisfies either p(0) < ∞, or else p(0) = ∞ and p(u) > −∞ for all
u ∈ ℜm . Then

sup inf φ(x, z) = inf sup φ(x, z)

z∈Z x∈X x∈X z∈Z

if and only if p is lower semicontinuous at u = 0.

Proposition 5.5.2: Assume that 0 ∈ ri dom(p) and p(0) > −∞.
Then
sup inf φ(x, z) = inf sup φ(x, z),
z∈Z x∈X x∈X z∈Z

and the supremum over Z in the left-hand side is finite and is attained.
Furthermore, the set of z ∈ Z attaining this supremum is compact if
and only if 0 lies in the interior of dom(p).

Proposition 5.5.3: (Classical Saddle Point Theorem) Let the

sets X and Z be compact. Then the set of saddle points of φ is
nonempty and compact.

To formulate more general saddle point theorems, we consider the

convex functions t : ℜn 7→ (−∞, ∞] and r : ℜm 7→ (−∞, ∞] given by

supz∈Z φ(x, z) if x ∈ X,
t(x) =
∞ if x ∈
/ X,

and
− inf x∈X φ(x, z) if z ∈ Z,
n
r(z) =
∞ if z ∈
/ Z.
Thus, by Prop. 3.4.1, (x∗ , z ∗ ) is a saddle point if and only if

sup inf φ(x, z) = inf sup φ(x, z),

z∈Z x∈X x∈X z∈Z

and x∗ minimizes t while z ∗ minimizes r.

The next two propositions provide conditions for the minimax equal-
ity to hold. These propositions are used to prove results about nonempti-
ness and compactness of the set of saddle points.
520 Chap. 5

Proposition
 5.5.4: Assume that t is proper and that the level sets
x | t(x) ≤ γ , γ ∈ ℜ, are compact. Then

sup inf φ(x, z) = inf sup φ(x, z),

z∈Z x∈X x∈X z∈Z

and the infimum over X in the right-hand side above is attained at a

set of points that is nonempty and compact.

Proposition 5.5.5: Assume that t is proper, and that the recession

cone and the constancy space of t are equal. Then

sup inf φ(x, z) = inf sup φ(x, z),

z∈Z x∈X x∈X z∈Z

and the infimum over X in the right-hand side above is attained.

Proposition 5.5.6: Assume that either t is proper or r is proper.

 
(a) If the level sets x | t(x) ≤ γ and z | r(z) ≤ γ , γ ∈ ℜ, of t
and r are compact, the set of saddle points of φ is nonempty and
compact.
(b) If the recession cones of t and r are equal to the constancy spaces
of t and r, respectively, the set of saddle points of φ is nonempty.

Proposition 5.5.7: (Saddle Point Theorem) The set of saddle

points of φ is nonempty and compact under any one of the following
conditions:
(1) X and Z are compact.
(2) Z is compact, and for some z ∈ Z, γ ∈ ℜ, the level set

x ∈ X | φ(x, z) ≤ γ

is nonempty and compact.

Sec. B.5.6 Theorems of the Alternative 521

(3) X is compact, and for some x ∈ X, γ ∈ ℜ, the level set


z ∈ Z | φ(x, z) ≥ γ

is nonempty and compact.

(4) For some x ∈ X, z ∈ Z, γ ∈ ℜ, the level sets
 
x ∈ X | φ(x, z) ≤ γ , z ∈ Z | φ(x, z) ≥ γ ,

are nonempty and compact.

Section 5.6. Theorems of the Alternative

Theorems of the alternative are important tools in optimization, which ad-

dress the feasibility (possibly strict) of affine inequalities. These theorems
can be viewed as special cases of MC/MC duality, as discussed in [Ber09].

Proposition 5.6.1: (Gordan’s Theorem) Let A be an m × n ma-

trix and b be a vector in ℜm . The following are equivalent:
(i) There exists a vector x ∈ ℜn such that

Ax < b.

(ii) For every vector µ ∈ ℜm ,

A′ µ = 0, b′ µ ≤ 0, µ≥0 ⇒ µ = 0.

(iii) Any polyhedral set of the form

{µ | A′ µ = c, b′ µ ≤ d, µ ≥ 0} ,

where c ∈ ℜn and d ∈ ℜ, is compact.

522 Chap. 5

Proposition 5.6.2: (Motzkin’s Transposition Theorem) Let A

and B be p × n and q × n matrices, and let b ∈ ℜp and c ∈ ℜq be
vectors. The system
Ax < b, Bx ≤ c
has a solution if and only if for all µ ∈ ℜp and ν ∈ ℜq , with µ ≥ 0,
ν ≥ 0, the following two conditions hold:

A′ µ + B ′ ν = 0 ⇒ b′ µ + c′ ν ≥ 0,

A′ µ + B ′ ν = 0, µ 6= 0 ⇒ b′ µ + c′ ν > 0.

Proposition 5.6.3: (Stiemke’s Transposition Theorem) Let A

be an m × n matrix, and let c be a vector in ℜm . The system

Ax = c, x>0

has a solution if and only if

A′ µ ≥ 0 and c′ µ ≤ 0 ⇒ A′ µ = 0 and c′ µ = 0.

The theorems of Gordan and Stiemke can be used to provide necessary

and sufficient conditions for the compactness of the primal and the dual
optimal solution sets of linear programs. We say that the primal linear
program
minimize c′ x
(B.27)
subject to a′j x ≥ bj , j = 1, . . . , r,

is strictly feasible if there exists a primal-feasible vector x ∈ ℜn with a′j x >

bj for all j = 1, . . . , r. Similarly, we say that the dual linear program

maximize b′ µ
r
X (B.28)
subject to aj µj = c, µ ≥ 0,
j=1

is strictly feasible if there exists a dual-feasible vector µ with µ > 0. We

have the following proposition.
Sec. B.5.6 Theorems of the Alternative 523

Proposition 5.6.4: Consider the primal and dual linear programs

(B.27) and (B.28), and assume that their common optimal value is
finite. Then:
(a) The dual optimal solution set is compact if and only if the primal
problem is strictly feasible.
(b) Assuming that the set {a1 , . . . , ar } contains n linearly indepen-
dent vectors, the primal optimal solution set is compact if and
only if the dual problem is strictly feasible.