Lecture 3

Convex Functions
September 2, 2008
Lecture 3
Outline
Convex Functions
Examples
Verifying Convexity of a Function
Operations on Functions Preserving Convexity
Convex Optimization 1
Lecture 3
Convex Functions
Informally: f is convex when for every segment [x
1
, x
2
], as x

=
x
1
+(1)x
2
varies over the line segment [x
1
, x
2
], the points (x

, f(x

))
lie below the segment connecting (x
1
, f(x
1
)) and (x
2
, f(x
2
))
Let f be a function from R
n
to R, f : R
n
R
The domain of f is a set in R
n
dened by
dom(f) = x R
n
[ f(x) is well dened (nite)
Def. A function f is convex if
(1) Its domain dom(f) is a convex set in R
n
and
(2) For all x
1
, x
2
dom(f) and (0, 1)
f(x
1
+ (1 )x
2
) f(x
1
) + (1 )f(x
2
)
Convex Optimization 2
Lecture 3
More on Convex Function
Def. A function f is strictly convex when dom(f) is convex and
f(x
1
+ (1 )x
2
) < f(x
1
) + (1 )f(x
2
)
for all x
1
, x
2
dom(f) and (0, 1)
Def. A function f is concave when f is convex, i.e.,
(1) Its domain dom(f) is a convex set in R
n
and
(2) For all x
1
, x
2
dom(f) and (0, 1)
f(x
1
+ (1 )x
2
) f(x
1
) + (1 )f(x
2
)
Def. A function f is strictly concave when f is strictly convex
Convex Optimization 3
Lecture 3
Examples on R
Convex:
Ane: ax + b over R for any a, b R
Exponential: e
ax
over R for any a R
Power: x
p
over (0, +) for p 1 or p 0
Powers of absolute value: [x[
p
over R for p 1
Negative entropy: xlnx over (0, +)
Concave:
Ane: ax + b over R for any a, b R
Powers: x
p
over (0, +) for 0 p 1
Logarithm: lnx over (0, +)
Convex Optimization 4
Lecture 3
Examples: Ane Functions and Norms
Ane functions are both convex and concave
Norms are convex
Examples on R
n
Ane function f(x) = a
t
x + b with a R
n
and b R
Euclidean, l
1
, and l

norms
General l
p
norms
|x|
p
=
_
n

i=1
[x
i
[
p
_
1/p
for p 1
Convex Optimization 5
Lecture 3
Examples on R
mn
The space R
mn
is the space of mn matrices
Ane function
f(X) = tr(A
T
X) + b =
m

i=1
n

j=1
a
ij
x
ij
+ b
Spectral (maximum singular value) norm
f(X) = |X|
2
=
max
(X) =
_

max
(X
T
X)
where
max
(A) denotes the maximum eigenvalue of a matrix A
Convex Optimization 6
Lecture 3
Verifying Convexity of a Function
We can verify that a given function f is convex by
Using the denition
Applying some special criteria
Second-order conditions
First-order conditions
Reduction to a scalar function
Showing that f is obtained through operations preserving convexity
Convex Optimization 7
Lecture 3
Second-Order Conditions
Let f be twice dierentiable and let dom(f) = R
n
[in general, it is
required that dom(f) is open]
The Hessian
2
f(x) is a symmetric n n matrix whose entries are the
second-order partial derivatives of f at x:
_

2
f(x)
_
ij
=

2
f(x)
x
i
x
j
for i, j = 1, . . . , n
2nd-order conditions: For a twice dierentiable f with convex domain
f is convex if and only if

2
f(x) 0 for all x dom(f)
f is strictly convex if

2
f(x) ~ 0 for all x dom(f)
Convex Optimization 8
Lecture 3
Examples
Quadratic function: f(x) = (1/2)x
t
Px + q
t
x + r with a symmetric
n n matrix P
f(x) = Px + q,
2
f(x) = P
Convex for P 0
Least-squares objective: f(x) = |Ax b|
2
with an mn matrix A
f(x) = 2A
T
(Ax b),
2
f(x) = 2A
T
A
Convex for any A
Quadratic-over-linear: f(x, y) = x
2
/y

2
f(x, y) =
2
y
3
_
y
x
_ _
y
x
_
T
0
Convex for y > 0
Convex Optimization 9
Lecture 3
Verifying Convexity of a Function
We can verify that a given function f is convex by
Using the denition
Applying some special criteria
Second-order conditions
First-order conditions
Reduction to a scalar function
Showing that f is obtained through operations preserving convexity
Convex Optimization 10
Lecture 3
First-Order Condition
f is dierentiable if dom(f) is open and the gradient
f(x) =
_
f(x)
x
1
,
f(x)
x
2
, . . . ,
f(x)
x
n
_
exists at each x domf
1st-order condition: dierentiable f is convex if and only if its domain is
convex and
f(x) +f(x)
T
(z x) f(z) for all x, z dom(f)
A rst order approximation is a global underestimate of f
Very important property used in algorithm designs and performance analysis
Convex Optimization 11
Lecture 3
Restriction of a convex function to a line
f is convex if and only if domf is convex and the function g : R R,
g(t) = f(x + tv), domg = t [ x + tv dom(f)
is convex (in t) for any x domf, v R
n
Checking convexity of multivariable functions can be done by checking
convexity of functions of one variable
Example f : S
n
R with f(X) = lndet X, domf = S
n
++
g(t) = lndet(X + tV ) = lndet X lndet(I + tX
1/2
V X
1/2
)
= lndet X
n

i=1
ln(1 + t
i
)
where
i
are the eigenvalues of X
1/2
V X
1/2
g is convex in t (for any choice of V and any X ~ 0); hence f is concave
Convex Optimization 12
Lecture 3
Operations Preserving Convexity
Positive Scaling
Sum
Composition with ane function
Pointwise maximum and supremum
Composition
Minimization
Convex Optimization 13
Lecture 3
Scaling, Sum, & Composition with Ane Function
Positive multiple For a convex f and > 0, the function f is convex
Sum: For convex f
1
and f
2
, the sum f
1
+ f
2
is convex
(extends to innite sums, integrals)
Composition with ane function: For a convex f and ane g [i.e.,
g(x) = Ax + b], the composition f g is convex, where
(f g)(x) = f(Ax + b)
Examples
Log-barrier for linear inequalities
f(x) =
m

i=1
ln(b
i
a
T
i
x), domf = x [ a
T
i
x < b
i
, i = 1, . . . , m
(Any) Norm of ane function: f(x) = |Ax + b|
Convex Optimization 14
Lecture 3
Pointwise maximum
For convex functions f
1
, . . . , f
m
, the pointwise-max function
F(x) = max f
1
(x), . . . , f
m
(x)
is convex (What is domain of F?)
Examples
Piecewise-linear function: f(x) = max
i=1,...,m
(a
T
i
x + b
i
) is convex
Sum of r largest components of a vector x R
n
:
f(x) = x
[1]
+ x
[2]
+ + x
[r]
is convex (x
[i]
is i-th largest component of x)
f(x) = max
(i
1
,...,i
r
)I
r
x
i
1
+ x
i
2
+ + x
i
r

I
r
= (i
1
, . . . , i
r
) [ i
1
< . . . < i
r
, i
j
1, . . . , m, j = 1, . . . , n
Convex Optimization 15
Lecture 3
Pointwise Supremum
Let , R
p
and f : R
n
R
p
R. Let f(x, z) be convex in x for
each z ,. Then, the supremum function over the set , is convex:
g(x) = sup
z,
f(x, z)
Examples
Set support function is convex for a set C R
n
,
S
C
: R
n
R, S
C
(x) = sup
zC
z
T
x
Set farthest-distance function is convex for a set C R
n
,
f : R
n
R, f(x) = sup
zC
|x z|
Maximum eigenvalue function of a symmetric matrix is convex

max
: S
n
R,
max
(X) = sup
|z|=1
z
T
Xz
Convex Optimization 16
Lecture 3
Composition with Scalar Functions
Composition of g : R
n
R and h : R R with dom(g) = R
n
and
dom(h) = R:
f(x) = h(g(x))
f is convex if
(1) g is convex, h is nondecreasing and convex
(2) g is concave, h is nonincreasing and convex
Examples
e
g(x)
is convex if g is convex

1
g(x)
is convex if g is concave and positive
Convex Optimization 17
Lecture 3
Composition with Vector Functions
Composition of g : R
n
R
p
and h : R
p
R with dom(g) = R
n
and
dom(h) = R
p
:
f(x) = h(g(x)) = h(g
1
(x), g
2
(x), . . . , g
p
(x))
f is convex if
(1) each g
i
is convex, h is convex and nondecreasing in each argument
(2) each g
i
is concave, h is convex and nonincreasing in each argument
Example

m
i=1
e
g
i
(x)
is convex if g
i
are convex
Convex Optimization 18
Lecture 3
Extended-Value Functions
A function f is an extended-value function if f : R
n
R , +
Example: consider f(x) = inf
y0
xy for x R
Def. The epigraph of a function f over R
n
is the following set in R
n+1
:
epif = (x, w) R
n+1
[ x R
n
, f(x) w
General Convex Function Def. A function f is convex if its epigraph
epif is a convex set in R
n+1
This denition is equivalent to the one we have used so far (when reduced
to the function class we have considered thus far). How?
For an f with domain domf, we associate an extended-value function

f
dened by

f(x) =
_
_
_
f(x) if x domf
+ otherwise
domf is the projection of epif on R
n
; convexity of f by letting w = f(x)
Convex Optimization 19
Lecture 3
Minimization
Let C R
n
R
p
be a nonempty convex set
Let f : R
n
R
p
R be a convex function [in (x, z) R
n
R
p
]. Then
g(x) = inf
zC
f(x, z) is convex
Example
Distance to a set: for a nonempty convex C R
n
,
dist(x, C) = inf
zC
|x z| is convex
Proof: Let x
1
, x
2
R
n
and (0, 1) be arbitrary. Let > 0 be arbitrarily
small. Then, there exist z
1
, z
2
C such that f(x
1
, z
1
) g(x
1
) + and
f(x
2
, z
2
) g(x
2
) + . Consider f(x
1
+ (1 )x
2
, z
1
+ (1 )z
2
)
and use convexity of f and C.
Convex Optimization 20