L.

Vandenberghe EE236C (Spring 2013-14)

8. Conjugate functions

• closed functions

• conjugate function

8-1
 Closed set

a set C is closed if it contains its boundary:

xk ∈ C, xk → x̄ =⇒ x̄ ∈ C

operations that preserve closedness

• the intersection of (finitely or infinitely many) closed sets is closed

• the union of a finite number of closed sets is closed

• inverse under linear mapping: {x | Ax ∈ C} is closed if C is closed

Conjugate functions 8-2

 Image under linear mapping

the image of a closed set under a linear mapping is not necessarily closed

example (C is closed, AC = {Ax | x ∈ C} is open):

R2+
 
C = {(x1, x2) ∈ | x1x2 ≥ 1}, A= 1 0 , AC = R++

sufficient condition: AC is closed if

• C is closed and convex

• and C does not have a recession direction in the nullspace of A. i.e.,

Ay = 0, x̂ ∈ C, x̂ + αy ∈ C ∀α ≥ 0 =⇒ y=0

in particular, this holds for any A if C is bounded

Conjugate functions 8-3

 Closed function

definition: a function is closed if its epigraph is a closed set

examples

• f (x) = − log(1 − x2) with dom f = {x | |x| < 1}

• f (x) = x log x with dom f = R+ and f (0) = 0
• indicator function of a closed set C: f (x) = 0 if x ∈ C = dom f

not closed

• f (x) = x log x with dom f = R++, or with dom f = R+ and f (0) = 1

• indicator function of a set C if C is not closed

Conjugate functions 8-4

 Properties

sublevel sets: f is closed if and only if all its sublevel sets are closed

minimum: if f is closed with bounded sublevel sets then it has a minimizer

common operations on convex functions that preserve closedness

• sum: f + g is closed if f and g are closed (and dom f ∩ dom g 6= ∅)

• composition with affine mapping: f (Ax + b) is closed if f is closed

• supremum: supα fα(x) is closed if each function fα is closed

Conjugate functions 8-5

 Outline

• closed functions

• conjugate function
 Conjugate function

f (x)
xy
the conjugate of a function f is

f ∗(y) = sup (y T x − f (x))

x∈dom f
x

(0, −f ∗(y))

f ∗ is closed and convex even if f is not

Fenchel’s inequality

f (x) + f ∗(y) ≥ xT y ∀x, y

(extends inequality xT x/2 + y T y/2 ≥ xT y to non-quadratic convex f )

Conjugate functions 8-6

 Quadratic function

1 T
f (x) = x Ax + bT x + c
2

strictly convex case (A ≻ 0)

1
f (y) = (y − b)T A−1(y − b) − c
∗
2

general convex case (A  0)

1
f ∗(y) = (y − b)T A†(y − b) − c, dom f ∗ = range(A) + b
2

Conjugate functions 8-7

 Negative entropy and negative logarithm

negative entropy

n
X n
X
f (x) = xi log xi f ∗(y) = eyi−1
i=1 i=1

negative logarithm

n
X n
X
f (x) = − log xi f ∗(y) = − log(−yi) − n
i=1 i=1

matrix logarithm

f (X) = − log det X (dom f = Sn++) f ∗(Y ) = − log det(−Y ) − n

Conjugate functions 8-8

 Indicator function and norm

indicator of convex set C: conjugate is support function of C


0 x∈C
f (x) = f ∗(y) = sup y T x
+∞ x ∈
6 C x∈C

norm: conjugate is indicator of unit dual norm ball


0 kyk∗ ≤ 1
f (x) = kxk f ∗(y) =
+∞ kyk∗ > 1

(see next page)

Conjugate functions 8-9

proof: recall the definition of dual norm:

kyk∗ = sup xT y
kxk≤1

∗ T


to evaluate f (y) = supx y x − kxk we distinguish two cases

• if kyk∗ ≤ 1, then (by definition of dual norm)

y T x ≤ kxk ∀x

and equality holds if x = 0; therefore supx (y T x − kxk) = 0

• if kyk∗ > 1, there exists an x with kxk ≤ 1, xT y > 1; then

f ∗(y) ≥ y T (tx) − ktxk = t(y T x − kxk)

and r.h.s. goes to infinity if t → ∞

Conjugate functions 8-10

 The second conjugate

f ∗∗(x) = sup (xT y − f ∗(y))

y∈dom f ∗

• f ∗∗ is closed and convex

• from Fenchel’s inequality (xT y − f ∗(y) ≤ f (x) for all y and x):

f ∗∗(x) ≤ f (x) ∀x

equivalently, epi f ⊆ epi f ∗∗ (for any f )

• if f is closed and convex, then

f ∗∗(x) = f (x) ∀x

equivalently, epi f = epi f ∗∗ (if f is closed convex); proof on next page

Conjugate functions 8-11

proof (f ∗∗ = f if f is closed and convex): by contradiction
suppose (x, f ∗∗(x)) 6∈ epi f ; then there is a strict separating hyperplane:
 T  
a z−x
≤c<0 ∀(z, s) ∈ epi f
b s − f ∗∗(x)

for some a, b, c with b ≤ 0 (b > 0 gives a contradiction as s → ∞)

• if b < 0, define y = a/(−b) and maximize l.h.s. over (z, s) ∈ epi f :

f ∗(y) − y T x + f ∗∗(x) ≤ c/(−b) < 0

this contradicts Fenchel’s inequality

• if b = 0, choose ŷ ∈ dom f ∗ and add small multiple of (ŷ, −1) to (a, b):
 T  
a + ǫŷ z−x ∗ T ∗∗


≤ c + ǫ f (ŷ) − x ŷ + f (x) < 0
−ǫ s − f ∗∗(x)

now apply the argument for b < 0

Conjugate functions 8-12

 Conjugates and subgradients

if f is closed and convex, then

y ∈ ∂f (x) ⇐⇒ x ∈ ∂f ∗(y) ⇐⇒ xT y = f (x) + f ∗(y)

proof: if y ∈ ∂f (x), then f ∗(y) = supu(y T u − f (u)) = y T x − f (x)

f ∗(v) = sup (v T u − f (u))

u

≥ v T x − f (x)
= xT (v − y) − f (x) + y T x
= f ∗(y) + xT (v − y)

for all v; therefore, x is a subgradient of f ∗ at y (x ∈ ∂f ∗(y))

reverse implication x ∈ ∂f ∗(y) =⇒ y ∈ ∂f (x) follows from f ∗∗ = f

Conjugate functions 8-13

 Some calculus rules

separable sum

f (x1, x2) = g(x1) + h(x2) f ∗(y1, y2) = g ∗(y1) + h∗(y2)

scalar multiplication: (for α > 0)

f (x) = αg(x) f ∗(y) = αg ∗(y/α)

addition to affine function

f (x) = g(x) + aT x + b f ∗(y) = g ∗(y − a) − b

infimal convolution

f (x) = inf (g(u) + h(v)) f ∗(y) = g ∗(y) + h∗(y)

u+v=x

Conjugate functions 8-14

 References

• J.-B. Hiriart-Urruty, C. Lemaréchal, Convex Analysis and Minimization

Algoritms (1993), chapter X

• D.P. Bertsekas, A. Nedić, A.E. Ozdaglar, Convex Analysis and

Optimization (2003), chapter 7.

• R. T. Rockafellar, Convex Analysis (1970)

Conjugate functions 8-15