Constrained Optimization

Dr Ramkrishna Pasumarthy

Professor, IIT Madras

Web MTech Program 2022

28 September, 2022

Dr Ramkrishna Pasumarthy (IIT Madras) Optimization 28 September, 2022 1 / 21

 Overview

1 Formulation

2 Example:Single Equality Constraint

3 Example:Single Inequality Constraint

4 Example:Single Inequality Constraint

5 Example:KKT Conditions

Dr Ramkrishna Pasumarthy (IIT Madras) Optimization 28 September, 2022 2 / 21

 Formulation

There are now equality and inequality constraints in the values that
the argument can take

minx∈Rn f (x) subject to ci (x) = 0, i ∈ E

ci (x) ≥ 0 i ∈ I

E: index for equality constraints

I : index for inequality constraints
The theory for constrained optimization is built by looking at three
examples

Dr Ramkrishna Pasumarthy (IIT Madras) Optimization 28 September, 2022 3 / 21

 Example 1: Single Equality Constraint

minx1 ,x2 (x1 + x2 ) subject to x12 + x22 − 2 = 0

f (x) = x1 + x2
I=ϕ
E = {1}
c1 (x) = x12 + x22 − 2
Intuition tells that the constraint set is a circle and the global
minimum is (−1, −1) as that is the point possible in the circle where
x1 + x2 is the most negative and hence lowest
From any other point, it is feasible to move in the circle to decrease f
- simply move anti-clockwise or clockwise in the direction towards
(−1, −1)
Dr Ramkrishna Pasumarthy (IIT Madras) Optimization 28 September, 2022 4 / 21
 Geometry of the Problem

Figure: ∇f parallel to ∇c1 at the minimum (−1, −1)

Dr Ramkrishna Pasumarthy (IIT Madras) Optimization 28 September, 2022 5 / 21
 Observations

One observes that at the minimum x ∗ = (−1, −1), the constraint

normal ∇c1 (x ∗ ) is parallel to ∇f (x ∗ )
Or in other words, there exists a scalar λ1 such that

∇f (x ∗ ) = λ1 ∇c1 (x ∗ )
1
It can be verified that λ1 = 2

Dr Ramkrishna Pasumarthy (IIT Madras) Optimization 28 September, 2022 6 / 21

 Motivation for the Lagrangian

One can derive the condition ∇f (x ∗ ) = λ1 ∇c1 (x ∗ ) from an

unconstrained optimization by considering the problem

minx1 ,x2 ,λ1 L = f (x) − λ1 c1 (x)

We have that the unconstrained first order conditions for this problem
are

∇x L = (∇x f − λ1 ∇x c1 ) = 0
∇λ1 L = −c1 (x) = 0

Both constraint and the gradient conditions come out of the function
L called the Lagrangian and λ1 , the Lagrange multiplier for c1
Warning: This being first order is only a necessary condition. Eg.
(1, 1) also satisfies the condition but is a maxium and not a minimum

Dr Ramkrishna Pasumarthy (IIT Madras) Optimization 28 September, 2022 7 / 21

 Proof of the First Order Condition
First Order Taylor Expansions:
c1 (x + s) ≈ c1 (x) + ∇c1 (x)T s
f (x + s) ≈ f (x) + ∇f (x)T s
For c to be feasible (remain in the circle) upto first order,
c1 (x + s) − c1 (x) ≈ 0 and hence ∇c1 (x)T s = 0. We require that any
feasible direction (satisfying ∇c1 (x)T s = 0) should lead to only
increase of the function f .
So, the component of the gradient along the direction where
∇c1 (x)T s = 0 must vanish
The feasible direction can be characterised as cfeas = {∇c1 (x)}⊥
Hence we must have
∇c1 (x)∇c1 (x)T
∇f (x) − ∇f (x) = 0
||∇c1 (x)||2
∇c1 (x)T ∇f (x)
Hence , λ1 = ||∇c1 (x)||2
Dr Ramkrishna Pasumarthy (IIT Madras) Optimization 28 September, 2022 8 / 21
 Example 2: Single Inequality Constraint

minx1 ,x2 (x1 + x2 ) subject to 2 − x12 − x22 ≥ 0

Same as before: Equality replaced by inequality

f (x) = x1 + x2
E =ϕ
I = {1}
c1 (x) = 2 − x12 − x22
Intuition tells that the constraint set is a disc and the global minimum
is still (−1, −1) as that is the point possible in the disc where x1 + x2
is the most negative and hence lowest
1
One notes that −λ1 = 2 ≥0

Dr Ramkrishna Pasumarthy (IIT Madras) Optimization 28 September, 2022 9 / 21

 Geometry of the Problem

Figure:
∇c1 is towards the interior of the feasible region
∇c1 is parallel to ∇f at (−1, −1)
s should also point in the interior of the feasible region
Dr Ramkrishna Pasumarthy
(IIT Madras) Optimization 28 September, 2022 10 / 21
 First Order Condition: In Interior

Consider the case when x is inside the interior of the circle - it is

possible to move in all directions from x. i.e. x + s ∈ disc , for all s
with ||s|| small enough
In this case, c1 (x) < 0
Then by choosing s = −∇f (x), one can decrease the value of the
function as for unconstrained first order condition
So, we must have

∇f (x) = 0 when c1 (x) < 0

Dr Ramkrishna Pasumarthy (IIT Madras) Optimization 28 September, 2022 11 / 21

 First Order Condition: In the Boundary

At the boundary, c1 (x) = 0

What is the restriction of s at the boundary?

0 ≤ c1 (x + s) ≈ c (x) +∇c1 (x)T s

|1{z }
=0 at boundary

So, s should satisfy ∇c1 (x)T s ≥ 0

f should be decreasing in some direction when x is not a minimum -
i.e. or

f (x + s) − f (x) ≈ ∇f (x)T s
| {z }
<0

Dr Ramkrishna Pasumarthy (IIT Madras) Optimization 28 September, 2022 12 / 21

 Geometry of the search direction constraints

Figure: Intersection of the spaces ∇c1 (x)T s ≥ 0 and ∇f (x)T s < 0

Dr Ramkrishna Pasumarthy (IIT Madras) Optimization 28 September, 2022 13 / 21

 First Order Condition at Boundary

The gray area where f can decrease if and only if ∇f (x) and ∇c1 (x)
are parallel - point in the same direction. i.e.

∇f (x) = λ1 c1 (x) where λ1 ≥ 0

So, we have a first order condition at the boundary c1 (x) = 0 that

c1 (x) = 0
∇f (x) = λ1 c1 (x) where λ1 ≥ 0

Dr Ramkrishna Pasumarthy (IIT Madras) Optimization 28 September, 2022 14 / 21

 Summary of First Order Conditions
At boundary:

c1 (x) = 0
∇f (x) = λ1 c1 (x) where λ1 ≥ 0

At Interior:

c1 (x) > 0
∇f (x) = 0

The interior condition can be rewritten as

At Interior:

c1 (x) > 0
∇f (x) = λ1 ∇c1 (x)
λ1 = 0

Dr Ramkrishna Pasumarthy (IIT Madras) Optimization 28 September, 2022 15 / 21

 Unified First Order Condition

Both the conditions can be summarized together as

c1 (x) ≥ 0
λ1 ≥ 0
∇f (x) = λ1 ∇c1 (x)
λ1 c1 (x) = 0

Dr Ramkrishna Pasumarthy (IIT Madras) Optimization 28 September, 2022 16 / 21

 Lagrangian

By defining once again the Lagrange function as L = f (x) − λ1 c1 (x),

the first order conditions can be summarized as

∇x,λ L = 0
for λ1 ≥ 0
λ1 c1 (x) = 0

λ1 c1 (x) = 0: complementary slackness condition

c1 (x) = 0 =⇒ constraint is active 1

λ1 > 0 =⇒ Lagrange Multiplier is active

Lagrange multiplier can be active only when the constraint is active -
complementary
NOTE: An equality constraint is always active !

1
at the boundary, one is just about to violate the constraint!
Dr Ramkrishna Pasumarthy (IIT Madras) Optimization 28 September, 2022 17 / 21
 Example 3: Two Inequality Constraints

minx1 ,x2 (x1 + x2 ) subject to

x12 + x22 − 2 = 0
x2 ≥ 0

f (x) = x1 + x2
I = {1, 2}
E =ϕ
c1 (x) = 2 − x12 − x22 and c2 (x) = x2
Intuition tells that the constraint
√ set is the upper semi-circlular disc
and the global minimum is (− 2, 0) as that is the point possible in
the circle where x1 + x2 is the most negative and hence lowest
√
NOTE: At (− 2, 0), both constraints are active - c1 (x) = c2 (x) = 0
Dr Ramkrishna Pasumarthy (IIT Madras) Optimization 28 September, 2022 18 / 21
 Geometry of the Problem

√
Figure: The gradient ∇f (x) at (− 2, 0) is a non-negative combination of the
gradients of c1 and c2 - points interior of the feasible set demanded by both the
constraints

Dr Ramkrishna Pasumarthy (IIT Madras) Optimization 28 September, 2022 19 / 21

 First Order Conditions - Generic Case - KKT Conditions

X
L = f (x) − λi ci (x)
i∈E∪I

KKT Conditions at a local minimum:

P
∇x L = ∇f (x) − i∈E∪I λi ∇ci (x) = 0
ci (x) ≥ 0 for all x ∈ I
ci (x) = 0 for all x ∈ E
λi ≥ 0 for all x ∈ I
λi ci (x) = 0 for all x ∈ I (complementary slackness)

Dr Ramkrishna Pasumarthy (IIT Madras) Optimization 28 September, 2022 20 / 21

 Second Order Sufficient Condition

For the unconstrained case, it was noted that ∇2 f (x) being positive
definite guaranteed a minimum. So, we have

w T (∇2 f )w > 0 for all w ̸= 0

Let C be the set of all possible search directions from x in the

constrained case
For constrained case we have,
Second Order Condition for constrained optimization:

w T (∇xx L)w > 0 for all w ∈ C

Dr Ramkrishna Pasumarthy (IIT Madras) Optimization 28 September, 2022 21 / 21