Introduction to Optimization

(iii) Lagrange Multipliers and

Kuhn-tucker Conditions

Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Objectives

 To study the optimization with multiple decision variables

and equality constraint : Lagrange Multipliers.

 To study the optimization with multiple decision variables

and inequality constraint : Kuhn-Tucker (KT) conditions

2 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Constrained optimization with equality
constraints
A function of multiple variables, f(x), is to be optimized subject to one or more
equality constraints of many variables. These equality constraints, gj(x), may or may

not be linear. The problem statement is as follows:

Maximize (or minimize) f(X), subject to gj(X) = 0, j = 1, 2, … , m

where  x1 
x 
  (1)
X  2 
 
 xn 

3 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Constrained optimization…

 With the condition that m n ; or else if m > n then the problem

becomes an over defined one and there will be no solution. Of the

many available methods, the method of constrained variation and the

method of using Lagrange multipliers are discussed.

4 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Solution by method of Lagrange
multipliers
 Continuing with the same specific case of the optimization problem with

n = 2 and m = 1 we define a quantity λ, called the Lagrange multiplier as

f / x2
  (2)
g / x2 (x1* , x 2* )
 Using this in the constrained variation of f [ given in the previous lecture]
 f g / x1 f 
df    dx1 0
 x1 g / x2 x2  (x1 *, x 2 *)

 f g 
And (2) written as   
(3) x  0

 1x 1  (x * , x * )
1 2

 f g 
    0
 x2 x2  (x * , x * ) (4)
1 2

5 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
Solution by method of Lagrange multipliers…
 Also, the constraint equation has to be satisfied at the extreme point

g ( x1 , x2 ) ( x * , x * )  0 (5)
1 2

 Hence equations (2) to (5) represent the necessary conditions for the point

[x1*, x2*] to be an extreme point.

g / x1
 λ could be expressed in terms of g / x1 as well has to be non-
zero.
 These necessary conditions require that at least one of the partial

derivatives of g(x1 , x2) be non-zero at an extreme point.

6 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
Solution by method of Lagrange multipliers…
 The conditions given by equations (2) to (5) can also be generated by

constructing a functions L, known as the Lagrangian function, as

L( x1 , x2 ,  )  f ( x1 , x2 )   g ( x1 , x2 ) (6)

 Alternatively, treating L as a function of x1,x2 and , the necessary

conditions for its extremum are given by

L f g
( x1 , x2 ,  )  ( x1 , x2 )   ( x1 , x2 ) 0
x1 x1 x1
L f g
( x1 , x2 ,  )  ( x1 , x2 )   ( x1 , x2 ) 0
x2 x2 x2
(7)
L
( x1 , x2 ,  )  g ( x1 , x2 ) 0


7 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Necessary conditions for a general problem
 For a general problem with n variables and m equality constraints the

problem is defined as shown earlier

 Maximize (or minimize) f(X), subject to gj(X) = 0, j = 1, 2, … , m

 x1 
x 
 
X  2 
where
 
 xn 

 In this case the Lagrange function, L, will have one Lagrange multiplier j
for each constraint as
L( x1 , x2 ,..., xn , 1 , 2 ,..., m )  f ( X)  1 g1 ( X)  2 g 2 ( X)  ...  m g m ( X)
(8)

8 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Necessary conditions for a general problem…
 L is now a function of n + m unknowns, x1 , x2 ,..., xn , 1 , 2 ,..., m , and the
necessary conditions for the problem defined above are given by
L f m g j
 ( X)    j ( X) 0, i 1, 2,..., n; j 1, 2,..., m
xi xi j 1 xi

L (9)
 g j ( X) 0, j 1, 2,..., m
 j

 which represent n + m equations in terms of the n + m unknowns, xi and j.

The solution to this set of equations gives us
 x1*   1* 
 *  *
x   
X  2   *  2 
  and   (10)
 xn*  m* 

9 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Sufficient conditions for a general problem
 A sufficient condition for f(X) to have a relative minimum at X* is that each

root of the polynomial in Є, defined by the following determinant equation

be positive.
L11   L12  L1n g11 g 21  g m1
L21 L22   L2 n g12 g 22 gm2
     
Ln1 Ln 2  Lnn   g1n g 2 n  g mn
0
(11) g11 g12  g1n 0   0
g 21 g 22 g2n   
    
g m1 gm2  g mn 0   0

10 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Sufficient conditions for a general problem…

where 2 L
Lij  ( X* ,  * ), for i 1, 2,..., n and j 1, 2,..., m
xi x j (12)
g p
g pq  ( X* ), where p 1, 2,..., m and q 1, 2,..., n
xq

 Similarly, a sufficient condition for f(X) to have a relative maximum at X*

is that each root of the polynomial in Є, defined by equation (11) be

negative.
 If equation (11), on solving yields roots, some of which are positive and

others negative, then the point X* is neither a maximum nor a minimum.

11 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Example
2 2
Minimize f ( X)  3x1  6 x1 x2  5 x2  7,x1  5 x2
Subject to x1  x2 5

Solution
g1 ( X)  x1  x2  5 0
L( x1 , x2 ,..., xn , 1 , 2 ,..., m )  f ( X)  1 g1 ( X)  2 g 2 ( X)  ...  m g m ( X)
with n = 2 and m = 1
L =3 x12  6 x1 x2  5 x22  7 x1  5 x2  1 ( x1  x2  5)
L
 6 x1  6 x2  7  1 0
x1
1
 x1  x2  (7  1 )
6
1
 5  (7  1 ) or1 23
6
12 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Example… L
 6 x1  10 x2  5  1 0
x2
1
 3 x1  5 x2  (5  1 )
2
1
 3( x1  x2 )  2 x2  (5  1 )
2
1 11
x2  x1 
2 2

 11  1 
Hence, X*  ,  ; λ*  23
2 2

 L11   L12 g11 

 
 21L L22   g 21  0
 g g12 0 
 11
13 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Example…
2 L 2 L 2 L
L11  2  6 L12 L21   6 L22  2  10
x1 ( X*,λ*) x1x2 ( X*,λ* )
x2 ( X*,λ*)

g1
g11  1
x1 ( X*,λ* )

g1
g12  g 21  1
x2 ( X*,λ* )

The determinant becomes   6  6 1

 
  6  10   1  0
 1 1 0 

or ( 6 )[ 1]  ( 6)[ 1]  1[ 6  10 ] 0
 2
Since  is negative, X*, λ * correspond to a maximum
14 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Kuhn – Tucker Conditions

 KT condition: Both necessary and sufficient if the objective function

is concave and each constraint is linear or each constraint function is

concave, i.e., the problems belongs to a class called the convex

programming problems.

15 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Kuhn-Tucker Conditions: Optimization
Model
Consider the following optimization problem
Minimize f(X)

subject to
gj(X) ≤ 0 for j=1,2,…,p

where the decision variable vector

X=[x1,x2,…,xn]

16 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Kuhn-Tucker Conditions
Kuhn-Tucker conditions for X* = [x1* , x2* , . . . xn*] to be a local minimum are

f m
g
  j 0 i 1, 2,..., n
xi j 1 xi
 j g j 0 j 1, 2,..., m
g j 0 j 1, 2,..., m
 j 0 j 1, 2,..., m

17 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Kuhn Tucker Conditions …

 In case of minimization problems, if the constraints are of

the form gj(X) ≥ 0, then λj have to be non-positive

 If the problem is one of maximization with the constraints

in the form gj(X) ≥ 0, then λj have to be nonnegative.

18 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Example 1
Minimize
2 2 2
f x  2 x  3x
1 2 3

subject to

g1  x1  x2  2 x3 12
g 2  x1  2 x2  3x3 8

19 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Example 1…
Kuhn – Tucker Conditions
f g g 2 x1  1  2 0 (14)
 1 1  2 2 0 4 x2  1  22 0 (15)
xi xi xi
6 x3  21  32 0 (16)

 j g j 0 1 ( x1  x2  2 x3  12) 0 (17)
2 ( x1  2 x2  3x3  8) 0 (18)

x1  x2  2 x3  12 0 (19)
g j 0
x1  2 x2  3x3  8 0 (20)

1 0 (21)
 j 0 2 0 (22)

20 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Example 1…
From (17) either 1 = 0 or x1  x2  2 x3  12 0 ,

Case 1: 1 = 0

From (14), (15) and (16) we have x1 = x2 =

 2 / 2 2 / 2
and x3 =
22  82 0,  2 0 or  8
Using these in (18) we get
2 0 2
From (22), , therefore, =0,
Therefore, X* = [ 0, 0, 0 ]

This solution set satisfies all of (18) to (21)

21 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Example 1…
Case 2: x1  x2  2 x3  12 0
 1  2 1  22 21  32
Using (14), (15) and (16), we have    12 0
2 4 3
or 171  122  144

But conditions (21) and (22) give us1 0 and2 0

17  12  144
simultaneously, which
1 cannot
2 be
possible with

Hence the solution set for this optimization problem is

X* = [ 0 0 0 ]
22 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Example 2
Minimize

f  x12  x22  60 x1

subject to
g1  x1  80 0
g 2  x1  x2  120 0

23 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Example 2…
Kuhn – Tucker Conditions

f g g 2 x1  60  1  2 0 (23)
 1 1  2 2 0
xi xi xi 2 x2  2 0 (24)

1 ( x1  80) 0 (25)
 j g j 0 2 ( x1  x2  120) 0 (26)

x1  80 0 (27)
g j 0 x1  x2  120 0 (28)

1 0 (29)
 j 0
2 0 (30)

24 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Example 2…

From (25) either 1 = 0 or ( x1  80) 0 ,

Case 1
2 2
From (23) and (24) we have x1  2
 30 andx2  2
Using these in (26) we get2 2  150  0
 2 0 or  150

Considering2 0 , X* = [ 30, 0]. But this solution set violates (27)

and (28)
For2  150 , X* = [ 45, 75]. But this solution set violates (27)

25 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Example 2…

Case 2: ( x1  80) 0

Using x1 80 in (23) and (24), we have

2  2 x2
1 2 x2  220 (31)

Substitute (31) in (26), we have 2 x2  x2  40  0

For this to be true, either x2 0 or x2  40 0

26 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Example 2…

Forx2 0 ,1  220

This solution set violates (27) and (28)

Forx2  40 0 ,1  140 and 2  80

This solution set is satisfying all equations from (27) to (31) and hence

the desired

Thus, the solution set for this optimization problem is

X* = [ 80, 40 ]

27 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 BIBLIOGRAPHY / FURTHER READING

1. Rao S.S., Engineering Optimization – Theory and Practice, Fourth

Edition, John Wiley and Sons, 2009.

2. Ravindran A., D.T. Phillips and J.J. Solberg, Operations Research –

Principles and Practice, John Wiley & Sons, New York, 2001.
3. Taha H.A., Operations Research – An Introduction, 8th edition, Pearson

Education India, 2008.

4. Vedula S., and P.P. Mujumdar, Water Resources Systems: Modelling

Techniques and Analysis, Tata McGraw Hill, New Delhi, 2005.

28 Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc
 Thank You

Water Resources Systems Planning and Management: M2L3 D Nagesh Kumar, IISc