MAEG4070 Engineering Optimization

Lecture 3 Dual Theory - Part I

Yue Chen
MAE, CUHK
email: [email protected]
Sep 14, 2021

1
Content of this course (tentative)
Linear programming
Lecture 2
Linear
Dual Theory – Part I
Lecture 3
Linearization
technique Programming
Lecture 7 Unconstrained optimization Lecture 13
Lecture 5 & 6

Optimization Non-Linear Constrained optimization

Lecture 8 & 9
Lecture 1 Lecture 4
Dual Theory – Part II Engineering examples
Lecture 10
Lecture 14

Distributed optimization Lecture 11

Multi-objective optimization Lecture 12

Theory
2
Introduction – Simple example
A company has some resources to produce three products (denoted as A, B, C). Each
product consumes a different mix of resources, and there will be a profit from selling
the product. The endowment of resources and its relationship with products are:

Production: 𝑦𝑦1 ≥ 0 𝑦𝑦2 ≥ 0 𝑦𝑦3 ≥ 0

A B C Endowment
Steel 3 4 2 ≤ 600
Wood 2 1 2 ≤ 400
Label 1 3 3 ≤ 300
Machine 1 4 4 ≤ 200
Profit 2 4 3
Maximize 2𝑦𝑦1 + 4𝑦𝑦2 + 3𝑦𝑦3
3
Introduction - General case

A company has 𝑅𝑅 kind of resources to produce 𝑁𝑁 items. Item 𝑛𝑛 ∈ {1, … , 𝑁𝑁} needs 𝑎𝑎𝑛𝑛𝑛𝑛
unit of resource 𝑟𝑟 ∈ {1, … , 𝑅𝑅}. The endowment of resource 𝑟𝑟 ∈ {1, … , 𝑅𝑅} is 𝑐𝑐𝑟𝑟 . The
price of item 𝑛𝑛 ∈ {1, … , 𝑁𝑁} is 𝑏𝑏𝑛𝑛 . How to maximize the total profit?

Variable: 𝑦𝑦𝑛𝑛 denotes the number of item 𝑛𝑛 produced

Objective: total profit ∑𝑁𝑁 𝑛𝑛=1 𝑏𝑏𝑛𝑛 𝑦𝑦𝑛𝑛
Constraints: 1) do not exceed the resource endowment ∑𝑁𝑁
𝑛𝑛=1 𝑎𝑎𝑛𝑛𝑛𝑛 𝑦𝑦𝑛𝑛 ≤ 𝑐𝑐𝑟𝑟 , ∀𝑟𝑟 = 1, … , 𝑅𝑅
2) 𝑦𝑦𝑛𝑛 ≥ 0, ∀𝑛𝑛 = 1, … , 𝑁𝑁

4
Introduction – General case
A company can also choose to sell all of its resource endowments 𝑐𝑐𝑟𝑟 , ∀𝑟𝑟 = 1, … , 𝑅𝑅 to
let another company to produce. Suppose the price for resource 𝑟𝑟 ∈ 1, … , 𝑅𝑅 is 𝑥𝑥𝑟𝑟 .
The company wants to ensure that the profit ∑𝑅𝑅𝑟𝑟=1 𝑎𝑎𝑛𝑛𝑛𝑛 𝑥𝑥𝑟𝑟 of selling 𝑎𝑎𝑛𝑛1 , … , 𝑎𝑎𝑛𝑛𝑛𝑛 unit of
resource 1, … , 𝑅𝑅 respectively, which are just adequate to produce one unit of product 𝑛𝑛,
is no less than the profit 𝑏𝑏𝑛𝑛 of producing and selling this unit of product 𝑛𝑛 by itself.
Meanwhile, it tries to minimize the total payment ∑𝑅𝑅𝑟𝑟=1 𝑐𝑐𝑟𝑟 𝑥𝑥𝑟𝑟 by the other company for
purchasing resources, so that the other company is willing to buy resources and
produce.

5
Basic models
Secondary problem, concerns primal problem
Major problem
Primal problem Dual problem
𝑻𝑻
𝒃𝒃 = 𝒃𝒃𝟏𝟏 , … , 𝒃𝒃𝑵𝑵
𝑻𝑻 𝑻𝑻
𝒚𝒚 = 𝒚𝒚𝟏𝟏 , … , 𝒚𝒚𝑵𝑵 𝒙𝒙 = 𝒙𝒙𝟏𝟏 , … , 𝒙𝒙𝑹𝑹

𝑨𝑨: ℝ𝑵𝑵×𝑹𝑹 , 𝒄𝒄: ℝ𝒓𝒓×𝟏𝟏

Compact form

Refer to previous page

6
Another Example

Can you tell me the optimal value of the following LP in 10 seconds?

Adding up all constraints, we have

We can easily find such a solution, so the optimal value is 0.8.

LP duality is a straightforward generalization of this simple trick!

7
Another Example

How to find a systematic way to estimate a lower bound of an LP’s optimal value?
Lambda convert signs:
Variables time lambda 1…n
to convert signs
× 𝝀𝝀𝟏𝟏 ≥ 𝟎𝟎
× 𝝀𝝀𝟐𝟐 ≤ 𝟎𝟎

…
lambda 1…n
× 𝝀𝝀𝑵𝑵

𝑁𝑁
𝑑𝑑𝑟𝑟 = � 𝑎𝑎𝑛𝑛𝑛𝑛 𝜆𝜆𝑛𝑛
𝑛𝑛=1

We want to determine the 𝜆𝜆𝑛𝑛 , ∀𝑛𝑛, so that 𝑐𝑐 = 𝑑𝑑 and ∑𝑁𝑁

𝑛𝑛=1 𝑏𝑏𝑛𝑛 𝜆𝜆𝑛𝑛 gives a lower bound
8
Another Example

To get the best (highest/tightest) lower bound, we maximize ∑𝑁𝑁

𝑛𝑛=1 𝑏𝑏𝑛𝑛 𝜆𝜆𝑛𝑛

Adding columns,
throughout all variables

9
Another Example

Primal problem Dual problem

Problems
connected
N
R
R

N
10
General Form
Primal problem Dual problem

11
Principles for LP duality

12
Example - 1

Primal problem Dual problem

min 5x1 + 4x2 + 3x3 max 4λ1 + 5λ2

s.t. x1 + x2 + x3 = 4 s.t. λ1 + 3λ2 ≤ 5
3x1 + 2x2 + x3 = 5 λ1 + 2λ2 ≤ 4
x1 ≥ 0, x2 ≥ 0, x3 ≥ 0 λ1+ λ2 ≤ 3

Refer to p12 for changing signs

13
Example - 2

Primal problem Dual problem

min x1 + 2x2 + x3 max 4λ1 + λ2 + λ3

s.t. x1 + x2 - x3 ≥ 4 s.t. λ1 + λ2 + 2λ3 ≤ 1
x1 - x2 + x3 = 1 λ1 - λ2 + λ3 ≥ 2
2x1 + x2 + x3 ≤ 1 -λ1 + λ2 + λ3 = 1
x1 ≥ 0, x2 ≤ 0 λ1 ≥ 0, λ3 ≤ 0

14
Example - 3

Primal problem

min 2x1 + 5x2 + 3x3

s.t. x1 + 3x2 - x3 ≥ 4
x1 - x2 + 2x3 = 1
2x1 + 4x2 + x3 ≤ 1
x1 ≤ 0, x3 ≥ 0

15
Example - 3

Primal problem Dual problem

min 2x1 + 5x2 + 3x3 max 4λ1 + λ2 + λ3

s.t. x1 + 3x2 - x3 ≥ 4 s.t. λ1 + λ2 + 2λ3 ≥ 2
x1 - x2 + 2x3 = 1 3λ1 - λ2 + 4λ3 = 5
2x1 + 4x2 + x3 ≤ 1 -λ1 + 2λ2 + λ3 ≤ 3
x1 ≤ 0, x3 ≥ 0 λ1 ≥ 0, λ3 ≤ 0

16
Example - 4

Primal problem

min 5x1 + x2 + 9x3

s.t. 3x1 + x2 + 2x3 = 27
2x1 + 2x2 + x3 ≥ 15
x1 + 5x2 + 7x3 ≤ 42
x1 ≤ 0, x3 ≥ 0

17
Example - 4

Primal problem Dual problem

min 5x1 + x2 + 9x3 max 27λ1 + 15λ2 + 42λ3

s.t. 3x1 + x2 + 2x3 = 27 s.t. 3λ1 + 2λ2 + λ3 ≥ 5
2x1 + 2x2 + x3 ≥ 15 λ1 + 2λ2 + 5λ3 = 1
x1 + 5x2 + 7x3 ≤ 42 2λ1 + λ2 + 7λ3 ≤ 9
x1 ≤ 0, x3 ≥ 0 λ2 ≥ 0, λ3 ≤ 0

18
Example - 5

Primal problem

max 2x1 + 3x2 + 7x3

s.t. x1 + 2x2 + x3 ≤ 60
2x1 + 3x2 + x3 ≥ 15
5x1 + x2 + 2x3 = 20
x1 ≥ 0, x2 ≤ 0

19
Example - 5

Primal problem Dual problem

max 2x1 + 3x2 + 7x3 min 60λ1 + 15λ2 + 20λ3

s.t. x1 + 2x2 + x3 ≤ 60 s.t. λ1 + 2λ2 + 5λ3 ≥ 2
2x1 + 3x2 + x3 ≥ 15 2λ1 + 3λ2 + λ3 ≤ 3
5x1 + x2 + 2x3 = 20 λ1 + λ2 + 2λ3 = 7
x1 ≥ 0, x2 ≤ 0 λ1 ≥ 0, λ2 ≤ 0

20
Why do we need dual problem?

1. When the primal problem has a lot of constraints and few variables, solving the
dual problem can reduce the computation time
(Because of the procedure of some well-known algorithms, e.g. Simplex)

2. Help to prove the primal problem is infeasible

(When the dual problem is unbounded, the primal problem is infeasible)

3. Sensitivity analysis
(Shadow price, the impact of right-hand side coefficients on the optimal value)

21
Duality Theorems

Symmetric: The dual of dual problem is the primal problem.

primal dual

dual primal

Proof:

22
Duality Theorems

Weak duality: Let 𝑥𝑥0 , 𝜆𝜆0 be a feasible solution of the primal problem and the
dual problem, respectively. We have 𝑐𝑐 𝑇𝑇 𝑥𝑥0 ≥ 𝑏𝑏 𝑇𝑇 𝜆𝜆0 .

Proof: since 𝑥𝑥0 is a feasible solution of the primal problem, we have

𝐴𝐴𝑥𝑥0 ≥ 𝑏𝑏, 𝑥𝑥0 ≥ 0
We also have 𝜆𝜆0 ≥ 0, therefore, 𝜆𝜆𝑇𝑇0 𝐴𝐴𝑥𝑥0 ≥ 𝜆𝜆𝑇𝑇0 𝑏𝑏. So
𝑏𝑏 𝑇𝑇 𝜆𝜆0 ≤ 𝑥𝑥0𝑇𝑇 𝐴𝐴𝑇𝑇 𝜆𝜆0 ≤ 𝑥𝑥0𝑇𝑇 𝑐𝑐 = 𝑐𝑐 𝑇𝑇 𝑥𝑥0

Dual opt = primal opt

Dual solutions Primal solutions

23
An example

Primal problem Dual problem

possible values

𝑇𝑇 6 1
For the primal problem, 𝑥𝑥0 = 1,1 is a feasible solution 𝑥𝑥 ∗ = , , 𝑐𝑐 𝑇𝑇 𝑥𝑥 ∗ = 28
𝑐𝑐 𝑇𝑇 𝑥𝑥0 = 40 is an upper bound of the objective value 5 5
𝜆𝜆∗ = 0,0,4,4 , 𝑑𝑑 𝑇𝑇 𝜆𝜆∗ = 28
For the dual problem, 𝜆𝜆0 = 1,1,1,1 𝑇𝑇 is a feasible solution
𝑑𝑑 𝑇𝑇 𝜆𝜆0 = 10 is an lower bound of the objective value optimal values for dual and primal
24
Duality Theorems

Weak duality: Let 𝑥𝑥0 , 𝜆𝜆0 be a feasible solution of the primal problem and
the dual problem, respectively. We have 𝑐𝑐 𝑇𝑇 𝑥𝑥0 ≥ 𝑏𝑏 𝑇𝑇 𝜆𝜆0 .

Primal (min) Dual (max)

Bounded below Bounded above

Infeasible Unbounded above

Unbounded below Infeasible

just an idea Feasible Feasible

25
Duality Theorems
𝑥𝑥2

1
Infeasible 𝑥𝑥1
1
both are infeasible as they do not intersect
(the change of being unbounded above is not valid)
𝜆𝜆2

1
Infeasible
1 𝜆𝜆1
26
Duality Theorems
𝑥𝑥2

Unbounded above
1 𝑥𝑥1

𝜆𝜆2

Just an idea Infeasible

0 𝜆𝜆1 27
Duality Theorems • This lecture: focus on linear programs.
• For nonlinear programs, strong duality needs
more conditions than feasibility (future lectures)

Optimality criterion: Let 𝑥𝑥0 , 𝜆𝜆0 be a feasible solution of the primal problem
and the dual problem, respectively. If 𝑐𝑐 𝑇𝑇 𝑥𝑥0 = 𝑏𝑏 𝑇𝑇 𝜆𝜆0 , then 𝑥𝑥0 , 𝜆𝜆0 are the
optimal solution of the primal and dual problems, respectively.
one is min and one is max, because
Proof: Cxo = Blambda, then optimal point
is touching and the same
According to weak duality, we have 𝑐𝑐 𝑇𝑇 𝑥𝑥 ≥ 𝑏𝑏 𝑇𝑇 𝜆𝜆0 , ∀𝑥𝑥 ∈ 𝑋𝑋
Since 𝑐𝑐 𝑇𝑇 𝑥𝑥0 = 𝑏𝑏 𝑇𝑇 𝜆𝜆0 , we can know 𝑥𝑥0 is the optimal solution.

Strong duality: If both the primal and dual problems are feasible, then both of
them have an optimal solution, i.e. 𝑥𝑥 ∗ , 𝜆𝜆∗ , and the optimal values are equal, i.e.
𝑐𝑐 𝑇𝑇 𝑥𝑥 ∗ = 𝑑𝑑 𝑇𝑇 𝜆𝜆∗ .

28
Duality Theorems

Primal problem Dual problem

Complementarity and slackness: Suppose 𝑥𝑥 ∗ , 𝜆𝜆∗ are the primal and dual
optimal solutions, respectively. Then, we have

𝑎𝑎𝑛𝑛𝑇𝑇 𝑥𝑥 ∗ > 𝑏𝑏 ⇒ 𝜆𝜆∗𝑛𝑛 = 0

𝜆𝜆∗𝑛𝑛 > 0 ⇒ 𝑎𝑎𝑛𝑛𝑇𝑇 𝑥𝑥 ∗ = 𝑏𝑏

29
An Example

Primal problem Dual problem

6 1
𝑥𝑥 ∗
If we know the optimal solution of the primal problem is = , ,
5 5
Try to determine the optimal solution of dual problem based on
Complementarity and slackness.

30
An Example
Primal problem

𝝀𝝀𝟏𝟏
𝝀𝝀𝟐𝟐
𝝀𝝀𝟑𝟑
𝝀𝝀𝟒𝟒

Dual problem

𝒙𝒙𝟏𝟏
𝒙𝒙𝟐𝟐

31
Economic Interpretation

According to Optimality criterion, we have

𝑓𝑓 ∗ = 𝑐𝑐 𝑇𝑇 𝑥𝑥 ∗ = 𝑏𝑏 𝑇𝑇 𝜆𝜆∗ = 𝜆𝜆1∗ 𝑏𝑏1 + ⋯ + 𝜆𝜆∗𝑁𝑁 𝑏𝑏𝑁𝑁

If parameter 𝑏𝑏𝑛𝑛 changes, what is the impact on the optimal value 𝑓𝑓 ∗ ?

Therefore, 𝜆𝜆∗𝑛𝑛 can be interpreted as the change of 𝑓𝑓 ∗ should there be 1

unit change of 𝑏𝑏𝑛𝑛 . We call it “shadow price” in economics.

The scarcer the resource, the greater the impact of its changes on the
objective function (cost), and therefore the higher the shadow price.
32
Thanks！

33