ME 310
Numerical Methods
Optimization
These presentations are prepared by
Dr. Cuneyt Sert
Mechanical Engineering Department
Middle East Technical University
Ankara, Turkey
[email protected]They can not be used without the permission of the author
1
Optimization
• Optimization is similar to root finding. Both involve guessing and searching for a point on a function.
Global maximum
f(x) f (x) = 0 , f (x) < 0
a b x
Global minimum
f (x) = 0 , f (x) > 0
• Optimum is the point where f (x) = 0. f (x) indicates whether it is a minimum or a maximum.
• In this range there can be only one global minimum and one global maximum. These can be at the
end points of the interval.
• There can be several local minimums and local maximums.
• Practical optimization problems are complicated and include several constraints.
• Minimization of cost of a manufactured part (time, quality, quantity, etc.).
• Maximization of efficiency of a cooling unit (size, material, safety, ergonomics, cost, etc.).
• Transportation problem (manage the shipping between several sources and several warehouses).
2
• We will study 1D (f=f(x)), unconstrained optimization using the following methods.
• Bracketing methods (Golden Section Search, Quadratic Interpolation)
• Open Methods (Newton’s Method)
Bracketing Methods
• Consider finding the maximum of a function f(x) in the interval [a,b].
• Consider a function that has only one maximum (or minimum) in [a,b].
• Similar to the bracketing methods used for root finding, we will iteratively narrow the interval [a,b]
to locate the minimum.
• Remember that in root finding (for example in the Bisection method), only one intermediate point
was enough to narrow the interval.
• In finding a maximum we need two intermediate points (x1 and x2).
• If f(x1) > f(x2) than the maximum is between [x2,b]. Otherwise it is between [a,x1]
a x2 x1 b a x2 x1 b a x2 x1 b a x2 x1 b 3
Golden Section Search
• There are several different ways in selecting the two intermediate points x1 and x2.
• In Golden Section Search these two points are selected as
x1 = a + d
x2 = b – d
a x2 x1 b where d = R (b-a).
d
5 1
• R 0.618034 . . . is called the golden-ratio. It is the positive root of r2 + r – 1 = 0.
2
• If f(x1) > f(x2) than continue with the interval [x2,b]. Otherwise continue with [a,x1]. This works to
locate a maximum. To locate a minimum do the opposite.
• Calculate two new intermediate points in the narrowed interval and iterate like this.
• At each iteration, the interval drops by a factor of R ( d i+1 = R d i ).
• Stop when (x1-x2) drops below the specified tolerance. See page 347 for an alternative stopping
criteria.
4
Golden Ratio
• What is the importance of the golden ratio, R = 0.618034 ?
• Consider the following case. Superscripts show the iteration number.
Iteration 0 • At iteration 0,
5 1 0 5 1 0
x 10 a 0 (b a 0 ) , x 02 b 0 (b a 0 )
2 2
• f(x10) < f(x20) , therefore continue with [x20,b0].
a0 x20 x10 b0
d0
d0 • At iteration 1, a1= x20 , b1= b0
5 1 1
x 11 a1 (b a1 )
2
Iteration 1 5 1 1
x 12 b1 (b a1 ) x 10
2
• Therefore there is no need to calculate x21 and f(x21).
This saves computations.
a1 x21 x11 b1
d1 Exercise: Show that x21 = x10
d1
5
Pseudocode for the Golden Section Search
R = 0.618033988
READ a, b, maxIter, tolerance
CALCULATE fa, fb
LOOP k from 1 to maxIter Exercise 21: This pseudocode is written to
x1 = a + R(b-a) ; f1 = func(x1) find a maximum. Modify it so that it can find a
minimum too.
x2 = b - R(b-a) ; f2 = func(x2)
IF ( f1 > f2) THEN Exercise 22: Change the stopping criteria
a = x2 ; x2 = x1 ; f2 = f1 with the one given in the book.
x1 = a + R(b-a) ; f1 = func(x1)
ELSE Exercise 23: What happens if
b = x1 : x1 = x2 ; f1 = f2 (i) the initial interval [a,b] contains more than
x2 = b - R(b-a) ; f2 = func(x2) one min or max?
ENDIF (ii) a is the maximum and b is the minimum,
or vice versa?
WRITE k, x1, x2
IF ( (x1 – x2) < tolerance) STOP
ENDLOOP 6
Quadratic Interpolation
• Based on the fact that a quadratic (2nd order) polynomial often provides a good approximation of a
function near an optimum point.
f(x)
quadratic polynomial
x0 x1 x3 x2
• Select 3 points (x0, x1 and x2) that contains only 1 optimum point of a function.
• Only one quadratic will pass through these points. Find the equation of this quadratic.
• Equate its first derivative to zero and find its optimum point, x3.
f (x 0 )( x 12 x 22 ) f (x 1 )( x 22 x 20 ) f (x 2 )( x 20 x 12 )
x3
2 f (x 0 )( x 1 x 2 ) 2 f (x 1 )( x 2 x 0 ) 2f (x 2 )( x 0 x 1 )
• Similar to the Golden Section Search, narrow the interval by discarding one of the points.
• Continue with the remaining 3 points and calculate a new optimum (x3).
• Iterate like this and stop when the approximate relative error drops below the tolerance value. 7
Newton’s Method
f (x i )
• Recall that Newton-Raphson method is used to find the root of f(x) =0 as xi1 xi
f (x i )
f (x i )
• Similarly the optimum points of f(x) can be found by applying N-R to f (x) = 0. x i1 x i
f (x i )
• This open method requires only one starting point.
• It also requires the 1st and 2nd derivative of f(x).
• It converges fast, but convergence is not guaranteed.
• At the end one can check the sign of f (x) to determine whether the optimum point is a minimum
or a maximum.
• If the derivatives are not known than their approximations can be used. This is similar to the
Secant method that we learned in root finding.
• To avoid divergence, it is a good idea to use this method when we are close enough to the
optimum point. So we can use a hybrid technique, where we start with a bracketing method and
safely narrow the interval and than continue with the Newton’s method.
8
Example 23:
Find the maximum of f(x) = 2 x - 1.75 x2 + 1.1 x3 – 0.25 x4 using
(a) Golden section search (a = -2 , b = 4, es = 1%)
(b) Quadratic interpolation (x0 = -1.75 , x1 = 2 , x2 = 2.25 , perform 5 iterations)
(c) Newton’s method (x0 = 2.5 , es = 1%)
(a) Golden Section Search
iter 1: a = -2 , b=4, x1 = a + R(b-a) = 1.708 , x2 = b - R(b-a) = 0.292
f(x1) = 1.664 , f(x2) = 0.460 f(x1) > f(x2) than continue with [x2 , b].
f(x1) > f(x2) than xopt = x1 = 1.708 , ea = (1-R) * (b-a) / |xopt|* 100 = 134 %
iter 2: a = 0.292 , b=4, x1 = a + R(b-a) = 2.584 , x2 = 1.708
f(x1) = 1.316 , f(x2) = 1.664 f(x1) < f(x2) than continue with [a , x1 ].
f(x1) < f(x2) than xopt = x2 = 1.708 , ea = (1-R) * (b-a) / |xopt|* 100 = 83 %
iter 3: a = 0.292 , b = 2.584 , x1 = 1.708 , x2 = b - R(b-a) = 1.167
f(x1) = 1.664 , f(x2) = 1.235 f(x1) > f(x2) than continue with [x2 , b].
f(x1) > f(x2) than xopt = x1 = 1.708 , ea = (1-R) * (b-a) / |xopt|* 100 = 51 %
......
iter 11: xopt = 2.073 , ea = 0.9 %
9
Example 23 (cont’d)
(b) Quadratic Interpolation
iter 1: x0 = -1.75 , x1 = 2.0 , x2 = 2.25
Calculate x3 = 2.0617
iter 2: x0 = 2.0 , x1 = 2.0617 , x2 = 2.25
Calculate x3 = 2.0741
iter 3: x0 = 2.0617 , x1 = 2.0741 , x2 = 2.25
Calculate x3 = 2.0779
iter 4: x0 = 2.0741 , x1 = 2.0779 , x2 = 2.25
Calculate x3 = 2.0791
iter 5: x0 = 2.0779 , x1 = 2.0791 , x2 = 2.25
Calculate x3 = 2.0786
ea = | (x3present - x3previous) / x3present |* 100 = 0.02 %
10
Example 23 (cont’d)
(c) Newton’s Method
f (x) = 2 – 3.5 x + 3.3 x2 – x3 , f (x) = -3.5 +6.6 x – 3 x2
x0 = 2.25
iter 1: x1 = x0 - f (x0) / f (x0) = 2.19565
ea = | (x1 - x0) / x1 |* 100 = 13.9 %
iter 2: x2 = x1 - f (x1) / f (x1) = 2.0917
ea = | (x2 - x1) / x2 |* 100 = 5.0 %
iter 3: x3 = x2 - f (x2) / f (x2) = 2.07951
ea = | (x3 - x2) / x3 |* 100 = 0.6 %
11
