Modeling, Simulation and Optimization for

Chemical Engineering
Part 3: Optimization for Chemical Engineering

Lecture 12: Single Variable Optimization and Methods for Optimization

Ho Chi Minh City University of Technology

 Contents
• Single Variable Single Variable Optimization is a process of finding the optimal
Optimization
• Optimization
Techniques
value (maximum or minimum) of a function that depends on a
• Example-
Analytical
single independent variable. This type of optimization is crucial in
Methods
• Newton’s engineering, economics, and science, where determining the
Method
• Equal Interval
Search Method
best outcome based on one variable can significantly impact
• Golden Section
Search Method
performance and efficiency.
• Quadratic
Interpolation

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents The general mathematical formulation for single variable optimization is:
• Single Variable
Optimization
• Optimization
Techniques
• Example-
Analytical
Methods
• Newton’s
Method
• Equal Interval
Search Method
• Golden Section
or
Search Method
• Quadratic
Interpolation

Where:
f(x) is the objective function.
x is the decision variable.
x∗ is the optimal solution.

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable First Derivative Test:
Optimization
• Optimization
Techniques To find the critical points of f(x), set the first derivative equal to zero
• Example-
Analytical
Methods
• Newton’s
Method
• Equal Interval
Search Method The solutions to this equation are called critical points.
• Golden Section
Search Method
• Quadratic
Interpolation
Second Derivative Test:
To determine if a critical point is a maximum or minimum, check the second derivative:

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable
Optimization
• Optimization Engineering Design: Finding optimal parameters for
Techniques
• Example-
Analytical
maximum efficiency.
Methods
• Newton’s
Method
• Equal Interval
Search Method Economics: Optimizing profit or cost functions.
• Golden Section
Search Method
• Quadratic

Physics: Finding equilibrium points in dynamic systems.

Interpolation

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable
Optimization
• Optimization
Techniques
• Example-
Analytical Methods
Analytical
Methods Use calculus (derivatives) to find critical points.
• Newton’s
Method
• Equal Interval
Search Method
• Golden Section
Search Method
• Quadratic
Interpolation Numerical Methods
Useful when analytical solutions are difficult to find.

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents Find the maximum or minimum of the function:
• Single Variable
Optimization
• Optimization
Techniques
• Example-
Analytical Step 1: Find the first derivative
Methods
• Newton’s
Method
• Equal Interval
Search Method
• Golden Section Step 2: Set the first derivative equal to zero to find critical points
Search Method
• Quadratic
Interpolation

Factorize the equation

Critical points: x = 1, x = 3

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable Step 3: Find the second derivative
Optimization
• Optimization
Techniques
• Example-
Analytical
Methods Step 4: Determine the nature of the critical points:
• Newton’s
Method f''(1) = -6 (local maximum) and f''(3) = 6 (local minimum)
• Equal Interval
Search Method
• Golden Section
Search Method Step 5: Calculate the function values at the critical points:
• Quadratic
Interpolation f(1) = 5, f(3) = 1

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable Maximize the function subject to the constraint
Optimization
• Optimization
Techniques
• Example-
Analytical
Methods
• Newton’s
Step 1: Define the Lagrangian
Method
• Equal Interval
Search Method
• Golden Section
Search Method
• Quadratic
Interpolation
Step 2: Compute partial derivatives

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable
Optimization
Step 3: Solve the system of equations:
• Optimization
Techniques
From equations
• Example-
Analytical
Methods
• Newton’s
Method
• Equal Interval
Search Method
• Golden Section
Search Method
• Quadratic
Interpolation After solution : (x, y) = (4, 4), (-4, -4), (4, -4), (-4, 4)

Step 4: Evaluate function values:

Max value: f(4, 4) = 16

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable
Optimization
• Optimization
Techniques
Minimize
• Example-
Analytical
Methods
• Newton’s
Step 1: Find the first derivative:
Method
• Equal Interval
Search Method
• Golden Section
Search Method
• Quadratic
Step 2: Set f'(x) = 0 to find critical point:
Interpolation

Step 3: Calculate the second derivative:

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable
Optimization Step 4: Apply second derivative test:
• Optimization
Techniques
• Example-
Analytical
the function has a local minimum at x=2.
Methods
• Newton’s
Method
• Equal Interval
Step 5: Evaluate function value:
Search Method
• Golden Section
Search Method
f(2) = 0
• Quadratic
Interpolation

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable
Optimization
• Optimization
Techniques
 Open search method
• Example-
Analytical
Methods  A good initial estimate of the solution is required
• Newton’s
Method
• Equal Interval
Search Method
 The objective function must be twice differentiable
• Golden Section
Search Method
• Quadratic  Unlike Golden Section Search method
Interpolation

• Lower and upper search boundaries are not required

(open vs. bracketing)
• May not converge to the optimal solution

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
f x  f x  
• Single Variable
'
Optimization
• Optimization
The derivative of the function, 0
Techniques
• Example-
Analytical
Methods
• Newton’s
Method
• Equal Interval
at the function’s maximum and minimum.
Search Method
• Golden Section
Search Method The minima and the maxima can be found by applying the Newton-
• Quadratic
Interpolation
Raphson method to the derivative, essentially obtaining

f ' ( xi )
x i 1  x i  ''
f ( xi )

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable
Optimization Initialization: Determine a reasonably good estimate for the maxima or the
• Optimization
Techniques
minima of the function f  x 
• Example-
Analytical

Step 1. Determine f '  x  and f ''  x 

Methods
• Newton’s
Method
• Equal Interval
Search Method
• Golden Section
Search Method
• Quadratic
x
Step 2. Substitute i (initial estimate 0 x for the first iteration)
Interpolation

and f ' x into f '' x  
f ' ( xi )
x i 1  x i  ''
f ( xi )

to determine xi 1 and the function value in iteration i.

Lecture 12: Single Variable Optimization and Methods for Optimization
 Contents
• Single Variable
Optimization Step 3.If the value of the first derivative of the function is zero then you have
• Optimization
Techniques
• Example- reached the optimum (maxima or minima). Otherwise, repeat Step 2 with
x
Analytical
Methods
• Newton’s the new value of i
Method
• Equal Interval
Search Method
• Golden Section
Search Method
• Quadratic
Interpolation

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable
Optimization
• Optimization
Techniques 2 2
• Example-
Analytical
Methods
• Newton’s  
Method 2
• Equal Interval
Search Method
• Golden Section The cross-sectional area A of a gutter with equal base and edge length of 2 is given
Search Method
• Quadratic by
Interpolation

A  4 sin  (1  cos  )

Find the angle  which maximizes the cross-sectional area

of the gutter.

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable
Optimization The function to be maximized is f ( )  4 sin  (1  cos  )
• Optimization
Techniques
• Example-
Analytical f ( )  4(cos   cos 2   sin 2  )
Methods
• Newton’s
Method f ( )  4 sin  (1  4 cos )
• Equal Interval
Search Method
• Golden Section
Search Method
• Quadratic
Interpolation
Iteration 1: Use  0   / 4 as the initial estimate of the solution
  
4(cos  cos 2  sin 2 )
 4 4 4  1.0466
1  
4  
 4 sin (1  4 cos )
4 4
f (1.0466)  5.196151

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable
Optimization
• Optimization
Iteration 2:
Techniques
• Example-
Analytical 4(cos 1.0466  cos 2 1.0466  sin 2 1.0466 )
Methods  2  1.0466   1.0472
• Newton’s  4 sin 1.0466 (1  4 cos 1.0466 )
Method
• Equal Interval
Search Method
• Golden Section
Summary of iterations
Search Method
• Quadratic Iteration 
Interpolation 1 0.7854 2.8284 -10.8284 1.0466 5.1962
2 1.0466 0.0062 -10.3959 1.0472 5.1962
3 1.0472 1.06E-06 -10.3923 1.0472 5.1962
4 1.0472 3.06E-14 -10.3923 1.0472 5.1962
5 1.0472 1.3322E-15 -10.3923 1.0472 5.1962

Remember that the actual solution to the problem is at 60

degrees or 1.0472 radians.
Lecture 12: Single Variable Optimization and Methods for Optimization
 Contents The Finite Difference Newton's Method is a variation of Newton's Method
• Single Variable
Optimization
• Optimization
Techniques
• Example-
Analytical
The iterative formula is the same as Newton's Method:
Methods
• Newton’s
Method
• Equal Interval
Search Method
• Golden Section
Search Method
• Quadratic
Interpolation

First Derivative Approximation Second Derivative Approximation

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable
Optimization
• Optimization How to Use These Formulas in Practice
Techniques
• Example-
Analytical
Methods
• Choose a step size h (commonly h=0.1).
• Newton’s
Method
• Equal Interval • Calculate the first and second derivatives using the finite
Search Method
• Golden Section
Search Method
• Quadratic
difference approximations.
Interpolation

• Update the value of x using the Newton's Method formula.

• Repeat for the specified number of iterations.

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable
Optimization
• Optimization Why It’s Called a Quasi-Newton Method
Techniques
• Example-
Analytical
Methods
• Newton’s
Method Quasi-Newton Methods approximate the second derivative using past
• Equal Interval
Search Method
• Golden Section information, rather than computing it directly.
Search Method
• Quadratic
Interpolation The provided formula fits into this category because it approximates the
second derivative using the change in the first derivative between two
points.
The method is a special case of a Quasi-Newton Method but is often
referred to as a Secant Method because it uses two points to estimate the
slope.
Lecture 12: Single Variable Optimization and Methods for Optimization
 Contents
• Single Variable
Optimization How to Use These Formulas in Practice
• Optimization
Techniques
• Example-
Step 1: Choose two initial estimates xq and xp
Analytical
Methods
• Newton’s Step 2: Calculate the first derivatives at these points:
Method
• Equal Interval
Search Method
• Golden Section
Search Method
• Quadratic
Interpolation
Step 3: Apply the formula: Step 4: Update the points:

•Use x~ as the new estimate.

•Repeat the process until convergence.

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable
Optimization
• Optimization Choose an interval [a, b] over which the optima occurs
Techniques

 ab   ab 
• Example-
 
Analytical
Methods Compute f   and f  2  2 
 2 2  
• Newton’s
Method
f(x)
• Equal Interval
Search Method  ab    ab  
• Golden Section If f
 2
  f
2  2
 
2
Search Method
• Quadratic then the interval in
Interpolation
which the maximum 
a  b   
occurs is  2  2 , b 2 2

otherwise it occurs in
 ab  
a b x
a, 2  2  (a+b)/2
Figure 1 Equal interval search method.

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable
Optimization • The Equal Interval method is inefficient when  is small.
• Optimization
Techniques
• Example-
• The Golden Section Search method divides the search more
Analytical
Methods
efficiently closing in on the optima in fewer iterations.
• Newton’s
Method
• Equal Interval
Search Method
• Golden Section f2
Search Method f1
• Quadratic
Interpolation
fl fu

Xl X2 X1 Xu

Figure 2. Golden Section Search method

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable
Optimization f2
• Optimization f1 f1
Techniques
• Example- fl fu fl fu
Analytical a-b
Methods
• Newton’s b
Method
• Equal Interval Xl X1 Xu Xl X2 X1 Xu
Search Method
a b a
• Golden Section
Search Method
• Quadratic Determining the first Determining the second
Interpolation intermediate point intermediate point
a b b ab
 
ab a a b

b
Golden Ratio=>  0.618...
a

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable
Optimization
• Optimization f2
Techniques f1
• Example-
Analytical fl fu
Methods
• Newton’s
Method
• Equal Interval
Search Method Xl X2 X1 Xu
• Golden Section
Search Method
• Quadratic
Interpolation
• If f ( x2 )  f ( x1 )then the new interval is [ xl , x2 , x1 ]

• If f ( x2 )  f ( x1 )then the new interval is [ x2 , x1 , xu ]

• All that is left to do is to determine the location of the second intermediate

point.
Lecture 12: Single Variable Optimization and Methods for Optimization
 Contents
• Single Variable
Optimization
• Optimization
Techniques 2 2
• Example-
Analytical
Methods
• Newton’s  
Method 2
• Equal Interval
Search Method
• Golden Section
Search Method The cross-sectional area A of a gutter with equal base and edge length of 2 is
• Quadratic
given by
Interpolation
A  4 sin  (1  cos  )
Find the angle  which maximizes the cross-sectional area of the gutter.
Using an initial interval of [0,  / 2] find the solution after 2 iterations.
Use an initial   0.05

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable
Optimization
The function to be maximized is f ( )  4 sin  (1  cos )
• Optimization
Techniques
• Example-
Analytical Iteration 1: Given the values for the boundaries of
Methods
• Newton’s xl  0 and xu   / 2 we can calculate the initial intermediate
Method
• Equal Interval points as follows:
Search Method
• Golden Section 5 1 5 1 f (0.97080)  5.1654
Search Method
x1  xl  ( xu  xl )  0  (1.5708)  0.97080
• Quadratic
2 2
Interpolation 5 1 5 1 f (0.60000)  4.1227
x2  xu  ( xu  xl )  1.5708  (1.5708)  0.60000
2 2
f1
f2

X1=?

Xl=X2 X2=X1 Xu
Xl X2 X1 Xu

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable
Optimization
• Optimization 5 1 5 1
Techniques x1  xl  ( xu  xl )  0.60000  (1.5708  0.60000)  1.2000
• Example- 2 2
Analytical
Methods
• Newton’s
Method
• Equal Interval To check the stopping criteria the difference between xu
Search Method
• Golden Section and xl is calculated to be
Search Method
• Quadratic
Interpolation
xu  xl  1.5708  0.60000  0.97080

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable Iteration 2
Optimization
• Optimization xl  0.60000
Techniques
• Example-
Analytical
xu  1.5708
Methods
• Newton’s x1  1.2000 f (1.2000)  5.0791
Method
• Equal Interval x2  0.97080 f (0.97080)  5.1654
Search Method
• Golden Section
Search Method
• Quadratic f ( x1 )  f ( x2 )
Interpolation
xl  0.60000
xu  1.2000
x1  0.97080
5 1 5 1
x2  xu  ( xu  xl )  1.2000  (1.2000  0.6000)  0.82918
2 2
xu  xl
Xl X2 X1 Xu  1.2000  0.6000  0.9000
2

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable
Optimization
Iteration xl xu x1 x2 f(x1) f(x2) 
• Optimization
Techniques 1 0.0000 1.5714 0.9712 0.6002 5.1657 4.1238 1.5714
• Example- 2 0.6002 1.5714 1.2005 0.9712 5.0784 5.1657 0.9712
Analytical 3 0.6002 1.2005 0.9712 0.8295 5.1657 4.9426 0.6002
Methods
• Newton’s 4 0.8295 1.2005 1.0588 0.9712 5.1955 5.1657 0.3710
Method 5 0.9712 1.2005 1.1129 1.0588 5.1740 5.1955 0.2293
• Equal Interval 6 0.9712 1.1129 1.0588 1.0253 5.1955 5.1937 0.1417
Search Method
7 1.0253 1.1129 1.0794 1.0588 5.1908 5.1955 0.0876
• Golden Section
Search Method 8 1.0253 1.0794 1.0588 1.0460 5.1955 5.1961 0.0541
• Quadratic 9 1.0253 1.0588 1.0460 1.0381 5.1961 5.1957 0.0334
Interpolation

xu  xl 1.0253  1.0588
  1.0420 f (1.0420)  5.1960
2 2

The theoretically optimal solution to the problem

happens at exactly 60 degrees which is 1.0472 radians
and gives a maximum cross-sectional area of 5.1962.

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable
Optimization
• Optimization • Quadratic Interpolation:
Techniques
• Example-
Analytical
Methods Equation would be of form:
• Newton’s
Method
• Equal Interval
Search Method
• Golden Section Search lies between three points
Search Method
• Quadratic
Interpolation
• Cubic Interpolation:

Equation would be of form:

Search lies between four points

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable
Optimization
• Optimization
Techniques
• Three points are considered; either equal distant or not namely x1, x2, x3
• Example-
Analytical • Assumption is that function is minimum, if derivative is zero and assumed as optimum point;
Methods
• Newton’s
Method estimated via
• Equal Interval
Search Method
• Golden Section • Suppose that f(x) is evaluated at x1, x2, x3 to yield f(x1) = f1, f(x2) = f2, and f(x3) = f3. The
Search Method
• Quadratic
Interpolation coefficients b and c can be evaluated from the solution of the three linear equations

Lecture 12: Single Variable Optimization and Methods for Optimization

 Contents
• Single Variable
Optimization
• Optimization
• Define b & c in terms x1, x2, x3, f1, f2 an
Techniques
• Example-
Analytical
f3, we get:
Methods
• Newton’s
Method
• Equal Interval
Search Method
• Golden Section
Search Method
• Quadratic
Interpolation

Stop if f(x*) < tolerance value, at that

point x* would be solution.
Graphical Representation of
Quadratic Interpolation
Lecture 12: Single Variable Optimization and Methods for Optimization
 Contents
• Single Variable
Optimization
• Optimization
Different Scenarios for
Techniques
• Example-
Moving to Next Iteration
Analytical
Methods
• Newton’s
Method
• Equal Interval
Search Method
• Golden Section
Search Method
• Quadratic
Interpolation

Lecture 12: Single Variable Optimization and Methods for Optimization