Course: AI Applications in Civil

Engineering (CE6003)

Single-variable Optimization Algorithms

Presented by: Dr. Shyamal Guchhait
• Since single-variable functions involve only one variable, the optimization
procedures are simple and easier to understand.
• Moreover, these algorithms are repeatedly used as a subtask of many multi-
variable optimization methods.
• Therefore, a clear understanding of these algorithms will help readers learn
complex algorithms discussed in subsequent chapters.
• The algorithms described in this chapter can be used to solve minimization
problems of the following type:

Minimize f(x),
where f(x) is the objective function and x is a real variable.

• The purpose of an optimization algorithm is to find a solution x, for which the

function f(x) is minimum.
• Two distinct types of algorithms are presented in this chapter: Direct search
methods use only objective function values to locate the minimum point, and
• Gradient-based methods use the first and/or the second-order derivatives of
the objective function to locate the minimum point.
 Optimality Criteria
Local Optimal Point
• A point x∗ is locally optimal if no nearby point has a better (lower for
minimization) function value.
• In minimization: f(x) ≥ f(x∗) for all x near x∗.

Global Optimal Point

• A point x∗∗ is globally optimal if no point in the entire search space has a
better function value.
• In minimization: f(x) ≥ f(x ∗∗) for all x.

Inflection Point
• A point x∗ is an inflection point if the function changes curvature —
increases then decreases or vice versa — without a local max/min.
 Optimality Criteria
Conditions for Optimality
For a function f(x):
• First derivative f′(x)=0 → stationary point.
• Second derivative f′′(x)>0 → local minimum.
• Second derivative f′′(x)<0 → local maximum.

Higher-Order Derivative Test

• If f′(x)=0 and f(n)(x)≠0:
• If n is odd → Inflection point.
• If n is even:
• f(n)(x)>0 → Local minimum.
• f(n)(x)<0 → Local maximum.
 Optimality Criteria
Exercise 1 – Check Optimality of x=0 for f(x)=x3

• Solution: Given Function: f(x)=x3

Observation from Graph:

• The function increases for x>0 and decreases for x<0.
• Hence, x=0 is not a max or min point.
• It is an inflection point (curve changes direction).

Verification using Derivatives:

• f′(x)=3x2
• At x=0, f′(0)=0.
• f′′(x)=6x
• At x=0, f′′(0)=0.
• f′′′(x)=6
• Nonzero value at x=0.

• Conclusion:
• The first two derivatives are zero.
• The third derivative (odd order) is non-zero.
• Therefore, x=0 is an inflection point.
 Optimality Criteria
Exercise 2 – Check Optimality of x=0 for f(x)=x4

Solution: Given Function: f(x)=x4

• Analysis:
• f′(x)=4x3 ⇒f′(0)=0
• f′′(x)=12x2 ⇒f′′(0)=0
• f′′′(x)=24x ⇒f′′′(0)=0
• f(4)(x)=24 ⇒f(4)(0)

• Conclusion:
• The first three derivatives are zero.
• The fourth derivative (even order) is positive.
• Hence, x=0 is a local minimum point.

• Note:
• These tests determine local optimality only.
• Global optimality requires checking all local minima and comparing function
values.
 Bracketing Methods
 Bracketing Methods
•The minimum of a function is found in two phases:
1. A crude technique to find lower and upper bounds of the minimum.
2. A more sophisticated method to search within these limits and find the optimal
solution with desired accuracy.
•This section: Methods for bracketing the minimum point.
•Next section: Finding the minimum point with desired accuracy.

 Bracketing Methods

1. Exhaustive Search Method

2. Bounding Phase Method
 Bracketing Methods
 Exhaustive Search Method
•Simplest of all methods.
•Optimum of a function is bracketed by calculating function values at equally spaced
points (Figure).
•Process:
[Link] from a lower bound.
[Link] three consecutive function values (assumes unimodality of the function).
[Link] on comparison:
•Either terminate search, or
•Replace one point with a new one and continue.
•Continue until the minimum is bracketed.

Figure: The exhaustive search method that uses equally spaced points.
 Bracketing Methods

Algorithm: Exhaustive Search Method

Step 1
• Set x1 = a, Δx = (b - a)/n (where n = number of intermediate points).
• Compute:- x2 = x1 + Δx, x3 = x2 + Δx
Step 2
• If f(x1) ≥ f(x2) ≤ f(x3), then the minimum lies in (x1, x3).→ Terminate.
• Else:- Set x1 = x2, x2 = x3, x3 = x2 + Δx
• - Go to Step 3
Step 3
Check if x3 ≤ b:- If yes, go to Step 2.
- If no, then no minimum exists in (a, b) or the boundary point (a or b) is the minimum.

Notes:- Final interval obtained: 2(b - a)/n

- On average, (n/2 + 2) function evaluations are needed to achieve desired accuracy.
 Bracketing Methods
Exhaustive Search Method
Problem
• Minimize: f(x) = x² + 54/x, over (0, 5)
Solution:
• Observation:
The function is unimodal ∗∗ in (0,5).
The minimum lies at x∗=3.
At this point: f(3)=27
• Verification:
First derivative: f′(3)=0
Second derivative: f′′(3)=6>0
→ Hence, x=3 is a local minimum. Fig. The unimodal, single-variable function
used in the exercise problems. The minimum
point is at x = 3.
∗∗ Unimodal
Function – A function is called unimodal on an interval if it has only one
extremum (minimum or maximum) in that interval. [U-shaped & inverted U-shaped]
 Bracketing Methods
Exhaustive Search Method
• Parameters
• a = 0, b = 5, n = 10 → Δx = (b − a)/n = 0.5
• Initialize: x₁ = a, x₂ = x₁ + Δx, x₃ = x₂ + Δx
• Iteration 1
• x₁=0.0, x₂=0.5, x₃=1.0
f(x₁)=∞, f(x₂)=108.25, f(x₃)=55.00
Check: f(x₁) ≥ f(x₂) ≤ f(x₃)? → No
Update → x₁=0.5, x₂=1.0, x₃=1.5
• Iteration 2
• x₁=0.5, x₂=1.0, x₃=1.5
f(x₁)=108.25, f(x₂)=55.00, f(x₃)=38.25
Check: 108.25 ≥ 55.00 ≤ 38.25? → No
Update → x₁=1.0, x₂=1.5, x₃=2.0
• Iteration 3
• x₁=1.0, x₂=1.5, x₃=2.0
f(x₁)=55.00, f(x₂)=38.25, f(x₃)=31.00
Check: 55.00 ≥ 38.25 ≤ 31.00? → No
Update → x₁=1.5, x₂=2.0, x₃=2.5
 Bracketing Methods
Exhaustive Search Method
• Iteration 4
• x₁=1.5, x₂=2.0, x₃=2.5
f(x₁)=38.25, f(x₂)=31.00, f(x₃)=27.85
Check: 38.25 ≥ 31.00 ≤ 27.85? → No
Update → x₁=2.0, x₂=2.5, x₃=3.0
• Iteration 5
• x₁=2.0, x₂=2.5, x₃=3.0
f(x₁)=31.00, f(x₂)=27.85, f(x₃)=27.00
Check: 31.00 ≥ 27.85 ≤ 27.00? → No
Update → x₁=2.5, x₂=3.0, x₃=3.5
• Iteration 6 (Bracket Found)
• x₁=2.5, x₂=3.0, x₃=3.5
f(x₁)=27.85, f(x₂)=27.00, f(x₃)=27.68
Check: 27.85 ≥ 27.00 ≤ 27.68? → Yes
Conclusion: Minimum is bracketed in (2.5, 3.5).
• Conclusion
• The exact minimizer is x* = 3 with f(x*) = 27, which lies inside the final
bracket.
 Bracketing Methods
Exhaustive Search Method
Accuracy of Exhaustive Search Method
• With n=10:
• Final interval length=2(b−a)/n=2(5−0)/10=1.0→
• Accuracy of solution = 1.0
• To achieve higher precision, we must increase n.
• More n → smaller interval → more function evaluations.
• For 3 decimal place accuracy: 2(b−a)/n=0.001.
• For a=0,b=5 → n≈10,000
• Function evaluations required: n/2+2=10,000/2+2=5,002
• Resulting interval:(2.9995, 3.0005)
 Region-Elimination Method

•Once the minimum point is bracketed, a more sophisticated algorithm needs to

be used to improve the accuracy of the solution.

•In this section, we describe algorithms that primarily work with the principle of
region elimination and require comparatively smaller function evaluations

•The idea: progressively eliminate parts of the search interval (a,b) that cannot
contain the minimum.

•Instead of evaluating the function everywhere (like exhaustive search), this

method uses selected interior points to shrink the interval step by step.

•It works only for unimodal functions (functions with one minimum).

1. Interval Halving Method

2. Fibonacci Search Method
3. Golden Section Search Method
 Region-Elimination Method
Basic Steps
• Step 1: Start with a search interval (a, b).
• Step 2: Divide the interval into equal sub-
intervals by choosing two or more interior
points.
• Step 3: Evaluate the function at these
interior points.
• Step 4: Compare function values:
- If f(x1) < f(x2), the minimum lies in (a, x2).
- If f(x1) > f(x2), the minimum lies in (x1, b). Fig. A typical single-variable
• Step 5: Eliminate the region that cannot unimodal function with function
contain the minimum. values at two distinct points.
• Step 6: Repeat the process with the
reduced interval until the interval length (or
desired accuracy) is achieved.
 Interval Halving Method
Idea
• A bracketing method to locate the minimum of a unimodal function.
• At each step, the search interval is halved by evaluating the function at selected
points.
• Much faster than exhaustive search because fewer function evaluations are needed.
Interval Halving Method – Algorithm
• Step 1: Start with an interval (a, b). Choose desired accuracy ε.
• Step 2: Compute the midpoint: xm = (a + b) / 2.
• Step 3: Compute two interior points: x1 = (a + xm) / 2, x2 = (xm + b) / 2.
• Step 4: Evaluate function values: f(x1), f(xm), f(x2).
• Step 5: Decision:
- If f(x1) < f(xm), new interval = (a, xm).
- Else if f(x2) < f(xm), new interval = (xm, b).
- Else, new interval = (x1, x2).
• Step 6: Check stopping criterion: (b − a)/2 ≤ ε.
- If satisfied → Terminate (minimum is bracketed).
- Else → Repeat from Step 2 with reduced interval. Fig. Three points x1, xm, and x2
used in the interval halving
method
 Interval Halving Method
Example
• Objective: minimize f(x) = x2 + 54/x on (0, 5). Unimodal on the interval.
• Rule each iteration: compute x1=(a+xm)/2, xm=(a+b)/2, x2=(xm+b)/2;
• compare f(x1), f(xm), f(x2) and pick (a,xm) or (xm,b) or (x1,x2).
• Iteration 1
• a = 0.000000 b = 5.000000
• x1 = 1.250000 xm = 2.500000 x2 = 3.750000
• f(x1) = 44.762500 f(xm) = 27.850000 f(x2) = 28.462500
• Decision: f(xm) ≤ f(x1), f(x2) → new interval = (x1, x2) ⇒ new interval = (1.250000, 3.750000)
• Iteration 2
• a = 1.250000 b = 3.750000
• x1 = 1.875000 xm = 2.500000 x2 = 3.125000
• f(x1) = 32.315625 f(xm) = 27.850000 f(x2) = 27.045625
• Decision: f(x2) < f(xm) → new interval = (xm, b) ⇒ new interval = (2.500000, 3.750000)
• Iteration 3
• a = 2.500000 b = 3.750000
• x1 = 2.812500 xm = 3.125000 x2 = 3.437500
• f(x1) = 27.110156 f(xm) = 27.045625 f(x2) = 27.525497
• Decision: f(xm) ≤ f(x1), f(x2) → new interval = (x1, x2) ⇒ new interval = (2.812500, 3.437500)
 Interval Halving Method
Iteration 4
• a = 2.812500 b = 3.437500
• x1 = 2.968750 xm = 3.125000 x2 = 3.281250
• f(x1) = 27.002950 f(xm) = 27.045625 f(x2) = 27.223744
• Decision: f(x1) < f(xm) → new interval = (a, xm) ⇒ new interval = (2.812500, 3.125000)
• Iteration 5
• a = 2.812500 b = 3.125000
• x1 = 2.890625 xm = 2.968750 x2 = 3.046875
• f(x1) = 27.036794 f(xm) = 27.002950 f(x2) = 27.006524
• Decision: f(xm) ≤ f(x1), f(x2) → new interval = (x1, x2) ⇒ new interval = (2.890625, 3.046875)
• Iteration 6
• a = 2.890625 b = 3.046875
• x1 = 2.929688 xm = 2.968750 x2 = 3.007813
• f(x1) = 27.015069 f(xm) = 27.002950 f(x2) = 27.000183
• Decision: f(x2) < f(xm) → new interval = (xm, b) ⇒ new interval = (2.968750, 3.046875)
• Iteration 7
• a = 2.968750 b = 3.046875
• x1 = 2.988281 xm = 3.007813 x2 = 3.027344
• f(x1) = 27.000413 f(xm) = 27.000183 f(x2) = 27.002230
• Decision: f(xm) ≤ f(x1), f(x2) → new interval = (x1, x2) ⇒ new interval = (2.988281, 3.027344)
Bracket after Iter 7: (2.988281, 3.027344).
As expected, the process tightens around x*=3 with f(3)=27.
 Gradient-Based Methods
• Gradient-based methods are iterative optimization techniques that use the
first derivative (gradient) of a function to find the minimum or maximum
point.
• They are applicable to continuous and differentiable functions.
• The next estimate is obtained by moving opposite to the gradient direction
(for minimization): xₖ₊₁ = xₖ − αₖ f′(xₖ)
• where:
• xₖ → current estimate of the variable
• αₖ → step size (learning rate)
• f′(xₖ) → gradient of f(x) at xₖ
• Stopping criterion: ∣f′(xₖ)∣ ≤ ε
 Newton–Raphson Method
• The Newton–Raphson method is derived from the Taylor series expansion
of f′(x) about xₖ:
f′(xₖ₊₁) = f′(xₖ) + f″(xₖ)(xₖ₊₁ − xₖ) + [f‴(xₖ)/2!](xₖ₊₁ − xₖ)²
• Neglecting higher-order terms:
f′(xₖ₊₁) ≈ f′(xₖ) + f″(xₖ)(xₖ₊₁ − xₖ)
• At the optimum, f′(xₖ₊₁) = 0, hence:
0 = f′(xₖ) + f″(xₖ)(xₖ₊₁ − xₖ)
• Rearranging gives the Newton–Raphson update formula:
xₖ₊₁ = xₖ − f′(xₖ)/f″(xₖ)
 Newton–Raphson Method
Example – Analytical Derivation
• Minimize: f(x) = x² + 54/x, for x > 0
• Step 1: Compute derivatives: f′(x) = 2x − 54/x² f″(x) = 2 + 108/x³
• Step 2: Start with x₀ = 2.500
• Iteration 1:
f′(2.500) = 2(2.500) − 54/(2.500)² = −3.640
f″(2.500) = 2 + 108/(2.500)³ = 8.912
x₁ = 2.500 − (−3.640 / 8.912) = 2.908
• Iteration 2:
f′(2.908) = 2(2.908) − 54/(2.908)² = −0.567
f″(2.908) = 2 + 108/(2.908)³ = 6.390
x₂ = 2.908 − (−0.567 / 6.390) = 2.997
• Iteration 3:
f′(2.997) ≈ 0.000 → Converged
Hence, x* = 3.000 and f(x*) = 27.000
 Newton–Raphson Method using Central Difference Technique
• When analytical derivatives are not available, numerical differentiation
using the central difference method can be applied.
• Central difference formulas:
f′(x) ≈ [f(x + h) − f(x − h)] / (2h)
f″(x) ≈ [f(x + h) − 2f(x) + f(x − h)] / h²
• Substituting these into the Newton–Raphson equation:
xₖ₊₁ = xₖ − [h(f(xₖ + h) − f(xₖ − h))] / [2(f(xₖ + h) − 2f(xₖ) + f(xₖ − h))]
 Newton–Raphson Method using Central Difference Technique
• Example – Using Central Difference
• Given f(x) = x² + 54/x, initial x₀ = 2.500, step size h = 0.25, tol = 0.0005.
• Solution
k xₖ f(xₖ−h) f(xₖ) f(xₖ+h) f′(xₖ) f″(xₖ) xₖ₊₁
0 2.500000 29.062500 27.850000 27.198864 -3.727273 8.981818 2.914980

1 2.914980 27.364935 27.022107 27.078817 -0.572236 6.392617 3.004495

2 3.004495 27.191562 27.000061 27.184173 -0.014778 6.009837 3.006954

3 3.006954 27.187630 27.000145 27.187658 0.000056 5.999962 3.006944

Notes: Derivatives approximated using central differences:

f'(xk) ≈ [f(xk+h) - f(xk−h)]/(2h),
f''(xk) ≈ [f(xk+h) − 2f(xk) + f(xk−h)]/h^2,
NR update: xk+1 = xk - h [f(xk+h) - f(xk−h)] / [2(f(xk+h) − 2f(xk) + f(xk−h))].
 Newton–Raphson Method using Central Difference Technique

• Conclusions:
• Newton–Raphson method is derived from Taylor series expansion.
• Both analytical and numerical (central difference) methods converge to the
same result.
• Central difference is especially useful when analytical derivatives are
difficult to compute.
• Choice of step size h affects accuracy and convergence speed.
 Bisection Method
Idea:
• Used to locate minimum of a unimodal function by finding root of derivative
f'(x)=0.
• At each iteration evaluate derivative at midpoint of given interval and
eliminate half-interval where minimum cannot lie.
• Both the function value and the sign of the first derivative at two points is
used to eliminate a certain portion of the search space. This method is
similar to the region-elimination methods
• The algorithm once again assumes the uni-modality of the function.
 Bisection Method
Algorithm
• Step 1: Choose initial bounds a and b such that f'(a) and f'(b) have opposite
signs.
• Step 2: Set tolerance ε (desired accuracy).
• Step 3: Compute midpoint z = (a + b)/2.
• Step 4: Compute derivative f'(z) (analytically or numerically, e.g., central
difference).
• Step 5: If |f'(z)| ≤ ε, stop — z approximates stationary point.
• Step 6: Else if f'(z) < 0, set a = z (minimum lies to the right).
• Step 7: Else (f'(z) > 0), set b = z (minimum lies to the left).
• Step 8: Repeat until (b − a) ≤ desired interval length or |f'(z)| ≤ ε.
Numerical derivative (central difference)
• For small h (e.g., h = 0.01 x):
• f'(x) ≈ [ f(x + h) − f(x − h) ] / (2h)
• Choose h proportional to x to maintain relative accuracy, e.g., h = 0.01 x.
 Bisection Method
Exercise- Solve using Bisection
• Problem: Minimize f(x) = x2 + 54/x on (0,5). Use bisection on f'(x)=0 with
numerical derivative (central difference).
Solution
Initial bounds: a = 2, b = 5. Tolerance: final interval length ≤ 0.001.
k a b z = (a+b)/2 f'(z) (central diff) Action / New interval

1 2.000000000 5.000000000 3.500000000 2.591395874 f'(z) > 0 → set b = z

2 2.000000000 3.500000000 2.750000000 -1.641209989 f'(z) < 0 → set a = z

3 2.750000000 3.500000000 3.125000000 0.719846985 f'(z) > 0 → set b = z

4 2.750000000 3.125000000 2.937500000 -0.383661176 f'(z) < 0 → set a = z

5 2.937500000 3.125000000 3.031250000 0.184985901 f'(z) > 0 → set b = z

6 2.937500000 3.031250000 2.984375000 -0.094848054 f'(z) < 0 → set a = z

 Bisection Method
Exercise- Solve using Bisection
k a b z = (a+b)/2 f'(z) (central diff) Action / New interval

7 2.984375000 3.031250000 3.007812500 0.046156405 f'(z) > 0 → set b = z

8 2.984375000 3.007812500 2.996093750 -0.024069696 f'(z) < 0 → set a = z

9 2.996093750 3.007812500 3.001953125 0.011111848 f'(z) > 0 → set b = z

10 2.996093750 3.001953125 2.999023438 -0.006461734 f'(z) < 0 → set a = z

11 2.999023438 3.001953125 3.000488281 0.002329346 f'(z) > 0 → set b = z

12 2.999023438 3.000488281 2.999755859 -0.002065121 f'(z) < 0 → set a = z

13 2.999755859 3.000488281 3.000122070 0.000132381 f'(z) > 0 → set b = z

• Final bracket after convergence (interval length ≤ 0.001):

(2.999755859, 3.000488281), midpoint ≈ 3.000122070
Exact minimizer: x* = 3.0 with f(x*) = 27.0
 Secant Method
Idea:
• Assume f'(x) is available (analytical or numerical). If f'(a) and f'(b) have
opposite signs, approximate the derivative between a and b as linear and
find z where derivative is zero using the secant formula.

Secant formula (to find z where f'(z)=0):

z = b - f'(b) * (b - a) / ( f'(b) - f'(a) )

• If f'(z) > 0 → minimum lies to left → set b = z;

• if f'(z) < 0 → minimum lies to right → set a = z.
• Repeat until desired accuracy.
 Secant Method
Exercise- Solve using Secant Method
Problem: Minimize f(x) = x2 + 54/x; initial a=2, b=5
Solution:
a (lower z (secant
Iter b (upper limit) f′(z) Action / Updated Interval
limit) point)
1 2.000000 5.000000 3.643599 3.219649 f′(z)>0 → eliminate (z,b) → new b = 3.643599

2 2.000000 3.643599 3.227564 1.271380 f′(z)>0 → eliminate (z,b) → new b = 3.227564

3 2.000000 3.227564 3.082671 0.482843 f′(z)>0 → eliminate (z,b) → new b = 3.082671

4 2.000000 3.082671 3.030305 0.180019 f′(z)>0 → eliminate (z,b) → new b = 3.030305

5 2.000000 3.030305 3.011145 0.066621 f′(z)>0 → eliminate (z,b) → new b = 3.011145

6 2.000000 3.011145 3.004103 0.024585 f′(z)>0 → eliminate (z,b) → new b = 3.004103

7 2.000000 3.004103 3.001511 0.009063 f′(z)>0 → eliminate (z,b) → new b = 3.001511

8 2.000000 3.001511 3.000557 0.003340 f′(z)>0 → eliminate (z,b) → new b = 3.000557

9 2.000000 3.000557 3.000205 0.001231 f′(z)>0 → eliminate (z,b) → new b = 3.000205

z = b - f'(b) * (b - a) / ( f'(b) - f'(a) )

References
• K. Deb, Optimization for Engineering Design: Algorithms &
Examples, Prentice Hall India
• Arora,J.S., Introduction to Optimum Design, 2nd Edition,
McGraw-Hill Book Company, 2000