Lecture Notes Template
22EE503 - OPTIMIZATION IN ENGINEERING
DESIGN
UNIT 1 & INTRODUCTION
1. Classification of optimization algorithms
The classical methods of optimization are useful in finding the optimum solution of continuous
and differentiable functions. These methods are analytical and make use of the techniques of
differential calculus in locating the optimum points. Since some of the practical problems
involve objective functions that are not continuous and/or differentiable, the classical
optimization techniques have limited scope in practical applications. However, a study of the
calculus methods of optimization forms a basis for developing most of the numerical
techniques of optimization presented in subsequent chapters. In this chapter we present the
necessary and sufficient conditions in locating the optimum solution of a single-variable
function, a multivariable function with no constraints, and a multivariable function with
equality and inequality constraints.
Optimization algorithms can be classified into several categories based on various criteria such
as the nature of the problem, the type of variables (continuous or discrete), and the approach
used to find the optimal solution. Here are some common classifications of optimization
algorithms:
1. Unconstrained Optimization Algorithms:
o These algorithms are designed to find the minimum (or maximum) of a function
without any constraints on the variables.
o Examples include gradient descent, Newton's method, and conjugate gradient
methods.
2. Constrained Optimization Algorithms:
o These algorithms optimize a function subject to constraints on the variables.
o Examples include linear programming (LP), quadratic programming (QP),
nonlinear programming (NLP), and integer programming (IP).
3. Global Optimization Algorithms:
o These algorithms aim to find the global minimum (or maximum) of a function
over the entire feasible region, rather than just a local optimum.
o Examples include genetic algorithms, simulated annealing, and particle swarm
optimization.
4. Local Optimization Algorithms:
o These algorithms search for a local optimum, meaning the best solution within
a local neighborhood of the starting point.
o Examples include gradient-based methods like gradient descent and Newton's
method.
5. Deterministic Optimization Algorithms:
o These algorithms produce the same solution for a given problem and initial
conditions every time they are run.
o Examples include gradient descent, simplex method (for linear programming),
and sequential quadratic programming (SQP).
6. Stochastic Optimization Algorithms:
o These algorithms introduce randomness or probabilistic elements to the
optimization process.
o Examples include genetic algorithms, simulated annealing, and evolutionary
algorithms.
7. Derivative-Free Optimization Algorithms:
o These algorithms do not require derivatives (gradients) of the objective function
and are suitable for problems where derivatives are unavailable, noisy, or
expensive to compute.
o Examples include genetic algorithms, particle swarm optimization, and Nelder-
Mead simplex method.
8. Heuristic Optimization Algorithms:
o These algorithms use heuristic methods or rules of thumb to find solutions that
may not be guaranteed to be optimal but are often good enough for practical
purposes.
o Examples include genetic algorithms, ant colony optimization, and tabu search.
9. Metaheuristic Optimization Algorithms:
o These are higher-level strategies that guide the search process of other
optimization algorithms.
o Examples include genetic algorithms, simulated annealing, and particle swarm
optimization.
10. Hybrid Optimization Algorithms:
o These algorithms combine different optimization techniques or algorithms to
leverage their respective strengths and overcome their weaknesses.
o Examples include genetic algorithm combined with local search methods, or
particle swarm optimization combined with gradient descent.
DISCUSSION QUESTIONS:
1. In what scenarios would you prefer to use a heuristic optimization algorithm (e.g., genetic
algorithms, simulated annealing) over a deterministic algorithm (e.g., Newton's method, Simplex
method), and what are the key factors influencing this choice?
2. Can you provide examples of real-world optimization problems where hybrid optimization
algorithms (combining elements of different algorithm classes) have been successfully applied,
and explain why a hybrid approach was beneficial in these cases?
3. How do gradient-based optimization algorithms differ from gradient-free optimization
algorithms, and what are the advantages and disadvantages of each in terms of computational
efficiency and applicability to various types of optimization problems?
