Seeding Methods for Solving General

Mixed Integer Nonlinear Optimization

Problems
George Apaaboah
August 27, 2018

Research project performed at

Laboratoire G-SCOP, Grenoble France

Under the supervision of:

Professor Pierre Genevois, G-SCOP

September 2018
 Abstract

A good initial feasible solution has been proven to be very important in achiev-
ing a good solution when solving general mixed integer nonlinear optimization
problems. This study focused on investigating the impact of seeding methods on
achieving good initial feasible solutions for population based algorithms. Four
seeding methods are investigated which include the random seeding method but
used as a benchmark. The other methods are feasibility algorithm, five step algo-
rithm and seeding repair algorithm. All the problems investigated except MAPSE
were obtained from MINLPlib.
A three step algorithm is used; which are: seeding method, population based
algorithm and sequential quadratic programming respectively. From the problems
investigated, the five step algorithm achieved the overall best solution on average
while the seeding random had the cheapest computational cost. The feasibility
seeding algorithm proved to be the most promising method yielding better solution
and computational cost compare with the other methods.

Résumé

Lors de la résolution d’un problème d’optimisation non linéaire mixte en nombre

entier, une bonne solution initiale faisable s’est avérée très importante pour obte-
nir une bonne solution. Cette étude à portée sur l’impact des méthodes seeding
sur la réalisation de bonnes solutions initiales réalisables pour les algorithmes gé-
nétiques. Quatre méthodes de seeding sont étudiées, avec la méthode du seeding
aléatoire utilisée comme référence. Les autres méthodes sont l’algorithme de fai-
sabilité, l’algorithme en cinq étapes et l’algorithme de réparation de seeding. Tous
les problèmes étudiés dans ce projet sauf MAPSE ont été obtenu de MINLPlib.
Un algorithme à trois étapes est utilisé à savoir : la méthode de seeding, algo-
rithme basé sur la population et programmation séquentielle quadratique respecti-
vement. Sur les problèmes traités, l’algorithme en cinq étapes atteint la meilleure
solution globale en moyenne alors que le seeding aléatoire avait le coût de calcul
le moins cher. L’algorithme du seeding faisable s’est avéré être la méthode la plus
prometteuse pour obtenir une meilleure solution et le coût du calcul est comparable
aux autres méthodes.
 Contents

Abstract i

Résumé i

1 Introduction 1
1.1 Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Convex and nonconvex optimization . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Mixed integer nonlinear optimization . . . . . . . . . . . . . . . . . . . . . . . 2
1.4 Feasibility problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.5 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.6 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.7 Organisation of report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 State-of-the-art 5
2.1 Literature review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

3 Presentation of methods 11
3.1 Three step algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.2 Augmented Lagrangian method . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.3 Genetic algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.4 Seeding methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.5 Sequential Quadratic Programming . . . . . . . . . . . . . . . . . . . . . . . . 17
3.6 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

4 Results and conclusion 21

4.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.2 Conclusion and future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Bibliography 29
4.3 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

ii
 1
Introduction

In this introduction, an overview of optimization will be given with focus on general problem
presentation, problems types and solution types. The problem statement and objectives of the
research will also be discussed.

1.1 Optimization
Optimization is a very important tool in mechanical system design, structural analysis and
decision processes for example. People naturally optimize and so does nature. The general
structure of an optimization problem is to maximize or minimize an objective subject to limited
resources. These resources could be human workforce, capital, design materials and capacity.
Solving optimization problems has proven to be difficult, some optimization algorithms are
very complicated and require several hours/ months to solve. Achieving an exact solution
for optimization problems has its own difficulty but this is also problem dependent. There is
no single algorithm to solve optimization problems but the choice of algorithm depends on
problem type. Over the years, several heuristic algorithms have been proposed which have
played a major role such as general purpose solvers.
Optimization problems can either be constraied or unconstrained. Unconstrained optimiza-
tion problem is finding X, a vector which has the function f (X) as its local minimum or maxi-
mum. Such a problem is of the form problem 1.1:

minimize
n
f (X) (1.1)
X∈R

These types of problem has the variable X unrestricted. Problems of this kind are present in
many real world applications. In most cases, unconstrained optimization problems are reformu-
lation of constrained optimization problems. Constrained optimization problems are problems
that have restrictions stated on the variables of the problem. Such a problem can be presented
as shown in problem 1.2:

minimize f (X)
subject to g j (X) ≤ 0, j = 1, 2, 3, ..., J
(1.2)
hk (X) = 0, k = 1, 2, 3, ..., K
where X ∈ RIx × NIy
In general, unconstrained problems are easier to solve than constrained problems and a bounded
constraint problem is also easier to solve than its unbounded version. Optimization problems
with linear objective functions and linear constraints are classified as linear programming prob-
lems. Problems with at least one of the constraint as a nonlinear function is classified as non-
linear programming problem. Nonlinear programming problems are the most present problem
type in general application of optimization. Solving an optimization problem yields either a
local or global solution. A local solution is a point at which the value of the objective function
is less than all other values within its neighborhood. A local solution may not be true for the
entire space of the optimization problem, a true solution satisfies the optimality conditions ev-
erywhere. A global solution of an optimization problem is the minimum value of the objective
function in the entire problem space. This is the solution that is always true. It can be computa-
tionally expensive to achieve a global solution compared with local solution but some problems
may require finding only a local solution.

1.2 Convex and nonconvex optimization

Optimization problems can generally be classified into either convex or nonconvex. The term
convex can be applied to both sets and functions. A convex set X is a set, that a straight line
segment joining any two point within the set lies completely in the same set. This can be written
as:
αx + (1 − α)y ∈ X, ∀α ∈ [0, 1] (1.3)
A function f is convex if a convex set is its domain and any two points x and y lie in this
domain. A convex function can be written as:

f (αx + (1 − α)y) ≤ α f (x) + (1 − α) f (y) ∈ X, ∀α ∈ [0, 1] (1.4)

The solution of a convex function yields a global minimum. If the conditions of a convex set is
not satisfied, then the set is nonconvex.

1.3 Mixed integer nonlinear optimization

The general formulation of a mixed integer nonlinear optimization problem is as shown below:

minimize f (X)
subject to g j (X) ≤ 0, j = 1, 2, 3, ..., J
(1.5)
hk (X) = 0, k = 1, 2, 3, ..., K
where X ∈ RIx × NIy

and F is the objective functional, the functionals g j and hk are the inequality and equality con-
straints respectively.The variable X could take both integer and real components. If the integrity
condition on X ∈ NIx is relaxed, the problem becomes a nonlinear optimization problem. If the
objective function and the constraints are linear, then the problem is a linear optimization prob-
lem and finally, if the objective function or one of the constraints is nonlinear, then the problem
is a nonlinear optimization problem.

2
1.4 Feasibility problems
In mathematical optimization, a feasibility problem is the type of problem that has the target
of finding a feasible solution without considering the objective function value. For this type
of problem, any solution obtained is regarded as good. A feasible solution is a solution that
satisfies all the (linear and nonlinear) constraints of a problem (Wright and Nocedal, 1999). A
major challenge in many optimization problems is finding an initial feasible point, feasibility
condition relaxation has been one method used to achieve this point.

1.5 Problem statement

In real world engineering design, most of the design optimization problems are either convex
or nonconvex mixed integer nonlinear optimization problems. These kinds of problems do not
have a single solution method and there exist several different methods for solving them. In
using population based algorithms such as genetic algorithm, it has been shown in research that
a good initial feasible solution could lead to achieving a better optimal solution and cheaper
computational cost (Dhavachelvan and Uma, 2005; Meadows et al., 2013; Paul et al., 2013;
Truemper, 2015; Diaz-Gomez and Hougen, 2007). Therefore, it is of great interest to develop
and investigate different seeding methods to identify the ones that achieve a good initial feasible
solution.

1.6 Objectives
The objective of this study is to propose algorithms for generating initial feasible solution for
population based algorithms and investigate the impact of initial feasible solution method on
the performance of the three step algorithm. The population based algorithm used in this study
is genetic algorithm and local search method is sequential quadratic programming.

1.7 Organisation of report

The report is oganised as follows; in chapter 1, an introduction to optimization is given with
problem statement and objectives of the study. The chapter 2 is the state-of-the-art, several
optimization tools are discussed. Then chapter 3 is the introduction of methods employed and
discussion of the structure of experiments. Finally, chapter 4 is the presentation of results,
discussion and conclusion. The problems investigated are presented in the appendix.
 2
State-of-the-art

2.1 Literature review

The general formulation of an engineering design optimization problem can be stated as in
problem 2.1 (Kannan and Kramer, 1994):

minimize f (X)
subject to g j (X) ≤ 0, j = 1, 2, 3, ..., J
(2.1)
hk (X) = 0, k = 1, 2, 3, ..., K
where X ∈ RIx × NIy

Where F(X) is the objective function and X are the design variables.
There are several methods to solve a constrained optimization problem. In this section,
we will review some of the methodologies already applied to finding solutions to nonlinear
constrained optimization problems and also some seeding methods.
The most popular method for solving the constrained optimization problem in engineering
design has been augmented Lagrangian or method of multiplies (Arora et al., 1991). This
method has the ability to reduce ill conditioning of subproblems which makes it superior to
the quadratic penalty method and log-barrier approach (Wright and Nocedal, 1999). Several
researchers have investigated the method from different perspectives. Andreani et al. (2007)
studied the augmented Lagrangian methods on problems where it is easier to fix one constraint
and solve the other. The problem was presented as

minimize f (x) subject to x ∈ Ω1 ∩ Ω2 (2.2)

x

Given that it is easier to solve problem 2.2 with Ω2 fixed. There exist several efficient al-
gorithms to solve this type of problems. In general, it is easier to solve the problem this way.
Boland and Eberhard (2014) also investigated augmented Lagrangian dual for integer program-
ming and provided a primal characterization for its bound. Given a mixed integer program
defined by:
zIP = sup{cx|Ax = b, x ∈ X} (2.3)
the Lagrangian relaxation can be written as:

zLR (λ ) := sup[cx + λ (b − Ax)] (2.4)

x∈X
with zLD := infλ zLR (λ ) as the Lagrangian dual associated to the primal. It is observed that
zLR (λ ) yields an upper bound with zLD , the best of such a bound on the mixed integer program.
Let the augmented Lagrangian relaxation be represented by:

zLR+
δ (λ ) := sup[cx + λ (b − Ax) − δ ρ(b − Ax)] (2.5)
x∈X

where ρ is a penalty term and δ ≥ 0 is a fixed parameter. Boland and Eberhard (2014) showed
that zLR+
δ is a strong duality for integer programming. In the paper of Feizollahi et al. (2017);
Boland and Eberhard (2014), they provided a different characterization for the duality of integer
programming.
The basic principle of the method is to convert a constrained optimization problem into
unconstrained optimization problems by adding a penalty term to the objective function. The
augmented Lagrangian of equation 2.1 can be written as:
J K
A(X, λ , r p ) = F(X) + ∑ [λ j g j (X) + r p g j (X)2 ] + ∑ {λk+t hk (X) + r p [hk (X)]2 } (2.6)
j=1 k=1

The λ j are the Lagrange multipliers while the r p are the parameters for the penalty. The aug-
mented Lagrangian algorithm can be state as shown in Algorithm 1 (Arora et al., 1991):

Algorithm 1 The general form of augmented Lagrangian

Step 1. Initialise k = 0, define λ (0) and r(0)
Step 2. Minimize A(X, λ (k) , r(k) ). Let x(k) be the solution
Step 3. Stop the iteration if convergence criteria is satisfied. Else go to step 4
Step 4. Update λ (k) and increase r(k)
Step 5. Set k = k + 1 and go to step 2

The penalty approach was further investigated by Puzzi and Carpinteri (2008). They pro-
posed a double-multiplicative dynamic penalty approach for constrained optimization prob-
lems. The penalty function was divided into two parts with one focusing on the problem with-
out equality constraints and the other case on the problem with equality constraints. The two
penalty functions are as follows:
J  
ĝ j (x) q + Q

P(x, q) = ∏ 1 + (2.7)
j=1 bj Q

J    K   
ĝ j (x) q + Q |hk (x)| q + Q
P(x, q) = ∏ 1 + ∏ 1 + ck (2.8)
j=1 bj Q k=1 Q
Where equations 2.7 and 2.8 represent the problem without equality and with equality con-
straints respectively. These two penalty functions were tested on five well known problems
that are used as benchmark for genetic algorithm approaches. These problems include; a ten-
variable function problem by Hock and Schittkowski (1981), the welded beam problem by Rao
(2009), a five variable function problem by Himmelblau (1972). Equation 2.7 performed bet-
ter for equations with inequality constraints but equation 2.8 underperformed compared with

6
existing strategy from Wu and Lin (2004). A modified version of equation 2.8 with better
performance was given as shown in equation 2.9:
J    K   
ĝ j (x) q + Q |hk (x)| q + Q
P(x, q) = ∏ 1 + ∏ 1 + K(q) ck (2.9)
j=1 bj Q k=1 Q

where
1
K(q) = (2.10)
r(Q − 1) [(r − 1)q + (Q − r)]
A detailed discussion on the different experiments can be found in Puzzi and Carpinteri (2008).
Kannan and Kramer (1994) studied an augmented Lagrangian method for mixed integer
discrete continuous optimization. They combined augmented Lagrangian with Powell’s method
and also with Fletcher and Reeves method to solve problems with discrete variables and also
problems with mixed discrete, integer, binary and continuous variables. The problems they
worked on had already been investigated by Sandgren (1990) using branch and bound method
together with exterior penalty function method and quadratic programming method. In all the
cases studied, the augmented Lagrangian method achieved a better solution than the branch and
bound method with the stopping criteria used.
Solving constrained optimization problems using augmented Lagrangian method can be
computationally expensive. The method requires several solutions of unconstrained optimiza-
tion problem. Deb and Srivastava (2012) proposed a genetic algorithm based parameter update
strategy to a specific augmented Lagrangian method. The method proposed by Deb and Sri-
vastava (2012) updates the critical parameters based on population statistics in a self-adaptive
manner. Convergence to the entire algorithm was achieved using local search. The genetic al-
gorithm augmented Lagrangian was shown to perform better and reliable compared with other
algorithms which were examined.
Araromi et al. (2015) applied the augmented Lagrangian genetic algorithm to decentralized
control configuration design for catalytic cracking units. Adeli and Cheng (1994) used the
augmented Lagrangian genetic algorithm for optimization of space structure. The augmented
Lagrangian genetic algorithm only require a few simple function evaluations compared with the
penalty function-based genetic algorithm. The study also showed that, the trial an error method
for coefficient of the starting penalty function and arbitrary adjustments can be avoided.
Benders decomposition is an algorithm for solving problems that have block structure. The
problem is divided into master problem (relaxation of integer variables) and subproblem. The
generalized Benders decomposition is a generalization of the Benders decomposition method
devised by Benders (1962). This generalization was done by Geoffrion (1972). The generaliza-
tion extends Benders decomposition to solution of mixed integer nonlinear convex optimization
problems (Geoffrion, 1972; De Wit, 2005). For the case where the given problem is noncon-
vex, Frank and Rebennack (2015) proposed an algorithm for solving it. The algorithm involve
convex relaxation of the main problem to achieve the lower bound problem which generates a
trial point y0 and associated lower bounds zi . The trial point is used to solve the upper bound
problem which yields a feasible solution x0 and associated upper bound z j . The complicating
variable y is relaxed in the upper bound problem. They applied the algorithm on optimal design
of mixed AC-DC distribution systems for commercial building.
The feasibility pump algorithm is a method for finding feasible solution for mixed inte-
ger optimization problems. The idea of feasibility pump was first proposed by Fischetti et al.
(2005). It is amongst the best known heuristics for mixed integer programming. The initial idea
for this heuristic was for 0 − 1 mixed integer linear programs. The approach has been extended
over the years to convex nonlinear problems and also nonconvex problems. Berthold et al.
(2017) provided an overview of the development of feasibility pump in the last ten years. Fea-
sibility pump algorithms are examples of a general class of algorithms called Successive Pro-
jection Methods. These types of algorithms target the problem of deciding emptiness of a set
intersection A ∩ B. For the case of nonconvex mixed integer nonlinear programs, D’Ambrosio
et al. (2012) were the first to extend the work of Bonami et al. (2009) from convex to nonconvex
problems. D’Ambrosio et al. (2012) presented the general form of the algorithm 2 for feasibil-
ity pump and addressed the challenges for cases with linear objective function and nonconvex
constraint. For the nonconvex problems, the algorithm alternates between solving a nonconvex
nonlinear program and a convex mixed integer nonlinear program. Solving (P1) which is ob-

Algorithm 2 The general form of feasibility pump

i=0
initialize (x̂0 , ŷ0 ) and (x̄0 , ȳ0 )
while (x̂i , ŷi ) 6= (x̄i , ȳi ) and CPU time < limit do
increase i
solve (P1)
solve (P2)
end while

tained by integrity relaxation yields (x̄i , ȳi ) while solving (P2) which is the original problem is
achieved by nonlinear constraints relaxation yields (x̂i , ŷi ).
The problem under investigation can sometimes have pseudo-convex ojective function and
pseudo-convex constraints. These kind of problems share same properties as convex functions
with reference to finding local extrema and derivative properties (Weisstein, 2014). The prin-
cipal difference is, they are strictly weaker than convex funtions and every convex function is
pseudo-convex but not vice versa. Pörn and Westerlund (2000) proposed a mixed integer non-
linear programming algorithm to minimise these kinds of problems. They extended the αECP
(Exteneded Cutting Planes) which was introduced by Westerlund et al. (1998). The principal
idea of the new method is to approximate the objective function at a certain boundary with an
improving cone of linearizations and all violated constraints by cutting planes. The method is
said to be the only one to achieve convergence to global optimum aside the branch and bound
methods for pseudo-convex problems.
It is known that the quality of the initial population for population based algorithms is very
important in achieving a good solution (Dhavachelvan and Uma, 2005; Meadows et al., 2013;
Paul et al., 2013; Truemper, 2015; Diaz-Gomez and Hougen, 2007). The initial population is
randomly generated as an example for population based algorithms. The population is gen-
erated by first sampling the search space. Ray et al. (2007) proposed the nearest neighbor
technique for travelling salesman problems (TSP) as an alternative to random seeding. They
achieve concluded that the TSP method was faster and achieved better results. (Dhavachelvan
and Uma, 2005) also proposed gene bank seeding method, this method uses TSP and quali-
fied individuals are grouped to form a gene bank. Using nearest neighbor, a feasible solution
is generated. Yang (1997) proposed selective initialization which uses the k-nearest neighbor
technique and assign priority to individuals that are most closed related. Yugay et al. (2008)
proposed sorted population method, this method sorts the population in ascending order based

8
on fitness and the best individuals are selected. Paul et al. (2013) investigated the performance
of these population seeding methods on genetic algorithms. They indicated that although the
random seeding is simple and easy to implement, the population generated may contain poor
fitness individuals which may result in higher computation time to converge to optimal solution
if any. They concluded that other algorithms had better initial population on the same problem
than random seeding, hence a better solution.
Maaranen et al. (2007) investigated spatial point process, specifically, simple sequential
inhabitation process (SSI). SSI generates uniformly random individuals that are at least a pre-
define minimum distance apart. That is, it selects only individuals that are greater than or
equal to the predefined distance. Individuals generated from SSI are far apart from adjacent
individuals which results in an evenly distributed population without clusters. Another method
also investigated was nonaligned systematic sampling (NSS). For NSS, a unit hypercube is
equally divided and one sample point is selected from each interval using prescribed rules.
Latin Hypercube Sampling (LHS) was investigated by Liefvendahl and Stocki (2006) for ge-
netic algorithms. The method generates a near-random sample of the population by searching
n dimensional space. They found that genetic algorithm performed better with LHS. Deb and
Tiwari (2008) also used LHS in generating initial population for genetic algorithm.
 3
Presentation of methods

In this part of the study, the methods that will be used are discussed and a concise structure
of their algorithms are presented. The genetic algorithm and augmented Lagrangian and se-
quential quadratic programming will be investigated. A combination of augmented Lagrangian
method and genetic algorithm (ALGA) will be applied to solving varying engineering prob-
lems. The main work of this study, which is seeding methods will be discussed. A detailed
description of these methods will be presented in the sections below.
A Mixed Integer Nonlinear Optimization Problem (MINLP) is a problem that can be repre-
sented in the form of problem 3.1:

min f (x, y)
subject to xi ∈ [lxi , uxi ] ⊂ Rl1 , i = 1...I1
yi ∈ [lyi , uyi ] ∩ Nl2 , i = 1...I2 (3.1)
g j (x, y) ≤ 0, j = 1...J
hk (x, y) = 0, k = 1...K

f : Rn → R is the objective function, the functions g j and hk are the inequality and equality
constraints respectively. Where x and y are variables, y an integer and lx, ux, ly, uy are lower(l)
and upper(u) bounds. By removing the integrity constraints on y, we obtain the integrity
constraint relaxed problem. The relaxed problem can be stated as follows:

min f (x, y)
subject to xi ∈ [lxi , uxi ] ⊂ Rl1 , i = 1...I1
yi ∈ [lyi , uyi ] ∩ Nl2 , i = 1...I2 (3.2)
g j (x, y) ≤ 0, j = 1...J
hk (x, y) = 0, k = 1...K

A straight-forward reformulation of problem 3.1 is to replace the condition y ∈ [ly, uy] ∩ Nl2
with y ∈ [ly, uy] and add an integrity constraint of the form |yi − int(yi )| ≤ ε, i = 1...I2 . Note
that these new constraints are not differentiable due to the int function.
There exist several algorithms that could be used to solve this kind of problem. In most
cases, different methods solve certain kinds of problems better than others. As an example,
there are methods that are very efficient for problems with only equality constraints while oth-
ers are efficient for inequality constraints and others for problems without constraints. In real
life engineering design problems, the problem mostly contains both types of constraints which
calls for the need to develop efficient methods to solve such problems. As mentioned in the
introductory section, it is easier to solve unconstrained problems and therefore it is interesting
to investigate methods that could convert constrained problems into unconstrained problems.
In this study, four seeding methods are studied, these are: random seeding, feasibility seed-
ing algorithm, a five step seeding algorithm and seeding by repair algorithm. These seeding
methods are applied to solving general constrained optimization problems using a three step
algorithm. The seeding method forms step of the algorithm, step 2 is to solve the problem
using genetic algorithm and step 3 is to refine the solution achieve at step 2 using sequential
quadratic programming. This is shown is algorithm 9.

3.1 Three step algorithm

The three step algorithm is the main algorithm used in this study, it is as presented as algorithm
3. The population based algorithm used is the genetic algorithm and the local search used is
sequential quadratic programming. The initial feasible solution is generated using the seeding
methods which will be described in the subsection.

Algorithm 3 The three step algorithm

Step 1. Generate initial feasible solution
Step 2. Solve the problem using augmented Lagrangian population based algorithm
Step 3. Refine solution using a local search method.

3.2 Augmented Lagrangian method

The augmented Lagrangian method is a method that is motivated by constructing an uncon-
strained optimization problem given that it is easier to solve the unconstrained problem. The
advantages of this method are: it reduces the possibility of ill-conditioning of the subproblems
(unconstrained problems), the minimizing function is continuously differentiable and the algo-
rithm also does not introduce non-smoothness (Wright and Nocedal, 1999). This is achieved
through the introduction of Lagrange multiplier estimates to the minimizing function. Aug-
mented Lagrangian can be applied to both equality and inequality constraints.
Given the equality constrained optimization problem 3.4 below:

minimize f (x)
x
subject to h(x) = 0
(3.3)
x∈X
for all f : Rn → R and h : Rm → R

A fundamental function associated with the solution to problem (3.4) is the Lagrangian. The
Lagrangian function L(x, λ ) can be written as

L(x, λ ) = f (x) − λ T h(x) (3.4)

12
where λ is the Lagrange multiplier. The augmented Lagrangian for the problem is presented
as:
t
1 t
LA (x, λ , ρ) = f (x) − ∑ λiT hi (x) + ρ ∑ c2j (x) (3.5)
i=1 2 j=1

Where c is the constraint violation, λ is the Lagrange multiplier, ρ is a positive penalty param-
eter. A quadratic term has been added to the Lagrangian function to achieve the augmented
Lagrangian. Given that the unique solution for equation 3.5 is x∗ , then both the quadratic
penalty of equation 3.4 and its gradient turns to 0 at x∗ . If λ = λ ∗ then x∗ is a stationary
point of equation 3.6. In this case, the constrained optimization problem has been transformed
into an unconstrained optimization problem using the augmented Lagrangian method. This is
different from the Lagrangian method by the introduction of the quadratic penalty term. To
solve the augmented Lagrangian problem, λ and ρ are chosen carefully so that LA (x, λ , ρ) is
minimized with respect to x. The resulting x is used to choose the values of the next λ and ρ.
If ρ is chosen to be reasonably large, then minimizing LA (x, λ , ρ) will cause c(x) to be small
and this is not dependent on the value of ρ. The λ can be updated using the formula:

λik+1 = λik − ρci (xk ) (3.6)

The ρ is increased if there has not been significant decrease in constraint violation at xk+1
compared with the one at xk The ρ can be updated using the formula:

ρ = βρ (3.7)

β must be chosen carefully so that ρ is neither too big or too small but large enough. Algorithm
3 shows the general form of augmented Lagrangian.

Algorithm 4 The general form of augmented Lagrangian

Step 1. Initialize k = 0, estimate λ (0) and ρ (0)
Step 2. Minimize LA (X, λ (k) , ρ (k) ). Let x(k) be the solution
Step 3. Stop the iteration if convergence criteria is satisfied. Else go to step 4
Step 4. Update λ (k) using equation 3.7 and increase ρ (k) using equation 3.8
Step 5. Set k = k + 1 and go to step 2

3.3 Genetic algorithm

A genetic algorithm is a search method for solving unconstrained optimization problems. The
method works by imitating the principle of natural selection that exist in evolutionary prin-
ciples. The genetic algorithm takes care of the problem of initial solution by starting with a
random set of solutions. Each solution is assigned a fitness value and this value is associated
with the problem and search objective function. The population is updated by applying three
operations similar to the natural selection principle, which are, reproduction, crossover and mu-
tation (Goldberg et al., 1991). The reproduction principle selects the individuals (parents), that
will contribute to the generation of the next population. The crossover principle puts together
two parents from the reproduction to create the next generation. The mutation principle applies
random changes to each parent to create new offspring. The genetic algorithm method was first
discovered by Professor John Holland and his students in 1965 (Goldberg and Holland, 1988).
An objective of their research was to develop an artificial system that mimics the natural se-
lection principle. The results of their study has led to several discoveries in the artificial and
natural systems and wide range of applications for these results. The principal objective of ge-
netic algorithm has been the robustness of the algorithm. It has been proven to provide robust
search in complicated systems (Goldberg and Holland, 1988). Algorithm 5 represents the gen-
eral form of genetic algorithm. A general diagram describing genetic algorithm can be found
in f igue 1 The termination criteria for genetic algorithm include: solution found, maximum

Algorithm 5 The general form of genetic algorithm

Step 1. Initialize the problem parameters, N, selection operator, crossover operator, and
mutation operator.
Step 2. Evaluate each string in the population N.
Step 3. Check if t > tmax or termination criteria is satisfied, STOP. Else go to step 4
Step 4. Reproduce a new population.
Step 5. Perform crossover on random pairs of strings.
Step 6. Mutate on selected string.
Step 7. Set t = t + 1 and go to step 2.

number of generations reached, fitness limit reached and no feasible point found.

3.3.1 Selection
During each successive generation, a proportion of the population is selected to produce new
generation. The individual solutions of the population are selected through a fitness method.
The fitness of each solution can be rated to select the best solution or only a random sample of
the population is rated. The fitness function could be stochastic to ensure that a large portion
of the fit individuals are selected.

3.3.2 Reproduction
After selecting individual solutions, the next step is to generate a new population of solutions.
The crossover and mutation operators are used. To produce a new solution, two parents are se-
lected from the group of previously selected individuals for breeding. An offspring is produced
using crossover and mutation and the process is repeated until a new population of solutions is
generated.

3.3.3 Crossover
This is the point where the alleles from one parent is switched with the other. The alleles are
the possible solution set of a problem. The offspring takes chromosomes from both parents

14
 Figure 3.1: Sketch of GA setting, operation and results. Source: Genevois (Unpublished)

and the crossover point is selected randomly. The chromosome is set of parameters defining a
potential solution to a genetic algorithm problem.

3.3.4 Mutation
Mutation is the operator that ensures diversity in the new population. To ensure that only a
small portion of the individuals are the same, allow only a small chance of mutation. This is
achieved by adjusting a selected alleles or replacing it with a new value.

3.4 Seeding methods

Seeding is a method of generating an initial population as input for genetic algorithm (Katayama
et al., 2000). It has been reported in some studies that, the method used to seed the initial
population affects the behavior of genetic algorithm and also the quality of the final solution
(Dhavachelvan and Uma, 2005). The most commonly used seeding method is Random Seeding
but there exist methods such as Nearest Neighbour, Selective Initialisation and Sorted Popula-
tion (Paul et al., 2013). As mentioned earlier, in this study, four seeding methods are investi-
gated, these are: random seeding, feasibility seeding algorithm, five step seeding algorithm and
seeding by repair algorithm.

3.4.1 Random seeding

This algorithm is used as a benchmark for the other seeding algorithms under investigation.
The random seeding method selects a population from a group of selected individuals generated
randomly inside bounds. The population size depends on the type of problem but traditionally,
it is generated to cover the entire search space. This group of individuals is generated by
sampling the search space through generating a number of individuals randomly.

3.4.2 Feasibility seeding algorithm

The feasibility seeding algorithm is a seeding method that generates x0 at random then apply ge-
netic algorithm as a search method to find initial feasible solution without objective. Algorithm
6 represents the feasibility seeding algorithm. The objective function is not considered as only
a feasible solution is needed.

Algorithm 6 Feasibility seeding algorithm

Step 0. Initialization: generate x0 ∈ [lx, ux] ∩ N, y0 ∈ [lx, ux] ∩ N at random
Step 1. While number of individuals is less than Npop
Step 2. Run genetic algorithm with constant objective value using the initial guesses x0 , y0
to obtain x, y

3.4.3 Five step seeding algorithm

The five step seeding algorithm is a seeding method that uses both random seeding and a search
method to obtain an initial feasible solution. The five step algorithm starts by randomly seeding
a population then an augmented Lagrangian genetic algorithm is used to search for the initial
feasible solution with relaxed integrity. If the stopping criterion is not satisfied, the integer vari-
ables in the problem are fixed and an augmented Lagrangian genetic algorithm is applied on the
generated point to achieve initial feasible solution. Algorithm 7 represents the five step seeding
algorithm. Equations 3.2 and 3.1 represent the relaxed and primal problems respectively.

Algorithm 7 Five step seeding algorithm

Step 0.Initialization: generate x0 ∈ [xl , xu ] ∩ R, y0 ∈ [yl , yu ] ∩ N
Step 1. Solve equation 3.2 using the initial guesses x0 , y0 to obtain x, y
Step 2. Test if max|int(y) − y| is within a prescribed tolerance ε, stop, else go to 3
Step 3. Round y to the nearest integers to obtain ỹ
Step 4.Solve equation 3.1 with y = ỹ held constant to obtain x̃
Step 5. Set x0 = x̃, y0 = ỹ and go to step 1

16
3.4.4 Seeding by repair
Numerical optimization for population based algorithms sometimes encounters challenges in
problems with non-trivial constraints. The solution of such problems sometimes lies on the
bounds of the feasible region. Schoenauer and Michalewicz (1996) discusses the importance
of searching for a solution to a constrained optimization problem at the frontier of the feasible
solution. Chootinan and Chen (2006) used this method inside genetic algorithm to repair a
percentage of solutions by projecting them on the frontier of the feasible space. The method
presented here involves every infeasible individual but only at the initial generation.
The seeding by repair method presented here initially generates randomly the initial pop-
ulation within their bounds then by projecting them on the frontier of the feasible space, the
infeasible ones are transformed into feasible individuals. Algorithm 8 represents the seeding
by repair method.

Algorithm 8 Seeding by repair

Step 1. Randomly generate initial population X
Step 2. Construct the column vector of the constraints

V = [X1 , ..., Xl , G1 (X), ..., G j (x), H1 (X), ..., Hk (X)]T

Step 3. Determine the degree of the constraint violation ∆V by

 
min{0,UX − X} + max{0, LX − X}
∆V =  min{0,UG − G(X)} + max{0,UX − G(X)} 
C − H(X) (I+J+k)×1

Step 4. Tolerance (tol) test, if |∆V | < tol, then accept X as solution and terminate
Step 5. Evaluate the derivatives of the constraints with respect to X
 
Id
∇V =  ∇G(X) 
∇H(X) (I+J+k)×1

Step 6. Select the elements of ∆V with values exceeding tol, ∆V and ∇V reduce in size
Step 7. Compute pseudoinverse ∇V p
Step 8. Update X with X + ∇V p × ∆V and return to step 2.

3.5 Sequential Quadratic Programming

Sequential quadratic programming (SQP) is a an effective method for solving constrained non-
linear optimization problems by solving quadratic subproblems. The SQP solves the quadratic
subproblem and use the solution to create a new iterate xk+1 . The new iterate is constructed in
such a way that the variables converge to a local minimum x∗ . Algorithm 9 represents a general
SQP algorithm.
Considering the equality constrained problem 3.4, fk and ∇ fk are the objective functional
and its derivative respectively, Bk is the definite Hessian approximation, ck is vector of con-
Algorithm 9 Sequential quadratic programming algorithm
Step 0. Initialize (x0 , λ0 )
Step 1. Choose an initial n × n symmetric positive definite Hessian approximation B0
Step 2. Evaluate f0 , ∇ f0 , c0 and A
Step 3. If termination criteria is satisfied, STOP, else go to 4
Step 4. Solve the subproblem to obtain pk , choose µk and set αk = 1
−1
Step 5. Update xk+1 = xk + αk pk , λk+1 = − Ak+1 ATk+1 Ak+1 ∇ fk+1


Step 6. Update Bk+1 by updating Bk using a quasi-Newton formula

straint, Ak denote the Jacobian matrix of the constraints and pk and αk are unique solutions.
Considering the mixed integer nonlinear problem of problem 1.2, the subproblem can be writ-
ten as:
1
minn pT Wk p + ∇ fkT p
x∈R 2
subject to (3.8)
T
∇gi (xk ) p + gi (xk ) ≤ 0, i ∈ ε
∇hi (xk )T p + hi (xk ) = 0, i ∈ I

3.6 Experiments
In this section of the study, the work is focused on performing experiments by generating good
initial feasible solutions for population based algorithms, these are called seeding methods. The
initial feasible solution is then used as initial solution for augmented Lagrangian population
based algorithms to solve the problem. Finally the solution obtained is refined with sequential
quadratic programming. A Matlab solver is used to be adapted to other variations of general
optimization problems. Experiments will be performed on different optimization problems to
investigate the performance of the seeding methods.
As mention before, the three step algorithm is the main algorithm used in this study.The
step 1 of this algorithm is the principal focus of this study as mentioned in the objectives.
The seeding algorithms presented in the former chapter are used to generate initial feasible
solutions for the problems under investigation. The initial feasible solution play an important
role in finding the optimal solution and a cheap computation cost as indicated in the literature
review. A good initial feasible solution will lead to a fast identification of the optimal solution.
The random seeding method is the default seeding method used in genetic algorithms. It
randomly generates the initial population by presampling the search space. In this study, the
random seeding method is used as a reference for the proposed seeding methods.
The feasibility seeding method is as described in subsection 3.4.2.The algorithm is run
several times and the best solution is selected as the initial feasible solution. The time required
to generate each solution is also measured to accessed the computational cost of the seeding
method. The five step seeding algorithm is as described in subsection 3.4.3. The seeding by
repair method is as described in subsection 3.4.4. It starts with random generation of initial
population and integer variables are fixed fixed to achieve the initial feasible population.
After obtaining an initial feasible solution with either of the seeding methods above, the
next step is to solve the problem to optimality using augmented Lagrangian genetic algorithm

18
(ALGA), algorithm 10. The solution obtained is further refined using sequential quadratic
programming (SQP). The algorithm for this procedure is called, the three step algorithm which
is presented in algorithm 3.

Algorithm 10 Augmented Lagragian genetic algorithm (ALGA)

Step 1. Initialise k = 0, estimate λ (0) and ρ (0) and generate the initial population N pop
Step 2. Minimise LA (X, λ (k) , ρ (k) ). Let x(k) be the solution
Step 3. Evaluate each member of the population
Step 4. Reproduce a new population
Step 5. Perform crossover on random pairs of strings
Step 6. Mutation
Step 7. Stop if t > tmax or the termination criteria is satisfied, STOP. Else go to step 4
Step 8. Update λ (k) using equation 3.6 and increase ρ (k)
Step 9. Set k = k + 1 and go to step 2

The problems under investigation are sourced from MINLPlib with the exception of MAPSE
(reformulated) and presented in the appendix of this report. The problems are; winFac, nvs20,
nvs21, and st_e38, all present on MINLPlib instances. Each problem has different number
of variables which could include integer variables.A summary of the problems description is
presented in Table 3.1

Problems description
No. Vari- No. Integer No. Con- No. Equal- No. In-
ables Variables tinuous ity Con- equality
Variables straints Constraints
MAPSE 7 1 6 3 8
windFac 14 3 11 14 0
nvs20 16 5 11 1 7
nvs21 3 2 1 0 2
st_e38 5 2 3 0 3
Table 3.1: Some properties of the problems under investigation

The experiments were run with a Matlab 2017a on a Dell OPTIPLEX 9020. Most of the
default settings for GA and SQL in Matlab were unchanged while the following were changed
to suit our experiments as shown in the Table 3.2. 100 runs is made for each experiment and
the best results selected. The stopping conditions used are the default conditions in Matlab GA
setup.
 GA and SQP settings
Properties value
Population size 20
Objective function scaling by rank
Elite count 2
Crossover fraction 0.8
Initial penalty 10
Penalty factor 100
Tolerance on seeding 10−3
Tolerance on constraints 10−6
Tolerance on objective function 10−6
Maximum generations 100
Table 3.2: Settings for GA and SQP

20
 4
Results and conclusion

4.1 Results
Experiments were performed for the different problems under investigation as mentioned in
section 3. The results obtained from running the main problems can be found in Table 4.1. In
Table 4.1, a comparison of the solution achieved and the solution in MINLPlib is presented.
The difference between our solutions and the solutions found on MINLPlib could be attributed
to the different tolerance levels used for the problems. Our experiments were conducted with
tolerance of 1e−6 for GA and SQP for all problems while this is different on MINLPlib. Wind-
Fac had a tolerance of 0, nvs20 had tolerance of 2e−16, nvs21 had tolerance of 3e−14 and
st_e38 had tolerance of 0. With our settings for GA and SQP, we could not achieve a solution
for windFac. For nvs20, nvs21, st_e38, we achieved better solution than the standard solutions.
For nvs21, our solution was the same as the one found by D’Ambrosio et al. (2012) except
for the seeding by repair which is almost the same as the solution on MINLPlib. None of the
algorithms implemented was successful in achieving a solution for the windFac problem. On
average, seeding random had the lowest seeding time in all problems while five step has the
highest seeding time. Considering the percentage of valid seeding, five step has the best per-
centage on average. With respect to good solution, the three step algorithm achieves the overall
best solution with five steps followed by feasibility and seeding random. On average, the three
step algorithm achieved the least computational cost with five step algorithm.
Figures 4.1, 4.2, 4.3 show the percentage of valid seeding, percentage of valid GA solution
and percentage of valid SQP solution respectively. The five step has on average the best per-
centage in all cases followed by feasibility and seeding random. Seeding by repair achieved
solutions for just nvs20 and nvs21 and at 100%. Seeding random is assumed to achieve zero
valid seeding.
From Figures 4.4, 4.5, 4.6, 4.7, 4.8, it is observed that on average, five step has a higher per-
centage of valid seeding and largest seeding time, followed by feasibility except for MAPSE
where feasibility has highest seeding time. Seeding random is almost zero due to the assump-
tion that the percentage of valid seeding is zero and the random seeding times for all the prob-
lems on average are also very close to zero. From Figures 4.5, windFac has no valid seeding
(zero percent) therefore the plot lies on the x-axis.
 Summary of Experiment Results
MAPSE WindFac nvs20 nvs21 st_e38
Average seeding time
Seeding Random 0.0001120 0.00011284 0.0001228 0.000101 0.000081
Feasibility Seed 1.214799 924.13645 0.8041256 2.146198 0.485609
5 Step Seeding 1.456601 1507.0353 1.0456947 0.945322 0.709330
Seeding by Repair 0.2997002 0.1027481 5.136527 0.002866
Average % valid seeding
Seeding Random 0 0 0 0 0
Feasibility Seed 98 0 100 100 100
5 Step Seeding 100 0 100 100 100
Seeding by Repair N/A N/A N/A N/A
Average % valid GA solution
Seeding Random 66 0 100 100 56
Feasibility Seed 54 0 100 100 93
5 Step Seeding 47 0 100 100 96
Seeding by Repair 0 100 100 0
Average no. GA generations
Seeding Random 44 100 4 7 46
Feasibility Seed 64 100 4 4 15
5 Step Seeding 64 100 5 4 7
Seeding by Repair 1 4 3 1
Average max constraint ViolationGA
Seeding Random 0.215781 93.111831 4.039E-14 8.441E-07 1.730E-08
Feasibility Seed 1.603E-6 56.656909 7.799E-09 4.312E-13 8.043E-09
5 Step Seeding 1.947E-6 2.2231011 0 5.624E-13 1.323E-08
Seeding by Repair 67.45067 0 9.115E-08 1.996718
Average % valid SQP solution
Seeding Random 95 0 100 100 81
Feasibility Seed 94 0 100 100 99
5 Step Seeding 97 0 100 100 77
Seeding by Repair 100 100 100 0
Total time
Seeding Random 0.432561 0.6305079 0.0265297 0.0308075 0.2957371
Feasibility Seed 0.572279 26.115511 0.0471982 0.073278 0.101852
5 Step Seeding 0.403433 42.481905 0.0581977 0.0400186 0.0803147
Seeding by Repair 0.024596 0.1160818 0.182188 0.0130273
Best solution value
Seeding Random 2.835E-4 None 170.676128 0.00 7124.7603
Feasibility Seed 3.101E-4 None 170.676082 0.00 7191.9368
5 Step Seeding 3.018E-4 22 None 170.675829 0.00 7115.13287
Seeding by Repair None 170.675974 -5.684781 None
MINLPlib solution 0.254487 230.922165 -5.684782 7197.72714
Table 4.1: Summary of results
Figure 4.1: Percentage of valid seeding for different seeding methods
Figure 4.2: Percentage of valid GA for seeding methods and problems

Figure 4.3: Percentage of valid SQP for seeding methods and problems

24
Figure 4.4: MAPSE: Plot of valid seeding against seeding times

Figure 4.5: WindFac: Plot of valid seeding against seeding times

Figure 4.6: nvs20: Plot of valid seeding against seeding times

Figure 4.7: nvs21: Plot of valid seeding against seeding times

26
Figure 4.8: st_e38: Plot of valid seeding against seeding times
4.2 Conclusion and future work
In conclusion, we have introduced three seeding methods for achieving good initial feasible
solution for general mixed integer nonlinear optimization problems with seeding random as
the benchmark. The experiments were conducted using the three step algorithm which is the
seeding methods, ALGA and SQP. Using seeding random as the benchmark, it can be seen
that it has the overall least computational cost followed by feasibility seeding and five step
seeding. With respect to good solution, five step seeding achieves the overall best solution for
all problems followed by feasibility seeding and seeding random. From this observation, the
most promising seeding algorithm on average is the feasible seeding algorithm compare with
seeding random and five step seeding. The seeding by repair seems promising too but in this
work, it failed to achieve solution for problems that feasibility seeding did achieve solution.
The methods investigated has been used to achieve some results but there still remain other
interesting research to be conducted in this direction. Future work could investigate different
global search methods to replace genetic algorithm for the seeding methods. Another inter-
esting results could be to investigated the performance of other population based algorithms
against the results achieved.

28
 Bibliography

Hojjat Adeli and Nai-Tsang Cheng. Augmented lagrangian genetic algorithm for structural
optimization. Journal of Aerospace Engineering, 7(1):104–118, 1994.

Roberto Andreani, Ernesto G Birgin, José Mario Martínez, and María Laura Schuverdt. On
augmented lagrangian methods with general lower-level constraints. SIAM Journal on Opti-
mization, 18(4):1286–1309, 2007.

Dauda Olurotimi Araromi, Kazzem Kolapo Salam, and Aminah Abolore Sulayman. Aug-
mented lagrangian genetic algorithm based decentralized control configuration design for
fluid catalytic cracking units. Advances in Chemical Engineering and Science, 6(01):1, 2015.

JS Arora, AI Chahande, and JK Paeng. Multiplier methods for engineering optimization. In-
ternational journal for numerical methods in engineering, 32(7):1485–1525, 1991.

Jacques F Benders. Partitioning procedures for solving mixed-variables programming prob-

lems. Numerische mathematik, 4(1):238–252, 1962.

Timo Berthold, Andrea Lodi, and Domenico Salvagnin. Ten years of feasi-
bility pump and counting. Preprint. http://cerc-datascience. polymtl. ca/wp-
content/uploads/2017/08/Technical-Report_DS4DM-2017-009. pdf, 2017.

N. L. Boland and A. C. Eberhard. On the augmented lagrangian dual for integer programming.
Mathematical Programming, 150(2):491–509, feb 2014.

Pierre Bonami, Gérard Cornuéjols, Andrea Lodi, and François Margot. A feasibility pump for
mixed integer nonlinear programs. Mathematical Programming, 119(2):331–352, 2009.

Piya Chootinan and Anthony Chen. Constraint handling in genetic algorithms using a gradient-
based repair method. Computers & operations research, 33(8):2263–2281, 2006.

Claudia D’Ambrosio, Antonio Frangioni, Leo Liberti, and Andrea Lodi. A storm of feasibility
pumps for nonconvex minlp. Mathematical programming, 136(2):375–402, 2012.

J De Wit. Evaluation of mixed integer nonlinear programming routines. Technical report, Tech-
nical report, Technische Universiteit Eindhoven, The Netherlands, 2005. Internal Traineeship
report DCT, 2005.
Kalyanmoy Deb and Soumil Srivastava. A genetic algorithm based augmented lagrangian
method for constrained optimization. Computational optimization and Applications, 53(3):
869–902, 2012.

Kalyanmoy Deb and Santosh Tiwari. Omni-optimizer: A generic evolutionary algorithm for
single and multi-objective optimization. European Journal of Operational Research, 185(3):
1062–1087, 2008.

P Dhavachelvan and GV Uma. Multi-agent-based integrated framework for intra-class testing

of object-oriented software. Applied Soft Computing, 5(2):205–222, 2005.

Pedro A Diaz-Gomez and Dean F Hougen. Initial population for genetic algorithms: A metric
approach. In GEM, pages 43–49, 2007.

Mohammad Javad Feizollahi, Shabbir Ahmed, and Andy Sun. Exact augmented lagrangian du-
ality for mixed integer linear programming. Mathematical Programming, 161(1-2):365–387,
2017. URL https://link.springer.com/article/10.1007/s10107-016-1012-8.

Matteo Fischetti, Fred Glover, and Andrea Lodi. The feasibility pump. Mathematical Pro-
gramming, 104(1):91–104, 2005.

Stephen M Frank and Steffen Rebennack. Optimal design of mixed ac–dc distribution sys-
tems for commercial buildings: A nonconvex generalized benders decomposition approach.
European Journal of Operational Research, 242(3):710–729, 2015.

Pierre Genevois. Influence of population seeding technique on the behaviour of a function

optimizer real-coded genetic algorithm (unpublished). Unpublished.

Arthur M Geoffrion. Generalized benders decomposition. Journal of optimization theory and

applications, 10(4):237–260, 1972.

David E Goldberg and John H Holland. Genetic algorithms and machine learning. Machine
learning, 3(2):95–99, 1988.

David E Goldberg, Kalyanmoy Deb, and James H Clark. Genetic algorithms, noise, and the
sizing of populations. Urbana, 51:61801, 1991.

David Mautner Himmelblau. Applied nonlinear programming. McGraw-Hill Companies,

1972.

W Hock and K Schittkowski. A collection of test problems for constrained global optimization
algorithms. Lecture Notes in Economics and Mathematical Systems, 187, 1981.

BK Kannan and Steven N Kramer. An augmented lagrange multiplier based method for mixed
integer discrete continuous optimization and its applications to mechanical design. Journal
of mechanical design, 116(2):405–411, 1994.

K Katayama, H Sakamoto, and H Narihisa. The efficiency of hybrid mutation genetic algorithm
for the travelling salesman problem. Mathematical and Computer Modelling, 31(10-12):
197–203, 2000.

30
AD Kone, B Nogarede, and M Lajoie Mazenc. Le dimensionnement des actionneurs élec-
triques: un problème de programmation non linéaire. Journal de Physique III, 3(2):285–301,
1993.

Mattias Liefvendahl and Rafał Stocki. A study on algorithms for optimization of latin hyper-
cubes. Journal of statistical planning and inference, 136(9):3231–3247, 2006.

Heikki Maaranen, Kaisa Miettinen, and Antti Penttinen. On initial populations of a genetic
algorithm for continuous optimization problems. Journal of Global Optimization, 37(3):
405, 2007.

Ben Meadows, Patricia Riddle, Cameron Skinner, and Michael M Barley. Evaluating the seed-
ing genetic algorithm. In Australasian Joint Conference on Artificial Intelligence, pages
221–227. Springer, 2013.

P Victer Paul, A Ramalingam, R Baskaran, P Dhavachelvan, K Vivekanandan, R Subramanian,

and VSK Venkatachalapathy. Performance analyses on population seeding techniques for
genetic algorithms. International Journal of Engineering and Technology (IJET), 5(3):2993–
3000, 2013.

Ray Pörn and Tapio Westerlund. A cutting plane method for minimizing pseudo-convex func-
tions in the mixed integer case. Computers & Chemical Engineering, 24(12):2655–2665,
2000.

Simone Puzzi and Alberto Carpinteri. A double-multiplicative dynamic penalty approach for
constrained evolutionary optimization. Structural and Multidisciplinary Optimization, 35
(5):431–445, 2008.

Singiresu S Rao. Engineering optimization: theory and practice. John Wiley & Sons, 2009.

Shubhra Sankar Ray, Sanghamitra Bandyopadhyay, and Sankar K Pal. Genetic operators for
combinatorial optimization in tsp and microarray gene ordering. Applied intelligence, 26(3):
183–195, 2007.

E Sandgren. Nonlinear integer and discrete programming in mechanical design optimization.

Journal of Mechanical Design, 112(2):223–229, 1990.

Marc Schoenauer and Zbigniew Michalewicz. Evolutionary computation at the edge of feasi-
bility. In International Conference on Parallel Problem Solving from Nature, pages 245–254.
Springer, 1996.

Klaus Truemper. Simple seeding of evolutionary algorithms for hard multiobjective minimiza-
tion problems, 2015.

Eric W Weisstein. Pseudoconvex function. 2014.

Tapio Westerlund, Hans Skrifvars, Iiro Harjunkoski, and Ray Pörn. An extended cutting plane
method for a class of non-convex minlp problems. Computers & Chemical Engineering, 22
(3):357–365, 1998.
Stephen Wright and Jorge Nocedal. Numerical optimization. Springer Science, 35(67-68):7,
1999.

Wen-Hong Wu and Chyi-Yeu Lin. The second generation of self-organizing adaptive penalty
strategy for constrained genetic search. Advances in Engineering Software, 35(12):815–825,
2004.

Rong Yang. Solving large travelling salesman problems with small populations. 1997.

Olga Yugay, Insoo Kim, Beomjune Kim, and Franz IS Ko. Hybrid genetic algorithm for solving
traveling salesman problem with sorted population. In Convergence and Hybrid Information
Technology, 2008. ICCIT’08. Third International Conference on, volume 2, pages 1024–
1028. IEEE, 2008.

32
4.3 Appendix
The problems under investigation are as follows:
Problem 1:
The model of the permanent magnet motor (MAPSE) Kone et al. (1993)
D
minimize Vu = π (D + E − e − λa ) (2C + E + e + λa )
λ (4.1)
3
with 0 < Vu [m ] < inf

Nonlinear inequality constraints:

0.001 ≤ k f ≤ 0.5[−] (4.2)

0.1 ≤ Be ≤ 1[T ] (4.3)

1 ≤ p ≤ 10[−] (4.4)
0.001 ≤ C ≤ 0.05[m] (4.5)
Nonlinear equality constraints:
Γem = 10[Nm] (4.6)
Ech = 1011 [A/m] (4.7)
int(p) − p = 0[−]
(4.8)
where p is an integer
Input variables and their bounds:

0.001 ≤ E ≤ 0.05[m] (4.9)

0.11 ≤ D ≤ 0.5[m] (4.10)

0.001 ≤ e ≤ 0.005[m] (4.11)
0.001 ≤ λa ≤ 0.05[−] (4.12)
0.004 ≤ L ≤ 0.5[m] (4.13)
0.8 ≤ β ≤ 1[−] (4.14)
105 ≤ JCu ≤ 107 [A/m2 ] (4.15)
Nonlinear inequality constraints

0.001 ≤ k f ≤ 0.5[−] (4.16)

0.1 ≤ Be ≤ 1[T ] (4.17)

1 ≤ p ≤ 10[−] (4.18)
0.001 ≤ C ≤ 0.05[m] (4.19)
Nonlinear equality constraints (p is an integer)

Γem = 10[Nm] (4.20)

 Ech = 1011 [A/m] (4.21)
int(p) − p = 0[−] (4.22)
with q
π
Γem = (1 − k f ) kr β Ech ED2 (D + E)Be (4.23)
2λ
2
Ech = kr EJcu (4.24)
e+E
k f = 1.5pβ (4.25)
D
2λa Ma
Be = h i (4.26)
D+2E
DLog D−2(λ a +e)

πβ Be
C= D (4.27)
4pB f er
πD
p= (4.28)
∆p
D
λ= (4.29)
L
and constants:
Kr = 0.7, Ma = 0.9T , ∆ p = 0.1m and B f er = 1.5T

Problem 2:
The windfac problem is as stated below. Source: http://www.minlplib.org/windfac.html. The
problem has 11 continuous variables and 3 integer variables.

minimize X(13)2 + X(14)2 (4.30)

subject to:
−12X(1) + X(2) = 0 (4.31)
−12.566370616
+ X(3) = 0 (4.32)
X(2)
−0.25X(2) + X(4) = 0 (4.33)
−X(4) + X(5) = −1 (4.34)
sin (0.5X(3)) X(1)X(6) − sin (0.5X(1)X(3)) = 0 (4.35)
 
1.570796327X(5)
−sin + X(9) = 0 (4.36)
X(4)
−X(9)X(6) + X(15) = 0 (4.37)
sin (1.5X(3)) X(1)X(7) − sin (1.5X(1)X(3)) = 0 (4.38)
 
4.712388981X(5)
−sin + X(10) = 0 (4.39)
X(4)

34
 −X(10)X(7) + X(13) = 0 (4.40)

sin (2.5X(3)) X(1)X(8) − sin(2.5X(1)X(3)) = 0 (4.41)

 
7.853981635X(5)
−sin + X(11) = 0 (4.42)
X(4)

−X(11)X(8) + X(14) = 0 (4.43)

Lower bounds:

X(1) = 1
X(2) = 1
(4.44)
X(5) = 1
X(15) = 0.8

Upper bounds:

X(1) = 10
X(2) = 100 (4.45)
X(5) = 100

where X(1), X(2), X(5) are integers.

Problem 3: The problem nvs20 is as stated below, source: http://www.minlplib.org/nvs20.html.

The problem has 11 continuous variables and 5 integer variables:

min
46 + 6X(4) + X(4)X(15) + X(4)X(15)2 + X(4)X(11) + X(4)X(11)2 + X(4)X(7) + X(4)X(7)2
+ X(4)X(1) + X(4)X(1)2 + 7X(4)2 + X(4)2 X(15) + X(4)2 X(15)2 + X(4)2 X(11) + X(4)2 X(11)2
+ X(4)2 X(7) + X(4)2 X(7)2 + X(4)2 X(1) + X(4)2 X(1)2 + 2X(4).3 + X(4).4 + 5X(13) + X(13)X(11)
+ X(13)X(11)2 + X(13)X(14) + X(13)X(14).2 + X(13)X(7) + X(13)X(7)2 + 6X(13)2 + X(13).2 X(11)
+ X(13)2 X(11)2 + X(13)2 X(14) + X(13)2 X(14)2 + X(13)2 X(7) + X(13)2 X(7)2 + 2X(13)3
+ X(13)4 + 5X(12) + X(12)X(5) + X(12)X(5)2 + X(12)X(14) + X(12)X(14)2 + X(12)X(9)
+ X(12)X(9)2 + 6X(12)2 + X(12)2 X(5) + X(12)2 X(5)2 + X(12)2 X(14) + X(12)2 X(14)2 + X(12)2 X(9)
+ X(12)2 X(9)2 + 2X(12)3 + X(12)4 + 5X(6) + X(6)X(5) + X(6)X(5)2 + X(6)X(15) + X(6)X(15)2
+ X(6)X(8) + X(6)X(8)2 + 6X(6)2 + X(6)2 X(5) + X(6)2 X(5)2 + X(6)2 X(15) + X(6)2 X(15)2
+ X(6)2 X(8) + X(6)2 X(8)2 + 2X(6)3 + X(6)4 + 6X(5) + X(5)X(10) + X(5)X(10)2 + X(5)X(16)
+ X(5)X(16)2 + 7X(5)2 + X(5)2 X(10) + X(5)2 X(10)2 + X(5)2 X(16) + X(5)2 X(16)2 + 2X(5)3
+ X(5)4 + 5X(15) + X(15)X(8) + X(15)X(8)2 + 6X(15)2 + X(15)2 X(8) + X(15)2 X(8)2 + 2X(15)3
+ X(15)4 + 5X(11) + X(11)X(7) + X(11)X(7)2 + 6X(11)2 + X(11)2 X(7) + X(11)2 X(7)2 + 2X(11)3
+ X(11)4 + 6X(14) + X(14)X(10) + X(14)X(10)2 + X(14)X(3) + X(14)X(3)2 + 7X(14)2
+ X(14)2 X(10) + X(14)2 X(10)2 + X(14)2 X(3) + X(14)2 X(3)2 + 2X(14)3 + X(14)4 + 5X(9)
+ X(9)X(3) + X(9)X(3)2 + X(9)X(16) + X(9)X(16)2 + 6X(9)2 + X(9)2 X(3) + X(9)2 X(3)2
+ X(9)2 X(16) + X(9)2 X(16)2 + 2X(9)3 + X(9)4 + 7X(10) + X(10)X(3) + X(10)X(3)2 + X(10)X(2)
+ X(10)X(2)2 + X(10)X(8) + X(10)X(8)2 + 8X(10)2 + X(10)2 X(3) + X(10)2 X(3)2 + X(10)2 X(2)
+ X(10)2 X(2)2 + X(10)2 X(8) + X(10)2 X(8)2 + 2X(10)3 + X(10)4 + 7X(3) + X(3)X(2) + X(3)X(2)2
+ X(3)X(7) + X(3)X(7)2 + 8X(3)2 + X(3)2 X(2) + X(3)2 X(2)2 + X(3)2 X(7) + X(3)2 X(7)2 + 2X(3)3
+ X(3)4 + 5X(2) + X(2)X(7) + X(2)X(7)2 + 6X(2)2 + X(2)2 X(7) + X(2)2 X(7)2 + 2X(2)3 + X(2)4
+ 5X(16) + X(16)X(1) + X(16)X(1)2 + 6X(16)2 + X(16)2 X(1) + X(16)2 X(1)2 + 2X(16)3 + X(16)4
+ 6X(8) + X(8)X(1) + X(8)X(1)2 + 7X(8)2 + X(8)2 X(1) + X(8)2 X(1)2 + 2X(8)3 + X(8)4 + 8X(7)
+ X(7)X(1) + X(7)X(1)2 + 9X(7)2 + X(7)2 X(1) + X(7)2 X(1)2 + 2X(7)3 + X(7)4 + 6X(1) + 7X(1)2
+ 2X(1)3 + X(1)4
(4.46)
subject to:

0.22X(1)+0.2X(2)+0.19X(3)+0.25X(4)+0.15X(5)+0.11X(6)+0.12X(7)+0.13X(8)+X(9) ≥ 2.5
(4.47)
−1.46X(1) − 1.3X(3) + 1.82X(4) − 1.15X(5) + 0.8X(7) + X(10) ≥ 1.1 (4.48)
1.29X(1) − 0.89X(2) − 1.16X(5) − 0.96X(6) − 0.49X(8) + X(11) ≥ −3.1 (4.49)
−1.1X(1)−1.06X(2)+0.95X(3)−0.54X(4)−1.78X(6)−0.41X(7)+X(12) ≥ −3.5 (4.50)
−1.43X(4) + 1.51X(5) + 0.59X(6) − 0.33X(7) − 0.43X(8) + X(13) ≥ 1.3 (4.51)

36
 −1.72X(2) − 0.33X(3) + 1.62X(5) + 1.24X(6) + 0.21X(7) − 0.26X(8) + X(14) ≥ 2.1 (4.52)
1.12X(1) + 0.31X(4) + 1.12X(7) − 0.36X(9) + X(15) ≥ 2.3 (4.53)
0.45X(2) + 0.26X(3) − 1.1X(4) + 0.58X(5) − 1.03X(7) + 0.1X(8) + X(16) ≥ −1.5 (4.54)
Bounds:
X(1) ≤ 200
X(2) ≤ 200
X(3) ≤ 200
X(4) ≤ 200
X(5) ≤ 200
X(6) ≤ 200
X(7) ≤ 200
X(8) ≤ 200
(4.55)
X(9) ≤ 200
X(10) ≤ 200
X(11) ≤ 200
X(12) ≤ 200
X(13) ≤ 200
X(14) ≤ 200
X(15) ≤ 200
X(16) ≤ 200
Where X(1), X(2), X(3), X(4), X(5) are integers.

Problem 4:
The problem nvs21 is as stated below, source: http://www.minlplib.org/nvs21.html. The prob-
lem has 1 continuous variables and 2 integer variables:
min
(4.56)
− 0.00201X(1)4 X(2)X(3)2
Subject to:
X(1)2 X(2) ≥ −675
(4.57)
− 0.1X(1)2 X(3)2 ≥ −0.419
Bounds:
X(1).up = 200; X(1).lo = 1
X(2).up = 200; X(2).lo = 1 (4.58)
X(3).up = 0.2; X(3).lo = 0.1
Problem 5:
The problem st_e38 is as stated below, source: http://www.minlplib.org/ st_e38.html. The
problem has 2 continuous variables and 2 integer variables:
min
0.0389X(1)X(3)X(4) + 0.1111312X(3)2 X(2) + 0.012348046875X(1)2 X(4) + 0.0775X(1)X(3)
(4.59)
Subject to:
− 0.0625X(1) + 0.0193X(3) ≤ 0
− 0.0625X(2) + 0.00954X(3) ≤ 0 (4.60)
2 3
3.1415927X(3) X(4) + 1.33333333333333X(3) ≥ 1296000
Bounds:
X(1).lo = 18; X(1).up = 100
X(2).lo = 10; X(2).up = 100
(4.61)
X(3).lo = 40; X(3).up = 80
X(4).lo = 20; X(4).up = 60

38