Lecture note series for

CSC 310: Algorithm and Complexity Analysis

UNIT 3: ALGORITHM EFFICIENCY

Dr Fatimah Adamu-Fika

Department of Cyber Security

Faculty of Computing
Air Force Institute of Technology
Kaduna, Nigeria

The version for 2021/2022 Academic Session

© Since 2020
 CONTENT

Overview................................................................................................................................... 1

Measuring Algorithm Efficiency ............................................................................................. 1

Measuring Input Size ............................................................................................................ 1

Estimating Running Times .................................................................................................... 2

Order of Growth ................................................................................................................... 3

Theoretical Analysis of Algorithms ......................................................................................... 6

Counting (Primitive) Operations Approach ..................................................................... 6

Identifying the Basic Operation Approach ...................................................................... 8

Worst-Case, Best-Case and Average-Case Efficiencies..................................................10

The Worst-Case Efficiency ................................................................................................10

The Best-Case Efficiency ...................................................................................................11

The Average-Case Efficiency...........................................................................................11

Asymptotic Notations............................................................................................................11

Big-Oh (or Oh-) Notation ( -Notation) ...........................................................................12

Formal Definition of Big-Oh Notation...........................................................................12

Big-Omega (or Omega-) Notation (𝛀-Notation) ..........................................................13

Formal Definition of Big-Omega Notation ..................................................................13

Big-Theta (or Theta-) Notation (𝚯-Notation) ..................................................................13

Formal Definition of Big-Theta Notation ......................................................................14

Which Notation to Use ......................................................................................................15

Comparison of Big-O and Big-Θ .................................................................................15

Some Properties of Asymptotic Order of Growth ..........................................................16

Summary of Asymptotic Notation Function Simplification Rules ............................... 1

Basic Efficiency Classes .....................................................................................................18

Understanding Code Complexity ...................................................................................19

Simple Statements..........................................................................................................20

If-Then-Else .......................................................................................................................20

Simple Loops ...................................................................................................................20

CSC 310 Unit 3 Dr F Adamu-Fika

ii Algorithm and Algorithm Analysis: Cyber Security Dept. AFIT
Complexity Analysis Algorithm Efficiency © Since 2020
 CSC 310: ALGORITHM AND COMPLEXITY ANALYSIS
Unit 1: Review and Introduction of Fundamentals
Nested Loops ..................................................................................................................20

Key Points to Remember ......................................................................................................21

Unit Goals................................................................................................................................22

Math You Need to Review ...................................................................................................22

Exercises ..................................................................................................................................23

Bibliography and Resources ................................................................................................24

Textbooks ............................................................................................................................24

Websites ..............................................................................................................................25
OVERVIEW

We already mentioned that an algorithm can be efficient in two ways: how fast a
given algorithm run determines its time efficiency, also called time complexity; the
amount of memory units the algorithm requires in addition to space needed by its
input and output indicates its space efficiency, also called space complexity. In the
past, both time and space were very expensive. With current technological
advancements, computer speed and memory size have been greatly improved. Now
the amount, of extra space required by an algorithm is not of much concern.
However, there exists still, a difference between the fast main memory, the slower
secondary memory, and the cache. The time issue has not diminished quite to the
same extent as the space issue. We shall primarily focus on time efficiency, but the
analytical framework we shall be exploring also applies to analysing space efficiency
as well. Also, when we consider space efficiency, we shall be focusing on main
memory usage rather than secondary memory. Therefore, until otherwise stated,
when we talk of efficiency, we are talking about time efficiency. And when we talk
of space efficiency, we are talking about how much main memory is needed by the
algorithm. Algorithms that have non-appreciable (not significant) space complexity
are said to be in place.

So now, the big question is “how do we measure efficiency”?

MEASURING ALGORITHM EFFICIENCY

Many algorithms transform input objects into output objects. For example, a correct
sorting algorithm that reorders a given list in non-decreasing order when applied to
the problem instance [5, 3, 4, 2, 1] will output [1, 2, 3, 4, 5].

Sorting
5 3 4 2 1 1 2 3 4 5
algorithm
input object output object

From the example above, we can fathom that the running time of the sorting
algorithm will grow as the number of elements to be sorted increases. This
characteristic is typical for most algorithms.

In some situations, even on the same size input, the same algorithm can have very
different running times. In these situations, the nature (or details) of the input affects
the process of solving the problem. For example, a searching algorithm that scans a
list from the first element to the last will run faster if the value being searched for is the
first element in the list. Whereas, the same algorithm, when applied to another
instance of the same size but with the value being searched for located at the last
location in the list, will take longer to run.

From these examples, it is apparent that to measure efficiency we need to be able

to: 1) measure the input size for the algorithm in question, and 2) measure the running
time of the algorithm relative to the input size.

Measuring Input Size

As algorithms generally run longer on larger inputs. For example, it will take longer to
sort a larger student list, compute the factorial of a larger number, multiply larger

Dr F Adamu-Fika Unit 3 CSC 310

Cyber Security Dept. AFIT Algorithm Analysis: Algorithm and 1
© Since 2020 Algorithm Efficiency Complexity Analysis
 matrices, and so on. So, it is reasonable to evaluate an algorithm’s efficiency as a
function of some parameter n indicating the algorithm’s input size (number of
elements in the input). However, some algorithms require more than one input to
indicate the size of their input. For instance, if the input to an algorithm is a graph, the
input size can be described by the number of vertices and edges in the graph.

In most cases, it is straightforward to select n. For example, for problems of sorting,

searching and most problems involving lists, n will be the size of the list. For problems
of evaluating a polynomial 𝑝(𝑥) = 𝑎𝑛 𝑥 𝑛 + . . . + 𝑎0 of degree n, the input size will be the
degree of the polynomial or the number of its coefficients, which is larger by 1 than its
degree.

However, there are cases where the choice of a parameter indicating an input size
does not matter. For example, when computing the product of two square (m × m)
matrices, the size of the problem is measured by the order m of the matrix, the order
of a matrix refers to its dimension, i.e., the number of rows and columns it has.

The selection of an appropriate input size metric can be influenced by the operations
of the given algorithm. For example, how should we decide an input’s size metric for
a spell-checking algorithm? If the algorithm examines individual characters of its input,
we should use the number of characters as the size metric; if it works by processing
words, we should take the number of words as the size metric.

There are cases, where the input is just one or two numbers. For example, when
checking the primality of a number (i.e., to check whether a number is prime) or when
multiplying two numbers. Here, the magnitude of the number(s) determines the input
size. In such scenarios, it is preferable to measure size by the number of bits (b) in the
number (n)’s binary representation:

𝑏 = [𝑙𝑜𝑔2 𝑛] + 1
his metric usually gives a better idea about the efficiency of the algorithm in question.
We shall indicate which input size measure is being used with each problem we study.

Estimating Running Times

Our initial thought would be to use empirical analysis, translate the algorithm into a
program and use some standard unit of time as a measurement. However, this is not
the best way for many reasons. To do so necessitates the implementation of the
algorithm, which may be difficult to do. The running time will be affected by factors
such as the speed of the computer, the programming language, the translation of
the language into machine code, and so on. The results of the analysis may not be
indicative of the running time on other inputs not included in the experiment. To
compare two or more algorithms, the same hardware and software environments
must be used. However, there are still times when a programmer finds it helpful to
measure the run time of an implemented algorithm. For example, when a particular
operation takes a surprisingly long amount of time, or maybe when there's some
aspect of the programming language that's slowing an algorithm down.

A better way, to measure efficiency is to use theoretical analysis. Using the

pseudocode of the algorithm we can characterise the running time as a function of
the input size. This allows us to consider all possible inputs. Also, it allows us to evaluate
the speed of the algorithm independent of the hardware and software environment.
And the most obvious benefit, we can analyse algorithms without implementing them
or running experiments on them.

CSC 310 Unit 3 Dr F Adamu-Fika

2 Algorithm and Algorithm Analysis: Cyber Security Dept. AFIT
Complexity Analysis Algorithm Efficiency © Since 2020
To analyse an algorithm theoretically, we can count the number of times one or more
operations or steps are executed. We shall find out how to compute such a count for
both non-recursive (or/and iterative) and recursive algorithms. We shall be learning
two methods of computing counts for non-recursive algorithms: 1. counts the number
of times each operation in an algorithm is executed or 2. count the number of times
the most time-consuming operation or step is executed.

Assuming that Cop is the execution time of an algorithm’s operation(s) on a particular

computer, and Cn is the number of times this(these) operation(s) is(are) executed. We
can then estimate the running time Tn of a program implementing the algorithm on
that computer by:

𝑇𝑛 ≈ 𝑐𝑜𝑝 𝐶𝑛

This formula gives a reasonable estimate can give a reasonable estimate of the
algorithm’s running time. It also makes it possible to answer such questions as how
much faster an algorithm would run on a faster computer as compared to when ran
on a slower machine. For example, an algorithm will obviously run 10 times faster on a
computer that is 10 times faster than another computer. Estimating the algorithm’s
running time could answer a more important of how much longer an algorithm will
1
run if its input size doubles. For example, suppose 𝐶𝑛 = 2 𝑛2 , let’s see how we can
answer how much longer it will take when the algorithm’s input size is doubled without
knowing the value of Cop.
1
𝑇2𝑛 𝐶𝑜𝑝 𝐶2𝑛 (2𝑛)2 4𝑛2
≈ ≈ 2 ≈ 2 ≈4
𝑇𝑛 𝐶𝑜𝑝 𝐶𝑛 1 𝑛
(𝑛)2
2
1
As you can notice Cop is neatly cancelled out in the ratio. Also note that 2, the
multiplicative factor, Cop, in the formula for the count was also cancelled out. This is
the reason why when we do efficiency analysis, we ignore multiplicative factors and
focus on the count’s order of growth to within a constant multiple for large-size inputs.

Order of Growth
Order of growth (or rate of growth) is a set of functions whose asymptotic growth
behaviour is considered equivalent. For example, 6n + 300, 100n, n + 1 and 2n all
belong to the same order of growth. They are linear functions. Functions that have n2
as a leading term are called quadratic. We are more interested in the order of growth
of the number of times the basic operation is executed on n (the input size of an
algorithm). Because, for smaller inputs, it is difficult to distinguish efficient algorithms
from inefficient ones. And for this reason, we are interested in the order of growth for
large input sizes.

For example, when we analyse two algorithms and expressed their running times in
terms of the size of the input, n: we found Algorithm A executed 100n operations to
solve a particular problem and Algorithm B executed n2 operations to solve the same
problem. The following table shows the running time for the two algorithms for different
input sizes.

Input size Running Time (Count)

Dr F Adamu-Fika Unit 3 CSC 310

Cyber Security Dept. AFIT Algorithm Analysis: Algorithm and 3
© Since 2020 Algorithm Efficiency Complexity Analysis
 Algorithm A Algorithm B

n 100n n2
1 100 1
10 1,000 100
100 10,000 10,000
10,000 1,000,000 100,000,000
100,000 10,000,000 10,000,000,000
1,000,000 100,000,000 1,000,000,000,000
For smaller n, Algorithm A takes longer to execute than Algorithm B: when n = 1, we
can see that Algorithm A takes 100 times longer than Algorithm A, and 10 times longer
when n = 10. However, when n = 100 they have the same execution time. For larger
n, Algorithm A runs much faster, the larger the value of n the better Algorithm A is
over Algorithm B.

We may guess, from the example above, that functions that have larger leading terms
will always grow faster than those with smaller leading terms. The leading term, in a
function, is the term with the highest exponent. For Algorithm A, the leading term has
a large coefficient, 100, which is why Algorithm B does better for smaller n. However,
regardless of the coefficients, there will always be some value of n where an < bn2.
This also applies to the non-leading terms. Even if the run time of Algorithm A were
100n + 1010, it would still be better than Algorithm B for a large enough n.

So, we expect an algorithm with a smaller leading term to be a better algorithm for
large problems. However, for smaller problems, there may be a crossover point where
another algorithm is better. The crossover point is the problem size where two
algorithms require the same run time or space. The location of the crossover point
depends on the details of the algorithms, the inputs, and the hardware. This is usually
ignored for algorithmic analysis purposes.

Similarly, when two algorithms have the same leading terms, it is difficult to say which
one is better. Thus, for algorithm analysis, functions with the same leading term are
considered equivalent and belong to the same order of growth, even if they have
different coefficients.

The following table shows some orders of growth (concerning the input size) that are
common for algorithm analysis.

Input Order of Growth

size, n n
log 𝑛 𝑛 log 𝑛 n2 n3 2n n!
2 1 2 2 4 8 4 2
4 2 4 8 16 64 16 24
8 3 8 24 64 512 256 40,320
16 4 16 64 256 4,096 65,536 2.09E+13
32 5 32 160 1,024 32,768 4.29E+09 2.63E+35
64 6 64 384 4,096 262,144 1.84E+19 1.27E+89

CSC 310 Unit 3 Dr F Adamu-Fika

4 Algorithm and Algorithm Analysis: Cyber Security Dept. AFIT
Complexity Analysis Algorithm Efficiency © Since 2020
 128 7 128 896 16,384 2,097,152 3.40E+38 3.86E+215
256 8 256 2,048 6.55E+04 1.68E+07 1.16E+77 too large
512 9 512 4,608 2.62E+05 1.34E+08 1.34E+154 too large
The table shows the significance of these orders of growth in algorithm analysis. The
function growing the slowest is the logarithmic function,
log 𝑛. It grows very slowly that on inputs of realistic sizes the execution time is expected
to be almost instantaneous. In the above table, the log base is 2, and the base of the
logarithm does not matter; changing bases is the equivalent of multiplying by a
constant, using the formula:

log 𝑎 𝑛 = log 𝑎 𝑏 log 𝑏 𝑛

Where a, is the new base and b the old. For example, to change the log function in
our example from base 2 to base 10, using the formula, we get:

log10 𝑛 = log10 2 log 2 𝑛

= 0.301 log 2 𝑛

∵ log 10 2 = 0.301

Therefore, we omit a logarithm’s base and write simply log 𝑛 in situations where we are
only interested in a function’s order of growth within a multiplicative constant. Similarly,
all exponential functions belong to the same order of growth irrespective of the base
of the exponent. Exponential functions are on the opposite end of the spectrum from
logarithmic functions, they grow very quickly that their values become extremely
large even for very small values of n. The factorial function is at this end of the
spectrum as well. Although, there is a huge difference between the orders of growth
of 2n and n!, n! is often also referred to as having exponential growth. We can see that
exponential algorithms are only useful for solving small problems.

Another observation we can make from the table above is the rate of growth of the
various functions when the input size doubles. Let’s summarise it in the following table:

When n doubles, i.e., becomes

Function Name 2n, function increase in value by Because
log 𝑛 logarithmic 1 log 2 2𝑛 = log 2 2 + log 2 𝑛
=1
+ log 2 𝑛
𝑛 linear twofold ( times) 2𝑛 = 2𝑛
𝑛 log 𝑛 Linearithmic slightly more than two folds 2𝑛 log 2 2𝑛 = 2𝑛 (1
(or log-linear) + log 2 𝑛)
𝑛2 quadratic fourfold ( times) (2𝑛)2 = 4𝑛2
𝑛3 cubic eightfold ( times) (2𝑛)3 = 8𝑛3
2𝑛 exponential square of 2n 22𝑛 = (2𝑛 )2 [= 4𝑛 ]
𝑛! factorial even math cannot help us to neatly summarise this one!
Thus, we can see an algorithm’s order of growth can help us evaluate how much
faster it will run on a faster computer (e.g., on a computer that is twice as fast) and

Dr F Adamu-Fika Unit 3 CSC 310

Cyber Security Dept. AFIT Algorithm Analysis: Algorithm and 5
© Since 2020 Algorithm Efficiency Complexity Analysis
 how much longer it will take to solve a problem when the input size grows in folds or
exponentially (e.g., when the input size doubles or triples).

THEORETICAL ANALYSIS OF ALGORITHMS

As we have mentioned, the relative performance of algorithms might depend on the

characteristics of the hardware, this means a particular algorithm might run faster on
Machine A and another one on Machine B. To circumvent this issue, first, we need to
specify a machine model of the implementation technology that we shall use,
including a model of the resources for that technology and the cost of using them.
We shall assume a generic one-processor computational model called Random
Access Machine (RAM) model (not to be confused with Random Access Memory).
The RAM model consists of a CPU, a potentially unbounded bank of memory cells,
each of which can hold an arbitrary number of characters, and memory cells that
are numbered and accessing any cell in memory take a unit of time. This means we
can index individual elements of an input. Although our model assumes a limitless
memory space, we, however, assume a limit on the size of each word of data. This
means that words do not grow arbitrarily

0 1 2 3 … … …
In the RAM model, instructions are executed one after the other, and no two are
executed concurrently. These instructions are simple ones found in real computers:
arithmetic (such as add, subtract, multiply, divide, modulus etc), data movement
(such as load, save etc) and control (such as conditional and unconditional branch,
subroutine call and return). Each instruction takes a constant amount of time to
execute.

Now that we have specified a model, our next step is to analyse the number of steps
or operations an algorithm requires under the given model.

Counting (Primitive) Operations Approach

One way to do it is to count the number of times each (primitive) operation is
executed. The operations are simple computations performed by an algorithm. These
operations are identifiable in pseudocode, independent from the programming
languages and the exact definition is not important.

Examples of primitive operations:

• Assigning a value to a variable e.g., j = 2

• Evaluating an expression e.g., 2k2 + 1
• Indexing into an array e.g., A[i]
• Comparing two values e.g., j ≤ n – 1
• Calling a routine or a subroutine e.g., BubbleSort[A]
• Returning from a routine or subroutine e.g., return A

We shall learn how to count primitive operations using the algorithm. Consider the
problem of finding the value of the largest element in a list of n numbers. For simplicity,
we assume that the list is implemented as an array. The following is a pseudocode of
a standard algorithm for solving the problem.

Algorithm ArrayMax(A)

CSC 310 Unit 3 Dr F Adamu-Fika

6 Algorithm and Algorithm Analysis: Cyber Security Dept. AFIT
Complexity Analysis Algorithm Efficiency © Since 2020
 Input: An array A[0 … n – 1] of n number
Output: the maximum element of A
START
1. max = A[0]
2. for( i = 1; i < n; i++ ) // for i = 1 to n – 1 do
3. if( A[i] > max )
4. max = A[i]
5. return max
END
Now, let’s start counting the operations in ArrayMax and the number of times each
operation will execute in a single run of the algorithm.

1. Line 1, max = A[0], initialises the variable max with the value A[0], that is two
primitive operations, indexing an array and assigning a value to a variable. This
is executed only once at the beginning of the algorithm. Thus, line 1 contributes
2 units to the count.

2. Line 2, for( i = 1; i < n; i++ ), at the beginning of the loop the counter i is initialised
0. This corresponds to executing the primitive operation of assigning a value to
a variable. So, i = 1 contributes unit to the count.

Before entering the loop, the condition i < n is verified. This corresponds to
executing the primitive operation of comparing two values. Since the counter
i starts at 1 and gets incremented by 1 at the end of each iteration of the loop,
the comparison i < n is executed n times. Thus, i < n contributes n units to count.

The body of the loop is executed n – 1 times, for values 1, 2, …, n – 1 of the

counter i.

At the end of each iteration of the loop, the counter i gets incremented by 1,
this corresponds to executing two primitive operations, evaluating an
expression, i.e., performing an arithmetic operation, and assigning a value to
a variable. This executes the same number of times the body of the loop
executes, i.e., n – 1 times. Thus, contributing 2(n – 1) units to the count.

Line 2, therefore, contributes a total of 1 + n + 2(n – 1) i.e., 3n – 1, units to count.

3. Line 3, if( A[i] > max ), compares A[i] to max. This corresponds to executing two
primitive operations, indexing an array and comparing two values. This is within
the body of the loop, so it executes n – 1 times. Thus, contributing 2(n – 1) units
to the count.

4. Line 4, max = A[i], only executes when line 3 executes, if it executes it updates
the variable max with A[i]. This corresponds to executing two primitive
operations, indexing an array and assigning a value to a variable.

How much line 4 contributes, depends on how many times it gets executed. In
the best-case (case a), when the first element of the array, i.e., A[0], is the
largest number of the array, line 4 will never get executed. Thus, it will contribute
0 units to the count. In the worst-case (case b), when the array is sorted in
increasing order, this means line 3 will always be true as every ith element of the
array is larger than the element before it. In this case, line 4 will be executed in
each iteration of the loop, i.e., line 4 will be executed n – 1 times. Thus,
contributing 2(n – 1) units to the count.

Dr F Adamu-Fika Unit 3 CSC 310

Cyber Security Dept. AFIT Algorithm Analysis: Algorithm and 7
© Since 2020 Algorithm Efficiency Complexity Analysis
 5. Line 5, return max, returns the value of variable max This corresponds to
executing one primitive operation, returning from a subroutine. Thus, it
contributes 1 unit to the count.

The following table summarises the primitive operations (Cn) performed by the
algorithm ArrayMax and the number of times they execute:

Count
at least at most
1 max = A[0] 2 2
2 for( i = 1; i < n; i++ ) 3n – 1 3n – 1
3 if A[i] > max 2(n – 1) 2(n – 1)
4 max = A[i] 0 2(n – 1)
5 return max 1 1
Total count (Cn) 5n 7n – 2
Thus, in the best case, 𝐶𝑛 = 2 + (3𝑛 – 1) + (2(𝑛 – 1)) + 0 + 1 = 𝟓𝒏 ∈ 𝑶(𝒏), it occurs
when A[0] is the largest element in the array. And in the worst-case, 𝐶𝑛 = 2 +
(3𝑛 – 1) + (2(𝑛 – 1)) + (2(𝑛 – 1)) + 1 = 𝟕𝒏 ∈ 𝑶(𝒏), this occurs when A is sorted in
increasing order, so that variable max is reassigned at each iteration of the loop.

From the above example, we can summarise the counting primitive operation
approach as:

1. Decide on a parameter (or parameters) indicating an input’s size.

2. Identify and count the primitive operations in each line of the algorithm.
3. Check whether the number of times each statement is executed depends only
on the size of an input. If it also depends on some additional property, the worst-
case, average-case, and, if necessary, best-case efficiencies must be
investigated separately.
4. Set up a count for each statement expressing the number of operations it
contains and the number of times each is executed.
5. Sum up all the counts for each case you encounter in step 3 above.
6. Establish the order of growth of the count(s).

This approach can be excessively difficult, especially when our algorithms have many
nested loops and are usually unnecessary. The thing to do is to identify the most
important operation of the algorithm, called the basic (or fundamental) operation,
and then compute the number of times the basic operation is executed.

Identifying the Basic Operation Approach

The basic operation is the operation contributing the most to the total running time.
Identifying the basic operation of an algorithm is not difficult: it is usually the most time-
consuming operation in the algorithm’s innermost loop. For example, most sorting
algorithms work by comparing elements (keys) of a list is sorted with each other; for
such algorithms, the basic operation is a key comparison. As another example,
algorithms for mathematical problems typically involve some or all the four
arithmetical operations: addition, subtraction, multiplication, and division. Of the four
arithmetic operations, the most time-consuming operation is division, followed by
multiplication and then addition and subtraction, with the last two usually considered
together.

CSC 310 Unit 3 Dr F Adamu-Fika

8 Algorithm and Algorithm Analysis: Cyber Security Dept. AFIT
Complexity Analysis Algorithm Efficiency © Since 2020
Using our previous algorithm ArrayMax above, let’s learn how to determine the cost
of running the basic operation. The obvious measure of an input’s size ArrayMax is the
number of elements in the array, i.e., n. The operations that are going to be executed
most often are in the body of its for loop. There are two operations in the loop’s body:
the comparison operation, A[i] > max and the value assignment operation, max =
A[i]. The basic operation of the algorithm is the comparison, A[i] > max, because it is
executed on each iteration of the loop and the assignment, max = A[i], is not, its
execution depends on A[i] > max. The number of comparisons will be the same for all
arrays of size n; therefore, in terms of this metric, there is no need to distinguish
between the worst, average, and best cases here.

Now that we have identified line 3 as the basic operation, next, we will find
summations to express the number of times, Cn, it is executed as a function of the input
size, n. ArrayMax executes A[i] > max (line 3) one time in each iteration of the loop,
the iterations are repeated i times, within the bound 1 to n – 1, i.e., for i = 1, 2, …, n
– 1.

Given a summation, we often want to substitute it with an algebraic equation with

the same value as the summation. This algebraic equation is known as a closed-form
solution and the process of replacing the summation with its closed-form solution is
known as solving the summation.

Number of times executed

1 max = A[0] 1

2 for( i = 1; i < n; i++ ) n

3 if( A[i] > max ) 1

n-1
4 max = A[i] x

5 return max 1
Therefore, we get the following summation from the basic operation:
𝑛 –1

𝐶𝑛 = ∑ 1
𝑖= 1

This is an easy summation to solve because it is simply the expression 1 summed

(repeated) n – 1 times. Thus,
𝑛 –1

𝐶𝑛 = ∑ 1 = 𝒏 − 𝟏 ∈ 𝑶(𝒏)
𝑖=1

From the above, we can summarise identifying basic operation approach as:

1. Decide on a parameter (or parameters), n, indicating an input’s size.

2. Identify the algorithm’s basic operation. (As a rule, it is in the inner-most loop.)
3. Check whether the number of times the basic operation is executed depends
only on the size of an input. If it also depends on some additional property, the

Dr F Adamu-Fika Unit 3 CSC 310

Cyber Security Dept. AFIT Algorithm Analysis: Algorithm and 9
© Since 2020 Algorithm Efficiency Complexity Analysis
 worst-case, average-case, and, if necessary, best-case efficiencies must be
investigated separately.
4. Set up a sum expressing the number of times the algorithm’s basic operation is
executed. Usually, each loop will be expressed as a summation, starting from
the outer-most loop to the inner-most loop. The lower index of the summation
will correspond to the value of the looping counter at initialisation, and the
upper index of the summation is the value of the looping counter at the last
iteration of the loop. The argument of the inner-most summation corresponds
to the number of times the basic operation is executed in each iteration of the
inner-most loop.
5. Using standard formulas and rules of sum manipulation, either find a closed-
form formula for the expression in step 4 or, at the very least, establish its order
of growth.

So far, the frameworks we have considered are for analysing the time efficiency of
non-recursive algorithms. To analyse the time efficiency of recursive algorithms we
replace identifying basic operations in step 3 with modelling recursions as recursive
relations. We then expand these relations and rewrite them as summations.

WORST-CASE, BEST-CASE AND AVERAGE-CASE EFFICIENCIES

We have established that it is logical to measure an algorithm’s efficiency as a

function of a parameter indicating the size of the algorithm’s input. And as we have
seen when counting primitive operations in ArrayMax, an algorithm’s running time
may depend not only on the input size but on the nature of the input, which makes it
likely to run faster on some inputs than it does on others.

The Worst-Case Efficiency

The worst-case efficiency of an algorithm is its efficiency for the worst-case input of
size n, which is an input (or inputs) of size n for which the algorithm runs the longest
among all possible inputs of that size. Meaning, the maximum running time an
algorithm can take for any input of size n.

The worst-case running time of an algorithm gives us an upper bound on the running
time for any input, that is, the longest time an algorithm can take to execute an input
of size n. For example, we say ArrayMax executes Cn = 7n – 2 primitive operations in
the worst case, meaning that the maximum number of primitive operations executed
by the algorithm, taken over all inputs of size n, is 7n – 2. Or we can similarly say that
its basic operation executes Cn = n – 1 times in the worst case, meaning that the
maximum number of times the basic operation is executed by the algorithm, taken
over all inputs of size n, is n – 1.

To perform a worst-case analysis, we need to identify the size n that an algorithm takes
the longest to execute. This type of analysis is undoubtedly important. It can lead to
better algorithms because for any instance of input n, it guarantees that the running
time of the algorithm will take longer than its running time in the worst case. This means,
that making the standard metric for efficiency be having an algorithm be efficient in
the worst case translates to requiring it to be efficient on every input.

We shall focus mainly on the worst-case analysis of algorithms, but we shall sometimes
mention other cases and their associated complexities.

CSC 310 Unit 3 Dr F Adamu-Fika

10 Algorithm and Algorithm Analysis: Cyber Security Dept. AFIT
Complexity Analysis Algorithm Efficiency © Since 2020
The Best-Case Efficiency
The best-case efficiency of an algorithm is its efficiency for the best-case input of size
n, which is an input (or inputs) of size for which the algorithm runs the fastest among
all possible inputs of that size. Meaning the minimum running time an algorithm can
take for any input of size n.

The best-case running time of an algorithm gives us a lower bound on the running time
for any input, that is, the fastest running time for any input of size . Thus, we can
analyse best-case efficiency by identifying the kind of inputs that would yield the
smallest count, Cn. This does not necessarily imply the smallest input but rather the
input of size n that the algorithm will execute the fastest. For example, when we use
the counting primitive approach on ArrayMax, in the best case where the array is
sorted and the first element is the largest element it contains, the algorithm executes
Cn = 5n primitive operations meaning that the minimum number of primitive
operations executed by the algorithm, taken over all inputs of size n, is 5n. We can
say, that ArrayMax executes its basic operations Cn = n – 1 in the best case, meaning
that the minimum number of times the basic operation is executed by the algorithm,
taken over all inputs of size n, is n – 1. Consider another example, in a sequential
search where the search value is the first element in the array, the basic operation
would be executed exactly one time.

The best-case analysis is not as important as the worst-case analysis. However, it can
still be useful. If an algorithm has an inefficient best-case, we can immediately discard
it without further analysis.

The Average-Case Efficiency

From our discussion, we have established may run faster on certain input than it does
on others. In such cases, ideally, we may wish to express the running time of such an
algorithm as an average taken over all possible inputs. Both the best-case and worst-
case analyses do not provide the necessary information about an algorithm’s
behaviour on a typical or a random input. This is the information that average-case
analysis seeks to yield. Analysing an algorithm’s average-case efficiency requires us
to define a probability distribution (make assumptions about) on the set of inputs
possible inputs of size, which typically is not an easy task. We shall mostly quote known
results about the average-case efficiency of algorithms under discussion.

ASYMPTOTIC NOTATIONS

We have seen that when we focus on just leading terms, we can focus on the
important part of the algorithm's running time, its order of growth. When we drop the
less significant terms and the constant-coefficient we use asymptotic notation. The
asymptotic behaviour of a function f(n) refers to the growth of f(n) as n gets large.

Essentially, the asymptotic notation is a convenient way of describing how fast a

function is growing. So, with asymptotic notation, we express an algorithm’s run time
in terms of how quickly it grows relative to the input, as the input gets larger. This
means, instead of talking about the algorithm’s running time directly, using asymptotic
notation we can tell how fast the running time is growing as the algorithm’s input size
increases.

We shall explain three forms of asymptotic notation to help us compare and rank
orders of growth: O (big-O), 𝛀 (big-omega) and 𝚯 (big-theta).

Dr F Adamu-Fika Unit 3 CSC 310

Cyber Security Dept. AFIT Algorithm Analysis: Algorithm and 11
© Since 2020 Algorithm Efficiency Complexity Analysis
 Big-Oh (or Oh-) Notation ( -Notation)
The big-O notation is used to describe the asymptotic upper bound. It bounds the
growth of an algorithm’s running time from above for sufficiently large input sizes.
is the set of functions that grow slower than or at the same rate as 𝑔(𝑛). It
indicates that the running time of an algorithm, 𝑡(𝑛), grows at most this much, this
means could grow slower than 𝑔(𝑛). If 𝑡(𝑛) is in 𝑂(𝑔(𝑛)) (denoted as 𝑡(𝑛) ∈ 𝑂(𝑔(𝑛)),
then for a sufficiently large n, the running time is at most 𝑐 ⋅ 𝑔(𝑛) for some constant .
We say that the running time is “big-O of 𝑔(𝑛)” or just “oh of 𝑔(𝑛)”.

Formal Definition of Big-Oh Notation

Function t(n) belongs to the set O(g(n)) if there exists a positive constant c and non-
negative integer n0 such that:

𝑡(𝑛) ≤ 𝑐 ⋅ 𝑔(𝑛) for all 𝑛 ≥ 𝑛0

image source: Levitin [1]

As an example, let’s prove the running time from our counting primitive operations
example: 7𝑛 – 2 ∈ 𝑂(𝑛).

By big-O definition, we need to find c > 0 and n0 ≥ 0 such that 7n – 2 ≤ c for every
integer n ≥ n0.

We try c = 5, and then look for n0 ≥ 0 such that:

7n – 2 ≤ 5 for every integer n ≥ n0.

An obvious choice is n0 = 1. Thus:

7n – 2 ≤ 5 for all n ≥ 1.

Note the definition gives us a lot of freedom in choosing specific values for the
constant and . For example, by choosing c = 7 and 2, we can also reason that:

7n – 2 ≤ 7n for every integer n ≥ 2.

The big-O notation gives the worst-case complexity of an algorithm. When we use big-
O notation we are essentially saying that this is the slowest an algorithm can run for
large enough input of all sizes. For example, for an algorithm with an order of growth
in O(n), we can say that its running time increases at most proportionally to the input
size . This means the running time can never be more than n.

CSC 310 Unit 3 Dr F Adamu-Fika

12 Algorithm and Algorithm Analysis: Cyber Security Dept. AFIT
Complexity Analysis Algorithm Efficiency © Since 2020
Big-Omega (or Omega-) Notation (𝛀-Notation)
The big-omega notation is used to describe the asymptotic lower bound. It bounds
the growth of an algorithm’s running time from below for sufficiently large input sizes.
Ω(𝑔(𝑛)) is the set of functions that grow faster than or at the same rate as g(n) It
indicates that the running time of an algorithm, t(n), grows at least this much, this
means could grow faster than g(n). If t(n) is in Ω(𝑔(𝑛)) (denoted as 𝑡(𝑛) ∈ Ω(𝑔(𝑛))),
then for a sufficiently large n, the running time is at least 𝑐 ⋅ 𝑔(𝑛) for some constant c.
We say that the running time is “big omega of g(n)” or just “Omega of g(n)”.

For example, for an algorithm with an order of growth in Ω(𝑛), we can say that its
running time increases at least proportionally to the size of the input.

Formal Definition of Big-Omega Notation

Function t(n) belongs to the set Ω(𝑔(𝑛)) if there exists a positive constant c and non-
negative integer n0 such that:

𝑡(𝑛) ≥ 𝑐 ⋅ 𝑔(𝑛) for all 𝑛 ≥ 𝑛0

image source: Levitin [1]

For example, let’s prove 𝑛3 ∈ Ω(𝑛2 ). We need to find c > 0 and n0 ≥ 0 such that 𝑛3 ≥
𝑐 ⋅ 𝑛2 for every integer 𝑛 ≥ 𝑛0 . Let’s try 𝑐 = 1, then we need to find n0 ≥ 0 such that:

𝑛3 ≥ 𝑛2 for all 𝑛 ≥ 𝑛0 .

choosing 𝑛 = 𝑛0 makes this true. Thus:

𝑛3 ≥ 𝑛2 for all n0 ≥ 0.

The big-omega notation gives the best-case complexity of an algorithm. The big-
omega notation essentially denotes that this is the fastest an algorithm can run for
arbitrary large inputs. For example, for an algorithm with an order of growth in Ω(𝑛2 ),
we can say that its running time increases at most proportionally to the square of the
input size n. This means the running time can never be less than 𝑛2 .

Big-Theta (or Theta-) Notation (𝚯-Notation)

The big-theta notation is used to describe asymptotically tight bound. It bounds the
growth of an algorithm’s running time from both above and below for sufficiently large
input sizes. Θ(𝑔(𝑛)) is the set of functions that grow at the same rate as 𝑔(𝑛). It indicates
that the running time of an algorithm, 𝑡(𝑛) grows at the same rate as 𝑔(𝑛). If 𝑡(𝑛) is in
Θ(𝑔(𝑛)) (denoted as 𝑡(𝑛) ∈ Θ(𝑔(𝑛)), then for a sufficiently large n, the running time is

Dr F Adamu-Fika Unit 3 CSC 310

Cyber Security Dept. AFIT Algorithm Analysis: Algorithm and 13
© Since 2020 Algorithm Efficiency Complexity Analysis
 sandwiched between 𝑐1 ⋅ 𝑔(𝑛) and 𝑐2 ⋅ 𝑔(𝑛), that means 𝑡(𝑛) is at least 𝑐2 ⋅ 𝑔(𝑛) and at
most 𝑐1 ⋅ 𝑔(𝑛) for some constants 𝑐1 and 𝑐2 . We say that the running time is “big theta
of 𝑔(𝑛)” or just “theta of 𝑔(𝑛)”.

Formal Definition of Big-Theta Notation

Function 𝑡(𝑛) belongs to the set Θ(𝑔(𝑛)) if there exist positive constants 𝑐1 and 𝑐2 , and
non-negative integer 𝑛0 such that:

𝑐2 ⋅ 𝑔(𝑛) ≤ 𝑡(𝑛) ≤ 𝑐1 ⋅ 𝑔(𝑛), for all 𝑛 ≥ 𝑛0 .

1image source: Levitin [1]

1
For example, let's show that 𝑛(𝑛 – 1) ∈ Θ(n2 ). We need to find c1 ≥ 0, c2 ≥ 0 and n0 ≥
2
1
0 such that 𝑐2 ⋅ 𝑛2 ≤ 𝑛(𝑛 – 1) ≤ 𝑐1 ⋅ 𝑛2 for every integer 𝑛 ≥ 𝑛0 .
2

First, we prove the upper bound, i.e., the right side of the inequality. Since we have
1
fractions, we will try fraction values for 𝑐2 . We try 𝑐2 = , now we find n0 ≥ 0 such that:
2

1 1 1
𝑛2 − 𝑛≤ 𝑛2 for all 𝑛 ≥ 𝑛0 .
2 2 2

1 1 1
[ 𝑛(𝑛 − 1) ≤ 𝑛2 − 𝑛]
2 2 2
1 1 1
An obvious choice is n0 = 0. Thus: 2 𝑛2 − 𝑛≤ 𝑛2 for all 𝑛 ≥ 0.
2 2

Second, we prove the lower bound, i.e., the left side of the inequality. We will also try
1 1
fraction values for . 𝑐1 < 2 for the definition to be true. So, we try 𝑐1 < 4 . Now, we
find n0 ≥ 0 such that:
1 1 1
𝑛2 ≤ 𝑛2 − 𝑛 for every integer 𝑛 ≥ 𝑛0 .
4 2 2

We can choose n0 = 2. Thus

1 1 1
𝑛2 ≤ 𝑛2 − 𝑛 for all 𝑛 ≥ 2.
4 2 2

1 1
Therefore, by the definition, we can select 𝑐2 = , 𝑐1 = and n0 = 2.
4 2

The big-theta notation gives the average-case complexity of an algorithm. When we

use the big-theta notation we are essentially saying that an algorithm will always run

CSC 310 Unit 3 Dr F Adamu-Fika

14 Algorithm and Algorithm Analysis: Cyber Security Dept. AFIT
Complexity Analysis Algorithm Efficiency © Since 2020
within a factor above and below for sufficiently large inputs. For example, for an
algorithm with an order of growth in Θ(log 𝑛), we can say that its running time increases
by a constant as the input size n grows. This means the running time will always be
log 𝑛, i.e., it can never be slower or faster than log 𝑛.

Which Notation to Use

Since big-O gives the slowest running time of an algorithm, it is widely used in
efficiency analysis, as we are always interested in the performance of an algorithm in
the worst-case scenario. We are sometimes interested in the average case too, big
theta. Since the best-case performance of an algorithm is generally not useful, big
omega notation is the least used notation among all three.

For example, let’s consider the running time for a sequential search algorithm. In the
best-case scenario, we find the search value at the first location and the running time
is 1. Whereas, in the worst-case scenario, the search value may either be at the end
of the list or not be in the list at all, the running time, in this case, will be the same as
the number of elements in the list. It would be true that the worst-case running time of
a sequential search is Θ(𝑛). However, it is incorrect to say that sequential search runs
in Θ(𝑛) all the time. Because in the best-case scenario, it runs in Θ(1). The running time
of sequential search is never worse than Θ(𝑛) but it is sometimes better.

Therefore, it would be more convenient and accurate to use big-O in such situations.
So, if we say the running time of sequential search is always 𝑂(𝑛) what we are saying
is: “its running time grows at most this much, but it could grow more slowly.”

We can make a stronger statement about the sequential search’s worst-case running
time, it is Θ(𝑛). However, for a blanket statement to cover all cases, the strongest
statement we can make is that sequential search runs in 𝑂(𝑛).

Because big-O is bounded above and big-theta is bounded both above and below,
the running time that is Θ(𝑔(𝑛)) in the worst-case scenario is also O(𝑔(𝑛)). But as we
have seen, the opposite is not always true. Similarly, running time that is Θ(𝑔(𝑛)) also
implies Ω(𝑔(𝑛)), because big omega is bounded below.

We can make correct but imprecise statements using big-O or big-theta. we can
correctly but imprecisely say that the best-case running time of sequential search is
𝑂(𝑛), because we know that it takes, at most, linear time. We can make another
correct and imprecise statement, that the worst-case running time of sequential
search is Ω(1), because we know that it takes at least constant time.

Comparison of Big-O and Big-Θ

• Use the notation Θ(𝑔(𝑛)), when the number of times the basic operation will be
executed is tightly bound to 𝑔(𝑛).
▪ When the number of executions is the same in all cases. Like in
ArrayMax - to find the maximum element in an array, the check to find
the largest element is always executed in all three cases.
▪ When the number of execution varies for the different cases, we can
express growth for each case as big-Θ.
• Use the notation 𝑂(𝑔(𝑛)), when the number of times the basic operation will be
executed is bounded at most (i.e., the upper bound) to 𝑔(𝑛) and the lower
bound could be anything arbitrary.

Dr F Adamu-Fika Unit 3 CSC 310

Cyber Security Dept. AFIT Algorithm Analysis: Algorithm and 15
© Since 2020 Algorithm Efficiency Complexity Analysis
 ▪ When the number of executions is different in the different cases. Like in
sequential search. We use big-𝑂 to generalise the growth of the
algorithm.
• Which is better?
▪ If two algorithms and for a particular problem incur overall time
complexities 𝑂(𝑔(𝑛)) and Θ(𝑔(𝑛)), then, it would be better to choose
algorithm A with complexity 𝑂(𝑔(𝑛)) as the algorithm incurs at most 𝑔(𝑛)
complexity for certain inputs, whereas algorithm B incurs 𝑔(𝑛)
complexity on all inputs. In other words, algorithm A has a worst-case
running time of 𝑔(𝑛) and there might be situations it would run faster.
Whereas algorithm B will always runs in 𝑔(𝑛), even in the best case.

Some Properties of Asymptotic Order of Growth

We can use the asymptotic notations to rank functions according to their growth rate.

Let 𝑓(𝑛) and 𝑔(𝑛) be any non-negative functions defined on a set of all real numbers.

We say 𝑓(𝑛) is 𝑂(𝑔(𝑛)) for all functions 𝑓(𝑛) that have a lower or the same order of
growth as 𝑔(𝑛), within a constant multiple as 𝑛 → ∞. Think of it as 𝑓(𝑛) ≤ 𝑔(𝑛)

For example:

• 𝑛 ∈ 𝑂(𝑛2 ) •
1
𝑛(𝑛 − 1) ∈ 𝑂(𝑛2 ) • 100𝑛 + 5 ∈ 𝑂(𝑛2 )
2

• 𝑛3 ∉ 𝑂(𝑛2 ) • 𝑛4 + 𝑛 + 1 ∉ 𝑂(𝑛2 ) • 0.001𝑛3 ∉ 𝑂(𝑛2 )

To summarise:

𝑓(𝑛) is 𝑂(𝑔(𝑛)) 𝑔(𝑛) is 𝑂(𝑓(𝑛))

g(n) grows more Yes No
𝑓(𝑛) grows more No Yes
Same growth Yes Yes
We say 𝑓(𝑛) is Ω(𝑔(𝑛)) for all functions 𝑓(𝑛) that have a lower or the same order of
growth as 𝑔(𝑛), within a constant multiple as 𝑛 → ∞. Think of it as 𝑓(𝑛) ≥ 𝑔(𝑛)

For example:

• 𝑛 ∈ Ω(𝑛2 ) • 100𝑛 + 5 ∈ Ω(log 𝑛) •

1
𝑛(𝑛 − 1) ∈ Ω(𝑛)
2
• 100𝑛 + 5 ∉ Ω(𝑛2 ) • 0.001𝑛3 ∉ Ω(𝑛4 ) • log 𝑛 ∉ Ω(𝑛)

To summarise:

𝑓(𝑛) is 𝑂(𝑔(𝑛)) 𝑔(𝑛) is 𝑂(𝑓(𝑛))

g(n) grows more No Yes
𝑓(𝑛) grows more Yes No
Same growth Yes Yes
We say 𝑓(𝑛) is Θ(𝑔(𝑛)) for all functions 𝑓(𝑛) that have a lower or the same order of
growth as 𝑔(𝑛), within a constant multiple as 𝑛 → ∞. Think of it as 𝑓(𝑛) = 𝑔(𝑛)

CSC 310 Unit 3 Dr F Adamu-Fika

16 Algorithm and Algorithm Analysis: Cyber Security Dept. AFIT
Complexity Analysis Algorithm Efficiency © Since 2020
For example:

• 𝑛 log 𝑛 ∈ Θ(log 𝑛) • 𝑛2 ∈ Θ(𝑛2 ) • 10𝑛 + 5 ∈ Θ(𝑛)

• 𝑛2 ∉ Θ(𝑛) • 100𝑛 + 5 ∉ Θ(𝑛2 ) • 𝑛2 ∉ Θ(𝑛3 )

To summarise:

𝑓(𝑛) is 𝑂(𝑔(𝑛)) 𝑔(𝑛) is 𝑂(𝑓(𝑛))

g(n) grows more No No
𝑓(𝑛) grows more No No
Same growth Yes Yes
Note:
• If 𝑓(𝑛) is 𝑂(𝑔(𝑛)) then 𝑔(𝑛) is Ω(𝑓(𝑛)).
▪ 𝑛 ∈ 𝑂(𝑛2 ) → 𝑛2 ∈ Ω(𝑛)
▪ log 𝑛 ∈ 𝑂(√𝑛) → √𝑛 ∈ Ω(log 𝑛)
• If 𝑓(𝑛) is Ω(𝑔(𝑛)) then 𝑔(𝑛) is O(𝑓(𝑛)).
▪ log 2 𝑛 ∈ Ω(log 𝑛2 ) → log 𝑛2 ∈ 𝑂(log 2 𝑛 )
• If 𝑓(𝑛) is a polynomial of degree 𝑑 then 𝑓(𝑛) is 𝑂(𝑛𝑑 ).
▪ 3𝑛4 + 5𝑛3 + 7𝑛2 + 𝑛 + 1 ∈ 𝑂(𝑛4 )
▪ This is also applicable to the Θ and Ω notations.
• If 𝑐 (𝑛) is 𝑂(𝑓(𝑛)) and 𝑑 (𝑛) is 𝑂(𝑔(𝑛)) then:
▪ 𝑐(𝑛) + 𝑑(𝑛) is 𝑂(𝑓(𝑛) + 𝑔(𝑛)).
o 6𝑛2 ∈ 𝑂(𝑛2 ) and 3𝑚3 ∈ 𝑂(𝑚3 ) → 6𝑛2 + 3𝑚3 ∈ 𝑂(𝑛2 + 𝑚3 )
▪ 𝑐(𝑛) ⋅ 𝑑(𝑛) is 𝑂(𝑓(𝑛) ⋅ 𝑔(𝑛)).
o 6𝑛2 ∈ 𝑂(𝑛2 ) and 3𝑚3 ∈ 𝑂(𝑚3 ) → 6𝑛2 ⋅ 3𝑚3 ∈ 𝑂(𝑛2 ⋅ 𝑚3 )
▪ This is also applicable to the Θ and Ω notations.
• If 𝑓(𝑛) is 𝑂(𝑔(𝑛)) and 𝑔(𝑛) is 𝑂(ℎ(𝑛)) then 𝑓(𝑛) is 𝑂(ℎ(𝑛)).
▪ 𝑛 ∈ 𝑂(𝑛2 ) and 𝑛2 ∈ 𝑂(𝑛3 ) → 𝑛 ∈ 𝑂(𝑛3 )
▪ This is also applicable to the Θ and Ω notations.
• If 𝜌(𝑛) is a polynomial in 𝑛 then log(𝜌(𝑛)) is 𝑂(log 𝑛).
▪ log(7𝑛5 ) ∈ 𝑂(log 𝑛)
▪ This is also applicable to the Θ and Ω notations.

Summary of Asymptotic Notation Function Simplification Rules

We can summarise the rules of simplifying running times to get their order of growth
as:

1. if 𝒇(𝒏) is a polynomial of degree 𝒅, then:

a. Drop lower-order terms (any terms (or factors) of degree less than 𝒅).
b. Drop constant factors.

Therefore,

• For big-O: 𝒇(𝒏) is in 𝑶(𝒏𝒅 ).

• For big-Omega: 𝒇(𝒏) is in 𝛀(𝒏𝒅 ).
• For big-Theta: 𝒇(𝒏) is in 𝚯(𝒏𝒅 ).
2. Use the smallest/closest possible class of functions.
• For example, we say “𝟓𝒏 is in 𝑶(𝒏)” instead of “𝟓𝒏 is in 𝑶(𝒏𝟑 )”
3. Use the simplest expression of the class.
• For example, we say “𝟕𝒏 – 𝟐 is in 𝑶(𝒏)” instead of “𝟕𝒏 – 𝟐 is in 𝑶(𝟕𝒏)”

Dr F Adamu-Fika Unit 3 CSC 310

Cyber Security Dept. AFIT Algorithm Analysis: Algorithm and 1
© Since 2020 Algorithm Efficiency Complexity Analysis
 Basic Efficiency Classes
All the functions whose orders differ by a multiplicative constant are put together to
form a complexity class. However, there are still infinitely many classes, for example,
the exponential function 𝑎 𝑛 has different orders of growth for different values of base
𝑎. Therefore, the time efficiency of a good number of algorithms falls into only a few
classes. Following is a list of these commonly encountered efficiency classes (in
relation to the input parameter 𝑛), in increasing order of their rate of growth, where 𝑐
is an arbitrary constant.

Class Name Example of running Description

time

1 constant 77 The running time will not change in

relation to the value of the input
parameter. This is typically
characterised by simple/basic
statements that will only run once in
each iteration of the algorithm.

log 𝑛 logarithmic log 5𝑛 The running time is proportional to

the number of times the input
log 𝑛3 (problem size) can be reduced by
some fraction. The running time will
change by a constant when
changes. When the doubles, the
running time will increase by 1.
Typically, a result of reducing a
problem’s size by a constant factor
in each iteration of the algorithm.

log 2 𝑛 polylogarithmic log 3 𝑛 = (log 𝑛)3

𝑐>1

𝑛 linear 100𝑛 The running time is directly

proportional to the value n. When n
doubles, the running time increases
twofold, (2n). Typically,
characterised by simple loops.

𝑛 log 𝑛 linearithmic 3𝑛 log 8𝑛 This is a combination of linear and

logarithmic time. Typically,
characterised by having n
logarithmic loops.

𝑛2 quadratic 9𝑛2 The running time is proportional to

the square of the input parameter.
When doubles, the running time
increases fourfold, (4n2). Typically,

CSC 310 Unit 3 Dr F Adamu-Fika

18 Algorithm and Algorithm Analysis: Cyber Security Dept. AFIT
Complexity Analysis Algorithm Efficiency © Since 2020
 characterised with two nested
loops.

𝑛3 cubic 4𝑛3 The running time is proportional to

the cube of the input parameter.
When doubles, the running time
increases eightfold, (8n3). Typically,
characterised with three nested
loops.

𝑛𝑐 polynomial 6𝑛4 The running time is proportional to

the cth power of the input size. When
𝑐≥1 𝑛7 the doubles, the running time
increases by 2c, (2cnc).

𝑐𝑛 exponential 2𝑛 Typical for algorithms that generate

all subsets of an n-element set.
𝑐≥1 3𝑛

𝑛! factorial (𝑛 – 1)! Typical for algorithms that generate

all permutations of an n-element set.

Note:

• Polynomial and exponential classes are very different. The exponential

functions grows much, much faster, no matter the magnitude of the constant
is. Factorial time is sometimes considered as belonging to the exponential
class. An exponential time in the form 𝑛𝑛 (pronounced n-to-the-n) is worse than
𝑛! time. Another order of growth called fractional power [𝑛𝑐 , 0 ≤ 𝑐 ≤ 1 .
1
Example: √𝑛 = 𝑛2 ] have better complexity than linear time. Quadratic and
cubic time are special cases of polynomial time.

• Order of growth of log 𝑛 is exactly the same as order of growth of

log 𝑛𝑐 . The logarithms differ only by a constant factor, and asymptotic notations
ignore that. Similarly, logs with different constant bases are equivalent. Order
of growth of log log 𝑛 (called loglogarithmic) is better than order of growth of
log 𝑛 .

• Asymptotic notations can also be used with multiple variables and with other
expressions on the right side of the equal sign. For example, the notation:
𝑓(𝑛, 𝑚) = 𝑛3 + 𝑚2 + 𝑂(𝑛 + 𝑚) represents the statement: there exists non-
negative constants 𝑐 and 𝑛0 such that 𝑓(𝑛, 𝑚) ≤ 𝑛3 + 𝑚2 + 𝑐 ⋅ (𝑛 + 𝑚) for all
𝑚, 𝑛 > 𝑛0 . We find complexity measures such as these in algorithms used
graphs.

Understanding Code Complexity

When given a segment of code, we can determine its efficiency from the kinds of
statements it consists of.

Dr F Adamu-Fika Unit 3 CSC 310

Cyber Security Dept. AFIT Algorithm Analysis: Algorithm and 19
© Since 2020 Algorithm Efficiency Complexity Analysis
 Simple Statements
statement (e.g., 𝑦 = 𝑥 + 50)
If a statement is "simple" (only involves primitive operations without loops) then the
execution time of the statement is constant.

If-Then-Else
if (condition) then
block 1 (sequence of statements)
else
block 2 (sequence of statements)
end if
Here, either block 1 will execute, or block 2 will execute.

Therefore, for worst-case analysis, the worst-case time is the slower of the two
branches:

maximum(time(block 1), time(block 2)).

Conversely, for best-case analysis, the best time is the faster of the two branches:

minimum(time(block 1), time(block 2)).

For example, if block 1 executes in constant time and block 2 executes in linear time,
the if-then-else statement would have linear complexity in the worst-case and
constant complexity in the best case.

Simple Loops
for i in 1 … n loop
sequence of statements
end loop
The loop executes n times, so the sequence of statements also executes times. If we
assume the statements have constant running time, the total time for the loop is n * 1,
which overall is linear.

Nested Loops
for i in 1 … n loop
for j in 1 … m loop
sequence of statements
end loop
end loop
The outer loop executes m times. Each time the outer loop executes, the inner loop
executes m times. As a result, the statements in the inner loop execute a total of 𝑛 × 𝑚
times. Thus, the complexity is 𝑂(𝑛 × 𝑚) (or Ω(𝑛 × 𝑚) or Θ(𝑛 × 𝑚)) depends on which
notation is being used). This can be extended for loop nesting consisting of k loops
(more than two loops). The total the statements in the innermost loop (loop k) would
execute is time(loop 1) * time(loop 2) * … * time(loop k) times.

In a common special case where the stopping condition of the inner loop is j < n
instead of j < m , i.e., the inner loop also executes n times, the total complexity for the
two loops is quadratic, O(n2).

Function/Procedure calls

CSC 310 Unit 3 Dr F Adamu-Fika

20 Algorithm and Algorithm Analysis: Cyber Security Dept. AFIT
Complexity Analysis Algorithm Efficiency © Since 2020
When a statement involves a procedure call, the complexity of the statement
includes the complexity of the procedure. Assume that you know that procedure f
takes constant time, and that procedure g takes time proportional to (linear in) the
value of its parameter k. Then the statements below have the time complexities
indicated.

𝑓(𝑘) has constant complexity

(𝑘) has linear complexity

When a loop is involved, the same rule applies. For example:

for j in 1 … n loop
𝑔(𝑗)
end loop
the statement ( ) has quadratic (𝑛 × 𝑛 complexity: the loop executes n times and
each procedure call 𝑔(𝑛) also has linear complexity relative to the parameter n.

Consecutive Code Fragments

When there is a consecutive or sequences of code fragments of varying complexities,

The total running time for the entire code fragment is the slowest piece of code, that
is maximum running time amongst the individual fragments.

sequence of statements
for i in 1 … n loop
sequence of statements
end loop
sequence of statements
for i in 1 … n loop
for j in 1 … 2n loop
sequence of statement
end loop
end loop
The sequence of statements outside the loops runs in constant time. The first loop runs
in linear time, while the second runs in quadratic time. The maximum time is quadratic.

KEY POINTS TO REMEMBER

• The efficiency of an algorithm depends on the amount of time, storage and

other resources required to execute the algorithm. The efficiency is measured
with the help of asymptotic notations.
• An algorithm may not have the same performance for different types of inputs.
With the increase in the input size, the performance will change.
• The study of change in performance of the algorithm with the change in the
order of the input size is defined as asymptotic analysis.
• Both time and space efficiencies are measured as functions of the algorithm’s
input size.
• Time efficiency is measured by counting either 1, the number of times the
algorithm’s primitive operations are executed or 2, the number of times the
algorithm’s basic operation is executed.

Dr F Adamu-Fika Unit 3 CSC 310

Cyber Security Dept. AFIT Algorithm Analysis: Algorithm and 21
© Since 2020 Algorithm Efficiency Complexity Analysis
 • The performance of some algorithms may differ significantly for inputs of the
same size. We need to distinguish between the worst-case, average-case, and
best-case efficiencies for such algorithms.
• The analysis framework’s primary interest lies in the order of growth of the
algorithm’s running time (or extra memory units consumed) as its input size goes
to infinity.
• The order of growth of the running time of an algorithm gives a simple
characteristic of the algorithm’s performance; it also allows us to compare the
relative performance of alternative algorithms.
• The asymptotic running time of an algorithm is defined in terms of functions.
• There are mainly three asymptotic notations: O (Big-Oh)—notation (asymptotic
upper bound, it gives the worst-case complexity of an algorithm), Ω (Big-
Omega)—notation (asymptotic lower bound it gives the best-case complexity
of an algorithm) and Θ (Big-Theta)-notation (asymptotic tight bound, it gives
the average-case complexity of an algorithm).
• Use Θ-notation to express the complexity of an algorithm: when its running time
is the same in the best-case, average- and worst-case scenarios.
• Use 𝑂-notation to express the complexity of an algorithm when its running time
is different in best-case, average-case and worst-case scenarios.
• Use Θ-notation to express the complexity of specific scenarios of an algorithm.

UNIT GOALS

At this point (after completing this unit) you should:

• Understand what it means for an algorithm to be efficient.

• Know how the size of the input is measured for certain basic problems.
• Be able to identify if (and when) the nature of the input will affect the efficiency
of the algorithm on the input of the same sizes. That is, knowing how to identify
the worst and best cases of an algorithm’s running time.
• Given an algorithm or fragment, be able to:
▪ identify its basic operation
▪ estimate the number of operations
▪ specify its order of growth using asymptotic notation
• Know how to rank algorithms based on their order of growth.
• Know which asymptotic notation to use.

MATH YOU NEED TO REVIEW

You need to revise and reacquaint yourselves with the following mathematical
concepts.

• Summations
▪ See appendix: summations for some common ones that you will need a
lot in algorithm analysis.
• Logarithms and Exponents
▪ Logarithm is the opposite of exponents.
o log 𝑎 𝑏 = 𝑐 ⟺ 𝑎 𝑐 = 𝑏
▪ Properties of Logarithms:
o log 𝑎 𝑏 ⋅ 𝑐 = log 𝑎 𝑏 + log 𝑎 𝑐
𝑏
o log 𝑎 𝑐 = log 𝑎 𝑏 – log 𝑎 𝑐

CSC 310 Unit 3 Dr F Adamu-Fika

22 Algorithm and Algorithm Analysis: Cyber Security Dept. AFIT
Complexity Analysis Algorithm Efficiency © Since 2020
 o log 𝑎 𝑏 𝑐 = 𝑐 ⋅ log 𝑎 𝑏
log 𝑏
o log 𝑎 𝑏 = log𝑐 𝑎
𝑐
o log 𝑎 𝑎 = 1
o log 𝑎 1 = 0
▪ Properties of Exponents:
o 𝑎 𝑏 ⋅ 𝑎 𝑐 = 𝑎 (𝑏+𝑐)
o (𝑎 𝑏 )𝑐 = 𝑎 𝑏𝑐
𝑎𝑏
o = 𝑎 (𝑏−𝑐)
𝑎𝑐
o 𝑎 log𝑎 𝑏 = 𝑏
o 𝑎 𝑐 ⋅log𝑎 𝑏 = 𝑏𝑐
• Algebraic manipulations

EXERCISES

1. For each of the following algorithms, indicate (i) a natural size metric for its
inputs, (ii) its basic operation, and (iii) whether the basic operation count can
be different for inputs of the same size:
a. computing the sum of n numbers;
b. computing n!;
c. finding the smallest element in a list of n numbers;
d. searching for a particular element in a list of n names.
2. For each of the following functions, indicate how much the function’s value will
change if its argument (n) is increased fourfold (increased four times).
a. log 2 𝑛
b. √𝑛
c. 𝑛
d. 𝑛2
e. 𝑛3
f. 2𝑛
3. For each of the following pairs of functions, indicate whether the first function
of each of the following pairs has a lower, same, or higher order of growth (to
within a constant multiple) than the second function.
a. 𝑛(𝑛 + 1) and 2000𝑛2
b. 100𝑛2 and 0.013𝑛3
c. log 2 (log 2 𝑛) and log 2 𝑛
d. 2𝑛−1 and 2𝑛
e. (𝑛– 1)! and 𝑛!
f. log 22 𝑛 and log 2 𝑛2
4. Express the function 𝑛3 ⁄1000 – 100𝑛2 + 100𝑛 + 3 in terms of 𝑂-notation.
5. Select the 𝑂-notation for the following pseudocode fragments
[the choices are 𝑂(1), 𝑂(log 𝑛), 𝑂(𝑛), 𝑂(𝑛 log 𝑛), 𝑂(𝑛2 ), 𝑂(𝑛3 ), 𝑂(2𝑛 ), 𝑂(𝑛!)]:
a. for i = 0 to n – 1
z=x+y
b. for i = 0 to n – 1
for j = 0 to i
for k = 0 to j
sum = A[i] + B[j] + C[k]
c. for i = 1; i ≤ n; i *= 2
for j = 0; j < n; j++

Dr F Adamu-Fika Unit 3 CSC 310

Cyber Security Dept. AFIT Algorithm Analysis: Algorithm and 23
© Since 2020 Algorithm Efficiency Complexity Analysis
 count = count + 1
6. Indicate the class Θ(𝑔(𝑛)) the following functions belong to. (Use the simplest
𝑔(𝑛) possible in your answer.
a. 𝑛3 + 𝑛2
b. 1000000𝑛3 + 𝑛2
c. 𝑛3 + 1000000𝑛 2
d. (𝑛2 + 𝑛) ⋅ (𝑛 + 1)
7. Rank these functions according to their growth, from slowest growing (at the
front) to fastest growing (at the rear): 𝑛3 , (5⁄3)3 , 1, 𝑛, 2𝑛, 𝑛2 .
8. For the functions, log 2 𝑛 and log 8 𝑛, what is the asymptotic relationship between
these functions?
Choose all answers that apply and justify your choice(s):
a. log 2 𝑛 is 𝑂(log 8 𝑛)
b. log 2 𝑛 is Ω(log 8 𝑛)
c. log 2 𝑛 is Θ(log 8 𝑛)
9. The ICT Centre issue students’ email addresses in the format
f(irst).m(iddle)[email protected] (f(irst) means the first letter of the student’s
first name and m(iddle), the first letter of the student’s middle name). No two
students can have the same email address. The Centre have a way to resolve
the issue of a new student having the same initials and surname as another
student that already assigned an email address. The first thing they do before
issuing a new email address is to search through the existing email addresses to
check whether it is available. So, they came up with this algorithm:
1) Start with a list of the addresses, sorted alphabetically in
ascending order.
2) Set min to 1 and max to the number of email addresses.
3) Set index to the average of min and max, round up so that
it is an integer.
4) Check the address located at that index in the list.
5) If the address equals the desired username, the search is
done! Exit the algorithm.
6) If the address is comparatively less than the desired
username, set min to be one larger than index.
7) If the address is comparatively greater than the desired
username, set max to be one smaller than index.
8) Go back to step 3.
Say whether the following statements are true or false and give justification:
a. The algorithm implements linear search.
b. The algorithm implements binary search.
c. The algorithm is more efficient than binary search.
d. The algorithm is more efficient than linear search.

BIBLIOGRAPHY AND RESOURCES

Textbooks

[1] A. Levitin, Introduction to the Design and Analysis of Algorithms, Boston: Pearson,
2012.

CSC 310 Unit 3 Dr F Adamu-Fika

24 Algorithm and Algorithm Analysis: Cyber Security Dept. AFIT
Complexity Analysis Algorithm Efficiency © Since 2020
[2] T. H. Cormen, C. E. Leiserson, R. L. Ronald and C. Stein, Introduction to Algorithms,
Cambridge, MA: MIT Press, 2009.

[3] R. Sedgewick and K. Wayne, Algorithms, Boston: Pearson, 2011.

Websites

[1] “Khan Academy,” [Online]. Available:

https://www.khanacademy.org/computing/ap-computer-science-
principles/algorithms-101. [Accessed 30 April 2022].

Dr F Adamu-Fika Unit 3 CSC 310

Cyber Security Dept. AFIT Algorithm Analysis: Algorithm and 25
© Since 2020 Algorithm Efficiency Complexity Analysis