CSE 225 Data Structures and Algorithms

Mirza Mohammad Lutfe Elahi

Department of Electrical and Computer Engineering

North South University
Fall 2016

Comparison of Algorithms

Chapter 02

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 1 / 36
 Comparison of Algorithms

Comparison of Algorithm?
There is more than one way to solve most problems.
The ways to solve the problem are not same.
But the solutions are functionally correct.

Figure: Direction to Joe’s Diner

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 2 / 36
 Comparison of Algorithms

Which Cost More to Feed?

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 3 / 36
 Comparison of Algorithms

Algorithm Efficiency

There are often many approaches (algorithms) to solve a problem.

- How do we choose between them?
At the heart of computer program design are two (sometimes
conflicting) goals to design an algorithm.
Easy to understand, code, debug.
Makes efficient use of the resources.
Goal one is the concern of Software Engineering.
Goal two is the concern of data structures and algorithm analysis.
When goal two is important,
- how do we measure an algorithm’s cost?

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 4 / 36
 Comparison of Algorithms

How to Measure Efficiency?

Critical resources:
- Time, memory, programmer effort, user effort
Factors affecting running time:
- For most algorithms, running time depends on ”size” of the input.
- Running time is expressed as T (n) for some function T on input
size n.

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 5 / 36
 Comparison of Algorithms

Analysis of Algorithms

An algorithm is a finite set of precise instructions for performing a

computation or for solving a problem.
What is the goal of analysis of algorithms?
- To compare algorithms mainly in terms of running time but also in
terms of other factors (e.g., memory requirements, programmer’s
effort etc.)
What do we mean by running time analysis?
- Determine how running time increases as the size of the problem
increases.

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 6 / 36
 Comparison of Algorithms

Input Size

Size of an array.
Polynomial degree.
Number of elements in a matrix.
Number of bits in the binary representation of the input.
Number of vertices and edges in a graph.

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 7 / 36
 Comparison of Algorithms

Time Analysis

Provides upper and lower bounds of running time.

Lower Bound ≤ Running Time ≤ Upper Bound

Different types of analysis:

Worst Case
Best Case
Average Case

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 8 / 36
 Comparison of Algorithms

Worst Case

Provides an upper bound on running time.

An absolute guarantee that the algorithm would not run longer, no
matter what the inputs are.

Lower Bound ≤ Running Time ≤ Upper Bound

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 9 / 36
 Comparison of Algorithms

Best Case

Provides an lower bound on running time.

An absolute guarantee that input is the one for which the algorithm
runs the fastest.

Lower Bound ≤ Running Time ≤ Upper Bound

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 10 / 36
 Comparison of Algorithms

Average Case

Provides an estimate of average running time.

Assumes that the input is random.
Useful when best or worst cases do not happen very often.

Lower Bound ≤ Running Time ≤ Upper Bound

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 11 / 36
 Comparison of Algorithms

How Do We Analyze an Algorithm?

Need to define objective measures.

1 Compare execution times?
- Empirical comparison (run programs)
- Not good: times are specific to a particular machine.
2 Count the number of statements?
- Empirical comparison (run programs)
- Not good: number of statements varies with programming language
and programming style.
3 Express running time t as a function of problem size n
- (i.e., t = f (n) )

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 12 / 36
 Comparison of Algorithms

Rate of Growth ≡ Asymptotic Analysis

Given two algorithms having running times f (n) and g (n), how do we
decide which one is faster?
Compare ”rates of growth” of f (n) and g (n).
Using rate of growth as a measure to compare different functions
implies comparing them asymptotically, i.e., as n → ∞
If f (n) is faster growing than g(x), then f(x) always eventually
becomes larger than g (n) in the limit, i.e., for large enough values of n

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 13 / 36
 Comparison of Algorithms

The Growth of Functions

A problem that can be solved with polynomial worst-case complexity

is called tractable.
Problems of higher complexity are called intractable.
Problems that no algorithm can solve are called unsolvable.

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 14 / 36
 Comparison of Algorithms

Asymptotic Notation - Big-O

Figure: g (n) is asymptotically upper bound for f (n)

Asymptotic less than.

O(g (n)) = f (n) : there exist positive constants c and n0 such that
0 ≤ f (n) ≤ cg (n) for all n ≥ n0
Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 15 / 36
 Comparison of Algorithms

Asymptotic Notation - Big-Ω

Figure: g (n) is asymptotically lower bound for f (n)

Asymptotic greater than.

Ω(g (n)) = f (n) : there exist positive constants c and n0 such that
0 ≤ cg (n) ≤ f (n) for all n ≥ n0
Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 16 / 36
 Comparison of Algorithms

Asymptotic Notation - Big-Θ

Figure: g (n) is asymptotically tight bound for f (n)

Asymptotic equality.

Θ(g (n)) = f (n) : there exist positive constants c1 , c2 and n0 such that
0 ≤ c1 g (n) ≤ f (n) ≤ c2 g (n) for all n ≥ n0
Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 17 / 36
 Comparison of Algorithms

Relation between Order of Growth Sets

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 18 / 36
 Comparison of Algorithms

Growth Functions

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 19 / 36
 Comparison of Algorithms

Growth Functions

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 20 / 36
 Comparison of Algorithms

Comparing Growth

N log2 N Nlog2 N N2 2N
1 0 0 1 2
2 1 2 4 4
4 2 8 16 16
8 3 24 64 256
16 4 64 256 65,536
32 5 160 1024 4.29X109
64 6 384 4096 1.84X1019
128 7 896 16,384 3.40X1038

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 21 / 36
 Comparison of Algorithms

Estimation Techniques

Known as ”back of the envelope” or ”back of the napkin” calculation

Determine the major parameters that effect the problem.
Derive an equation that relates the parameters to the problem.
Select values for the parameters, and apply the equation to yield and
estimated solution.

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 22 / 36
 Comparison of Algorithms

Estimation Example

Known as ”back of the envelope” or ”back of the napkin” calculation

How many library bookcases does it take to store books totaling one
million pages?
Estimate:
- Pages/inch
- Feet/shelf
- Shelves/bookcase

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 23 / 36
 Comparison of Algorithms

Understanding the Growth Rate

Consider the example of buying elephants and goldfish:

Cost: (cost of elephants) + (cost of goldfish)

Approximation:

Cost ' cost of elephants

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 24 / 36
 Comparison of Algorithms

Understanding the Growth Rate

How to Find f (n)?

Associate a ”cost” with each statement.
Find total number of times each statement is executed.
Add up the costs.
The low order terms of a function are relatively insignificant for large
n.
n4 + 100n2 + 10n + 50
Approximately
n4
Highest order term determines rate of growth!

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 25 / 36
 Comparison of Algorithms

Running Time Examples

Algorithm 1 Cost
arr[0] = 0; c1
arr[1] = 0; c1
arr[2] = 0; c1
...
arr[N-1] = 0; c1
——
c1 + c1 + ... + c1 = c1 ∗ N

Algorithm 2 Cost
for(i = 0; i < N; i++) c2
arr[1] = 0; c1
——
(N + 1) ∗ c2 + N ∗ c1 = (c2 + c1 ) ∗ N + c2

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 26 / 36
 Comparison of Algorithms

Running Time Examples

Algorithm Cost
sum = 0; c1
for(i = 0; i < N; i++) c2
for(j = 0; j < N; i++) c2
sum += arr[i][j]; c3
——
c1 + c2 ∗ (N + 1) + c2 ∗ N ∗ (N + 1) + c3 ∗ N ∗ N
= (c2 + c3 ) ∗ N 2 + 2c2 ∗ N + (c1 + c2 )

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 27 / 36
 Comparison of Algorithms

Big-O Visualization

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 28 / 36
 Comparison of Algorithms

Running Time Examples

Algorithm Cost in Big − O

i = 0;
while(i<N)
{
X = X + Y; O(1)
result = mystery(X); O(N)
i++; O(1)
}
Cost of the body of the while loop: O(N)
Loop executed N times
Total cost: N ∗ O(N) = O(N 2 )

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 29 / 36
 Comparison of Algorithms

Running Time Examples

Algorithm Cost in Big − O

if(i<j)
for(i = 0; i < N; i++) O(N)
X = X + i;
else
X = 0; O(1)

MAX(O(N), O(1)) = O(N)

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 30 / 36
 Comparison of Algorithms

Running Time Examples

What does the following algorithm compute?

i n t max_diff1 ( i n t a [ ] , int n)
{
int m = 0;

f o r ( i n t i = 0 ; i < n ; i++)
f o r ( i n t j = i + 1 ; j < n ; j++)
i f ( abs ( a [ i ] a [ j ]) > m)
m = abs ( a [ i ] a[j]) ;

return m ;
}

It returns the maximum difference between any two numbers in the

input array.
Comparisons:
(n − 1) + (n − 2) + (n − 3) + ... + 1 = (n−1)
n = 0.5n2 − 0.5n
Time Complexity: O(n ) 2

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 31 / 36
 Comparison of Algorithms

Running Time Examples

Another algorithm solving the same problem.

i n t max_diff2 ( i n t a [ n ] , i n t n )
{
i n t min = a [ 0 ] ;
i n t max = a [ 0 ] ;

f o r ( i n t i = 1 ; i < n ; i++)
i f ( a [ i ] < min )
min = a [ i ] ;
e l s e i f ( a [ i ] > max )
max = a [ i ] ;

r e t u r n max − min ;
}

Comparisons: 2n − 2
Time Complexity: O(n)

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 32 / 36
 Comparison of Algorithms

Orders of Magnitude

Listed from slowest to fastest growth:

O(1) Constant time
O(log2 N) Logarithmic time
O(N) Linear time
O(Nlog2 N) N logarithmic time
O(N 2 ) Quadratic time
O(N 3 ) Cubic time
O(2N ) Exponential time

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 33 / 36
 Comparison of Algorithms

Examples

Rank the following functions by order of growth from the slowest to the
fastest. (notation: lgn = log2 n)

4n2 , 10lgn, 1000, n2 , 2n , 100n, 2lgn

1000 10lgn 2lgn 100n n2 4n2 2n

Slowest =⇒ Fastest

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 34 / 36
 Comparison of Algorithms

Analyze Complexity of Code Segments

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 35 / 36
 Comparison of Algorithms

Analyze Complexity of Code Segments

Mirza Mohammad Lutfe Elahi (ECE@NSU) C++ Plus Data Structures, Nell Dale (5th) Chapter 02 36 / 36