IT623 Algorithms & Data

Structures
Asymptotic Notation Q, O, W
Analysis of Algorithm for Time and Space
• To analyze an algorithm means:
• developing a formula for predicting how fast an algorithm is, based on the
size of the input (time complexity), and/or
• developing a formula for predicting how much memory an algorithm requires,
based on the size of the input (space complexity)
• Usually time is our biggest concern
• Most algorithms require a fixed amount of space and/or
• Memory is not as expensive as time
Problem Size Matters in Calculating Time or
Space Complexity
• Time and Space complexity will depend on the Problem Size, why?
• If we are searching an array, the “size” of the input array could decide who
long it will take to go through the entire array
• If we are merging two arrays, the “size” could be the sum of the two array
sizes
• If we are computing the nth Fibonacci number, or the nth factorial, the “size” is
n
• We choose the “size” to be a parameter that determines the actual
time (or space) required
• It is usually obvious what this parameter is
• Sometimes we need two or more parameters
Characteristic Operation
• In computing time complexity, one good approach is to count
characteristic operations
• What a “characteristic operation” is depends on the particular problem
• If searching, it might be comparing two values
• If sorting an array, it might be:
• comparing two values
• swapping the contents of two array locations
• both of the above
• Sometimes we just look at how many times the innermost loop is executed
How many times innermost loop will execute?
Algorithm Array_Sum(A): Algorithm Array2D_Sum(A):
for i := 0 to n-1 do for i := 0 to n-1 do
sum := sum + A[i][j] for j := 0 to n-1 do
return sum; sum := sum + A[i][j]
return sum;
Sequential Search Analysis
Input: An array A storing n items, target x
Output: true if x is in A, false otherwise
Algorithm Sequential_Search(A, x):
for i := 0 to n-1 do
if (A[i] = x)
return true;
return false;

• Identify “Characteristic Operations” in this code?

• How many times will the loop actually execute?
• that depends on how many times comparison operations are require to execute
Exact vs Simplified Analysis
• It is sometimes possible, in assembly language, to compute exact time
and space requirements
• We know exactly how many bytes and how many cycles each machine
instruction takes
• For a problem with a known sequence of steps (factorial, Fibonacci), we can
determine how many instructions of each type are required
• However, often the exact sequence of steps cannot be known in
advance
• The steps required to sort an array depend on the actual numbers in the array
(which we do not know in advance)
Exact vs Simplified Analysis
• In a higher-level language (such as Java), we do not know how long
each operation takes
• Which is faster, x < 10 or x <= 9 ?
• We don’t know exactly what the compiler does with this
• The compiler almost certainly optimizes the test anyway (replacing the slower
version with the faster one)
• In a higher-level language we cannot do an exact analysis
• Our timing analyses will use major oversimplifications
• Nevertheless, we can get some very useful results
Average, Best or Worst Case
• Usually we would like to find the average time to perform an algorithm
• However,
• Sometimes the “average” isn’t well defined
• Example: Sorting an “average” array
• Time typically depends on how out of order the array is
• How out of order is the “average” unsorted array?
• Sometimes finding the average is too difficult
• Often we have to be satisfied with finding the worst (longest) time required
• Sometimes this is even what we want (say, for time-critical operations)
• The best (fastest) case is seldom of interest
Sequential Search Analysis – Best and Worst
Cases
Input: An array A storing n items, target • Value of x is found at the last index in
x an array i.e. A[n-1] → Worst Case,
Output: true if x is in A, false otherwise require n comparison/iteration

Algorithm Sequential_Search(A, x): • Worse Case Time Complexity: k*n + c

for i := 0 to n-1 do • N is Problem Size i.e. Array Size
if (A[i] = x) • k time taken for loop execution
(increment i, check if i < N-1) and
return true; comparison
return false; • c is time taken for loop init and return
• Value of x is found at the first index in • Best Case Time Complexity: k*1 + c
an array i.e. A[0] → Best Case, • This is nothing but a special case of worst
required only 1 comparison/iteration case equation
More Examples
Algorithm Is_Array_Sorted(A): Algorithm Swap(x, y):
for i := 0 to n-1 do temp := x
if A[i] > A[i+1] x := y
return false y := temp
return true;
• Best Case →
• Best Case → __________________?
__________________? • Worst Case →
• Worst Case → __________________?
__________________?
More Examples
Algorithm Array2D_Sum(A): Algorithm Array2D_Sum():
for i := 0 to n-1 do for i := 0 to n-1 do
for j := 0 to n-1 do for j := 0 to n-1 do
sum := sum + A[i][j] A[i][j] := i*j
return sum; for i := 0 to n-1 do
for j := 0 to n-1 do
sum := sum + A[i][j]
return sum;
Algorithm Analysis
• Represent Algorithm Analysis (i.e. Runtime) as function of input n i.e.
f(n) e.g. For Sequential Search f(n) = k*n + c
• what we care about: orders of growth
• Ex: 0.5n2 and 700n2 are no different!
• when input size doubles, the running time quadruples
• constant factors do NOT matter
Constant Factors are Ignored, Why?
• Algo1: f1(n) = 5n2
• Algo2: f2(n) = n2
• f1(n)/f2(n) = 5n2/ n2 = 5
• If we double the data size i.e. n = 2n
• f1(2n)/f2(2n) = 5(2n2)/ (2n2) = 10n2/ 2n2 = 5

• Algo: f(n) = kn2+c

• As n → ∞, c becomes less and less dominating

• Therefore, n2 is the dominating factor.

Only Highest Order Term Matters
• We don't need to study algorithms when we want to sort two
elements, because different algorithms make no difference
• we care about algorithm performance when the input size n is very
large
• Ex: n2 and n2 + n + 2 are no different, because when n is really large, n + 2 is
negligible compared to n2
• only the highest-order term matters
Only Highest Order Term Matters
• f1(n) = n2, f2(n) = n2+n+10
• With n=10
• 10) = (100) and
• f2(10) = (100 + 10 + 10) = 120
• Difference of 16.67%

• With n = 50000
• f1(50000) = (2,500,000,000) and
• f2(50000) = (2,500,000,000 + 50000 + 10) = 2,500,050,010 ~ 2,500,000,000
• Difference of 0.002%
What is the time complexity ?
Algorithm Dummy(A): Algorithm Dummy1(A):
for i := 1 to n step 2*i do for i := n to 1 step –i/2 do
// do something // do something
What is the time complexity?
Algorithm Algo1(): Algorithm Algo2():
for i := 0 to n-1 do for i := 0 to n-1 do
// do something for j := 1 to n step j*2 do
sum := sum + A[i][j]
for j := 0 to m-1 do
return sum;
// do something
Big O notation – Simplifying f(n)
• Throwing out the constants is one of two things we do in analysis of
algorithms
• By throwing out constants, we simplify 12n2 + 35 to just n2
• Our timing formula is a polynomial, and may have terms of various
orders (constant, linear, quadratic, cubic, etc.)
• We usually discard all but the highest-order term
• We simplify n2 + 3n + 5 to just n2
• We call this a Big O notation
Asymptotic Notation

f(n)  O(g(n))
Usually written as
f(n) = O(g(n)) or
f(n) is O(g(n))
Can we Justify Asymptotic Notation?
• Consider f(n) = n2 + 3n + 5 as n varies:
n=0 n2 = 0 3n = 0 f(n) = 5
n = 10 n2 = 100 3n = 30 f(n) = 135
n = 100 n2 = 10000 3n = 300 f(n) = 10,305
n = 1000 n2 = 1000000 3n = 3000 f(n) = 1,003,005
n = 10,000 n2 = 108 3n = 3*104 f(n) = 100,030,005
n = 100,000 n2 = 1010 3n = 3*105 f(n) = 10,000,300,005
f(n) = x2 + 3x + 5, for x=1..10
Common time complexities
BETTER • O(1) constant time
• O(log n) log time
• O(n) linear time
• O(n log n) log linear time
• O(n2) quadratic time
• O(n3) cubic time
• O(nk) polynomial time
• O(2n) exponential time
WORSE
Common Time
Complexities
Big O Visual - O(g(n)) is the set of functions
with smaller or same order of growth as g(n)
O(NlogN) O(N2)
O(N) 20NlogN+3N+100 5N2 +3N+100
3N+100 NlogN+100N 15N2 +100
100N 100NlogN-0.5N-20 N2 + 100NlogN-20
0.5N-20

O(N3)
O(1) 5N3 +3N+100
100 23N3 + 5N2 +100N
5 N3 + 100NlogN-20
0.5
Algorithm Examples for Time Complexity
Big O Notation Name Example(s)
# Odd or Even number,
O(1) Constant
# Swapping two numbers
# Finding element on sorted array
O(log n) Logarithmic
with binary search
# Find max element in unsorted
array,
O(n) Linear
# Duplicate elements in array with
Hash Map

# Sorting elements in array with

O(n log n) Linearithmic
merge sort

# Duplicate elements in array

2
O(n ) Quadratic **(naïve)**,
# Sorting array with bubble sort
O(n3) Cubic # 3 variables equation solver
O(2n) Exponential # Find all subsets
# Find all permutations of a given
O(n!) Factorial
set/string
Examples:
• 2n2 = O(n3): 2n2 ≤ cn3  2 ≤ cn  c = 1 and n0= 2

• n2 = O(n2): n2 ≤ cn2  c ≥ 1  c = 1 and n0= 1

• 1000n2 + 1000n = O(n2): 1000n2 + 1000n ≤ 1000n2 + n2 = 1001n2 

c=1001 and n0 = 1000

• n = O(n2): n ≤ cn2  cn ≥ 1  c = 1 and n0= 1

Exercise
1. n2 + 3n + 5 = O(n2): Find c and n0
n2 + 3n + 5 <= n2 + 3n2 + 5  4n2 + 5  c=4 and n0=1

• Which of the below Big O notation is correct.

1. n4 + 100n2 + 10n + 50 = O(n3)
2. 10n3 + 2n2 = O(n3)
3. n3 - n2 = O(n4)
4. nlogn+ n2 = O(nlogn)
5. 10 = O(n)
6. 1273 = O(1)
More Examples
• Show that 30n + 8 is O(n).
• Show c,n0: 30n + 8  cn, n ≥ n0 .
• Let c=31, n0 = 8. Then
cn = 31n = 30n + n ≥ 30n + 8, so 30n + 8 ≤ cn
Big-O example, graphically
• Note 30n + 8 isn’t less than n
anywhere (n>0). cn =
• It isn’t even less than 31n 31n 30n+8

Value of function →
everywhere.
• But it is less than 31n 30n+8
everywhere to the right of n=8. n
O(n)
n ≥ n0=8 →
Increasing n →
No Uniqueness
• There is no unique set of values for n0 and c in proving the asymptotic bounds

• Prove that 100n + 5 = O(n2)

• 100n + 5 ≤ 100n + n = 101n ≤ 101n2
for all n ≥ 5
n0 = 5 and c = 101 is a solution
• 100n + 5 ≤ 100n + 5n = 105n ≤ 105n2
for all n ≥ 1
n0 = 1 and c = 105 is also a solution
Must find SOME constants c and n0 that satisfy the asymptotic notation relation
Big W (Omega)

W(g(n)) is the set of functions with

larger or same order of growth as g(n)
Examples
• 5n2 = W(n) • n = W(2n)
 c, n0 such that: 0  cn  5n2  cn  5n2  c = 1 and • n3 = W(n2)
n0 = 1 • n = W(logn)
• 100n + 5 ≠ W(n2)

 c, n0 such that: 0  cn2  100n + 5

100n + 5  100n + 5n ( n  1) = 105n

cn2  105n  n(cn – 105)  0

Since n is positive  cn – 105  0  n  105/c

 contradiction: n cannot be smaller than a constant

Big Q (theta)

Q(g(n)) is the set of functions with

the same order of growth as g(n)
Examples

• n2/2 –n/2 = Q(n2)

• ½ n2 - ½ n ≤ ½ n2 n ≥ 0  c2= ½

• ½ n2 - ½ n ≥ ½ n2 - ½ n * ½ n ( n ≥ 2 ) = ¼ n2  c1= ¼

• n ≠ Q(n2): c1 n2 ≤ n ≤ c2 n2

 only holds for: n ≤ 1/c1

More Examples

• 6n3 ≠ Q(n2): c1 n2 ≤ 6n3 ≤ c2 n2

 only holds for: n ≤ c2/6

• n ≠ Q(logn): c1 logn ≤ n ≤ c2 logn

 c2 ≥ n/logn,  n ≥ n0 – impossible
Asymptotic Notation Sets

O(n) Q(n) W(n)

For each of the following pairs of functions, either
f(n) is O(g(n)), f(n) is Ω(g(n)), or f(n) = Θ(g(n)).
Determine which relationship is correct.
• f(n) = log n2; g(n) = log n + 5 -> f(n) = Q (g(n))
• f(n) = n; g(n) = log n2 -> f(n) = W(g(n))
• f(n) = log log n; g(n) = log n -> f(n) = O(g(n))
• f(n) = n; g(n) = log2 n -> f(n) = W(g(n))
• f(n) = n log n + n; g(n) = log n -> f(n) = W(g(n))
• f(n) = 10; g(n) = log 10 -> f(n) = Q(g(n))
• f(n) = 2n; g(n) = 10n2 -> f(n) = W(g(n))
• f(n) = 2n; g(n) = 3n -> f(n) = O(g(n))
Properties
• Theorem:
f(n) = Q(g(n))  f = O(g(n)) and f = W(g(n))
• Transitivity:
• f(n) = Q(g(n)) and g(n) = Q(h(n))  f(n) = Q(h(n))
• Same for O and W
• Reflexivity:
• f(n) = Q(f(n))
• Same for O and W
• Symmetry:
• f(n) = Q(g(n)) if and only if g(n) = Q(f(n))
• Transpose symmetry:
• f(n) = O(g(n)) if and only if g(n) = W(f(n))