Program: III B. Tech.

CSE
Course : Design & Analysis of Algorithms
Topic: Algorithms, Complexity and Analysis

Dr. P E S N Krishna Prasad

Professor & head
Department of CSE
Anil Neerukonda Institute of Technology & Sciences
Visakhapatnam – 531162
Email: [email protected]
 Contents
1. Introduction to Algorithms
2. What is Complexity?
3. Order and Order of Magnitude
4. Analysis of Algorithms
5. Some Mathematical Background

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Introduction to Algorithms
An algorithm is a sequence of unambiguous instructions for solving a
problem, i.e., for obtaining a required output for any legitimate input
in a finite amount of time.
problem

algorithm

input “computer” output

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Euclid’s Algorithm
Problem: Find gcd(m,n), the greatest common divisor of two
nonnegative, not both zero integers m and n
Examples: gcd(60,24) = 12, gcd(60,0) = 60, gcd(0,0) = ?

Euclid’s algorithm is based on repeated application of equality

gcd(m,n) = gcd(n, m mod n)
until the second number becomes 0, which makes the problem
trivial.
Example: gcd(60,24) = gcd(24,12) = gcd(12,0) = 12

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Descriptions of Euclid’s Algorithm
Step 1 If n = 0, return m and stop; otherwise go to Step 2
Step 2 Divide m by n and assign the value fo the remainder to r
Step 3 Assign the value of n to m and the value of r to n. Go to
Step 1.

while n ≠ 0 do
r ← m mod n
m← n
n←r
return m
Department of CSE, ANITS –CSE316 Design & Analysis of
Algorithms
 Other Approach to Compute gcd(m,n)

Consecutive integer checking algorithm

Step 1 Assign the value of min{m,n} to t
Step 2 Divide m by t. If the remainder is 0, go to
Step 3;
otherwise, go to Step 4
Step 3 Divide n by t. If the remainder is 0, return
t and stop;
otherwise, go to Step 4
Step 4 Decrease t by 1 and go to Step 2

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 One more Approach
Middle-school procedure
Step 1 Find the prime factorization of m
Step 2 Find the prime factorization of n
Step 3 Find all the common prime factors
Step 4 Compute the product of all the common
prime factors
and return it as gcd(m,n)

Is this an algorithm?

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Another Approach
Input: Integer n ≥ 2
Output: List of primes less than or equal to n
for p ← 2 to n do A[p] ← p
for p ← 2 to ⎣√n⎦ do
if A[p] ≠ 0 //p hasn’t been previously eliminated from the list
j ← p* p
while j ≤ n do
A[j] ← 0 //mark element as eliminated
j←j+p

Example: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Department of CSE, ANITS –CSE316 Design & Analysis of
Algorithms
 Why Study Algorithms?

• Theoretical importance

– the core of computer science

• Practical importance

– A practitioner’s toolkit of known algorithms

– Framework for designing and analyzing algorithms for new problems

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Major Issues related to Algorithms

• How to design algorithms

• How to analyze algorithm efficiency

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Algorithm Design Techniques
❖ Brute force Greedy approach

❖ Divide and conquer Dynamic programming

Iterative improvement
❖ Decrease and conquer
Backtracking
❖ Transform and conquer
Branch and bound
❖ Space and time tradeoffs

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Analysis of Algorithms
• How good is the algorithm? • Issues:
– time efficiency – correctness
– space efficiency – time efficiency
– space efficiency
– optimality
• Does there exist a better algorithm?
– lower bounds
– optimality • Approaches:
– empirical analysis – less useful
– theoretical analysis – most important

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 What is Complexity?
The concept of complexity is used to quantify the efficiency of an
algorithm. There are two types of complexities:
•Time: How many operations are made?
•Space: How much extra space is needed?

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
Notations

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Theoritical analysis of Time Efficiency

Time efficiency is analyzed by determining the number of repetitions of the

basic operation as a function of input size
Basic operation: the operation that contributes most towards the running
time of the algorithm

input size

T(n) ≈ copC(n)
running time execution time Number of times
for basic operation basic operation
is executed

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
Order of Magnitude

Department of CSE, ANITS –CSE316 Design & Analysis of Algorithms

 Order
Usually we use the frequency count to compare algorithms.
Consider the following 3 programs:

(a) (b) (c)

x←x+y for i ← 1 to n do for i ← 1 to n do
x←x+y for j ← 1 to n do
end x←x+y
end
end

The frequency count of stmt x ← x + y is 1, n, n2.

No matter which machine we use to run these programs, we know that
the execution time of (b) is n times the execution time of (a).
Department of CSE, ANITS –CSE316 Design & Analysis of
Algorithms
 Notations Informal Definitions

f(n) is in O(g(n)) if order of growth of f(n) ≤ order of growth of

g(n) (within constant multiple and for large n).
f(n) is in Ω(g(n)) if order of growth of f(n) ≥ order of growth of
g(n) (within constant multiple and for large n).

f(n) is in Θ(g(n)) if order of growth of f(n) = order of growth of

g(n) (within constant multiples and for large n).

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Notations: Formal Definitions

f(n) ∈ O(g(n)) iff there exist positive constant c and non-negative

integer n0 such that
f(n) ≤ c g(n) for every n ≥ n0
f(n) ∈ Ω(g(n)) iff there exist positive constant c and non-negative
integer n0 such that
f(n) ≥ c g(n) for every n ≥ n0
f(n) ∈ Θ(g(n)) iff there exist positive constants c1 and c2 and
non-negative integer n0 such that
c1 g(n) ≤ f(n) ≤ c2 g(n) for every n ≥ n0

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Why Constant is Needed
What is gained by the constant c: What is gained by the constant c:
f(n) ≤ c g(n) for every n ≥ n0
f(n) ≤ c g(n) for every n ≥ n0
What do you think?
What do you think? Allows c g(n) to exceed f(n)
Allows us to ignore leading constants in f(n)

For example, 0.1n^2, n^2, and 10n^2

All have same growth rate: n^2
Meaning what: as input size doubles, time is 4x
increase

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Big-O Definition
Informal Definition: f(n) is in O(g(n)) if order of growth of f(n) ≤ order
of growth of g(n) (within constant multiple),
Definition: f(n) ∈ O(g(n)) iff there exist positive constant c and
non-negative integer n0 such that
f(n) ≤ c g(n) for every n ≥ n0
Examples:
• 10n is O(n2)
– [Can choose c and n0. Solve for 2 different c’s]
• 5n + 20 is O(n) [Solve for c and n0]

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Big-O Notation
Def f(n) = O(g(n)) if and only if ∃ 2 positive constants c and n0,
such that
|f(n)| ≤ c · |g(n)| ∀ n ≥ n0.
So, g(n) actually is the upper bound of f(n).

n0
Figure 1. Illustrating “big O”

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Big-O Examples
•Is 17 n2 – 5 = O(n2)?

•Is 35 n3 + 100 = O(n3)?

•Is 6 · 2n + n2 = O(2n)?

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Complexity Classes

f(n) n! 2n n3
n2

n log n

n (linear time)

log n

n
Figure Growth rates of some important complexity classes
Department of CSE, ANITS –CSE316 Design & Analysis of
Algorithms
Note:
log n
n(linear)
n log n polynomial time (easy or tractable)
n2
n3

2n
n! exponential time (hard or intractable)

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Big-Omega Definition
Informal Definition: f(n) is in Ω (g(n)) iff order of growth of f(n) ≥ order
of growth of g(n) (within constant multiple),
Definition: f(n) ∈ Ω(g(n)) iff there exist positive constant c and
non-negative integer n0 such that
f(n) ≥ c g(n) for every n ≥ n0
Examples:
• 10n2 is Ω(n)
• 5n + 20 is Ω(n)

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Ω notation
Def iff ∃ two positive constants c and n0,
such that

So, g(n) is the lower bound of f(n).

n0

Figure . Illustrating Ω

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
Examples:
Examples
•Is ?

•Is ?

value of n0
•Is ?

When n is bigger, 2n will grow faster than n100. ( Yes, you can find n0 )

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Big Theta Definition
Informal Definition: f(n) is in Θ(g(n)) if order of growth of f(n) =
order of growth of g(n) (within constant multiple),
Definition: f(n) ∈ Θ(g(n)) iff there exist positive constants c1 and c2
and non-negative integer n0 such that
c1 g(n) ≤ f(n) ≤ c2 g(n) for every n ≥ n0
Examples:
• 10n2 is Θ(n2)
• 5n + 20 is Θ(n)

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Θ notation
Def iff ∃ three positive constants, c1, c2, n0 ,

such that

n0
Figure. Illustrating Θ

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Examples
Examples:
• Is 3n + 2 = Θ(n)?

• Is 3n + 2 = Θ(n2)?

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
• Is ?

• Is ?

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Property of Order

1) Transitive:
If and then

If and then

If and then
2) Reflexive:

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Property of Order

3) Symmetric:

iff

iff

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Analysis of Algorithms

a) Best Case, Worse Case Analysis

The problem:
Sort n keys in nondecreasing sequence.

Solving strategy:
Insertion sort – insert the ith item into a sorted list of length (i – 1) by looking at numbers
one at a time, ∀ i >1.
Example: sort 3,5,4,1
step 1 step 2 step 3 step 4
3 3 3 1
5 5 4 3
4 4 5 4
1 1 1 5

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Algorithm
Algorithm:

A(0) ← -maxint; // for efficient to stop the loop

for i ← 2 to n do
target ← A(i);
j ← i –1;
while (target < A(j))
A( j + 1) ← A(j);
j ← j –1;
A( j + 1) ← target;

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Analysis

Best Case performance is Θ(n) or Ω(n), but not Ω(n2)

since insertion sort runs in Θ(n) when the input is sorted.

Worst Case performance is Θ(n2) or O(n2)

The running time of insertion sort is between Ω(n) to O(n2)

The running time is bound to O(n2)

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Average Analysis
Algorithm:
b) Average Case Analysis
Procedure search ( n, S, x, location) // n is total # of keys,
The problem: S is an array,
x is the key,
Is the key x in the array S of n keys?
location is output parameter
Solving strategy: begin
location ← 1;
Sequential search while (location ≤ n) and (S[location] ≠ x)
location ++;
if location > n location ← 0;
end;

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Average Case Analysis

Average Case Analysis:

When ‘x is in the array’, we assume that

Probability prob
1) When x = kth, # of key comparisons = k

about half array is searched.

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Average Case Analysis
2) When ‘x may not be in the array’

Assume prob (x in array) = p,

prob (x not in array) = 1 – p, and it takes

n comparisons to know that ‘is not in the array’

If p = 1, (as calculated in case (1))

If , (about ¾ of the array is

searched)

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Some Mathematical Background
Definition. Logb x = y if b y = x b > 1

1) Logb is a strictly increasing function,

∴if x1 < x2 then Logb x1 < Logb x2 .

2) Logb is a one-to-one function,

if Logb x1 = Logb x2 then x1 = x2 .

3) Logb 1 = 0 ∀b

4) Logb xa = a · Logb x

5) Logb(x1 · x2) = Logb x1 + Logb x2

Department of CSE, ANITS –CSE316 Design & Analysis of
Algorithms
 Cont…
6) x1Logb x2 = x2Logb x1

7) to convert from one base to another,

8) sum of consecutive integers

9) geometric sums

10) Suppose f is a continuous, decreasing function, a, b are integers, then

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Cont…
11) Suppose f is a continuous, increasing function, a, b are
integers,
then

12)

Note:
Log2 = Lg
Loge = Ln

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Example: Sequential Search

• Worst case: Omega, Theta, O?

• Best case: Omega, Theta, O?
• Average case: Omega, Theta, O?

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
Maximum Element

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
Element Uniqueness Problem

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
Matrix Multiplication

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
Counting Binary Digits

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Gaussian Elimination

Algorithm GaussianElimination(A[0..n-1,0..n])
//Implements Gaussian elimination of an n-by-(n+1) matrix A
for i ← 0 to n - 2 do
for j ← i + 1 to n - 1 do
for k ← i to n do
A[j,k] ← A[j,k] - A[i,k] * A[j,i] / A[i,i]

Find the efficiency class and a constant factor improvement.

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms
 Some Classical Problems
❖ Travelling Salesman
❖ Knapsack problem
❖ N-Queens
❖ Coin Change
❖ Load Balancing
❖ Interval Covering
❖ Longest Increasing
Sequence
❖ Range Sum (Min and Max)
❖ Constrained Subset Sum
Department of CSE, ANITS –CSE316 Design & Analysis of
Algorithms
 Some Mathematical Problems
❖ Fibonacci Series
❖ Binomial Coefficient
❖ Pigeonhole principle
❖ Permutations and Combinations
❖ Lexicographic Ordering
❖ Bit Manipulation
❖ Towers of Hanoi
❖ Integer Sequence
❖ Matrix Multiplications
❖ And so on.
Department of CSE, ANITS –CSE316 Design & Analysis of
Algorithms
What is Complexity?

Department of CSE, ANITS –CSE316 Design & Analysis of

Algorithms