Design and Analysis

of Algorithms

CS-332
(3-0-3)
Books / Reference Materials:
Textbook:
1. Introduction to Algorithms, Cormen, T. H.,Leiserson, C.E., Rivest, R.L.&
Stein, C., 4th Edition (2009), MIT Press.
Reference Books:
2. An Introduction to the Analysis of Algorithms, Sedgewick, R.& Flajolet,
P., 2nd Edition (2012), Addison-Wesley.
3. Foundation of Algorithms Using C++ Pseudocode, Neapolitian, R., 6th
Edition (2013), Jones & Bartlett Learning.
4. Introduction to the Design and Analysis of Algorithms, Levitin, A., 3rd
Edition (2012), Pearson
 Course Objectives

Analyze the Asymptotic Performance of

Analyze Algorithms.

Write rigorous Correctness proofs for

Write algorithms.
Demonstrat Demonstrate a familiarity with major
algorithms and data structures.
e
Apply important algorithmic design
Apply paradigms and methods of analysis.

Synthesize efficient algorithms in common

Synthesize design situations.
  An effective, efficient and best
method which can be used to
express solution of any problem
within a finite amount of space and
time and in a well-defined formal
language
1. Step by step procedure for solving
any problem is known as algorithm.
2. An algorithm is a finite set of
Algorithm instructions that, if followed,
accomplishes a particular task.
3. An algorithm is a sequence of
computational steps that transform
the input into a valuable or required
output.
4. Any special method of solving a
certain kind of problem is known as
algorithm.
Analysis of Algorithms
• The theoretical study of computer-program
performance and resource usage. What’s more
important than performance?
• User-friendliness
• Modularity • Programmer time
• Correctness • Simplicity
• Maintainability • Extensibility
• Functionality • Reliability
• Robustness
Why Study
Algorithms
and
Performance?
What kinds of problems are
solved by algorithms?
• Sorting is by no means the only computational
problem for which algorithms have been
developed. Practical applications of algorithms are
ubiquitous and include the following examples:
• Human Genome Project
• Internet
• Electronic commerce
• Manufacturing and other commercial enterprises
 1) Input
There are more quantities that are
extremely supplied.

2) Output
At least one quantity is produced.
Algorithms
must satisfy 3) Definiteness
Each instruction of the algorithm
the following should be clear and unambiguous.
criteria 4) Finiteness
The process should be terminated
after a finite number of steps.

5) Effectiveness
Every instruction must be basic
enough to be carried out theoretically
or by using paper and pencil.
Properties of Algorithm
Non-Ambiguity
Each step in an algorithm should be non-ambiguous. That means each instruction should be
clear and precise. The instruction in any algorithm should not denote any conflicting
meaning.
Range of Input
The range of input should be specified. This is because normally the algorithm is input driven
and if the range of input is not being specified then an algorithm can go into an infinite state.
Multiplicity
The same algorithm can be represented in several different ways. That means we can write
in simple English the sequence of instruction, or we can write it in form of pseudo code.
Similarly, for solving the same problem we can write several different algorithms.
Speed
The algorithm is written using some specified ideas. But such an algorithm should be
efficient and should produce the output with fast speed.
Finiteness
The algorithm should be finite. That means after performing required operations it should be
terminated.
Fundamentals of Algorithmic
Problem Solving
Understanding the Problem
Ascertaining the capabilities of a computational Design
Choosing between exact and approximate problem
solving
Algorithm Design Techniques
Designing an appropriate data structure
Method of specifying an algorithm
Proving an Algorithm’s correctness
Analyzing an algorithm
Coding an algorithm
Fundamenta
ls of
Algorithmic
Problem
Solving
Preconditions and Postconditions
Preconditions and Postconditions

Frequently a programmer must communicate

precisely what a function accomplishes, without
any indication of how the function does its
work.

Can you think of a situation

where would this occur ?
Preconditions, Postconditions and
Invariant
• One way to specify such requirements is with a pair of
statements about the function.
• The precondition statement indicates what must be
true before the function is called.
• The postcondition statement indicates what will be
true when the function finishes its work.
• An invariant is a statement placed in a code segment
that is repeated should be true each time the loop
execution reaches that point. Invariants are often used
in a loops and recursions
Example

void write_sqrt( double x)

// Precondition: x >= 0.
// Postcondition: The square root of x has
// been written to the standard output.
• In this example, the
precondition requires that
x >= 0

...be true whenever the

function is called.

}
Example: The Graph Search Algorithm
• visit v //retrieve and remove edge e
• put all of the edges that leave v from the dispenser
into the dispenser • let w be the head of e
// Precondition: • if w has not been visited
// v is visited. Every other vertex • visit e
in the graph is unvisited. • visit w
// Every edge in the graph that put all of the edges that
goes from v to another vertex is in leave w into the dispenser
the dispenser.
• endif
• while the dispenser is not empty
• endwhile
// Invariant:
// Postcondition:
// Every edge in the graph that
goes from a visited vertex to an // If there is a directed path from
unvisited vertex is in the dispenser. v to a vertex x then x has been
visited.
Example
struct Node { // SINGLY-linked list
int value;
struct Node* next;
};
// Insert a new node with the given value before an existing node.
// If the list is empty, create a new node, which becomes the
head.
void insert_after(struct Node* existing, int value, struct Node** a_head)
{ // … }
// Create a list with the numbers from start to stop. Return the
head.
struct Node* create_count_list(int start, int stop) {
struct Node* head = NULL; // start as empty list
struct Node* tail = NULL;
for(int value = start; value <= stop; value++) {
insert_after(tail, value, &head);
tail = tail -> next;
Loop Invariant
Loop Invariant Condition with Examples
• A loop invariant is a condition that is true at the beginning and end of
every loop iteration.
• The concept is similar to a class invariant, which must be true at the
beginning and end of every public method.
• A loop invariant is some predicate (condition) that holds for every loop
iteration.
• When you write a loop that works correctly, you are at least implicitly
relying on a loop invariant.
• For example, let’s look at a simple for loop that looks like this:
int j = 9;
for(int i=0; i<10; i++)
j--;
A good loop invariant should
satisfy three properties:
• Initialization: The loop invariant must be true before the first
execution of the loop.
• Maintenance: If the invariant is true before an iteration of
the loop, it should be true also after the iteration.
• Termination: When the loop is terminated the invariant
should tell us something useful, something that helps us
understand the algorithm.

On the other hand, the loop conditional must be false after the
loop terminates, otherwise, the loop would never terminate.
A good loop invariant should
satisfy three properties:
Loop Invariant:
• max is the maximum value in a[0…i-1]
Initialization
•i=1 and the scope of the array is a[0…i-
1]=a[0..1-1]=a[0], max = a[0]
•Thus, invariant holds
Maintenance:
•Assuming loop invariant holds at iteration k, then
max is the maximum value in a[0..k-1]
•In the next iteration k=1, the code compares and
possibly updates max with a[k], max is the
maximum value in scope a[0..k]
•Thus, invariant holds
Termination
• i=len(a), max is the maximum value in
a[0..len(a)-1]
• Thus, invariant holds
Asymptotic Analysis
Sorting
Algorithms
Insertion
Sort
 Key
Questions
●
How do you fi nd effi cient
solutions to seemingly hard
problems?
●
How do you prove that an
a l g o r i t h m is correct?
●
How do you analyze an
algorithm’s runtime?
Insertion Sort
• Idea: like sorting a hand of playing cards
• Start with an empty left hand and the cards facing
down on the table.
• Remove one card at a time from the table, and insert
it into the correct position in the left hand
• compare it with each of the cards already in the
hand, from right to left
• The cards held in the left hand are sorted
• these cards were originally the top cards of the
pile on the table
25
Insertion Sort

To insert 12, we need

to make room for it by
moving first 36 and
6 10 2 4 36 then 24.

12

26
Insertion Sort

6 10 24 36

12

27
Insertion Sort

6 10 24 3
6

12

28
Insertion Sort
input array

5 2 4 6 1
3
at each iteration, the array is divided in two sub-arrays:

left sub-array right sub-array

sorted unsorted

29
Insertion Sort

30
Another Example

31
 Insertion Sort

7 4 2 1 6
 Insertion Sort

7 4 2 1 6
 Insertion Sort

7 4 2 1 6
 Insertion Sort

7 4 2 1 6
 Insertion Sort

4 7 2 1 6
 Insertion Sort

4 7 2 1 6
 Insertion Sort

4 7 2 1 6
 Insertion Sort

4 7 2 1 6
 Insertion Sort

4 2 7 1 6
 Insertion Sort

4 2 7 1 6
 Insertion Sort

2 4 7 1 6
 Insertion Sort

2 4 7 1 6
 Insertion Sort

2 4 7 1 6
 Insertion Sort

2 4 7 1 6
 Insertion Sort

2 4 1 7 6
 Insertion Sort

2 4 1 7 6
 Insertion Sort

2 1 4 7 6
 Insertion Sort

2 1 4 7 6
 Insertion Sort

1 2 4 7 6
 Insertion Sort

1 2 4 7 6
 Insertion Sort

1 2 4 7 6
 Insertion Sort

1 2 4 7 6
 Insertion Sort

1 2 4 6 7
 Insertion Sort

1 2 4 6 7
INSERTION-SORT
Alg.: INSERTION-SORT(A) 1 2 3 4 5 6 7 8

for j ← 2 to n a1 a2 a3 a4 a5 a6 a7 a8

do key ← A[ j ] key
Insert A[ j ] into the sorted sequence A[1 . . j -1]

i←j-1
while i > 0 and A[i] > key
do A[i + 1] ← A[i]
i←i–1
A[i + 1] ← key
• Insertion sort – sorts the elements in place
56
Loop Invariant for Insertion Sort
Alg.: INSERTION-SORT(A)
for j ← 2 to n
do key ← A[ j ]
Insert A[ j ] into the sorted sequence A[1 . . j -1]

i←j-1
while i > 0 and A[i] > key
do A[i + 1] ← A[i]
i←i–1
A[i + 1] ← key
Invariant: at the start of the for loop the elements in A[1 . . j-1]
are in sorted order
57
Analysis of Insertion Sort
INSERTION-SORT(A) cost
for j ← 2 to n times
do key ← A[ j ] c1 n
Insert A[ j ] into the sorted sequence A[1 . . j -1] c2 n-1
i←j-1 0 n-1
n-1
n
t
while i > 0 and A[i] > key c4 j 2 j

 (t  1)
n

do A[i + 1] ← A[i] c5 j 2 j

 (t  1)
n

i←i–1 c6
j 2 j

A[i + 1] ← key
c7
tj: # of times the while statement is executed at iteration j
n n c
n 8 n-
T (n) c1n  c2 (n  1)  c4 (n  1)  c5  t j  c6  t j  1 c7 1t j  1 c8 (n  1)
j 2 j 2 j 2
58
Best Case Analysis
• The array is already sorted “while i > 0 and A[i] > key”

• A[i] ≤ key upon the first time the while loop test is run
(when i = j -1)

• tj = 1

• T(n) = c1n + c2(n -1) + c4(n -1) + c5(n -1) +

c8(n-1) = (c1 + c2 + c4 + c5 + c8)n + (c2 + c4

+ c5 + c8) n n n
T (n) c1n  c2 (n  1)  c4 (n  1)  c5  t j  c6  t j  1 c7  t j  1 c8 (n  1)
j 2 j 2 j 2
= an + b = (n) 59
Worst Case Analysis
• The array is in reverse sorted order “while i > 0 and A[i] > key”
• Always A[i] > key in while loop test
• Have to compare key with all elements to the left of the j-th position
 compare with j-1 elements  tj = j
n
n(n  1) n
n(n  1) n
n(n  1)
using j 1
j
2
  j 
j 2 2
 1   ( j  1) 
j 2 2
we have:

 n( n  1)  n( n  1) n( n  1)
T ( n ) c1n  c2 ( n  1)  c4 ( n  1)  c5   1  c6  c7  c8 ( n  1)
 2  2 2

an 2  bn  c a quadratic function of n

• T(n) = (n2) order

n
ofn
growth in nn
2

T (n) c1n  c2 (n  1)  c4 (n  1)  c5  t j  c6  t j  1 c7  t j  1 c8 (n  1)

j 2 j 2 j 2 60
Comparisons and Exchanges in Insertion
Sort

INSERTION-SORT(A) cost
for j ← 2 to n times
do key ← A[ j ] c1 n

Insert A[ j ] into the sorted sequence A[1 . . j -1] c2 n-

 n2/2 comparisons 1
i←j-1
 n-t
n
0 j 2 j
while i > 0 and A[i] > key
 (t
n
1 j 2 j  1)
do A[i + 1] ← A[i]2
 n-(t
n
 n /2 exchanges c4 j 2 j  1)
i←i–1 1
A[i + 1] ← key c5 61
Running time

The running time depends on the input: an

already sorted sequence is easier to sort.

Parameterize the running time by the size of the

input, since short sequences are easier to sort
than long ones.
Generally, we seek upper bounds on the
running time, because everybody likes a
guarantee.
Kinds of Analyses
• T(n) = maximum time of an
Worst-case: algorithm on any input of
(usually) size n.
Average- • T(n) = expected time of algorithm over
all inputs of size n.
case: • Need assumption of statistical
(sometimes) distribution of inputs.
• Cheat with a slow algorithm
Best-case: that works fast on some
input.
Loop invariants and the
correctness of Insertion sort
• Initialization: j=2 (prior to the 1st Iteration). Therefore, A[1….j-
1] = A[1]
//if only an array of single elements is always sorted
• Maintenance: Similar to mathematical Induction
• Line 5 to 8 -> find the correct position of A[j] and when to
placing A[j], A[1….j-1] is sorted.
• For a particular j; A[1….j-1] is sorted.
• While loop  A[j] in the correct position and A[1…..j-1] is now
sorted.
• At the beginning of the next iteration, j is j+1 (j is incremented)
• A[1….j] is sorted, which means loop Invariant holds.
Termination: The “for” loop terminates when j>n (i.e., j=n+1). The
subarray is A[1….n], by default, A[1……n] is in sorted order
Insertion Sort - Summary

• Advantages
• Good running time for “almost sorted”
arrays (n)
• Disadvantages
• (n2) running time in worst and average
case
•  n2/2 comparisons and exchanges
66