Unit-I

IntroductiontoAlgorithms
Prepared by,
Prof. Tarannum Kumasgi
Algorithm :
 An algorithm is any well-defined computational
procedure that takes some value as input and
produces some value as output in a finite amount of
time.
An algorithm is a sequence of computational steps
that transform the input into the output.
Needof Algorithm:
 To understand the problem
 Construction of the list of the variables
 To design the output
 Problem Development
 Testing the program
 Validating the program
 Characteristics/Properties of Algorithm

1. Definiteness (Unambiguous) − Algorithm should

be clear and unambiguous.
2. Input − An algorithm should have 0 or more well-
defined inputs.
3. Output − An algorithm should have 1 or more well-
defined outputs and should match the desired
output.
Properties of Algorithm contd..
4. Finiteness − If we trace out the instructions of an
algorithm, then for all cases the algorithm will
terminate after a finite number of steps.
5. Feasibility − It should be feasible with the available
resources.
6. Effectiveness: Every step must be accurate.
Don’t write unnecessary step which are not required.
 Expressing an Algorithm :
 We can express an algorithmin many ways,
including natural language, flowcharts, pseudocode,
actual programming languages.
 Natural language is a popular choice, since it
comes so naturally to us and can convey the steps
of an algorithm to a wide audience.
Expressing an Algorithm contd..
 When we're developing algorithms, we are often
working both with people who know programming
and those who don't—but they all know
natural language.
 Flowchart :
 “A flowchart is a graphical representation of an algorithm.
Programmers often useit as a program- planning tool to
solve a problem".
 It makes use of symbols which are connected among
them to indicate the flow of information and
processing.
 The process of drawing a flowchart for an
algorithm is known as “flowcharting”.
 Flowchart Symbols:
SymbolsUsedInFlowchart
Rules For Creating a Flowchart :
 We should follow some rules while creating a flowchart.
 Rule 1: Flowchart opening statement must be ‘start’ keyword.
 Rule 2: Flowchart ending statement must be ‘end’ keyword.
 Rule 3: All symbols in the flowchart must be connected with an
arrow line.
 Rule 4: Each decision point should have two or more distinct
outcomes.
 Rule 5: Flow should generally move from top to bottom or left
to right.
Example 1. Add two numbers entered by
the user.
Example 2.
Find the largest
among three
different
numbers entered
by the user.
Solve the following[Practice Examples].

1. Draw a flowchart to find the sum of first 100 natural

numbers.
2. Draw a flowchart to find the smallest of three numbers x, y
and z.
3. Draw flowchart for the problem of determining prime
number?
Algorithm Design Techniques :
1. Divide and Conquer Approach

2. Greedy Technique

3. Dynamic Programming

4. Branch and Bound

5. Randomized Algorithms

6. Backtracking Algorithm
1. Divide and Conquer Approach :
 It is a top-down approach. The algorithms which follow the
divide & conquer techniques involve three steps:
1) Divide the problem :Divide the original problem into a set of
subproblems.
2) Conquer the sub-problems :Solve every subproblem
individually, recursively.
3) Combine the solutions :Combine the solution of the
subproblems (top level) into a solution of the whole original
problem.
4) Example: Merge sort, Quicksort.
 2. Greedy Technique :
● Greedy method is used to solve the optimization problem.
● An optimization problem is one in which we are given a set of input
values, which are required either to be maximized or minimized
(known as objective), i.e. some constraints or conditions.
Areas of Application :
● Greedy approach is used to solve many problems, such as
Finding the shortest path between two vertices using Dijkstra’s
algorithm.
● Finding the minimal spanning tree in a graph using Prim’s
/Kruskal’s algorithm, etc.
 3. Dynamic Programming :
• Dynamic Programming is a bottom-up approach.
• The approach of Dynamic programming is similar to
divide and conquer.

• The difference is that whenever we have recursive

function calls with the same result, instead of calling
them again we try to store the result in a data structure
in the form of a table and retrieve the results from the
table.
3. Dynamic Programming contd.. :
• Thus, the overall time complexity is reduced.
“Dynamic” means we dynamically decide, whether
to call a function or retrieve values from the table.

• Example: 0-1 Knapsack, subset-sum problem

 4.Branch andBound:
• This technique is very useful in solving combinatorial
optimization problem that have multiple solutions and we
are interested in find the most optimum solution.
• In this approach, the entire solution space is represented
in the form of a state space tree.
• As the program progresses each state combination is
explored, and the previous solution is replaced by new
one if it is not the optimal than the current solution.
• Example: Job sequencing, Travelling salesman problem.
 5. Randomized Algorithms :
● An algorithm that uses random numbers to decide
what to do next anywhere in its logic is called
Randomized Algorithm.
● For example, in Randomized Quick Sort, we use
random number to pick the next pivot (or we
randomly shuffle the array).
● Typically, this randomness is used to reduce time
complexity or space complexity
 6.BacktrackingAlgorithm:
● Backtracking Algorithm tries each possibility until they
find the right one. It is a depth-first search of the set
of possible solution.
● This technique is very useful in solving combinatorial
problems that have a single unique solution.
● During the search, if an alternative doesn't work, then backtrack to the choice
point, the place which presented different alternatives, and tries the next
alternative.
● Example: N-queen problem, maize problem.
Performance Analysis of Algorithms :
● If we want to go from city "A" to city "B", there can be
many ways of doing this. We can go by flight, by bus,
by train and also by bicycle. Depending on the
availability and convenience, we choose the one
which suits us.
● Similarly, in computer science, there are multiple
algorithms to solve a problem.
● When we have more than one algorithm to solve a
problem, we need to select the best one.
Performance Analysis of Algorithms :
● Performance analysis helps us to select the
best algorithm from multiple algorithms to
solve a problem.

● When there are multiple alternative

algorithms to solve a problem, we analyze
them and pick the one which is best suitable for
our requirements.
 Definition :
● Performance of an algorithm is a processof
making evaluative judgement about algorithms.

● Performance of an algorithm means

predicting the resources which are required to an
algorithm to perform its task.
● Generally, the performance of an algorithm depends on the
following elements...
Whether that algorithm is providing the exact solution for the problem?
Whether it is easy to understand?
Whether it is easy to implement?
How much space (memory) it requires to solve the problem?
How much time it takes to solve the problem? Etc.,

Performance analysis of an algorithm is the process of

calculating space and time required by that algorithm.
 Time Complexity:
● It is function describing the amount of time an
algorithm takes in term of the amount of input to the
algorithm.i.e. it signifies the total time required by the
program to run till its completion.
● “Time can mean the number of memory access
performed, the number of comparisons between integers,
the number of times some inner loop is executed, or some
other natural unit related to the amount of real time the
algorithm will take.
Space Complexity
● It is function describing the amount of memory (Space) an
algorithm takes in term of the amount of input to the algorithm
i.e Space complexity of an algorithm is total space taken by the
algorithm with respect to the input size.
● Instruction space: Instruction space is the space needed to
store the compiled version of the program instructions.
● Data space: Data space is the space needed to store all constant
and variable values.
● Environment stack space: The environment stack is used to save
information needed to resume execution of partially completed
functions.
 ASymptotic Notation :
● Asymptotic Notations are the expressions that are
used to represent the complexity of an algorithm.
● Types of Data Structure Asymptotic Notation

1. Big-O Notation (Ο)

2. Omega Notation (Ω)
3. Theta Notation (θ)
Types of Asymptotic Notation :
● Big-O Notation (Ο) :
Big O notation specifically describes worst case scenario. It represents the upper bound running time
complexity of an algorithm.
O(g(n)) = { f(n): there exist positive constants c and n0 such that 0 <= f(n) <= c*g(n) for

all n >= n0}

● Omega Notation (Ω):
Omega notation specifically describes best case scenario. It represents the lower
bound running time complexity of an algorithm.

Ω (g(n)) = {f(n): there exist positive constants c and n0

such that 0 <= c*g(n) <= f(n) for all n >= n0}.
● Theta Notation (θ):
This notation describes both upper bound and lower bound of an algorithm so we
can say that it defines exact asymptotic behaviour.

Θ(g(n)) = {f(n): there exist positive constants c1, c2

and n0 such that 0 <= c1*g(n) <= f(n) <= c2*g(n) for all
n >= n0}
 AnalysisofAlgorithm
● The analysis of algorithm is to compare the various algorithms
to solve a same problem. This is done to analyse which
algorithm takes less resources such as time, effort and memory
to solve a particular problem.
 Types of analysis of algorithm :
To analyze a particular algorithm, we need to understand for which
input the algorithm takes less time and for which input it takes more
time. Based on this, we divide the inputs in three cases:
1. Best case where we assume the input, for which algorithm
takes less time.
2. Worst case where we assume the input, for which algorithm
takes long time.
3. Average case where the input lies in between best and worst
case.
Order of Growth :
● An order of growth is a set of functions whose
asymptotic growth behavior is considered
equivalent.
● For example, 2n, 100n and n + 1 belong to the
same order of growth, which is written O(n) in
Big-Oh notation and often called linear because
every function in the set grows linearly with n.
OrderofGrowth
MergeSort(arr[],l,r)
If r > l
1. Find the middle point to divide the array into two halves:
middle m = l+ (r-l)/2
2.Call mergeSort for first half: Call
mergeSort(arr, l, m)
3. Call mergeSort for second half: Call
mergeSort(arr, m+1, r)
4. Merge the two halves sorted in step 2 and 3: Call
merge(arr, l, m, r)
O(1)- Big O notation O(1) represents the complexity of an algorithm that always
execute in same time or space regardless of the input data.
O(1) example: The following step will always execute in same time(or space)
regardless of the size of input data.
Accessing array index(int num = arr[5])

O(n) - Big O notation O(N) represents the complexity of an algorithm, whose

performance will grow linearly (in direct proportion) to the size of the input
data.
O(n) example: The execution time will depend on the size of array. When the size of
the array increases, the execution time will also increase in the same proportion
(linearly)
Traversing an array

O(n^2)- Big O notation O(n^2) represents the complexity of an algorithm, whose

performance is directly proportional to the square of the size of the input data.
O(n^2) example: Traversing a 2D array

logarithmic growth O(log n), log-linear growth O(n log n), exponential growth
O(2^n) and factorial growth O(n!).
O(1)<O(logn)<O(n)<O(nlogn)<O(n^2)<O(n^3)<O(2^n)<O(n!)

Complexity----->

Input Data----->
Recursion :
Some computer programming languages allow a module or
function to call itself. This technique is known as recursion.
Properties of Recursion :
A recursive function can go infinite like a loop. To avoid infinite
running of recursive function, there are two properties that a
recursive function must have −

● Base criteria −There must be at least one base criteria or

condition, such that, when this condition is met the
function stops calling itself recursively.

● Progressive approach −The recursive calls should progress

in such a way that each time a recursive call is made it
comes closer to the base criteria.
Implementation of Recursion :
Analysis of Recursion :
Time Complexity :
● A call made to a function is Ο(1), hence the (n) number of
times a recursive call is made makes the recursive function Ο(n).
Space Complexity :

● Space complexity is counted as what amount of extra space is

required for a module to execute.
● space complexity of recursive function may go higher than that
of a function with iteration.
Recurrence Relations :
● Recurrence : A new occurrence of something that happened or
appeared before : a repeated occurrence.
● A Recurrence Relation is a functional relation between the
independent variable x, dependent variable f(x) and the
differences of various order of f (x).
● The simplest form of a recurrence relation is the case where the next
term depends only on the immediately previous term.
● Recurrence relations are used to reduce complicated problems to an
iterative process based on simpler versions of the problem.
● An example problem in which this approach can be used is the Tower
of Hanoi puzzle.
Recurrence Relation Methods :
Example : The Worst Case Running Time T(n) of the MERGE SORT
Procedures is described by the recurrence.
T (n) = θ (1) if n=1
2T(n/2)+θ (n) if n>1
We have to show that it is asymptotically bound by O(nlog n)(Best Case).
There are four methods for solving Recurrence:
1. Substitution Method
2. Iterative Method
3. Recursion Tree Method
4. Master Method
Recurrence Relation Merge Sort :
1. Merge Sort is a Divide and Conquer algorithm.
2. It divides the input array into two halves, calls the
itself for two halves, and then merges the two
sorted halves.
3. The merge() function is used for merging two halves.
4. The merge(arr, l, m, r) is a key process that assumes that arr[l..m]
and arr[m+1..r] are sorted and merges the two sorted subArrays
into one.
 The operation of merge sort on the
array A with length 8 that initially
contains the
 sequence (12,3,7, 9, 14, 6, 11, 2).
The indices p, q, and r into each
subarray appear above their values.
 Numbers in italics indicate the order
in which the MERGE-SORT and
MERGE procedures are called
following the initial call of MERGE-
SORT.(A, 1, 8).
Analysis of Merge sort :
1. Substitution method for solving
recurrences :
The substitution method comprises two steps:
1. Guess the form of the solution using symbolic
constants.
2. Verify mathematical induction to show that the
solution works.
3. Solve for the constants.
 We can use the substitution method to establish
either an upper or a lower bound on a recurrence.
 It’s usually best not to try to do both at the same
time. That is, rather than trying to prove a ‚Theta-
bound directly, first prove an O-bound, and then
prove an Omega-bound.
2. Recursion Tree Method :
 The Recursion Tree Method is a way of
solving recurrence relations.
 In this method, a recurrence relation is converted
into recursive trees. Each node represents the cost
incurred at various levels of recursion. To find the
total cost, costs of all levels are summed up.
Steps to solve recurrence relation using recursion
tree method:
1. Draw a recursive tree for given recurrence relation.
2. Calculate the cost at each level and count the total no
of levels in the recursion tree.
3. Count the total number of nodes in the last level and
calculate the cost of the last level
4. Sum up the cost of all the levels in the recursive tree
Note: If summing up all the levels becomes complex,
we can find an upper bound by considering a
perfectly full tree and / or an infinite geometrical
series (the ratio is typically less than 1).
Example 1
W(n) = 2W(n/2) + n2

Subproblem size at level i is: n/2i

Subproblem size hits 1 when 1 = n/2i  i = lgn
Cost of the problem at level i = (n/2i)2 No. of nodes at level i = 2i
Total cost: W (n)  lg n1n 2  2lg n W (1)  n 2 lg n1 1   n  n 2   1   O(n) n 2
i i
1
 i i 0 2
 
i 0 2
 
i 0 2 1 1
 O ( n)  2 n 2
2

 W(n) = O(n2) 58
Example 2
E.g.: T(n) = 3T(n/4) + cn2

• Subproblem size at level i is: n/4i

• Subproblem size hits 1 when 1 = n/4i  i = log4n
• Cost of a node at level i = c(n/4i)2
• Number of nodes at level i = 3i  last level has 3log4n = nlog43 nodes
• Total cost:
log4 n 1 i

     
 i
3 2 3 2 1
T ( n)     cn   n 4     cn   n 4 
 16 
log 3

i 0  16 
log 3
3
cn 2   n log4 3  O(n 2 )
i 0
1
16
 T(n) = O(n ) 2 59
Example 2 - Substitution
T(n) = 3T(n/4) + cn2
Guess: T(n) = O(n2)
◦ Induction goal: T(n) ≤ dn2, for some d and n ≥ n0
◦ Induction hypothesis: T(n/4) ≤ d (n/4)2

Proof of induction goal:

T(n) = 3T(n/4) + cn2

≤ 3d (n/4)2 + cn2

= (3/16) d n2 + cn2

≤ d n2 if: d ≥ (16/13)c

Therefore: T(n) = O(n2)

60
Example 3 (simpler proof)
W(n) = W(n/3) + W(2n/3) + n

• The longest path from the root

to a leaf is:
n  (2/3)n  (2/3)2
n…1

• Subproblem size hits 1 when

1 = (2/3)in  i=log3/2n

• Cost of the problem at level i =

n lg n
W (n)  n  n  ...  n(log 3/ 2 n)  n  O(n lg n)
• Total cost: 3
lg
2
61
Example 3
W(n) = W(n/3) + W(2n/3) + n

• The longest path from the root

to a leaf is:
n  (2/3)n  (2/3)2
n…1

• Subproblem size hits 1 when

1 = (2/3)in  i=log3/2n

• Cost of the problem at level i =

(log3 / 2 n ) 1
n W (n)  n  n  ...  i 0
n  2(log3 / 2 n )W (1) 

•
log3 / 2 n
lg n 1
n  1  nlog
Total cost: 3/ 2 2
 n log3/ 2 n  O(n)  n  O(n)  n lg n  O(n)
i 0 lg 3/ 2 lg 3/ 2
62
 Example 3 - Substitution
W(n) = W(n/3) + W(2n/3) + O(n)

Guess: W(n) = O(nlgn)

◦ Induction goal: W(n) ≤ dnlgn, for some d and n ≥ n0
◦ Induction hypothesis: W(k) ≤ d klgk for any K < n (n/3, 2n/3)

Proof of induction goal:

Try it out as an exercise!!

T(n) = O(nlgn)

63
3. Master Method / Master Theorem :
The Master Theorem is a tool used to solve recurrence
relations that arise in the analysis of divide-and-
conquer algorithms. The Master Theorem provides a
systematic way of solving recurrence relations of the
form:
T(n) = aT(n/b) + f(n)
where n = size of the problem ,a = number of subproblems in the recursion and a >= 1
n/b = size of each subproblem ,b > 1, k >= 0 and p is a real number.
 Heap Sort :
• The (binary) heap data structure is an
array object that we can view as a nearly
complete binary tree, as shown in Figure.
• Each node of the tree corresponds to an
element of the array. The tree is
completely filled on all levels except
possibly the lowest, which is filled from
the left up to a point.
● Length [A],number of elements in array .
● Heap-Size[A], number of elements in a heap stored within
array A.
● The root of tree A [1] and gives index 'i' of a node that indices of
its parent(i), left child(i), and the right child(i) can be computed.
● PARENT (i)
return (i/2)
LEFT (i)
return 2i

RIGHT (i)
return 2i+1
Representation of an array
The index of 20 is 1
To find the index of the left child, we calculate 1*2=2

This takes us (correctly) to the 14.

Now, we go Right of (index 2), so we calculate 2*2+1=5

This takes us (again, correctly) to the 6.

Now, 4's index is 7, we want to go to the parent, so we

calculate 7/2 =3 which takes us to the 17.
Heap Property:
• There are two kinds of binary heaps:
1. Max Heap
2. Min-Heap
1. Max Heap:
● In a Binary Heap, for every node i
other than the root, the value of the
node is greater than or equal to the
value of its highest child

A [PARENT (i) ≥A[i]

● Thus, the highest element in a heap is
stored at the root.
 2.MIN-HEAP:
● In MIN-HEAP, the value of
the node is lesser than or
equal to the value of its
lowest child.

A [PARENT (i) ≤A[i]

● Thus, the s m a l l e s t
element in a heap is
stored at the root.
Heapify
• Heapify is the process of creating a heap data structure from a
binary tree. It is used to create a Max-Heap. heapify(array).

• Its inputs are an array A with the heap-size attribute and an index i
into the array. When it is called, MAX-HEAPIFY assumes that the
binary trees rooted at LEFT(i) and RIGHT(i) are max-heaps.

• MAX-HEAPIFY lets the value at A[i] “float down” in the max-heap

so that the subtree rooted at index i obeys the maxheap property.
Build Heap
We can use the procedure 'Heapify' in a bottom-up fashion to
convert an array A[1 . . n] into a heap.
Since the elements in the subarray A[n/2 +1 . . n] are all leaves,
the procedure BUILD_HEAP goes through the remaining nodes of
the tree and runs 'Heapify' on each one.
 The bottom-up order of processing node guarantees that the
subtree rooted at children are heap before 'Heapify' is run at their
parent.
Buildheap(A)
{
heapsize[A] length[A]
for i |length[A]/2 //down to 1
do Heapify(A, i)
}
Heap Sort Algorithm :
The heap sort algorithm starts by using procedure BUILD-HEAP to build a heap
on the input array A[1 . . n]. Since the maximum element of the array stored at the
root A[1], it can be put into its correct final position by exchanging it with A[n]
(the last element in A). If we now discard node n from the heap than the remaining
elements can be made into heap. Note that the new element at the root may violate
the heap property. All that is needed to restore the heap property.
Heapsort(A)
{
Buildheap(A)
for i  length[A] //down to 2
do swap A[1]  A[i]
heapsize[A]  heapsize[A] - 1
Heapify(A, 1)
}
How to heapify root element when its subtrees are already max
heaps??
Example:
1. Let the input array be

2. Create a complete binary tree from the array

3.Start from the first index of non-leaf node whose index
is given by n/2 - 1.
4. Set current element i as largest.

5. The index of left child is given by 2i + 1 and the right child is

given by 2i + 2.

6. If leftChild is greater than currentElement (i.e. element at ith

index), set leftChildIndex as largest.

7. If rightChild is greater than element in largest, set

rightChildIndex as largest.

8. Swap largest with currentElement

Repeat steps 3-7 until the subtrees are also heapified.
ExampleMaxHeap
// Build heap (rearrange array)
for (int i = n / 2 - 1; i >= 0; i--)

heapify(arr, n, i);
As shown in the
diagram, we start by
heapifying the lowest
smallest trees and
gradually move up
until we reach the root
element.
Working of Heap Sort
1.Since the tree satisfies Max-Heap property, then the largest
item is stored at the root node.
2.Swap: Remove the root element and put at the end of the
array (nth position) Put the last item of the tree (heap) at the
vacant place.
3.Remove: Reduce the size of the heap by 1.
4.Heapify: Heapify the root element again so that we have
the highest element at root.
5.The process is repeated until all the items of the list are
sorted.
Algorithm: Heapsort(A)
BUILD-MAX-HEAP(A)
for i = A. length downto 2
exchange A[1] with A[i]
A.heap-size = A.heap-size - 1
MAX-HEAPIFY(A, 1)
Algorithm:
// Heap sort
for (int i = n - 1; i >= 0; i--) {
swap(&arr[0], &arr[i]);

// Heapify root element to get highest element at root

again
heapify(arr, i, 0);
}
ComplexityAnalysisofHeapSort:
● Worst,Best,Average Case Time Complexity: O(n*log n)
➢ The height of a complete binary tree containing n
elements is log n
➢ need to move an element from the root to the leaf node

making a multiple of log(n) comparisons and swaps.

➢ build_max_heap stage, we do that for n/2 elements so the
worst case complexity of the build_heap step is n/2*log
n ~ nlog n
➢ Also since the build_max_heap and heap_sort steps are
executed one after another, the algorithmic complexity
is not multiplied and it remains in the order of nlog n.
● Space Complexity : O(1)
Time Analysis
Build Heap Algorithm will run in O(n) time
There are n-1 calls to Heapify each call requires O(log n)
time
Heap sort program combine Build Heap program and
Heapify, therefore it has the running time of O(n log n) time
Total time complexity: O(n log n)
HeapSortApplications:
● Systems concerned with security and embedded systems
such as Linux Kernel use Heap Sort because of the O(n
log n) upper bound on Heap Sort running time and
constant O(1) upper bound on its auxiliary storage.

● Although Heap Sort has O(n log n) time complexity even

for the worst case, it doesn't have more applications (
compared to other sorting algorithms like Quick Sort,
Merge Sort ).
Thank You