Algorithms and Problem Solving (15B11CI411)

EVEN 2022

Module 1: Lecture 5-6

Jaypee Institute of Information Technology (JIIT)

A-10, Sector 62, Noida
Outline
Review of Searching Techniques and analyzing their time complexity
• Linear Search and its analysis
• Binary Search and its analysis
• Median Search and its analysis
Review of Sorting Techniques and analyzing their time complexity
• Selection Sort, Bubble Sort and Insertion Sort analysis
• Merge Sort and its analysis
• Quick Sort and its analysis
• Heap Sort and its analysis
• Radix Sort and Count Sort Analysis

2
Searching Techniques

3
Median Search
• The median of a list of N values is found by sorting the input array in increasing
order, and taking the middle value.
• This will take O(n log n) time in average/best case.
• Median search algorithm is mostly used to find kth smallest/kth largest element
in a given set of values.
procedure select(k,S) \\S is a list of numerical values and k is the rank of the kth smallest element in S
if |S|=1 then return single element in S
else
choose an element ‘a’ randomly from S;
let S1, S2, S3 be the sequences of elements in S respectively less, equal to, and greater than a;
if (|S1| >= k) then return select(k,S1); Condition 1
else if (|S1|+|S2| >= k) then return a; Condition 2
else return select(k-|S1|-|S2|, S3); Condition 3

4
 7 10 4 3 20 15 8 12 6

Find the 4th median (k=4)

Randomly choose an element a = 10

Now divide the array into three part S1, S2, S3
S1= {7,4,3,8,6}
S2={10}
S3={20,15,12}
Now |S1|>= k; first condition true  select(4,S1) \\select(k, S1)
Check in the array S1={7,4,3,8,6}
Randomly choose element a = 3
Now divide the array S1 into three parts S11,S12,S13
S11={}
S12={3}
S13={7,4,8,6}
Here first and second condition fail third condition true  select(3, S13) \\select(k-|S11|-|S12|,S13)

5
 7 10 4 3 20 15 8 12 6

Here k=3
S13= {7,4,8,6}
Randomly choose a=7
Now divide S13 into three parts such as: S131,S132,S133
S131={4,6}
S132={7}
S133={8}
Here condition 1 fails
Condition 2 true as |S131| + |S132| >= K
Return median a =7

6
Recursive Implementation
#include<iostream> if(j>=k)
using namespace std; {
int median(int A[],int l,int r,int k) median(S1,0,j-1,k);
{ }
int ind = r-l+1; else if((j+n)>=k)
int index = rand() % ind; return A[index];
int i,j=0,m=0,n=0; else
int S1[10],S3[10],S2[1]; median(S3,0,m-1,k-(j+n));
for(i=0;i<=r;i++) }
{
if(A[i]<A[index]) // Driver program to test above methods
{ int main()
S1[j]=A[i]; {
j++; int arr[] = {7, 10, 4, 3, 20, 15, 8, 12,6};
} int n = sizeof(arr)/sizeof(arr[0]), k = 4;
int element=median(arr, 0, n-1, k);
else if(A[i]==A[index]) cout << "K'th smallest element is "<<element;
{ return 0;
S2[n]=A[i]; }
n++;
}
else
{
S3[m]=A[i];
m++;
} }
7
Complexity Analysis of Median Search
Best Case
• Condition 2 is met in the first call to median search. No further recursive calls are made.
• T(n)= O(n)

Worst Case
• Condition 1 or 3 are met (recursive call for any one subpart S1 or S3).
• But it is possible that the subpart S1 or S3 contains n-1 elements.
• T(n)= T(n-1)+n , On solving T(n)= O(n2)
Sorting Techniques

9
Selection Sort : Review
• The selection sort algorithm sorts an array by repeatedly finding the minimum element (considering
ascending order) from unsorted part and putting it at the beginning. The algorithm maintains two
subarrays in a given array.

1)The subarray which is already sorted.

2) Remaining subarray which is unsorted.

• In every iteration of selection sort, the minimum element (considering ascending order) from the
unsorted subarray is picked and moved to the sorted subarray.

10
 Time Complexity Analysis:
void selectionSort(int arr[], int n) Two nested loops with no break statements:
{
int i, j, min_idx; Outer loop runs n-1 times:
Inner loop runs as follows:
// One by one move boundary of unsorted subarray
for (i = 0; i < n-1; i++) Outerloop Inner loop runs :
{ 1st run n-1
// Find the minimum element in unsorted array 2nd run n-2
min_idx = i;
3rd run n-3
for (j = i+1; j < n; j++)
if (arr[j] < arr[min_idx]) . .

min_idx = j; . .
n-1th run 1
// Swap the found minimum element with the first element
swap(&arr[min_idx], &arr[i]);
} Total time complexity: (Worst, Best, Average all cases)
} (n-1) + (n-2) + (n-3) + ….. + 1 = O(n2)

11
 Bubble Sort : Review
• Bubble Sort is the simplest sorting algorithm that works by repeatedly swapping the adjacent elements if they
are in wrong order.

void bubbleSort(int arr[], int n)

{
int i, j; Time Complexity Analysis:
bool swapped;
for (i = 0; i < n-1; i++) Worst and Average Case Time Complexity: O(n2). Worst case
{ occurs when array is reverse sorted.
swapped = false;
for (j = 0; j < n-i-1; j++)
Best Case Time Complexity: O(n).
{ Best case occurs when array is already sorted.
if (arr[j] > arr[j+1]) The for nested loop breaks when the array is already sorted.
{
swap(&arr[j], &arr[j+1]);
swapped = true;
}
}
// IF no two elements were swapped by inner loop, then break
if (swapped == false)
break;
}
} 12
 Insertion Sort
• Insertion sort is a simple sorting algorithm that works similar to the way you sort playing cards in your
hands. The array is virtually split into a sorted and an unsorted part. Values from the unsorted part are
picked and placed at the correct position in the sorted part.

void insertionSort(int arr[], int n) Time Complexity Analysis:

{
int i, key, j; Outer for loop runs n-1 times always:
for (i = 1; i < n; i++)
{ While loop inside the for loop runs as follows:
key = arr[i];
j = i - 1; Best case: while loop doesn’t run at all (arr[j]<key)
Only one comparison statement executed
while (j >= 0 && arr[j] > key)
{ Worst case: while loop runs 1 time in 1st pass of for loop , 2 times in 2nd pass of
arr[j + 1] = arr[j]; for loop and n-1 times in last pass of for loop.
j = j - 1;
}
arr[j + 1] = key;
} Hence, Best case complexity of Insertion Sort : O(n)
} Average and Worst case complexity of Insertion Sort : O(n 2)
13
 Divide-and-Conquer Based Sorting Techniques

Divide and conquer is a strategy to solve a large problem as follows:

• Divide the problem into a number of sub-problems

– Similar sub-problems of smaller size

• Conquer the sub-problems

– Solve the sub-problems recursively

– Lowest level Sub-problem size is small enough to solve it in straight forward manner

• Combine the solutions of the sub-problems

– Obtain the solution for the original problem

14
14
 Merge Sort : REVIEW

• Merge sort work on a principle of divide-and-conquer algorithm.

• Merge sort repeatedly breaks down an array into several sub array
until each sub array consists of a single element
• Merge the sub-problem solutions together:
• Compare the sub-array’s first elements
• Remove the smallest element and put it into the result array
• Continue the process until all elements have been put into the
result array

15
Merge Sort Algorithm
INPUT: a sequence of n numbers stored in array A
OUTPUT: an sorted sequence of n numbers
l points first element index of array and r points the
index of last element

MergeSort (A, p, r) // sort A[p..r] by divide & conquer

❑ Check for base case
if p< r
then q (p+r)/2 ❑ Divide

MergeSort (A, p, q) ❑ Conquer

MergeSort (A, q+1, r) ❑ Conquer

Merge (A, p, q, r) // merges A[p..q] with A[q+1..r]
❑ Combine

Initial Call: MergeSort(A, 1, n)

16
 Merge(A, p, q, r) Procedure Merge
n1 = q – p + 1
n2 = r – q Input: Array containing sorted subarrays A[p..q] and A[q+1..r].
for i = 1 to n1 Output: Merged sorted subarray in A[p..r].
do L[i] = A[p + i – 1]
for j = 1 to n2
do R[j] = A[q + j]
L[n1+1] = ∞
R[n2+1] = ∞
i=1
O(n)
j=1
for k =p to r Sentinels, to avoid having to check if either subarray is fully copied at
each step.
do if L[i] ≤ R[j]
then A[k] = L[i]
i=i+1
else A[k] = R[j]
j=j+1
Source: Cormen, Thomas H.; Leiserson, Charles E.; Rivest, Ronald L.; Stein, Clifford (2009) [1990]. Introduction to Algorithms (3rd ed.).
MIT Press and McGraw-Hill. ISBN 0-262-03384-4. 1320 pp 17
Time Complexity Analysis of Merge Sort
MergeSort (A, p, r) // sort A[p..r] by divide & conquer
1 if p < r
then q = floor((p+r)/2)
O(1)
2
T(n/2)
3 MergeSort (A, p, q) O(n)
T(n/2)
4 MergeSort (A, q+1, r)
5 Merge (A, p, q, r) // merges A[p..q] with A[q+1..r] `

So, we can write the overall running time of MERGE-SORT function in the form of recurrence relation as:

T(n)=2*T(n/2)+ Θ(n)+ Θ(1)

So Solving above equation using any of the method: T(n)= Θ(nlog2n)

Best case , Worst Case and average case : O(nlog2n)

18
 Quicksort Algorithm: Review

Given an array of n elements (e.g., integers):

• If array contains only one element, return
• Else
– pick one element to use as pivot.
– Partition elements into two sub-arrays:
• Left sub-array contains elements less than or equal to pivot
• Right sub-array contains elements greater than pivot
– Then repeatedly apply above two steps on the two sub-arrays until array become sorted
– Return results

19
 Partitioning Array
Given a pivot, partition the elements of the array such that the resulting array ,consists of:
1. One sub-array that contains elements > pivot
2. Another sub-array that contains elements <= pivot

The sub-arrays are stored in the original data array.

Partitioning loops through, swapping elements below/above pivot.

20
 Quicksort Code:

void quicksort(int a[], int p, int r)

{
if(p < r)
{ int q; O(n)
q = partition(a, p, r);
T(i)
quicksort(a, p, q-1);
T(n-i-1)
quicksort(a, q+1, r);
}
}

Source: Cormen, Thomas H.; Leiserson, Charles E.; Rivest, Ronald L.; Stein, Clifford (2009) [1990]. Introduction to Algorithms (3rd ed.).
MIT Press and McGraw-Hill. ISBN 0-262-03384-4. 1320 pp 21
 int partition (int arr[], int low, int high)
{

int pivot = arr[high]; // selecting last element as pivot

int i = (low - 1); // index of smaller element
for (int j = low; j <= high- 1; j++)
{
// If the current element is smaller than or equal to pivot
if (arr[j] <= pivot)
{ i++; // increment index of smaller element
swap(&arr[i], &arr[j]);
}
}
swap(&arr[i + 1], &arr[high]);
return (i + 1);
}

Source: Cormen, Thomas H.; Leiserson, Charles E.; Rivest, Ronald L.; Stein, Clifford (2009) [1990]. Introduction to Algorithms (3rd ed.).
MIT Press and McGraw-Hill. ISBN 0-262-03384-4. 1320 pp 22
Running time analysis
❑ The advantage of this quicksort is that we can sort “in-place”, i.e., without the need for a
temporary buffer depending on the size of the inputs.

❑ Partitioning Step: Time Complexity is O(n).

❑ Recall that quicksort involves partitioning, and 2 recursive calls. Thus, giving the basic
quicksort relation:
T(n) = O(n) + T(i) + T(n-i-1) = cn+ T(i) + T(n-i-1)
where i is the size of the first sub-block after partitioning.

We shall take T(0) = T(1) = 1 as the initial conditions.

To find the solution for this relation, we’ll consider three cases:
1. The Worst-case
2. The Best-case
3. The Average-case
All above cases depends on the value of the pivot!!
23
Running time analysis
Worst-Case (Data is sorted already)
■ When the pivot is the smallest (or largest) element at partitioning on a block
of size n, the result
◆ yields one empty sub-block, one element (pivot) in the “correct” place and
one sub-block of size (n-1)
◆ takes O(n) times.
■ Recurrence Equation:
T(1) = 1
T(n) = T(n-1) + cn
Solution: O(n2 )
Worse than Mergesort!!!

24
Running time analysis
Best case:
■ The pivot is in the middle (median) (at each partition step), i.e. after each partitioning,
on a block of size n, the result
◆ yields two sub-blocks of approximately equal size and the pivot element in the
“middle” position
◆ takes n data comparisons.
■ Recurrence Equation becomes
T(1) = 1
T(n) = 2T(n/2) + cn
Solution: O(n logn)
Comparable to Mergesort!!

25
Running time analysis
Average case:

The average case input would be any randomly generated arrays and the pivot is at the random position every step.

It turns out the average case running time also is O(n logn).
(Detailed discussion out of scope in this course)

26
Different Comparison based sorting algorithms
Sorting Worst-case Best-case Space
Algorithm time time overhead
Selection sort Ο (n2) Ο(n2) Ο(1)

Bubble Sort Ο (n2) Ο (n) Ο(1)

Insertion Ο (n2) Ο (n) Ο(1)

Sort
Merge Sort Ο (n log n) Ο (n log n) Ο(n)

Quick Sort Ο (n2) Ο (n log n) Ο(1)

27
Heap Sort : REVIEW
A sorting algorithm that works by first organizing the data to be sorted into
a special type of binary tree called a heap

Procedures on Heap:
• Heapify
• Build Heap
• Heap Sort
• Heapify() picks the largest child key and compare it to the parent key. If parent key is larger
than heapify quits, otherwise it swaps the parent key with the largest child key. So that the
parent is now becomes larger than its children.
void heapify(int arr[], int N, int i)
T(n) <= T(2n/3) + O(1) = O(logn)
{ int largest = i;
int left = 2 * i + 1; For a complete binary tree of height h, number of
nodes is f(h) = 2^(h+1) - 1.
int right = 2 * i + 2; We have nearly complete binary tree with bottom
if (left < N && arr[left] > arr[largest]) half full.
If height of original tree is h, then height of left
largest = left;
is h - 1 and right is h - 2.
if (right < N && arr[right] > arr[largest]) n = 1 + f(h-1) + f(h-2)
largest = right; f(h-2) = 2^(h-1) - 1 = (f(h-1) - 1)/2
So, n = 1 + f(h-1) + (f(h-1) - 1)/2
if (largest != i) { = 1/2 + 3*f(h-1)/2
swap(&arr[i], &arr[largest]); f(h-1) = (2*n-1)/3
Which when n approaches to high values can be
heapify(arr, N, largest); }
written as O(2n/3).
}
• We can use the procedure 'Heapify' in a bottom-up fashion to convert an array
A[1 . . n] into a heap. Since the elements in the subarray A[n/2 +1 . . n] are all
leaves, the procedure buildheap() goes through the remaining nodes of the tree
and runs 'Heapify' on each one. The bottom-up order of processing node
guarantees that the subtree rooted at children are heap before 'Heapify' is run at
their parent.

void buildheap(int arr[], int N) T(n) = O(n)

{
for (int i = N / 2 - 1; i >= 0; i--)
heapify(arr, N, i);
}
• The heap sort algorithm starts by using procedure buildheap() to build a heap on
the input array A[1 . . n]. Since the maximum element of the array stored at the
root A[1], it can be put into its correct final position by exchanging it with A[n]
(the last element in A). If we now discard node n from the heap than the
remaining elements can be made into heap. Note that the new element at the
root may violate the heap property. All that is needed to restore the heap
property.

void heapSort(int arr[], int N) T(n) = O(nlogn)

{
buildheap(int arr[], int N)
for (int i = N - 1; i >= 0; i--) {
swap(&arr[0], &arr[i]);
heapify(arr, i, 0); }
}
 Time Complexity Analysis
• Build Heap Algorithm will run in O(n) time
• There are n-1 calls to Heapify each call requires O(log n) time
• Heap sort program combine Build Heap program and Heapify,
therefore it has the running time of O(n log n) time
• Total time complexity: O(n log n)
Counting Sort
• Counting sort is a sorting algorithm that sorts the elements of an array by
counting the number of occurrences of each unique element in the array.
• The count is stored in an auxiliary array and the sorting is done by mapping
the count as an index of the auxiliary array.
How Counting Sort Works?
1. Find out the maximum element (let it be max) from the given array:

33
2. Initialize an array of length max+1 with all elements 0. This array is used for storing
the count of the elements in the array.

3. Store the count of each element at their respective index in count array. For
example: if the count of element 3 is 2 then, 2 is stored in the 3rd position
of count array. If element "5" is not present in the array, then 0 is stored in 5th
position.

34
4. Store cumulative sum of the elements of the count array. It helps in placing the
elements into the correct index of the sorted array.

5. Find the index of each element of the original array in the count array. This gives the
cumulative count. Place the element at the index calculated as shown in figure on next
slide

35
 \\ Find the index of each element of the original array
in count array
\\ place the elements in output array
i = size – 1
while i >= 0:
output[count[array[i]] - 1] = array[i]
count[array[i]] -= 1
i -= 1

6. After placing each element at its correct position, decrease its count by one.

36
 Counting Sort Algorithm
countingSort(array, size)
max <- find largest element in array
initialize count array with all zeros
for j <- 0 to size
find the total count of each unique element and store the count at jth index
in count array
for i <- 1 to max
find the cumulative sum and store it in count array itself
for j <- size down to 1
restore the elements to array decrease count of each element restored by 1

37
Counting sort analysis
• Time Complexity: O(n+k) where n is the number of elements in input array and
k is the range of input.
• Auxiliary Space: O(n+k)

38
 Radix Sort
• This sort is unusual because it does not directly compare any of the elements

• We instead create a set of buckets and repeatedly separate the elements into the buckets

• On each pass, we look at a different part of the elements.

• Assuming decimal elements and 10 buckets, we would put the elements into the bucket
associated with its units digit

• The buckets are actually queues so the elements are added at the end of the bucket

• At the end of the pass, the buckets are combined in increasing

39
 Radix Sort

• On the second pass, we separate the elements based on the “tens” digit, and on the
third pass we separate them based on the “hundreds” digit

• Each pass must make sure to process the elements in order and to put the buckets
back together in the correct order.

• Radix sort use counting sort in the intermediate steps.

40
void radixsort(int array[], int size)
{ // Get maximum element
int max = getMax(array, size);
// Apply counting sort to sort elements based on place value.
for (int place = 1; max / place > 0; place *= 10)
countingSort(array, size, place);
}

void countingSort(int array[], int size, int place)

{
const int max = 10;
int output[size]; int count[max];
for (int i = 0; i < max; ++i)

count[i] = 0; // Calculate count of elements

for (int i = 0; i < size; i++)
count[(array[i] / place) % 10]++;
for (int i = 1; i < max; i++)
count[i] += count[i - 1];
for (int i = size - 1; i >= 0; i--)
{ output[count[(array[i] / place) % 10] - 1] = array[i];
count[(array[i] / place) % 10]--;
}
for (int i = 0; i < size; i++)
array[i] = output[i];
}
41
 Radix Sort Complexity Analysis

• Each element is examined once for each of the digits it contains, so if the elements have at
most M digits and there are N elements this algorithm has complexity O(M*N)

• This means that sorting is linear based on the number of elements.

• Though this is a very time efficient algorithm it is not space efficient

• If an array is used for the buckets and we have B buckets, we would need N*B extra
memory locations because it’s possible for all of the elements to wind up in one bucket

• If linked lists are used for the buckets you have the overhead of pointers

42
References
• Cormen, Thomas H.; Leiserson, Charles E.; Rivest, Ronald L.; Stein, Clifford (2009) [1990]. Introduction to
Algorithms (3rd ed.). MIT Press and McGraw-Hill. ISBN 0-262-03384-4.
• Alfred V. Aho, J.E. Hopcroft, Jeffrey D. Ullman, Data Structures and Algorithms, Addison-Wesley Series in
Computer Science and Information Processing, 1983
• https://www.geeksforgeeks.org/linear-search/
• https://www.tutorialspoint.com/data_structures_algorithms/binary_search_algorithm.htm
• https://www.cpp.edu/~ftang/courses/CS241/notes/sorting.htm
• https://blog.51cto.com/wintys/102975
• https://www.programiz.com/dsa/counting-sort

43