CC104

Data Structures and Algorithms

Chapter 10 Learning Module

ANALYSIS OF SORTING ALGORITHMS

Disclaimer:

This learning material is provided in accordance with the modular learning approach adopted by the
University in response to the disruptions caused by Typhoon Pepito, which affected the delivery of
education in the province. The authors and publisher of the content are duly acknowledged. The
college and its faculty do not claim ownership of the sourced information. This learning material is
intended solely for instructional purposes and is not for commercial use.
Introduction

Sorting algorithms are a set of instructions that take an array or list as an input and arrange the items
into a particular order.

Sorts are most commonly in numerical or a form of alphabetical (or lexicographical) order, and can
be in ascending (A-Z, 0-9) or descending (Z-A, 9-0) order.

Why Sorting Algorithms are Important

Since they can often reduce the complexity of a problem, sorting algorithms are very important in
computer science. These algorithms have direct applications in searching algorithms, database
algorithms, divide and conquer methods, data structure algorithms, and many more.

Trade-Offs of Sorting Algorithms

When choosing a sorting algorithm, some questions have to be asked – How big is the collection being
sorted? How much memory is available? Does the collection need to grow?

The answers to these questions may determine which algorithm is going to work best for each
situation. Some algorithms like merge sort may need a lot of space or memory to run, while insertion
sort is not always the fastest, but doesn't require many resources to run.

You should determine what your requirements are, and consider the limitations of your system before
deciding which sorting algorithm to use.

Classification of a Sorting Algorithm

Sorting algorithms can be categorized based on the following parameters:

1. The number of swaps or inversions required: This is the number of times the algorithm swaps
elements to sort the input. Selection sort requires the minimum number of swaps.

2. The number of comparisons: This is the number of times the algorithm compares elements
to sort the input. Using Big-O notation, the sorting algorithm examples listed above require at
least O(nlogn) comparisons in the best case, and O(n^2) comparisons in the worst case for
most of the outputs.

3. Whether or not they use recursion: Some sorting algorithms, such as quick sort, use recursive
techniques to sort the input. Other sorting algorithms, such as selection sort or insertion sort,
use non-recursive techniques. Finally, some sorting algorithms, such as merge sort, make use
of both recursive as well as non-recursive techniques to sort the input.

4. Whether they are stable or unstable: Stable sorting algorithms maintain the relative order of
elements with equal values, or keys. Unstable sorting algorithms do not maintain the relative
order of elements with equal values / keys.

For example, imagine you have the input array [1, 2, 3, 2, 4]. And to help differentiate between the
two equal values, 2, let's update them to 2a and 2b, making the input array [1, 2a, 3, 2b, 4].

Stable sorting algorithms will maintain the order of 2a and 2b, meaning the output array will be [1, 2a,
2b, 3, 4]. Unstable sorting algorithms do not maintain the order of equal values, and the output array
may be [1, 2b, 2a, 3, 4].

Insertion sort, merge sort, and bubble sort are stable. Heap sort and quick sort are unstable.
 5. The amount of extra space required: Some sorting algorithms can sort a list without creating
an entirely new list. These are known as in-place sorting algorithms, and require a
constant O(1) extra space for sorting. Meanwhile, out of place sorting algorithms create a new
list while sorting.

Insertion sort and quick sort are in place sorting algorithms, as elements are moved around a pivot
point, and do not use a separate array.

Merge sort is an example of an out of place sorting algorithm, as the size of the input must be allocated
beforehand to store the output during the sort process, which requires extra memory.

Lesson 1: Bubble Sort

Just like the way bubbles rise from the bottom of a glass, bubble sort is a simple algorithm that sorts
a list, allowing either lower or higher values to bubble up to the top. The algorithm traverses a list and
compares adjacent values, swapping them if they are not in the correct order.

With a worst-case complexity of O(n^2), bubble sort is very slow compared to other sorting algorithms
like quicksort. The upside is that it is one of the easiest sorting algorithms to understand and code from
scratch.

From technical perspective, bubble sort is reasonable for sorting small-sized arrays or specially when
executing sort algorithms on computers with remarkably limited memory resources.

Example:

First pass through the list:

• Starting with [4, 2, 6, 3, 9], the algorithm compares the first two elements in the array, 4 and
2. It swaps them because 2 < 4: [2, 4, 6, 3, 9]

• It compares the next two values, 4 and 6. As 4 < 6, these are already in order, and the algorithm
moves on: [2, 4, 6, 3, 9]

• The next two values are also swapped because 3 < 6: [2, 4, 3, 6, 9]

• The last two values, 6 and 9, are already in order, so the algorithm does not swap them.

Second pass through the list:

• 2 < 4, so there is no need to swap positions: [2, 4, 3, 6, 9]

• The algorithm swaps the next two values because 3 < 4: [2, 3, 4, 6, 9]

• No swap as 4 < 6: [2, 3, 4, 6, 9]

• Again, 6 < 9, so no swap occurs: [2, 3, 4, 6, 9]

The list is already sorted, but the bubble sort algorithm doesn't realize this. Rather, it needs to
complete an entire pass through the list without swapping any values to know the list is sorted.

Third pass through the list:

• [2, 4, 3, 6, 9] => [2, 4, 3, 6, 9]

 • [2, 4, 3, 6, 9] => [2, 4, 3, 6, 9]

• [2, 4, 3, 6, 9] => [2, 4, 3, 6, 9]

• [2, 4, 3, 6, 9] => [2, 4, 3, 6, 9]

Clearly bubble sort is far from the most efficient sorting algorithm. Still, it's simple to wrap your head
around and implement yourself.

Properties

• Space complexity: O(1)

• Best case performance: O(n)

• Average case performance: O(n*n)

• Worst case performance: O(n*n)

• Stable: Yes

Example in C++

Lesson 2: Insertion Sort

Insertion sort is a simple sorting algorithm for a small number of elements.

Example:

In Insertion sort, you compare the key element with the previous elements. If the previous elements
are greater than the key element, then you move the previous element to the next position.

Start from index 1 to size of the input array.

[835142]
Step 1 :

key = 3 //starting from 1st index.

Here `key` will be compared with the previous elements.

In this case, `key` is compared with 8. since 8 > 3, move the element 8

to the next position and insert `key` to the previous position.

Result: [ 3 8 5 1 4 2 ]

Step 2 :

key = 5 //2nd index

8 > 5 //move 8 to 2nd index and insert 5 to the 1st index.

Result: [ 3 5 8 1 4 2 ]

Step 3 :

key = 1 //3rd index

8 > 1 => [ 3 5 1 8 4 2 ]
5 > 1 => [ 3 1 5 8 4 2 ]
3 > 1 => [ 1 3 5 8 4 2 ]
Result: [ 1 3 5 8 4 2 ]

Step 4 :

key = 4 //4th index

8 > 4 => [ 1 3 5 4 8 2 ]
5 > 4 => [ 1 3 4 5 8 2 ]
 3 > 4 ≠> stop
Result: [ 1 3 4 5 8 2 ]

Step 5 :

key = 2 //5th index

8 > 2 => [ 1 3 4 5 2 8 ]
5 > 2 => [ 1 3 4 2 5 8 ]
4 > 2 => [ 1 3 2 4 5 8 ]
3 > 2 => [ 1 2 3 4 5 8 ]
1 > 2 ≠> stop
Result: [1 2 3 4 5 8]

The algorithm shown below is a slightly optimized version to avoid swapping the key element in every
iteration. Here, the key element will be swapped at the end of the iteration (step).

Properties:

• Space Complexity: O(1)

• Time Complexity: O(n), O(n n), O(n n) for Best, Average, Worst cases respectively.
• Best Case: array is already sorted
• Average Case: array is randomly sorted
• Worst Case: array is reversely sorted.
• Sorting In Place: Yes
• Stable: Yes

Lesson 3: Quick Sort

Quick sort is an efficient divide and conquer sorting algorithm. Average case time complexity of
Quick Sort is O(nlog(n)) with worst case time complexity being O(n^2) depending on the selection of
the pivot element, which divides the current array into two sub arrays.

For instance, the time complexity of Quick Sort is approximately O(nlog(n)) when the selection of
pivot divides original array into two nearly equal sized sub arrays.
On the other hand, if the algorithm, which selects of pivot element of the input arrays, consistently
outputs 2 sub arrays with a large difference in terms of array sizes, quick sort algorithm can achieve
the worst case time complexity of O(n^2).

The steps involved in Quick Sort are:

• Choose an element to serve as a pivot, in this case, the last element of the array is the pivot.

• Partitioning: Sort the array in such a manner that all elements less than the pivot are to the
left, and all elements greater than the pivot are to the right.

• Call Quicksort recursively, taking into account the previous pivot to properly subdivide the
left and right arrays.

The space complexity of quick sort is O(n). This is an improvement over other divide and conquer
sorting algorithms, which take O(nlong(n)) space.

Quick sort achieves this by changing the order of elements within the given array. Compare this with
the merge sort algorithm which creates 2 arrays, each length n/2, in each function call.

However there does exist the problem of this sorting algorithm being of time O(n*n) if the pivot is
always kept at the middle. This can be overcomed by utilizing a random pivot

Complexity

Best, average, worst, memory: n log(n)n log(n)n 2log(n). It's not a stable algorithm, and quicksort is
usually done in-place with O(log(n)) stack space.

The space complexity of quick sort is O(n). This is an improvement over other divide and conquer
sorting algorithms, which take O(n log(n)) space.
Lesson 4: Merge Sort
Merge Sort is a Divide and Conquer algorithm. It divides input array in two halves, calls itself for the
two halves and then merges the two sorted halves. The major portion of the algorithm is given two
sorted arrays, and we have to merge them into a single sorted array. The whole process of sorting an
array of N integers can be summarized into three steps-

• Divide the array into two halves.

• Sort the left half and the right half using the same recurring algorithm.

• Merge the sorted halves.

There is something known as the Two Finger Algorithm that helps us merge two sorted arrays together.
Using this subroutine and calling the merge sort function on the array halves recursively will give us
the final sorted array we are looking for.

Since this is a recursion based algorithm, we have a recurrence relation for it. A recurrence relation is
simply a way of representing a problem in terms of its subproblems.

T(n) = 2 * T(n / 2) + O(n)

Putting it in plain English, we break down the subproblem into two parts at every step and we have
some linear amount of work that we have to do for merging the two sorted halves together at each
step.

Complexity

The biggest advantage of using Merge sort is that the time complexity is only n*log(n) to sort an entire
Array. It is a lot better than n^2 running time of bubble sort or insertion sort.

Before we write code, let us understand how merge sort works with the help of a diagram.
 • Initially we have an array of 6 unsorted integers Arr(5, 8, 3, 9, 1, 2)

• We split the array into two halves Arr1 = (5, 8, 3) and Arr2 = (9, 1, 2).

• Again, we divide them into two halves: Arr3 = (5, 8) and Arr4 = (3) and Arr5 = (9, 1) and Arr6 =
(2)

• Again, we divide them into two halves: Arr7 = (5), Arr8 = (8), Arr9 = (9), Arr10 = (1) and Arr6 =
(2)

• We will now compare the elements in these sub arrays in order to merge them.

Properties:

• Space Complexity: O(n)

• Time Complexity: O(n*log(n)). The time complexity for the Merge Sort might not be obvious
from the first glance. The log(n) factor that comes in is because of the recurrence relation we
have mentioned before.
 • Sorting In Place: No in a typical implementation

• Stable: Yes

• Parallelizable: yes

C++ Implementation

Explanation:

Merge Process:

The function follows these steps:

1. Iterate through A and B simultaneously:

• A while loop runs as long as both token_a and token_b are within the bounds of their
respective arrays (A and B).

• At each step, compare the current element from A (A[token_a]) with the current element from
B (B[token_b]).

• If A[token_a] <= B[token_b]:

o Place A[token_a] into C[token_c].

 o Increment token_a (move to the next element in A).

o Increment token_c (move to the next position in C).

• Else:

o Place B[token_b] into C[token_c].

o Increment token_b (move to the next element in B).

o Increment token_c.

2. Copy Remaining Elements:

Once either A or B is fully traversed, the remaining elements of the other array are appended to C:

• If there are remaining elements in A:

o Use a while loop to copy them directly into C.

• If there are remaining elements in B:

o Use a while loop to copy them into C.

Example Walkthrough:

Given:

• A = {2, 5, 7, 8, 9, 12, 13}

• B = {3, 5, 6, 9, 15}

Process:

• Start with token_a = 0, token_b = 0, token_c = 0.

Step Compare A[token_a] and B[token_b] Add to C[token_c] Resulting C

1 2 <= 3 C[0] = 2 {2}
2 5>3 C[1] = 3 {2, 3}
3 5 <= 5 C[2] = 5 {2, 3, 5}
4 7>6 C[3] = 6 {2, 3, 5, 6}
5 7 <= 9 C[4] = 7 {2, 3, 5, 6, 7}
6 8 <= 9 C[5] = 8 {2, 3, 5, 6, 7, 8}
7 9 <= 9 C[6] = 9 {2, 3, 5, 6, 7, 8, 9}
8 Remaining elements in A: {12, 13} Copy to C {2, 3, 5, 6, 7, 8, 9, 12, 13}
9 Remaining element in B: {15} Copy to C {2, 3, 5, 6, 7, 8, 9, 12, 13, 15}
Lesson 5: Heap Sort
Heapsort is an efficient sorting algorithm based on the use of max/min heaps. A heap is a tree-based
data structure that satisfies the heap property – that is for a max heap, the key of any node is less than
or equal to the key of its parent (if it has a parent).

This property can be leveraged to access the maximum element in the heap in O(logn) time using the
maxHeapify method. We perform this operation n times, each time moving the maximum element in
the heap to the top of the heap and extracting it from the heap and into a sorted array. Thus, after n
iterations we will have a sorted version of the input array.

The algorithm is not an in-place algorithm and would require a heap data structure to be constructed
first. The algorithm is also unstable, which means when comparing objects with same key, the original
ordering would not be preserved.

This algorithm runs in O(nlogn) time and O(1) additional space [O(n) including the space required to
store the input data] since all operations are performed entirely in-place.

The best, worst and average case time complexity of Heapsort is O(nlogn). Although heapsort has a
better worse-case complexity than quicksort, a well-implemented quicksort runs faster in practice. This
is a comparison-based algorithm so it can be used for non-numerical data sets insofar as some relation
(heap property) can be defined over the elements.

Heapify Function

The heapify function is the core of heap operations in the Heap Sort algorithm. Its purpose is to
maintain the max heap property for a subtree rooted at a specific index i.

Example of heapify function:

Key Steps in heapify:

1. Identify the largest element:

o Start by assuming the root node (index i) is the largest.

o Compare it with its left child (index 2*i + 1) and right child (index 2*i + 2).

o Update largest to hold the index of the child if it is greater than the root.

2. Swap and recurse:

o If the largest element is not the root, swap the root with the largest child.

o Recursively call heapify on the affected subtree to ensure the heap property is
maintained.

3. Base case:

o If the current node is already larger than its children (or is a leaf node), no further
action is required, and the recursion terminates.

Example Walkthrough:

Input Array: arr = {4, 10, 3, 5, 1}

Call: heapify(arr, 5, 1)

1. Initial state:

o i = 1, root value = 10, left child = 5, right child = 1.

o No child is greater than the root, so no changes.

Call: heapify(arr, 5, 0)

1. Initial state:

o i = 0, root value = 4, left child = 10 (index 1), right child = 3 (index 2).

o Largest is updated to 1 (value 10).

2. Swap:

o Swap root 4 with 10.

3. Recursive call:

o After the swap, array becomes {10, 4, 3, 5, 1}.

o Call heapify(arr, 5, 1) to reheapify the affected subtree.

4. Reheapify:

o Root = 4, left child = 5 (index 3), right child = 1 (index 4).

o Largest is updated to 3 (value 5).

o Swap 4 and 5, resulting in {10, 5, 3, 4, 1}.

Heap Sort Function

Step 1: Build a max heap.

• Start from the last non-leaf node (n / 2 - 1) and heapify each node in reverse order.

• After this step, the largest element is at the root of the heap (arr[0]).

Step 2: Sort the array.

• Swap the root (largest element) with the last element of the heap.

• Reduce the heap size by 1 and reheapify the reduced heap.

• Repeat until the entire array is sorted.

Example Execution:

Input Array: arr = {4, 10, 3, 5, 1}

Build Max Heap:

• Initial array: {4, 10, 3, 5, 1}

• After heapify: {10, 5, 3, 4, 1}

Sort:

• Swap root (10) with last element (1): {1, 5, 3, 4, 10}

• Reheapify: {5, 4, 3, 1, 10}

• Swap root (5) with last element (1): {1, 4, 3, 5, 10}

 • Reheapify: {4, 1, 3, 5, 10}

• Continue until fully sorted: {1, 3, 4, 5, 10}

Lesson 6: Counting Sort

The counting sort algorithm works by first creating a list of the counts or occurrences of each unique
value in the list. It then creates a final sorted list based on the list of counts.

One important thing to remember is that counting sort can only be used when you know the range
of possible values in the input beforehand.

Example:

Say you have a list of integers from 0 to 5:

input = [2, 5, 3, 1, 4, 2]

First, you need to create a list of counts for each unique value in the input list. Since you know the
range of the input is from 0 to 5, you can create a list with five placeholders for the values 0 to 5,
respectively:

count = [0, 0, 0, 0, 0, 0]
// val: 0 1 2 3 4 5

Then, you go through the input list and iterate the index for each value by one.

For example, the first value in the input list is 2, so you add one to the value at the second index of
the count list, which represents the value 2:

count = [0, 0, 1, 0, 0, 0]
// val: 0 1 2 3 4 5

The next value in the input list is 5, so you add one to the value at the last index of the count list,
which represents the value 5:
count = [0, 0, 1, 0, 0, 1]
// val: 0 1 2 3 4 5

Continue until you have the total count for each value in the input list:
count = [0, 1, 2, 1, 1, 1]
// val: 0 1 2 3 4 5

Finally, since you know how many times each value in the input list appears, you can easily create a
sorted output list. Loop through the count list, and for each count, add the corresponding value (0 -
5) to the output array that many times.
For example, there were no 0's in the input list, but there was one occurrence of the value 1, so you
add that value to the output array one time:
output = [1]
Then there were two occurrences of the value 2, so you add those to the output list:

output = [1, 2, 2]

And so on until you have the final sorted output list:

output = [1, 2, 2, 3, 4, 5]

Properties

• Space complexity: O(k)

• Best case performance: O(n+k)

• Average case performance: O(n+k)

• Worst case performance: O(n+k)

• Stable: Yes (k is the range of the elements in the array)

C++ Implementation

Lesson 7: Radix Sort

QuickSort, MergeSort, and HeapSort are comparison-based sorting algorithms. CountSort is not. It has
the complexity of O(n+k), where k is the maximum element of the input array. So, if k is O(n), CountSort
becomes linear sorting, which is better than comparison based sorting algorithms that have O(nlogn)
time complexity.
The idea is to extend the CountSort algorithm to get a better time complexity when k goes O(n2). Here
comes the idea of Radix Sort.
Radix Sort is a sorting algorithm that organizes numbers by processing each digit one at a time, starting
from the least significant digit (rightmost digit) to the most significant digit (leftmost digit).

How Radix Sort Works:

1. Group by Digits:
o Imagine the numbers as written vertically, one below the other.
o Start by sorting them based on the units place (last digit). Group numbers with the
same digit together, keeping their order intact.
2. Sort the Next Digit:
o Move to the tens place (second-to-last digit) and sort the numbers again, but keep the
previous order intact for numbers with the same tens digit.
3. Continue for All Digits:
o Keep repeating this process for the hundreds place, thousands place, and so on, until
all digits have been processed.
4. Sorted Array:
o After sorting by all digits, the numbers will be in ascending order.

Radix Sort uses stable sorting for each digit, ensuring that the order of equal digits is preserved
between passes. By progressively sorting each digit from the least significant to the most significant,
the entire array becomes sorted.

Simple Analogy:
Think of sorting a pile of envelopes by postal codes:
1. First, sort by the last digit of the postal code.
2. Then, sort by the second-to-last digit while keeping the previous sorting intact.
3. Repeat until all digits are sorted.

The Algorithm:
For each digit i where i varies from the least significant digit to the most significant digit of a number,
sort input array using countsort algorithm according to ith digit. We used count sort because it is a
stable sort.

Example: Assume the input array is:

10, 21, 17, 34, 44, 11, 654, 123

Based on the algorithm, we will sort the input array according to the one's digit (least significant
digit).

0: 10
1: 21 11
2:
3: 123
4: 34 44 654
5:
6:
7: 17
8:
9:

So, the array becomes 10, 21, 11, 123, 24, 44, 654, 17

Now, we'll sort according to the ten's digit:

0:
1: 10 11 17
2: 21 123
3: 34
4: 44
5: 654
6:
7:
8:
9:

Now, the array becomes : 10, 11, 17, 21, 123, 34, 44, 654

Finally, we sort according to the hundred's digit (most significant digit):

0: 010 011 017 021 034 044

1: 123
2:
3:
4:
5:
6: 654
7:
8:
9:
The array becomes : 10, 11, 17, 21, 34, 44, 123, 654
which is sorted. This is how our algorithm works.

Lesson 8: Bucket Sort

Bucket sort is a comparison sort algorithm that operates on elements by dividing them into different
buckets and then sorting these buckets individually. Each bucket is sorted individually using a separate
sorting algorithm like insertion sort, or by applying the bucket sort algorithm recursively.
Bucket sort is mainly useful when the input is uniformly distributed over a range. For example, imagine
you have a large array of floating point integers distributed uniformly between an upper and lower
bound.
You could use another sorting algorithm like merge sort, heap sort, or quick sort. However, those
algorithms guarantee a best case time complexity of O(nlogn).
Using bucket sort, sorting the same array can be completed in O(n) time.
Pseudo Code for Bucket Sort:

How Bucket Sort Works:

1. Create Buckets:
o Divide the range of the input values into intervals or "buckets."
o Each bucket holds values that fall into its range.
2. Distribute Elements:
o Go through the input array and assign each element to its appropriate bucket based
on its value.
3. Sort Buckets:
o Sort the elements in each bucket using a suitable sorting algorithm (e.g., Insertion
Sort, Quick Sort, etc.).
4. Merge Buckets:
o Concatenate the sorted elements from all buckets into the final sorted array.
Code Explanation:
1. Bucket Creation:
o Each bucket is represented as a vector.
o The input range (e.g., 0.0 to 1.0) is divided into equal intervals corresponding to the
number of buckets.
2. Element Distribution:
o Each element is assigned to a bucket based on its value.
o For instance, if arr[i] = 0.565 and there are 6 buckets, 0.565 * 6 = 3.39, so the element
goes into bucket index 3.
3. Sort Buckets:
o Each bucket is sorted individually using std::sort.
4. Merge Buckets:
o The elements from all sorted buckets are concatenated back into the original array.

Example Walkthrough:
Input Array: {0.897, 0.565, 0.656, 0.123, 0.665, 0.343}
Step 1: Create Buckets
Divide into 6 buckets (as n = 6):
• Bucket 0: [ ]
• Bucket 1: [0.123]
• Bucket 2: [ ]
• Bucket 3: [0.343, 0.565]
• Bucket 4: [0.656, 0.665]
• Bucket 5: [0.897]
Step 2: Sort Buckets
Sort each bucket:
 • Bucket 1: [0.123]
• Bucket 3: [0.343, 0.565]
• Bucket 4: [0.656, 0.665]
• Bucket 5: [0.897]
Step 3: Merge Buckets
Concatenate the sorted buckets: Output Array: {0.123, 0.343, 0.565, 0.656, 0.665, 0.897}

Additional Source:
FreeCodeCamp - https://www.freecodecamp.org/news/sorting-algorithms-explained-with-examples-
in-python-java-and-c/

Programiz.com - https://www.programiz.com/dsa/sorting-algorithm

GeekforGeeks.org - https://www.programiz.com/dsa/sorting-algorithm