Heapsort

Lecturer: Dr. Keyhanipour

 Outline
• Iterative sorting algorithms (comparison based)
• Selection Sort
• Bubble Sort
• Insertion Sort
• Recursive sorting algorithms (comparison based)
• Quick Sort
• Merge Sort
• Heap Sort
• Radix sort (non-comparison based)
• Properties of Sorting:
• In-place sort, stable sort

2
 Heap
• A heap is a data structure that allows you to quickly retrieve
the largest (or smallest) element
• It takes time Θ(𝑛) to build the heap
• If you need to retrieve largest element, second largest, third
largest …, in long run the time taken for building heaps will be
rewarded

3
 Heapsort: Idea
HeapSort (𝐴[1. . 𝑛])
Build a heap from A
For i = n down to 1
Retrieve largest element from heap
Put element at end of A
Reduce heap size by one
End

Key:
1. Build a heap in linear time
2. Retrieve largest element (and make it ready for next retrieval) in 𝑂(log 𝑛) time
4
 Heaps
• A heap can be seen as a complete binary tree:

Perfect Binary Tree 16

14 10

8 7 9 3

2 4 1

• A complete binary tree is a binary tree in which every level, except

possibly the last, is completely filled, and all nodes are as far left as
possible
5
 Heaps
• In practice, heaps are usually implemented as arrays:
16

14 10

8 7 9 3

2 4 1

16 14 10 8 7 9 3 2 4 1

6
 Heaps
• To represent a complete binary tree as an array:
• The root node is 𝐴[1]
16
• Node 𝑖 is 𝐴[𝑖]
14 10
• The parent of node 𝑖 is 𝐴[𝑖/2]
(note: integer divide) 8 7 9 3

• The left child of node 𝑖 is 𝐴[2𝑖] 2 4 1

• The right child of node 𝑖 is 𝐴[2𝑖 + 1]

16 14 10 8 7 9 3 2 4 1
1 2 3 4 5 6 7 8 9 10
7
 Referencing Heap Elements
• So…
Parent(i)
{return i/2;}

Left(i)
{return 2*i;}

right(i) 1 2 3 4 5 6 7 8 9 10
{return 2*i + 1;} 16 15 10 8 7 9 3 2 4 1
Level: 3 2 1 0
8
 Heap Height
• Definitions:
• The height of a node in the tree = the number of edges on the longest
downward path to a leaf
• The height of a tree = the height of its root
h=3
16

h=2 h=1
14 10

h=1 8 7 9 3 h=0

2 4 1 h=0

9
 Heap Height
• What is the height of an n-element heap? Why?
• log2(𝑛) .
• Basic heap operations take at most time proportional to the
height of the heap.

10
 Different Heap Types
• There are two kinds of binary heaps:
• Max-heaps
• Min-heaps
• In both kinds, the values in the nodes satisfy a heap property,
the specifics of which depend on the kind of heap.

11
 The Max-Heap Property
• Max-Heaps satisfy the max-heap property:
𝐴[𝑃𝑎𝑟𝑒𝑛𝑡(𝑖)] ≥ 𝐴 𝑖 for all nodes 𝑖 > 1
• In other words, the value of a node is at most the value of its parent
• The value of a node should be greater than or equal to both its left
and right children
• And all of its descendants
• Where is the largest element in a max-heap?
• Root

12
 The Min-Heap Property
• Min-Heaps satisfy the min-heap property:
𝐴[𝑃𝑎𝑟𝑒𝑛𝑡(𝑖)] ≤ 𝐴 𝑖 for all nodes 𝑖 > 1
• In other words, the value of a node is at least the value of its parent
• The value of a node should be less than or equal to both its left and
right children
• And all of its descendants
• Where is the smallest element in a min-heap?
• Root

13
 Max-Heaps
• For the heapsort algorithm, we use max-heaps.
• Min-heaps commonly implement priority queues.

14
 Are they max-heaps?
• Violation to max-heap property: a node has value less than one
of its children
16

4 10

14 7 9 3
16 4 10 14 7 9 3 2 8 1 =
2 8 1

16

10 14

7 8 9 3
16 10 14 7 8 9 3 2 4 1 =
2 4 1

15
 Max-Heap Operations
• The Max-Heapify procedure, which runs in 𝑂(log 𝑛) time, is
the key to maintaining the max-heap property.
• The Build-Max-Heap procedure, which runs in linear time,
𝑂(𝑛), produces a maxheap from an unordered input array.
• The Heapsort procedure, which runs in 𝑂(𝑛 log 𝑛) time, sorts
an array in place.
• The Max-Heap-Insert, Heap-Extract-Max, Heap-Increase-Key
and Heap-Maximum procedures, which run in 𝑂(log 𝑛) time,
allow the heap data structure to implement a priority queue.

16
 Max-Heap Operations: Heapify()
• Heapify(): maintain the max-heap property
• Given: a node 𝑖 in the heap with children 𝑙 and 𝑟
• Given: two subtrees rooted at 𝑙 and 𝑟, assumed to be heaps
• Problem: The subtree rooted at 𝑖 may violate the max-heap
property
• Action: let the value of the parent node “sift down” so subtree at 𝑖
satisfies the heap property
• Fix up the relationship between 𝑖, 𝑙, and 𝑟 recursively

17
 Max-Heap Operations: Heapify()
Heapify(A, i)
{ // precondition: subtrees rooted at l and r are heaps
l = Left(i); r = Right(i);
if (l <= heapSize(A) && A[l] > A[i])
largest = l;
Among A[l], A[i], A[r],
else
which one is largest?
largest = i;
if (r <= heapSize(A) && A[r] > A[largest])
largest = r;
if (largest != i) {
Swap(A, i, largest);
If violation, fix it.
Heapify(A, largest);
}
} // postcondition: subtree rooted at i is a max-heap

18
 Heapify: Example

16

4 10

14 7 9 3

2 8 1

A = 16 4 10 14 7 9 3 2 8 1

19
 Heapify: Example (cont.)

16

4 10

14 7 9 3

2 8 1

A = 16 4 10 14 7 9 3 2 8 1

20
 Heapify: Example (cont.)

16

4 10

14 7 9 3

2 8 1

A = 16 4 10 14 7 9 3 2 8 1

21
 Heapify: Example (cont.)

16

14 10

4 7 9 3

2 8 1

A = 16 14 10 4 7 9 3 2 8 1

22
 Heapify: Example (cont.)

16

14 10

4 7 9 3

2 8 1

A = 16 14 10 4 7 9 3 2 8 1

23
 Heapify: Example (cont.)

16

14 10

8 7 9 3

2 4 1

A = 16 14 10 8 7 9 3 2 4 1

24
 Heapify: Example (cont.)

16

14 10

8 7 9 3

2 4 1

A = 16 14 10 8 7 9 3 2 4 1

25
 Analyzing Heapify()
• The running time of Heapify on a subtree of size 𝑛 rooted at
node 𝑖 is:
• determining the relationship between elements: Θ(1)
• plus the time to run Heapify on a subtree rooted at one of the children
of 𝑖, where 2𝑛/3 is the worst-case size of this subtree (the worst case
occurs when the bottom level of the tree is exactly half full).
𝑇(𝑛) ≤ 𝑇(2𝑛/3) + (1)
• By case 2 of the Master Theorem:
𝑇(𝑛) = 𝑂(lg 𝑛)
• Alternatively, the running time on a node of height h: 𝑂(ℎ)

26
 Review: Master Method
𝑇 𝑛 = 𝑎𝑇 𝑛Τ𝑏 + 𝑓(𝑛)

• Compare 𝑛log𝑏 𝑎 vs. 𝑓(𝑛):

• If 𝑓 𝑛 = 𝑂(𝑛log𝑏 𝑎−𝜖 ) for some constant 𝜖 > 0, then:
𝑇 𝑛 = Θ(𝑛log𝑏 𝑎 ).
• If 𝑓 𝑛 = Θ(𝑛log𝑏 𝑎 ), then: 𝑇 𝑛 = Θ(𝑛log𝑏 𝑎 lg 𝑛).
• If 𝑓 𝑛 = Ω(𝑛log𝑏 𝑎+𝜖 ) for some constant 𝜖 > 0, and if
𝑎𝑓(𝑛Τ𝑏) ≤ 𝑐𝑓(𝑛) for some constant 𝑐 < 1 and all sufficient
large 𝑛, then: 𝑇(𝑛) = Θ(𝑓(𝑛)).

27
 The Master method: Case 2
𝑇 𝑛 = 𝑎𝑇 𝑛Τ𝑏 + 𝑓(𝑛)

• Compare 𝑛log𝑏 𝑎 vs. 𝑓(𝑛):

• If 𝑓 𝑛 = Θ(𝑛log𝑏 𝑎 ), then: 𝑇 𝑛 = Θ(𝑛log𝑏 𝑎 lg 𝑛).

2𝑛 3
•𝑇 𝑛 = 𝑇 +  1 → a = 1, b =
3 2

• 𝑓 𝑛 = Θ 1 = Θ 𝑛log𝑏 𝑎 → 𝑇 𝑛 = Θ(lg 𝑛)

28
 Building a Heap
• We can use the procedure Heapify in a bottom-up manner to
convert an array 𝐴[1 . . 𝑛], where 𝑛 = 𝐴. 𝑙𝑒𝑛𝑔𝑡ℎ, into a max-heap.
• The elements in the subarray 𝐴[( 𝑛/2 + 1) . . 𝑛] are all leaves of
the tree, and so each is a 1-element heap to begin with. The
procedure BUILD-MAX-HEAP goes through the remaining
nodes of the tree and runs MAX-HEAPIFY on each one.
• So, walk backwards through the array from 𝑛/2 to 1, calling
Heapify() on each node.

29
 Building a Heap (cont.)
• Fact: for array of length n, all elements in range 𝐴[ 𝑛/2 + 1 . . 𝑛]
are heaps (1-element heaps).
Heap size # Leaves # Internal Nodes
1 1 0
2 1 1
3 2 1
4 2 2
5 3 2

• 0 <= (# 𝑙𝑒𝑎𝑣𝑒𝑠 − # 𝑖𝑛𝑡𝑒𝑟𝑛𝑎𝑙𝑁𝑜𝑑𝑒𝑠) <= 1

• # 𝑖𝑛𝑡𝑒𝑟𝑛𝑎𝑙𝑁𝑜𝑑𝑒𝑠 = 𝑛/2
30
 BuildHeap()
// given an unsorted array A, make a heap
BuildHeap(A)
{
heapSize(A) = length(A);
for (i = length[A]/2 down to 1)
Heapify(A, i);
}

31
 BuildHeap(): Example
• Work through example: 𝐴 = {4, 1, 3, 2, 16, 9, 10, 14, 8, 7}

1 3

2 16 9 10

14 8 7

A= 4 1 3 2 16 9 10 14 8 7

32
BuildHeap(): Example (cont.)

1 3

2 16 9 10

14 8 7

A= 4 1 3 2 16 9 10 14 8 7

33
BuildHeap(): Example (cont.)

1 3

2 16 9 10

14 8 7

A= 4 1 3 2 16 9 10 14 8 7
34
BuildHeap(): Example (cont.)

1 3

14 16 9 10

2 8 7

A= 4 1 3 14 16 9 10 2 8 7
35
BuildHeap(): Example (cont.)

1 3

14 16 9 10

2 8 7

A= 4 1 3 14 16 9 10 2 8 7

36
BuildHeap(): Example (cont.)

1 10

14 16 9 3

2 8 7

A= 4 1 10 14 16 9 3 2 8 7
37
BuildHeap(): Example (cont.)

1 10

14 16 9 3

2 8 7

A= 4 1 10 14 16 9 3 2 8 7
38
BuildHeap(): Example (cont.)

16 10

14 1 9 3

2 8 7

A= 4 16 10 14 1 9 3 2 8 7
39
BuildHeap(): Example (cont.)

16 10

14 7 9 3

2 8 1

A= 4 16 10 14 7 9 3 2 8 1
40
BuildHeap(): Example (cont.)

16 10

14 7 9 3

2 8 1

A= 4 16 10 14 7 9 3 2 8 1
41
BuildHeap(): Example (cont.)

16

4 10

14 7 9 3

2 8 1

A = 16 4 10 14 7 9 3 2 8 1
42
BuildHeap(): Example (cont.)

16

14 10

4 7 9 3

2 8 1

A = 16 14 10 4 7 9 3 2 8 1
43
BuildHeap(): Example (cont.)

16

14 10

8 7 9 3

2 4 1

A = 16 14 10 8 7 9 3 2 4 1
44
 Analyzing BuildHeap()
• Each call to Heapify() takes 𝑂(lg 𝑛) time
• There are 𝑂(𝑛) such calls (specifically, 𝑛/2 )
• Thus the running time is 𝑂(𝑛 lg 𝑛)
• A tighter bound is 𝑂(𝑛)
• Details are at the textbook

45
 Heapsort
• Given BuildHeap(), an in-place sorting algorithm is easily
constructed:
• Maximum element is at A[1]
• Discard by swapping with element at A[n]
• Decrement heapSize[A]
• A[n] now contains correct value
• Restore heap property at A[1] by calling Heapify()
• Repeat, always swapping A[1] for A[heapSize(A)]

46
 Heapsort
Heapsort(A)
{
BuildHeap(A);
for (i = length(A) down to 2){
Swap(A[1], A[i]);
heapSize(A) -= 1;
Heapify(A, 1);
}
}

47
 Heapsort Example
• Work through example: A = {4, 1, 3, 2, 16, 9, 10, 14, 8, 7}

1 3

2 16 9 10

14 8 7

A= 4 1 3 2 16 9 10 14 8 7

48
 Heapsort Example (cont.)
• First: build a heap

16

14 10

8 7 9 3

2 4 1

A = 16 14 10 8 7 9 3 2 4 1
49
 Heapsort Example (cont.)
• Swap last and first

14 10

8 7 9 3

2 4 16

A= 1 14 10 8 7 9 3 2 4 16
50
 Heapsort Example (cont.)
• Last element sorted

14 10

8 7 9 3

2 4 16

A= 1 14 10 8 7 9 3 2 4 16
51
 Heapsort Example (cont.)
• Restore heap on remaining unsorted elements

14

8 10

4 7 9 3

2 1 16 Heapify

A = 14 8 10 4 7 9 3 2 1 16
52
 Heapsort Example (cont.)
• Repeat: swap new last and first

8 10

4 7 9 3

2 14 16

A= 1 8 10 4 7 9 3 2 14 16
53
 Heapsort Example (cont.)
• Restore heap

10

8 9

4 7 1 3

2 14 16

A = 10 8 9 4 7 1 3 2 14 16
54
 Heapsort Example (cont.)
• Repeat

8 3

4 7 1 2

10 14 16

A= 9 8 3 4 7 1 2 10 14 16
55
 Heapsort Example (cont.)
• Repeat

7 3

4 2 1 9

10 14 16

A= 8 7 3 4 2 1 9 10 14 16
56
 Heapsort Example (cont.)
• After some intermediate steps …

2 3

4 7 8 9

10 14 16

A= 1 2 3 4 7 8 9 10 14 16
57
 Analyzing Heapsort
• The call to BuildHeap() takes 𝑂(𝑛) time
• Each of the 𝑛 − 1 calls to Heapify() takes 𝑂(lg 𝑛) time
• Thus the total time taken by HeapSort()
= 𝑂(𝑛) + (𝑛 − 1) 𝑂(lg 𝑛)
= 𝑂(𝑛) + 𝑂(𝑛 lg 𝑛)
= 𝑂(𝑛 lg 𝑛)

58
Heapsort: Animated Illustration 1

59
Heapsort: Animated Illustration 2

60
 Heapsort: Summary
• Heapsort uses a heap data structure to improve selection sort
and make the running time asymptotically optimal
• Running time is 𝑂(𝑛 log 𝑛) – like merge sort, but unlike
selection, insertion, or bubble sorts
• Sorts in place – like insertion, selection or bubble sorts, but
unlike merge sort

61
Summary

62
 Sorting Algorithms: Summary
Time Complexity Space Complexity
Sorting
In-place? Stable?
Algorithm Average Worst
Best Case Worst Case
Case Case
Selection
Ω(𝑛2 ) Θ(𝑛2 ) 𝑂(𝑛2 ) 𝑂(1) Yes No
Sort

Bubble Sort Ω(𝑛) Θ(𝑛2 ) 𝑂(𝑛2 ) 𝑂(1) Yes Yes

Insertion
Ω(𝑛) Θ(𝑛2 ) 𝑂(𝑛2 ) 𝑂(1) Yes Yes
Sort

Merge Sort Ω(𝑛 log 𝑛) Θ(𝑛 log 𝑛) 𝑂(𝑛 log 𝑛) 𝑂(𝑛) No Yes

Radix Sort Ω(𝑛𝑑) Θ(𝑛𝑑) 𝑂(𝑛𝑑) 𝑂(𝑛 + 𝑑) No Yes

Quicksort Ω(𝑛 log 𝑛) Θ(𝑛 log 𝑛) 𝑂(𝑛2 ) 𝑂(𝑛 log 𝑛) Yes No

Heapsort Ω(𝑛 log 𝑛) Θ(𝑛 log 𝑛) 𝑂(𝑛 log 𝑛) 𝑂(1) Yes No

63