CMP3005

Analysis of Algorithms
Lecture Notes 5
- Divide and Conquer Algorithms
 Divide-and-Conquer
The most-well known algorithm design strategy:
1. Divide instance of problem into two or more
smaller instances

2. Solve smaller instances recursively

3. Obtain solution to original (larger) instance by

combining these solutions

2
Divide-and-Conquer Technique (cont.)

a problem of size n

subproblem 1 subproblem 2
of size n/2 of size n/2

a solution to a solution to
subproblem 1 subproblem 2

a solution to
the original problem
A. Levitin “Introduction to the Design &
Analysis of Algorithms,” 3rd ed., Ch. 5
3
©2012 Pearson Education, Inc. Upper
 Divide-and-Conquer Examples
• Binary search: decrease-by-half (or degenerate
divide&conq.)

• Sorting: mergesort and quicksort

• Binary tree traversals

• Multiplication of large integers

• Matrix multiplication: Strassen’s algorithm

• Closest-pair algorithm
A. Levitin “Introduction to the Design &
Analysis of Algorithms,” 3rd ed., Ch. 5
4
©2012 Pearson Education, Inc. Upper
 General Divide-and-Conquer Recurrence
T(n) = aT(n/b) + f (n) where f(n)  O(nd), d  0

Master Theorem: If a < bd, T(n)  O(nd)

If a = bd, T(n)  O(nd log n)
If a > bd, T(n)  O(nlog b a )

Examples: T(n) = 4T(n/2) + n  T(n)  ?

T(n) = 4T(n/2) + n2  T(n)  ?
T(n) = 4T(n/2) + n3  T(n)  ?

5
 Fake-Coin Problem
• Problem: Among n identical-
looking coins, one is fake
• Have a balance scale that can
compare any two sets of coins
• Suppose that fake one is lighter
than the genuine one
• Which coin is the fake one?
 Fake-Coin Problem
• Decrease-by-a-constant-factor strategy
– Divide n coins into two piles of n/2 coins each
– Leave one extra coin aside if n is odd
– Put the two piles on the scale
– If the piles weigh the same, the coin put aside
must be fake
– Otherwise, proceed in the same manner with
the lighter pile
 Fake-Coin Problem
• Input size: n (number of coins)
• Basic operation: Weighting
• Worst-case

T(n) = aT(n/b) + f (n) where f(n)  O(nd), d  0

Master Theorem: If a < bd, T(n)  O(nd)
If a = bd, T(n)  O(nd log n)
If a > bd, T(n)  O(nlog b a )
Complexity: ???
 Fake-Coin Problem
• A better solution?
 Non-Recursive Binary Search
Searching for 19
 Recursive Binary Search
int bsearch(int a[], int lo, int h, int k)
{
int m;
if(h < lo)
return -1;
m = (lo + h) / 2;
if(a[m] == k)
return m;
if(k < a[m])
return bsearch(a, lo, m-1, k);
else
return bsearch(a, m+1, h, k);
}
 Analysis of Binary Search
Recurrence Relation ?
T(n)=O(1) + T(n/2)
T(1)=1
Above is an example of recurrence relation and the way to
solve it is by Substitution.
T(n)=T(n/2) +1
= T(n/22)+1+1
= T(n/23)+1+1+1
= logn
T(n) O(logn)
One can also use master theorem to find the complexity
 Merge Sort
• An example of divide and conquer approach
• Used to sort a list of numbers
– Divide the list into two parts and sort each part
separately using the same approach until we
reach a single element and, thus, cannot divide it
into two (a recursive solution).
– Merge the two lists into a final sorted list (bottom
up, merge single element lists into lists with two
elements, merge two 2-element lists into a four
element list etc…).
 Mergesort
• Split array A[0..n-1] in two about equal halves and
make copies of each half in arrays B and C
• Sort arrays B and C recursively
• Merge sorted arrays B and C into array A as
follows:
– Repeat the following until no elements remain in one of the
arrays:
• compare the first elements in the remaining unprocessed
portions of the arrays
• copy the smaller of the two into A, while incrementing the
index indicating the unprocessed portion of that array
– Once all elements in one of the arrays are processed, copy the
remaining unprocessed elements from the other array into A.
14
Pseudocode of Mergesort

A. Levitin “Introduction to the Design &

Analysis of Algorithms,” 3rd ed., Ch. 5
15
©2012 Pearson Education, Inc. Upper
Pseudocode of Merge

A. Levitin “Introduction to the Design &

Analysis of Algorithms,” 3rd ed., Ch. 5
16
©2012 Pearson Education, Inc. Upper
 Mergesort Example
8 3 2 9 7 1 5 4

8 3 2 9 7 1 5 4

8 3 2 9 71 5 4

8 3 2 9 7 1 5 4

3 8 2 9 1 7 4 5

2 3 8 9 1 4 5 7

1 2 3 4 5 7 8 9
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 5 ©2012 Pearson
Education, Inc. Upper Saddle River, NJ. All Rights Reserved. 17
 Analysis of Mergesort
• All cases have same efficiency: Θ(n log n) -> Show by
master theorem and also using backward substitution
method

• Number of comparisons in the worst case is close to

theoretical minimum for comparison-based sorting:
log2 n! ≈ n log2 n - 1.44n

• Space requirement: Θ(n)

• Can be implemented without recursion (bottom-up)

A. Levitin “Introduction to the Design &
Analysis of Algorithms,” 3rd ed., Ch. 5
18
©2012 Pearson Education, Inc. Upper
 Quicksort
• Select a pivot (partitioning element) – here, the first
element
• Rearrange the list so that all the elements in the first s
positions are smaller than or equal to the pivot and all the
elements in the remaining n-s positions are larger than or
equal to the pivot (see next slide for an algorithm)
p

A[i]p A[i]p
• Exchange the pivot with the last element in the first (i.e., )
subarray — the pivot is now in its final position
• Sort the two subarrays recursively
A. Levitin “Introduction to the Design &
Analysis of Algorithms,” 3rd ed., Ch. 5
19
©2012 Pearson Education, Inc. Upper
Quicksort Algorithm
Hoare’s Partitioning Algorithm

A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 5
©2012 Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved.
 Quicksort Example
5 3 1 9 8 2 4 7

22
 Analysis of Quicksort
• Best case: split in the middle — Θ(n log n)
• Worst case: sorted array! — Θ(n2)
• Average case: random arrays — Θ(n log n)

• Improvements:
– better pivot selection: median of three partitioning
– switch to insertion sort on small subfiles
– elimination of recursion
These combine to 20-25% improvement

• Considered the method of choice for internal

sorting of large files (n ≥ 10000)
23
 Binary Tree Algorithms
Binary tree is a divide-and-conquer ready structure!

Ex. 1: Classic traversals (preorder, inorder, postorder)

Algorithm Inorder(T)
if T  
Inorder(Tleft)
print(root of T)
Inorder(Tright)
T(n) = aT(n/b) + f (n) where f(n)  O(nd), d  0
Master Theorem: If a < bd, T(n)  O(nd)
If a = bd, T(n)  O(nd log n)
Efficiency: Θ(n) Complexity: ???
If a > bd, T(n)  O(nlog b a )
A. Levitin “Introduction to the Design &
Analysis of Algorithms,” 3rd ed., Ch. 5
24
©2012 Pearson Education, Inc. Upper
 Binary Tree Algorithms (cont.)

Ex. 2: Computing the height of a binary tree

T(n) = aT(n/b) + f (n) where f(n)  O(nd), d  0

Master Theorem: If a < bd, T(n)  O(nd)
If a = bd, T(n)  O(nd log n)
TL TR If a > bd, T(n)  O(nlog b a )
Complexity: ???

h(T) = max{h(TL), h(TR)} + 1 if T   and h() = -1

Efficiency: Θ(n)

A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 5 ©2012 Pearson
Education, Inc. Upper Saddle River, NJ. All Rights Reserved. 25
 Binary Tree Algorithms (cont.)

Ex. 2: Computing the height of a binary tree

TL TR

26
 Multiplication of Large Integers
Consider the problem of multiplying two (large) n-digit
integers represented by arrays of their digits such as:
A = 12345678901357986429 B = 87654321284820912836
The grade-school algorithm:
a1 a2 … an
b1 b2 … bn
(d10) d11d12 … d1n
(d20) d21d22 … d2n
…………………
(dn0) dn1dn2 … dnn

Efficiency: n2 one-digit multiplications

27
First Divide-and-Conquer Algorithm

Faster Integer Multiplication – First Try

• Divide : Split I and J into high-order and low-order digits.
– ex: I = 61438521 is divided into Ih= 6143 and Il = 8521
– i.e. I = 6143  104 + 8521 n/2
I = I h 10 + Il
J = J h 10 n / 2 + J l

• Conquer : define I  J by multiplying the parts and adding

( ) (
I  J = I h 10 n / 2 + I l  J h 10 n / 2 + J l )
= (I h  J h )10 n + (I h  J l ) + (I l  J h )10 n / 2 + (I l  J l )
subProblem1 subProblem3 subProblem4 subProblem2

Complexity: T(n) = 4T(n/2) + cn  T(n) is O(n2).

28
 Second Divide-and-Conquer Algorithm

Faster Integer Multiplication – Second Try

• Improved Algorithm

I  J = (I h  J h )10 n + (I h  J l ) + (I l  J h )10 n / 2 + (I l  J l )

= (I h  J h )10 n + (I h − I l ) ( J l − J h ) + (I h  J h ) + (I l  J l )10 n / 2 + (I l  J l )

subProblem1 subProblem3 subProblem1 subProblem2 subProblem2

• Complexity: T(n) = 3T(n/2) + cn,

 T(n) is O(nlog23), by the Master Theorem,
from part 1 with =1.
• Thus, T(n) is O(n1.585).

29
 Integer Multiplication
• Note that in these solutions multiplying by
powers of 10 are not considered to be
separate multiplications because they are
simply shift operations.
 Divide and Conquer Algorithms
• For many problems both brute force and
divide&conquer solutions exist.
• For example, consider computing nth power of
a given number a.
 Computing integer power an
• Brute force algorithm
Algorithm power(a,n)
value  1 a n = a  a  a
for i  1 to n do
value  value  a
return value
• Complexity: O(n)
 First try: Computing integer power an

int power(int a, int n)

{
if(n == 0)
return 1;
if(n == 1)
return a;
else if(n % 2 == 1)
return power(a, n / 2) * power(a, n / 2) * a;
else
return power(a, n / 2) * power(a, n / 2);
}
Complexity = ? Can we improve it?
Second try: Computing integer power an
int power(int a, int n)
{
if(n == 0)
return 1;
if(n == 1)
return a;
int partial = power(a, n/2);

if(n % 2 == 1)
return partial * partial * a;
else
return partial * partial;
}
Complexity = ?
 Strassen’s Matrix Multiplication
Strassen observed [1969] that the product of
two matrices can be computed as follows:
Let A and B be two n × n matrices where n is a
power of 2.

35
 Formulas for Strassen’s Algorithm
M1 = (A00 + A11)  (B00 + B11)

M2 = (A10 + A11)  B00

M3 = A00  (B01 - B11)

M4 = A11  (B10 - B00)

M5 = (A00 + A01)  B11

M6 = (A10 - A00)  (B00 + B01)

M7 = (A01 - A11)  (B10 + B11)

36
 Analysis of Strassen’s Algorithm
If n is not a power of 2, matrices can be padded
with zeros.

Number of multiplications:
M(n) = 7M(n/2), M(1) = 1
Solution: M(n) = 7log 2n ≈ n2.807 vs. n3 of brute-force
alg.
Algorithms with better asymptotic efficiency are
known but they are even more complex.
37
 Closest-Pair Problem
• Remember the problem
• Brute-force solution ?
• Efficiency of brute force approach ?
 Closest-Pair Problem by Divide-and-Conquer
Step 1 Divide the points given into two subsets Pl
and Pr by a vertical line x = m so that half the
points lie to the left or on the line and half the
points lie to the right or on the line.
x=m

dl
d
r

d d 39
 Closest Pair by Divide-and-Conquer (cont.)
Step 2 Find recursively the closest pairs for the left and right
subsets.
Step 3 Set d = min{dl, dr}
We can limit our attention to the points in the symmetric
vertical strip S of width 2d as possible closest pair. (The
points are stored and processed in increasing order of
their y coordinates.)
Step 4 Scan the points in the vertical strip S from the lowest up.
For every point p(x,y) in the strip, inspect points in
in the strip that may be closer to p than d.

A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 5 ©2012 Pearson Education, Inc.
Upper Saddle River, NJ. All Rights Reserved.
40
 Efficiency of the Closest-Pair Algorithm

Running time of the algorithm is described by

T(n) = 2T(n/2) + M(n), where M(n)  O(n)

By the Master Theorem (with a = 2, b = 2, d = 1)

T(n)  O(n log n)

41
 Summary

Divide-and-Conquer
• Divide-and conquer is a general
algorithm design paradigm:
– Divide: divide the input data S in
two or more disjoint subsets S1, S2,
…
– Recur: solve the subproblems
recursively
– Conquer: combine the solutions for
S1, S2, …, into a solution for S
• The base case for the recursion
are subproblems of constant size
• Analysis can be done using
recurrence equations