Divide and Conquer - Part 1

Design and
Analysis of
Algorithms
DIVIDE AND CONQUER
PART I
GENERAL TEMPLATE
BINARY SEARCH
MERGE SORT & QUICK SORT
SOLVING RECURRENCE RELATIONS

 Instructor
Prof. Amrinder Arora
amrinder@gwu.edu
Please copy TA on emails
Please feel free to call as well

 Available for study sessions
Science and Engineering Hall
GWU
Algorithms Divide and Conquer - Part I 2
LOGISTICS

Algorithms
Analysis
Asymptotic
NP-
Completeness
Design
D&C
DP
Greedy
Graph
B&B
Applications
WHERE WE ARE

 A technique to solve complex problems by breaking into
smaller instances of the problem and combining the results
 Recursive methodology – Smaller instances of the same type of
problem
 Typically used accompaniments
 Induction for proving correctness
 Recurrence relation solving for computing time (and/or space)
complexity
DIVIDE AND CONQUER

By definition: For D&C, sub
problems must be of same
type.
[The phrase “D&C” is also used
in other contexts. It may refer
to breaking down a task, but in
Computer Science, D&C is a
formal paradigm]
D&C – CS, NOT MANAGEMENT/POLITICS

 A recursive algorithm is an algorithm that calls itself on
smaller input.
 Algorithm sort (Array a)
Begin
sort (subarray consisting of first half of a)
sort (subarray consisting of second half of a)
do_something_else();
End
RECURSION

 Recurrence Relation is a recursive formula, commonly used to
analyze the time complexity of recursive algorithms
 For example
 T(n) = T(n/2) + T(n/2) + n2
 T(n) = a T(n/b) + f(n)
 Note: Recurrence Relations have uses outside of time
complexity analysis as well (for example in combinatorics),
but for the purpose of this lecture, this is the main use case.
RECURRENCE RELATIONS

 Wikipedia says: “…it is often necessary to replace the original
problem by a more general or complicated problem in order to
get the recursion going, and there is no systematic method for
finding the proper generalization.”
 Refer to this as the “generalization” step
 Sometimes counterintuitive that making a “generalization”, that is,
making the problem harder actually helps in solving it!
HOW TO D&C

divide_conquer(input J)
{
// Base Case
if (size of input is small enough) {
solve directly and return
}
// Divide Step
divide J into two or more parts J1, J2,...
// Recursive Calls
call divide_conquer(J1) to get a subsolution S1
...
// Merge Step
Merge the subsolutions S1, S2,...into a global solution S
return S
}
GENERAL TEMPLATE

 Number of subproblems that you create in the “divide” step
 This plays a role in the recurrence relation that is created for
analysis
 T(n) = a T(n/b) + f(n)
Here “a” branches, each with size “n/b”, and f(n) time spent in
dividing and merging
 Example: T(n) = T(n/2) + 1
1 branch, size half and constant time spent in dividing and merging
NUMBER OF BRANCHES

divide_conquer(input J)
{
// Base Case
if (size of input is small enough) {
solve directly and return
}
// Divide Step
divide J into two or more parts J1, J2,...
// Recursive Calls
...
// Merge Step
Merge the subsolutions S1, S2,...into a global solution S
return S
}
GENERAL TEMPLATE – TIME COMPLEXITY
VIEW
Combined time
in steps other
than recursive
calls: f(n)
a recursive calls of size
n/b each. Total time:
a T(n/b)

 Binary Search
 Merge Sort
 Quick Sort
D&C – EXAMPLE ALGORITHMS

 Search (A, low, high, key)
 Mid = (low + high) / 2
 Compare A[mid] to key, and look either in left half or in right half
 T(n) = T(n/2) + 1
 T(n) = O(log n)
BINARY SEARCH

 Classic problem: Given an array, to sort it
 Generalization step: Given an array and indexes i and j (start and end)
to sort that portion of it
 Algorithm MergeSort (input: A,i,j) {
// Divide portion
if (j – i < THRESHOLD) {
InsertionSort(A,i,j)
Return
}
int k=(i+j)/2
// Recursive Calls
MergeSort(A,i,k)
MergeSort(A,k+1,j)
// Merge Calls
Merge(A,i,k,k+1,j)
}
MERGE SORT

 How to merge two lists effectively?
MERGING

 T(n) = 2T(n/2) + (n)
 Need some methods for solving such recurrence equations
 Substitution method
 Recursion tree method (unfolding)
 Master theorem
 T(n) = (n log n)
TIME COMPLEXITY OF MERGE SORT

EXAMPLES OF RECURRENCE RELATIONS

 Examples:
 T(n) = 2 T(n/2) + cn
T(n) = O (n log n)
 T(n) = T(n/2) + n
T(n) = O (n)
 3 General Approaches:
 Substitution method (Guess and Prove)
 Recursion tree method (unfold and reach a pattern)
 Master theorem
SOLVING RECURRENCE RELATIONS

 Given T(n) = 2 T(n/2) + cn
 We first “guess” that the solution is O(n log n)
 To prove this using induction, we first assume T(m) <= km log
m for all m < n
 Then T(n) = 2 T(n/2) + cn
<= 2 kn/2 log (n/2) + cn
= kn log n – (k – c)n // log (n/2) = log n – 1
<= k n log n, as long as k >= c
SUBSTITUTION METHOD FOR MERGE SORT

MASTER THEOREM FOR SOLVING
RECURRENCE RELATIONS
Only applies to Recurrence Relations of following type
T(n) = aT(n/b) + f(n)
 Case 1. If f(n) = O(nc) where c < logb a, then T(n) = θ(n^logb a)
 Case 2. If it is true, for some constant k ≥ 0, that f(n) = θ(nc
logk n) where c = logb a, then T(n) = θ(nc logk+1 n)
 Case 3. If it is true that f(n) = Ω(nc) where c > logb a, then T(n)
= θ(f(n))

T(n) = a T(n/b) + f(n)
If we unfold this, we get an expression like:
T(n) = ak T(n/bk) + f(n) + a f(n/b) + … + ak f(n/bk)
Then, for k ≈ logbn, T(n/bk) will be a small constant, and we can
assume T(n/bk) = 1.
Then, T(n) = a^(logbn) + f(n) + af(n/b) + … + ak f(n/bk)
= n^(logba) + f(n) + af(n/b) + … + ak f(n/bk)
We note that there are about logbn terms.
MASTER THEOREM – INTUITION
Intuition != Formal Proof

T(n) = n^(logba) + f(n) + af(n/b) + … + ak f(n/bk)
We observe that:
• If f(n) is very small, say a constant, then the first term dominates
• If f(n) =  (n^(logba)), then the T(n) = f(n) log n.
// The log n factor arises because there are ~ log n terms
• If f(n) is too large, then f(n) terms dominate
MASTER THEOREM – INTUITION (CONT.)

T(n) = 2 T(n/2) + c n
In this case:
• a = b = 2
• f(n) = c n
• logba = 1
• n^(logba) = n
So, f(n) =  (n^(logba))
Therefore, by Master Theorem,
T(n) = (f(n) log n)
That is, T(n) = (n log n)
APPLYING MASTER THEOREM TO MERGE
SORT RECURRENCE

 Select a “partition” element
 Partition the array into “left” and “right” portions (not
necessarily equal) based on the partition element
 Sort the left and right sides
 An inverted view of mergesort – spend time upfront
(partition), no need to merge later.
QUICKSORT

 quicksort(A,p,r)
if (p < r) {
q = partition (A,p,r)
quicksort(A,p,q-1)
quicksort(A,q+1,r)
}
QUICKSORT – THE PSEUDO CODE

 Invented in 1960 by C. A. R. Hoare
 More widely used than any other sort
 A well-studied, not difficult to implement algorithm
 R. Sedgewick – 1975 Ph.D. thesis at Stanford Univ. –
Analysis and Variations of QuickSort
QUICKSORT (THE TRIVIA CLUB VIEW)
Who said: “Elementary, My Dear Watson”?

 “There are two ways of constructing a software design: One
way is to make it so simple that there are obviously no
deficiencies, and the other way is to make it so complicated
that there are no obvious deficiencies. The first method is far
more difficult.”
 “We should forget about small efficiencies, say about 97% of
the time: premature optimization is the root of all evil.”
QUOTES, QUOTES

 How to find a good partition element
 How to partition (efficiently)
 Partition array so that:
 Some partitioning element (q) is its final position
 Every element smaller than q is to the left of q
 Every element larger than q is to the right of q
 Sedgwick states that “improving QuickSort is the better
mousetrap of computer science”
CENTRAL PROBLEM IN QUICKSORT

 T(n) = T(n1) + T(n2) + O(n)
Where n1 + n2 = n – 1
 So it all depends upon the kind of the split, and split will
likely not be the same each time.
 Worst case – very bad split: O(n2)
 Best case – good split: O(n log n)
 Average case – where does that fit?
http://mathworld.wolfram.com/Quicksort.html
QUICKSORT – TIME COMPLEXITY ANALYSIS

How long will the algorithm take?
Function sum (integer a) {
if (a == 1) exit;
if (a is odd) {
a = 3a + 1
} else {
a = a/2
}
}
Trichotomy – Extended
Given two functions f(n) and g(n), both strictly increasing with n,
is it possible that f(n) and g(n) cannot be compared
asymptotically?
OPEN QUESTIONS

1. Median Finding
(Textbook § 4.6)
2. Closest pair of
points algorithm
(Textbook § 4.7)
3. Strassen’s
algorithm for
matrix
multiplication
(Textbook § 4.8)
https://youtu.be/1AIvlizGo7Y
READING ASSIGNMENTS
We already have quite a few people who know how
to divide. So essentially we are now looking for
people who know how to conquer.

Algorithms
Analysis
Asymptotic
NP-
Completeness
Design
D&C
DP
Greedy
Graph
B&B
Applications
WHERE WE ARE

Divide and Conquer - Part 1

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Divide and Conquer - Part 1 (20)

More from Amrinder Arora (20)

Recently uploaded (20)

Divide and Conquer - Part 1