Introduction to Algorithms

6.046J

Lecture 2
Prof. Shafi Goldwasser
 Solving recurrences
• The analysis of integer multiplication
from Lecture 1 required us to solve a
recurrence.
• Recurrences are a major tool for
analysis of algorithms
• Today: Learn a few methods.
• Lecture 3: Divide and Conquer
algorithms which are analyzable by
recurrences.
L2.2
 Recall: Integer Multiplication

• Let X = A B and Y = C D where A,B,C

and D are n/2 bit integers
• Simple Method: XY = (2n/2A+B)(2n/2C+D)
• Running Time Recurrence
T(n) < 4T(n/2) + 100n

How do we solve it?

L2.3
 Substitution method
The most general method:
1. Guess the form of the solution.
2. Verify by induction.
3. Solve for constants.
Example: T(n) = 4T(n/2) + 100n
• [Assume that T(1) = Θ (1).]
• Guess O(n3) . (Prove O and Ω separately.)
• Assume that T(k) ≤ ck3 for k < n .
• Prove T(n) ≤ cn3 by induction.
L2.4
 Example of substitution
T ( n) = 4T ( n / 2) + 100n
≤ 4c(n / 2)3 + 100n
= (c / 2) n3 + 100n
= cn3 − ((c / 2) n3 −100n )
desired – residual
≤ cn 3 desired
whenever (c/2)n3 – 100n ≥ 0, for
example, if c ≥ 200 and n ≥ 1.
residual

L2.5
 Example (continued)
• We must also handle the initial conditions,
that is, ground the induction with base
cases.
• Base: T(n) = Θ (1) for all n < n0, where n0
is a suitable constant.
• For 1 ≤ n < n0, we have “Θ (1)” ≤ cn3, if
we pick c big enough.

This bound is not tight!

L2.6
 A tighter upper bound?
We shall prove that T(n) = O(n2).
Assume that T(k) ≤ ck2 for k < n:
T ( n) = 4T (n / 2) +100n
≤ cn2 +100n
2
≤ cn

for no choice of c > 0. Lose!

L2.7
 A tighter upper bound!
IDEA: Strengthen the inductive hypothesis.
• Subtract a low-order term.
Inductive hypothesis: T(k) ≤ c1k2 – c2k for k < n.
T ( n) = 4T ( n / 2) + 100n
≤ 4(c1 (n / 2) 2 − c2 (n / 2))+ 100n
= c1n 2 − 2c2 n + 100n
= c1n 2 − c2 n − (c2 n −100n)
≤ c1n 2 − c2 n if c > 100.
2
Pick c1 big enough to handle the initial conditions.
L2.8
 Recursion-tree method
• A recursion tree models the costs (time) of a
recursive execution of an algorithm.
• The recursion tree method is good for
generating guesses for the substitution
method.
• The recursion-tree method can be unreliable,
just like any method that uses ellipses (…).
• The recursion-tree method promotes intuition,
however.
L2.9
 Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:

L2.10
 Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:
T(n)

L2.11
 Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2
T(n/4) T(n/2)

L2.12
 Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2 (n/2)2

T(n/16) T(n/8) T(n/8) T(n/4)

L2.13
 Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2 (n/2)2

(n/16)2 (n/8)2 (n/8)2 (n/4)2

…

Θ (1)

L2.14
 Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2 n2
(n/4)2 (n/2)2

(n/16)2 (n/8)2 (n/8)2 (n/4)2

…

Θ (1)

L2.15
 Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2 n2
5 n2
(n/4) 2 (n/2) 2
16
(n/16)2 (n/8)2 (n/8)2 (n/4)2
…

Θ (1)

L2.16
 Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2 n2
5 n2
(n/4) 2 (n/2) 2
16
25 n 2
(n/16)2 (n/8)2 (n/8)2 (n/4)2 256

…
…

Θ (1)

L2.17
 Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2 n2
5 n2
(n/4) 2 (n/2) 2
16
25 n 2
(n/16)2 (n/8)2 (n/8)2 (n/4)2 256

…
…

Θ (1) Total = n
2
( 5 5
1 + 16 + 16
2
( ) +( )5 3
16
+ )
= Θ (n2) geometric series
L2.18
 Appendix: geometric series

n +1
1 − x
1 + x + x2 +  + xn = for x ≠ 1
1− x

2 1
1+ x + x + = for |x| < 1
1− x

Return to last
slide viewed.

L2.19
 The master method

The master method applies to recurrences of

the form
T(n) = a T(n/b) + f (n) ,
where a ≥ 1, b > 1, and f is asymptotically
positive.

L2.20
 Idea of master theorem
Recursion tree:
f (n) f (n)
a
f (n/b) f (n/b) … f (n/b) a f (n/b)
h = logbn a
f (n/b2) f (n/b2) … f (n/b2) a2 f (n/b2)

…
…

#leaves = ah
= alog bn nlog baΤ (1)
Τ (1)
= nlog ba

L2.21
 Three common cases
Compare f (n) with nlog ba:
1. f (n) = O(nlog ba – ε ) for some constant ε > 0.
• f (n) grows polynomially slower than nlog ba
(by an nε factor).
Solution: T(n) = Θ (nlog ba) .

L2.22
 Idea of master theorem
Recursion tree:
f (n) f (n)
a
f (n/b) f (n/b) … f (n/b) a f (n/b)
h = logbn a
f (n/b2) f (n/b2) … f (n/b2) a2 f (n/b2)

…
…

C ASE 1:
CASE 1: The
The weight
weight increases
increases
geometrically
geometrically from
fromthethe root
root to the nlog baΤ (1)
to the
Τ (1) leaves.
leaves. The
The leaves
leaves hold
hold aa constant
constant
fraction
fraction of
of the
the total
total weight.
weight. Θ (nlog ba)
L2.23
 Three common cases
Compare f (n) with nlog ba:

2. f (n) = Θ (nlog ba lgkn) for some constant k ≥ 0.

• f (n) and nlog ba grow at similar rates.
Solution: T(n) = Θ (nlog ba lgk+1 n) .

L2.24
 Idea of master theorem
Recursion tree:
f (n) f (n)
a
f (n/b) f (n/b) … f (n/b) a f (n/b)
h = logbn a
f (n/b2) f (n/b2) … f (n/b2) a2 f (n/b2)

…
…

CCASEASE 2:
2: (k
(k == 0)
0) The
The weight
weight
isis approximately
approximately the the same
same on
on
nlog baΤ (1)
Τ (1) each
each ofof the
the log
logbbnn levels.
levels.
Θ (nlog balg n)
L2.25
 Three common cases (cont.)
Compare f (n) with nlog ba:
3. f (n) = Ω (nlog ba + ε ) for some constant ε > 0.
• f (n) grows polynomially faster than nlog ba (by
an nε factor),
and f (n) satisfies the regularity condition that
a f (n/b) ≤ c f (n) for some constant c < 1.
Solution: T(n) = Θ ( f (n) ) .

L2.26
 Idea of master theorem
Recursion tree:
f (n) f (n)
a
f (n/b) f (n/b) … f (n/b) a f (n/b)
h = logbn a
f (n/b2) f (n/b2) … f (n/b2) a2 f (n/b2)

…
…

C ASE 3:
CASE 3: The
The weight
weight decreases
decreases
geometrically
geometrically from
fromthethe root
root to the nlog baΤ (1)
to the
Τ (1) leaves.
leaves. The
The root
root holds
holds aa constant
constant
fraction
fraction of
of the
the total
total weight.
weight. Θ ( f (n))
L2.27
 Examples

Ex. T(n) = 4T(n/2) + n

a = 4, b = 2 ⇒ nlog ba = n2; f (n) = n.
CASE 1: f (n) = O(n2 – ε ) for ε = 1.
∴ T(n) = Θ (n2).

Ex. T(n) = 4T(n/2) + n2

a = 4, b = 2 ⇒ nlog ba = n2; f (n) = n2.
CASE 2: f (n) = Θ (n2lg0n), that is, k = 0.
∴ T(n) = Θ (n2lg n).

L2.28
 Examples
Ex. T(n) = 4T(n/2) + n3
a = 4, b = 2 ⇒ nlog ba = n2; f (n) = n3.
CASE 3: f (n) = Ω (n2 + ε ) for ε = 1
and 4(cn/2)3 ≤ cn3 (reg. cond.) for c = 1/2.
∴ T(n) = Θ (n3).

Ex. T(n) = 4T(n/2) + n2/lg n

a = 4, b = 2 ⇒ nlog ba = n2; f (n) = n2/lg n.
Master method does not apply. In particular,
for every constant ε > 0, we have nε
= ω (lg n).
L2.29
 Conclusion

• Next time: applying the master method.

• For proof of master theorem, goto section

L2.30