Recursion tree method

Solve T(n) = 2 ⋅ T(n /2) + Θ(n)

T (n) inputs of length n

inputs of length n/2

T (n / 2) T (n / 2)

T (n / 4) T (n / 4) T (n / 4) T (n / 4) inputs of length n/4

T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8)

⋮
T(1) T(1) ………. base case: input n=1

22
 Recursion tree method — TopHat Question 2

Question. How many layers?

Solve T(n) = 2 ⋅ T(n /2) + Θ(n)
A. constant
B. log2 n
T (n) C. log4 n
D. n
E. n2
T (n / 2) T (n / 2)

T (n / 4) T (n / 4) T (n / 4) T (n / 4)

T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8)

⋮
T(1) T(1) ……….

23
 Recursion tree method — TopHat Question 3

Question. How many leaves?

Solve T(n) = 2 ⋅ T(n /2) + Θ(n)
A. constant
B. log2 n
T (n) C. log4 n
D. n
E. n2
T (n / 2) T (n / 2)

T (n / 4) T (n / 4) T (n / 4) T (n / 4)

T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8)

⋮
T(1) T(1) ……….

24
 Recursion tree method

Solve T(n) = 2 ⋅ T(n /2) + Θ(n)

T (n) inputs of length n

inputs of length n/2

T (n / 2) T (n / 2)

log 2 n T (n / 4) T (n / 4) T (n / 4) T (n / 4) inputs of length n/4

T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8)

⋮
T(1) T(1) ………. #leaves = n base case: input n=1

25
Recursion tree method — analysis idea

• charge each operation to the function call (i.e. tree node) at which it is executed
• for Mergesort on a subarray of length k we do O(k) work to compute the split point
and do the merge
• work in further recursive calls is counted separately

T(n) = 2 ⋅ T(n /2) + cn where c > 0

T (n)
use cn instead of Θ(n)

T (n / 2) T (n / 2)

T (n / 4) T (n / 4) T (n / 4) T (n / 4)

T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8)

T(1) T(1) ……….

⋮
Recursion tree method — analysis idea

• charge each operation to the function call (i.e. tree node) at which it is executed
• for Mergesort on a subarray of length k we do O(k) work to compute the split point
and do the merge
• work in further recursive calls is counted separately

T(n) = 2 ⋅ T(n /2) + cn where c > 0

cn(n)
T
use cn instead of Θ(n)

Tcn/2
(n / 2) T cn/2
(n / 2)

T (n / 4) T (n / 4) T (n / 4) T (n / 4)

T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8) T (n / 8)

T(1) T(1) ……….

⋮
Recursion tree method — analysis idea

• charge each operation to the function call (i.e. tree node) at which it is executed
• for Mergesort on a subarray of length k we do O(k) work to compute the split point
and do the merge
• work in further recursive calls is counted separately
amount of work per level:
T(n) = 2 ⋅ T(n /2) + cn where c > 0

cn(n)
T c⋅n
use cn instead of Θ(n)

Tcn/2
(n / 2) T cn/2
(n / 2) 2 ⋅ c ⋅ n /2

Tcn/4
(n / 4) T cn/4
(n / 4) Tcn/4
(n / 4) Tcn/4
(n / 4) 4 ⋅ c ⋅ n /4

T (n / 8) T (n / 8) T (n / 8) T (n / 8) cn/2
Tk(n / 8) T (n / 8) T (n / 8) T (n / 8) 2k ⋅ c ⋅ n /2k

Θ(1) Θ(n)

Total: Θ(n log n)

binary search

Input: sorted array A, integer x

Output: the index in A where the int x is located or “x is not in A”

1.Divide: compare x to middle element

2.Conquer: recursively search one subarray
3.Combine: trivial

example: find 16

2 3 7 10 11 14 16 18 20 23

29
binary search - running time

Algorithm 1: BinarySearch( sorted array A, int p, int r, query )

/* find the index of query in A (if present) */
1 q b p+r
2 c;
2 middle A[q];
3 if query == middle then
4 return q;
5 if query < middle then
6 BinarySearch(A, p, q, query);
7 else
8 BinarySearch(A, p, q, query);
9 return q not in A;

30
binary search — running time

Algorithm 1: BinarySearch( sorted array A, int p, int r, query )

/* find the index of query in A (if present) */
1 q b p+r
2 c;
2 middle A[q];
3 if query == middle then O(c)
4 return q;
5 if query < middle then
6 BinarySearch(A, p, q, query);
recursive call on array half the size
7 else
8 BinarySearch(A, p, q, query);
9 return q not in A;

T(n) = 1 ⋅ T(n /2) + Θ(1)

#of subproblem work dividing

subproblems size and combining

31
find closed form by “telescoping”
Reminder: our goal is to find a closed form formula for the recurrence

method:
• substitute the recursive formula until we get a good guess
• use induction to prove the formula

T(n) = T(n /2) + c = T(n /4) + 2c = …

= T(n /23) + 3c = … = T(n /2log n) + (log n)c = Θ(log n)

proof by “telescoping”

claim. T(n) = Θ(log n)

proof. by induction on n.
• base case: for n = 2 (or any constant) it takes const comparisons and log n = const
• suppose that the formula is true for every k < n
• proof for n: substitute k = n/2

T(n) = T(n /2) + Θ(1) = Θ(log(n /2)) + Θ(1) = Θ(log n)

Proof by “telescoping” for mergesort

Proposition. If T (n) satisfies the following recurrence, then T (n) = n log2 n.

assuming n
( is a power of 2
1 n == 1
T (n) = n
2T 2 + cn otherwise

Pf. T (n)  2 · T n
+ cn  4 · T n
+ 2cn 
2 4

n 4 n
Substitute the formula in to T(n/2)
8·T 8 + 3cn  2 · T 24 + 4cn 

···

n
 2k T 2k
+ kcn

The last line corresponds to the base case. We know that the base case is T(1) = 0

T (1) = T ( 2nk ) =) k = log(n)

base case

by substituting log(n) for k we get

n
T (n)  2log(n) · T 2log(n)
+ log(n) · n = n + O(n log n)

34
Master Theorem
For recurrences defined as

T (n) = aT (n/b) + f (n) in which a > 0, b ≥ 1 are constants

Then the overall computational cost is given by

logb n
%
logb a
T (n) = Θ(n )+ ak −1 f (n/bk −1 )
! "# $
leaves !k =1 "# $
internal nodes

Such that

log a if f (n) = O(nlogb a−! )
Θ(n b )

T (n) = Θ(nlogb a log n) if f (n) = Θ(nlogb a )


Θ(f (n)) if f (n) = Ω(nlogb a+! ) and af (n/b) ≤ cf (n)

in which a ≥ 1, b > 1, c < 1 and ! > 0 are constants.

Solve the following recurrences

1. T (n) = 4T (n/2) + n
2. T (n) = 4T (n/2) + n2
3. T (n) = 4T (n/2) + n3
Asymptotic time to execute integer operations

Basic operations:
adding, subtracting, multiplying, dividing

Complexity of mathematical operations:

• Usually in this class we count them as constant time
• Today: we look at algorithms for efficiently implementing these basic operations.
• Today: in our analysis one computational step is counted as one bit operation.

Unit of measurement will be bit operations.

Input: two n-bit long integers a , b

Output: arithmetic operation on the two integers

2
Integer addition

Addition. Given two n-bit integers a and b, compute a + b.

Subtraction. Given two n-bit integers a and b, compute a – b.

Grade-school algorithm. Θ(n) bit operations.

2 1 3
1 1 0 1 0 1 0 1
+ 1 2 5
+ 0 1 1 1 1 1 0 1

3
Integer addition

Addition. Given two n-bit integers a and b, compute a + b.

Subtraction. Given two n-bit integers a and b, compute a – b.

Grade-school algorithm. Θ(n) bit operations.

1 1 1 1 1 1 0 1
2 1 3
1 1 0 1 0 1 0 1
+ 1 2 5
+ 0 1 1 1 1 1 0 1
3 3 8
1 0 1 0 1 0 0 1 0

4
Integer addition

Addition. Given two n-bit integers a and b, compute a + b.

Subtraction. Given two n-bit integers a and b, compute a – b.

Grade-school algorithm. Θ(n) bit operations.

an-1 an-2…a2 a1 a0
+ bn-1 bn-2…b2 b1 b0

bits of a + b

Remark. Grade-school addition and subtraction algorithms are asymptotically

optimal.

5
Integer multiplication

Multiply 21310 x 12510 = 110101012 x 011111012

・(subscript refers to decimal and binary representation)
Grade-school algorithm:

2 1 3

x 1 2 5

6
Integer multiplication

Multiply 21310 x 12510 = 110101012 x 011111012

・(subscript refers to decimal and binary representation)
Grade-school algorithm:

2 1 3

x 1 2 5

1 0 6 5

4 2 6

+ 2 1 3

2 6 6 2 5

7
 TopHat

Multiplication. Given two n-bit integers a and b, compute a × b.

Grade-school algorithm.

Question: What is the running time? 1 1 0 1 0 1 0 1

A. O(1) × 0 1 1 1 1 1 0 1

B. O(log n) 1 1 0 1 0 1 0 1
0 0 0 0 0 0 0 0
C. O(n)
1 1 0 1 0 1 0 1
D. D. (n2)
1 1 0 1 0 1 0 1
1 1 0 1 0 1 0 1
1 1 0 1 0 1 0 1
1 1 0 1 0 1 0 1
0 0 0 0 0 0 0 0
0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 1

8
Integer multiplication

Multiplication. Given two n-bit integers a and b, compute a × b.

Grade-school algorithm. Θ(n2) bit operations.

1 1 0 1 0 1 0 1
× 0 1 1 1 1 1 0 1
1 1 0 1 0 1 0 1
0 0 0 0 0 0 0 0
n bits each 1 1 0 1 0 1 0 1 n intermediate
total of O(n2) bits 1 1 0 1 0 1 0 1 products
1 1 0 1 0 1 0 1
1 1 0 1 0 1 0 1
1 1 0 1 0 1 0 1
0 0 0 0 0 0 0 0
0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 1

2n additions

9
Integer multiplication

Multiplication. Given two n-bit integers a and b, compute a × b.

Grade-school algorithm. Θ(n2) bit operations.

an-1 an-2…a2 a1 a0
+ bn-1 bn-2…b2 b1 b0

n bits
n bits each n intermediate
n bits
total of O(n2) bits products
… n bits
+

2n additions a x b 2n bits

Conjecture. [Kolmogorov 1952] Grade-school algorithm is optimal.

Theorem. [Karatsuba 1960] Conjecture is wrong. (his result O(nlog2 3 ) = O(n1.59... ) ) 10
Multiplying large integers

Input: n bit integers a and b

Compute: ab

computing ab = multiplying two n bit integers ~ O(n2) time

Divide-and-conquer:
・subproblem: same problem on smaller input
- input size here is the number of bits in the integers
- goal: compute ab by multiplying n/2 bit integers and then combine them

11
Divide: n bits to n/2
Input: n-bit integers a , b
Compute: ab

How to divide n-bit integers into n/2-bit ints?

355710 = 3500 + 57 = 35 x 102 + 57

355710 = 1101111001012 = 1101110000002 + 1001012 = 1101112 x 26 + 1001012

Note: in decimal multiplying by 10k shifts the number k bits to the left. Same in
binary, multiplying by 2k shifts k bits to the left (glues k 0s at the end.)
12
TopHat

Question. Write the representation of the base-3 number 1201203 as two parts of
n/2 digit base-3 integers.

A. 2 ⋅ 1203

B. 3 ⋅ 1203 + 1203

3
C. 120 3 ⋅ 3 + 1203

6 3
D. 120 3 ⋅ 3 + 120 3 ⋅ 3
Divide: n bits to n/2

Input: integers a ,b of length n bits

How to “divide”? What are the subproblems?

・split a in to two substrings A1, A0 of length n/2

a = an 1 an 2 . . . a n2 a n2 1 . . . a2 a1 a0
A1 = an−1an−2…a n2 A0 = a n2 −1…a1a0

・then a = A12n/2 + A0
・same for b: b = B12n/2+B0

14
Multiplying large integers using D&C

Input: n bit integers a and b

Compute: ab
・A1, A0, B1, B0 are n/2 bit integers
ab consists of multiplying 2 n bit integers ~ O(n2) time

a = A12n/2 + A0 b = B12n/2 + B0

Compute:
ab = (A12n/2 + A0)(B12n/2 + B0)=

15
Multiplying large integers using D&C

Input: n bit integers a and b

Compute: ab
・A1, A0, B1, B0 are n/2 bit integers
ab consists of multiplying 2 n bit integers ~ O(n2) time

a = A12n/2 + A0 b = B12n/2 + B0

Compute ab = A1B12n + (A1B0 + A0B1)2n/2 + A0B0

・this formula consists of 4 multiplications of n/2 bit numbers and 3 additions

Divide and conquer approach: multiply n/2 bit integers recursively

16
Multiplying large integers using D&C

(reminder: ab = A1B12n + (A1B0 + A0B1)2n/2 + A0B0)

Algorithm 1: Multiply(ints a, b, n)
1 if n == 1 then
2 return ab;
3 m b n2 c;
4 A0 ba/2m c /* integer division */
5 B0 bb/2m c /* integer division */
6 A1 a mod 2m ;
7 B1 b mod 2m ;
8 x Multiply(A1 , B1 , m);
9 y Multiply(A1 , B0 , m); recursive call on input of size n/2
10 z Multiply(A0 , B1 , m);
11 w Multiply(A0 , B0 , m);
12 return x22m + (y + z)2m + w;

Note that lines 4-7 can be implemented by simple bit-shifts — no real division
is taking place!

Recurrence:

17
Solve the recurrence
(
1 n=1
T (n) 
4T ( n2 ) + O(n) otherwise

18
Solve the recurrence
(
1 n=1
T (n) 
4T ( n2 ) + O(n) otherwise

n
Use telescoping and solve T(n) = 4 ⋅ T( ) + cn
2

( 2)
n n n 2 2 n
T(n) = 4 ⋅ T( ) + cn = 4 4 ⋅ T( ) + c + cn = (2 ) ⋅ T( 2 ) + 3cn = …
2 4 2

2 3 n 2 k n
= (2 ) ⋅ T( 3 ) + 5cn = … = (2 ) ⋅ T( k ) + (k + 1)cn = …
2 2

we want to substitute the base case T(1), which happens when k =

log n

= (22)log nT(1) + (log n)cn = n 2 + Θ(1)n log n = Θ(n 2)

Now we can prove by induction that T(n) = Θ(n 2)

19
General formula for recurrences of specific form

Recurrence:

(2)
n
T(n) = 4T + cn = cn log24 = Θ(n 2)

General form:

(2)
n
T(n) = qT + cn = cn log2 q for q > 2

Discussed in detail: Kleinberg-Tardos pages 214 - 218

20
Multiplying large integers using D&C

(reminder: ab = A1B12n + (A1B0 + A0B1)2n/2 + A0B0)

Algorithm 1: Multiply(ints a, b, n)
1 if n == 1 then
2 return ab;
3 m b n2 c;
4 A0 ba/2m c /* integer division */
5 B0 bb/2m c /* integer division */
6 A1 a mod 2m ;
7 B1 b mod 2m ;
8 x Multiply(A1 , B1 , m);
9 y Multiply(A1 , B0 , m); recursive call on input of size n/2
10 z Multiply(A0 , B1 , m);
11 w Multiply(A0 , B0 , m);
12 return x22m + (y + z)2m + w;

Note that lines 4-7 can be implemented by simple bit-shifts — no real division
is taking place!

Recurrence: T (n) = 4T ( n2 ) + cn T (n) = ⇥(n2 )

21
Karatsuba’s algorithm

Input: n bit integers a and b

Compute: ab
a = A12n/2 + A0
b = B12n/2 + B0

Compute ab = A1B12n + (A1B0 + A0B1)2n/2 + A0B0

Karatsuba’s trick: (A1+A0)(B1+B0) = A1B1 + A0B0 + (A1B0 + A0B1)

22
Karatsuba’s algorithm

Input: n bit integers a and b

Compute: ab
a = A12n/2 + A0
b = B12n/2 + B0

Compute ab = A1B12n + (A1B0 + A0B1)2n/2 + A0B0

Karatsuba’s trick: (A1+A0)(B1+B0) = A1B1 + A0B0 + (A1B0 + A0B1)

23
Karatsuba’s algorithm

Input: n bit integers a and b

Compute: ab
a = A12n/2 + A0
b = B12n/2 + B0

Compute ab = A1B12n + (A1B0 + A0B1)2n/2 + A0B0

Karatsuba’s trick: (A1+A0)(B1+B0) = A1B1 + A0B0 + (A1B0 + A0B1)

Thus we get x = A1B1 , y = A0B0, z = (A1+A0)(B1+B0) multiply n/2 bit integers

n/2 bits!
ab = A1B12n + (A1B0 + A0B1)2n/2 + A0B0 = x2n + (z - x - y)2n/2 + y

The new formula contains only 3 multiplications of n/2 bits.

24
Karatsuba’s algorithm

Algorithm 1: Karatsuba(ints a, b, n)
1 if n == 1 then
2 return ab;
3 m b n2 c;
4 A0 ba/2m c /* integer division */
5 B0 bb/2m c /* integer division */
6 A1 a mod 2m ;
7 B1 b mod 2m ;
8 x Karatsuba(A1 , B1 , m);
9 y Karatsuba(A0 , B0 , m);
10 z Karatsuba(A1 + A0 , B1 + B0 , m);
11 return x22m + (z x y)2m + y;

25
Karatsuba’s algorithm - TopHat

Algorithm 1: Karatsuba(ints a, b, n)
1 if n == 1 then
2 return ab;
3 m b n2 c;
4 A0 ba/2m c /* integer division */
5 B0 bb/2m c /* integer division */
6 A1 a mod 2m ;
7 B1 b mod 2m ;
8 x Karatsuba(A1 , B1 , m);
9 y Karatsuba(A0 , B0 , m);
10 z Karatsuba(A1 + A0 , B1 + B0 , m);
11 return x22m + (z x y)2m + y;

Question: What is the recurrence for this algorithm?

n
A. T(n) = Θ(n) C. T(n) = 3 ⋅ T( ) + Θ(n)
2
n n
B. T(n) = 2 ⋅ T( ) + Θ(n) D. T(n) = 3 ⋅ T( ) + Θ(1)
3 3
26
Karatsuba’s algorithm

Algorithm 1: Karatsuba(ints a, b, n)
1 if n == 1 then
2 return ab;
3 m b n2 c;
4 A0 ba/2m c /* integer division */
5 B0 bb/2m c /* integer division */
6 A1 a mod 2m ;
7 B1 b mod 2m ;
8 x Karatsuba(A1 , B1 , m);
9 y Karatsuba(A0 , B0 , m);
10 z Karatsuba(A1 + A0 , B1 + B0 , m);
11 return x22m + (z x y)2m + y;

T (n) = 3T ( n2 ) + cn

T (n) = ⇥(nlog2 3 ) = ⇥(n1.59... )

27
Recursion tree method
work per procedure
n
Solve T (n) = 3T ( 2 ) + cn where c 2 const

cn cn

3
c n2 2 cn
c n2 c n2

9
c n4 c n4 c n4 n cn
c4 4 c n4 4 cn

( 32 )k cn

28
 Recursion tree method - TopHat
work per procedure
n
Solve T (n) = 3T ( 2 ) + cn where c 2 const

cn cn

3
c n2 2 cn
c n2 c n2

9
c n4 c n4 c n4 n cn
c4 4 c n4 4 cn

Question: what is the hight of the tree and number of nodes at ( 32 )k cn

layer k? (you may assume the root is at layer 0)

A. log3 n & n k +
B. log2 n & 3k
C. log2 n & n 3
D. log3 n & k 3
29
 Recursion tree method
work per procedure
n
Solve T (n) = 3T ( 2 ) + cn where c 2 const

cn cn

3
c n2 2 cn
h = log2 n c n2 c n2

9
c n4 c n4 c n4 n cn
c4 4 c n4 4 cn

( 32 )k cn

log3 n
( 32 )= 1 log2 n
n3 =
1
+
= 3log3 n ⇤ 3 log3 2
= n3log2 3
( 32 )log3 n cn = cnlog2 3 = cn1.59...

30
General formula for recurrences of specific form

Recurrence for Karatsuba’s:

T (n) = 3T ( n2 ) + cn = cnlog2 3
General form:

T (n) = qT ( n2 ) + cn = cnlog2 q for q > 2

Discussed in detail: Kleinberg-Tardos pages 214 - 218

31
 History of asymptotic complexity of integer multiplication

year algorithm order of growth

? grade-school Θ(n2)
Faster than grade-

school algorithm for 1962 Karatsuba–Ofman Θ(n1.585)

about 320–640 bits. 1963 Toom-3, Toom-4 Θ(n1.465), Θ(n1.404)

1966 Toom–Cook Θ(n1 + ε)

1971 Schönhage–Strassen Θ(n log n log log n)

2007 Fürer n log n 2 O(log*n)

2019 Harvey, van der Hoeven ⇥(n log n)

number of bit operations to multiply two n-bit integers

Remark. GNU Multiple Precision Library uses one of five

different algorithms depending on size of operands.

32