Marco A.

Fernandez

Stanford Research Institute

6 77
Theoretical Press
 This series of lectures is warmly dedicated
to the Burroughs B5000 computer once installed at
Instituto de Ciências Exatas,
in memory of countless sleepless nights.
Volume 1 / Fundamental Data Structures

THE ART OF
ALGORITHM DESIGN
THIRD EDITION

São Paulo · Rio de Janeiro · Brasília · Porto Alegre

Buenos Aires · Santiago · Bogotá · Lima
Lisboa · Madrid · Paris · Berlin
 The author and publisher have taken care in the preparation of this book, but
make no expressed or implied warranty of any kind and assume no responsibility
for errors or omissions. No liability is assumed for incidental or consequential
damages in connection with or arising out of the use of the information or pro-
grams contained herein.

Visit us on the Web: [Link]

Library of Congress Cataloging-in-Publication Data

Fernandez, Marco A., 1975
The art of algorithm design / Marco A. Fernandez.
xv, 580 p. 24 cm.
Includes bibliographical references and index.
Contents: v. 1. Fundamental data structures.
ISBN 978-0-000-00000-0 (v. 1, 3rd ed.)
1. Computer algorithms. 2. Data structures. I. Title.
QA76.9.A43F47 2024 24-00000

Copyright © 2024 by Theoretical Press

All rights reserved. Printed in Brazil. This publication is protected by copyright,
and permission must be obtained from the publisher prior to any prohibited
reproduction, storage in a retrieval system, or transmission in any form or by any
means.

ISBN-13 978-0-000-00000-0
ISBN-10 0-000-00000-0
First digital release, January 2024
 PREFACE

Here is your book, the one your thousands of letters have asked
us
to publish. It has taken us years to do, checking and rechecking
countless results to bring you only the best, only the elegant,
only the correct.

 Imaginary Publisher's Note (2024)

The process of designing algorithms for digital computers is especially at-

tractive, not only because it can be economically and scientically reward-
ing, but also because it can be an aesthetic experience much like composing
poetry or music. This book is the rst volume of a multi-volume set that
has been designed to train the reader in the fundamental skills that go into
an algorithm designer's craft.

The following chapters are not meant to serve as an introduction to pro-

gramming; the reader is supposed to have had some previous experience.
The prerequisites are actually quite simple, but a beginner requires time
and practice in order to internalize the concept of algorithmic thinking.
The reader should possess:

a) Some idea of how a stored-program digital computer works; not nec-

essarily the electronics, but the manner in which instructions can be
kept in memory and successively executed.
b) An ability to put the solutions to problems into such explicit terms
that a computer can understand them. (These machines have no
common sense; they do exactly as they are told, no more and no less.)
c) Some knowledge of the most elementary programming techniques, such
as looping, recursion, and the use of indexed variables.
d) A little knowledge of common algorithmic jargoncomplexity, in-
variant, big-O notation, amortized cost.

v
vi Preface

These four prerequisites can perhaps be summed up into the single require-
ment that the reader should have already written and tested at least four
non-trivial programs.
The complete set of books, entitled The Art of Algorithm Design, has the
following general outline:

Volume 1. Fundamental Data Structures

Chapter 1. Basic Concepts
Chapter 2. Linear and Hierarchical Structures

Volume 2. Graph Algorithms

Chapter 3. Traversals and Connectivity
Chapter 4. Shortest Paths and Flows

Volume 3. Algorithmics
Chapter 5. Sorting and Selection
Chapter 6. Searching and Hashing

Volume 4. Complexity and Computation

Chapter 7. NP-Completeness
Chapter 8. Approximation Algorithms

São Paulo, Brazil M. A. F.

January 2024

Preface to the Third Edition

After having spent ve years rening the exposition of the rst two editions,
I am now able to fulll a long-held dream: the entire text has been re-
typeset using LATEX, and every algorithm has been veried against a modern
reference implementation.
In this new edition I have gone over every word of the text, adding dozens
of new exercises and improving answers to many existing ones. The Art
of Algorithm Design is, however, still a work in progress. Therefore some
parts of this book are headed by an under construction icon, indicating
material not yet fully up-to-date.

Stanford, California D. E. K.
April 1997

Things have changed in the past two decades.

 Bill Gates (1995)
 NOTES ON THE EXERCISES

The exercises in this set of books have been designed for self-study as well
as for classroom study. It is dicult, if not impossible, for anyone to learn
a subject purely by reading about it, without applying the information to
specic problems and thereby being encouraged to think about what has
been read.
Rating numbers have been provided to indicate the level of diculty. These
numbers have the following general signicance:

Rating Interpretation
00 An extremely easy exercise answerable immediately.
10 A simple problem; about one minute.
20 An average problem; about fteen to twenty minutes.
30 A moderately dicult problem; possibly two or more hours.
40 A term-project-level problem.
50 An unsolved research problem.

The rating is preceded by M if the exercise involves signicant mathemat-

ics, and by HM if it requires higher mathematics (calculus or beyond).
Exercises preceded by → are especially recommended.

Summary of codes:
00 Immediate
10 Simple (one minute)
→ Recommended 20 Medium (quarter hour)
M Mathematically oriented 30 Moderately hard
HM Requiring higher math 40 Term project
50 Research problem

Exercises
1. [00] What does the rating M20 mean?
2. [10] Of what value can exercises in a textbook be to the reader?

vii
viii Notes on the Exercises

3. [14] Prove that 73 = 343. Generalize your answer. [This is an exam-

ple of a trivial problem the author has tried to avoid.]
4. [HM45] Prove that when n is an integer, n > 2, the equation xn +y n =
z n has no solution in positive integers x, y, z .
We can face our problem.
We can arrange such facts as we have with order and method.
 Hercule Poirot, in Murder on the Orient Express
(1934)
 CONTENTS

Preface v
Notes on the Exercises vii
1 Basic Concepts 1
1.1 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 Computational Complexity . . . . . . . . . . . . . . 2
1.1.2 Exercises . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Mathematical Preliminaries . . . . . . . . . . . . . . . . . . 3
1.2.1 Mathematical Induction . . . . . . . . . . . . . . . . 3
1.2.2 Exercises . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Linear and Hierarchical Structures 5

2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Linear Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.1 Stacks and Queues . . . . . . . . . . . . . . . . . . . 5
2.2.2 Exercises . . . . . . . . . . . . . . . . . . . . . . . . 6

Answers to Exercises 7
Tables of Numerical Quantities 9
.1 Fundamental Constants (Decimal) . . . . . . . . . . . . . . 9
.2 Fibonacci Numbers . . . . . . . . . . . . . . . . . . . . . . . 9

Index to Notations 11

ix
x CONTENTS
 CHAPTER 1
BASIC CONCEPTS

Many persons who are not conversant with mathematical stud-

ies imagine that because the business of a computer is to give
its results in numerical notation, the nature of its processes
must consequently be arithmetical, rather than algebraical
and analytical. This is an error.

 Augusta Ada, Countess of Lovelace (1843)

1.1. Algorithms
The notion of an algorithm is basic to all of computer science, so we should
begin with a careful analysis of this concept.
The word algorithm itself is quite interesting; at rst glance it may
look as though someone intended to write logarithm but jumbled up the
rst four letters. In fact, the word traces back to a celebrated ninth-
century Persian mathematician, Abu 'Abd Allah [Link] ibn Musa
al-Khwarizmi, whose name, Latinized, yielded algorismus and eventually
algorithm.
It will be instructive to begin with a concrete example. Consider the
following procedure, known since antiquity:
Algorithm 1.1 (Euclid's Algorithm). Given two positive integers m and
n, nd their greatest common divisor.

E1. [Find remainder.] Divide m by n and let r be the remainder. (We

will have 0 ≤ r < n.)
E2. [Is it zero?] If r = 0, the algorithm terminates; n is the answer.
E3. [Reduce.] Set m ← n, n ← r, and go back to step E1. □

The format above illustrates the style in which all algorithms throughout
this book will be presented. Each algorithm is given an identifying letter

1
2 CHAPTER 1. BASIC CONCEPTS

(E in the example above), and its steps are labeled by this letter followed
by a number (E1, E2, E3).
A few properties distinguish a genuine algorithm from a mere set of
rules:

Finiteness. An algorithm must always terminate after a nite number of

steps.
Deniteness. Each step must be precisely dened; the actions to be car-
ried out must be rigorously and unambiguously specied.
Input. An algorithm has zero or more inputs, quantities given to it initially
before the algorithm begins.
Output. An algorithm has one or more outputs, quantities that have a
specied relation to the inputs.
Eectiveness. Every operation performed must be suciently basic that
it can in principle be done exactly in a nite length of time by a person
using pencil and paper.

1.1.1. Computational Complexity

We say that Algorithm 1.1 has time complexity O(log min(m, n)). More
formally, we use the big-O notation :
Denition 1.1.1. Let f and g be functions from N to R+ . We say that
f (n) = O(g(n)) if there exist positive constants c and n0 such that

f (n) ≤ c · g(n) for all n ≥ n0 .

The following theorem establishes the correctness of Euclid's algorithm:

Theorem 1.1.1 (Correctness of Algorithm E). Algorithm
√ 1.1 terminates
after at most 2 logϕ min(m, n) steps, where ϕ = (1 + 5)/2 is the golden
ratio.
Proof. At each iteration, the pair (m, n) is replaced by (n, r) where r =
m mod n < n. By the properties of Fibonacci numbers (see Section 1.2.8),
the values strictly decrease and the algorithm must terminate. The log-
arithmic
√ bound follows from the identity gcd(Fk+1 , Fk ) = 1 and Fk ∼
ϕk / 5.

1.1.2. Exercises
1. [00] What is the value of gcd(12, 8)?
2. [10] Trace through Algorithm E with m = 35, n = 15, listing the
values of m, n, and r at each step.
 1.2. MATHEMATICAL PRELIMINARIES 3

→ [20] Prove that Algorithm E always terminates in a nite number of

steps.
3. [M25] Show that gcd(Fn+1 , Fn ) = 1 for all n ≥ 1, where Fn denotes
the n-th Fibonacci number.
4. [HM40] Determine the exact average number of steps required by
Algorithm E when m and n are chosen uniformly at random from
{1, . . . , N }, and analyze the behavior as N → ∞.

1.2. Mathematical Preliminaries

Algorithms are most precisely described and analyzed using the language
of mathematics. This section reviews the key concepts we shall need.

1.2.1. Mathematical Induction

Mathematical induction is the primary tool for proving properties of algo-
rithms that operate on integer-parameterized inputs. The principle states:
Theorem 1.2.1 (Principle of Mathematical Induction). Let P (n) be a
statement about a positive integer n. If

(i) P (1) is true (base case), and

(ii) P (k) ⇒ P (k + 1) for all k ≥ 1 (inductive step),

then P (n) is true for all positive integers n.

As a rst application, we prove the well-known summation formula:
Theorem 1.2.2. For all n ≥ 1,
n
X n(n + 1)
k= .
2
k=1

Proof. Base case. For n =P 1

1: k=1 k = 1 = 1·2
2 . ✓
P
Inductive step. Assuming k=1 k = n(n + 1)/2, we have
n

n+1
X n(n + 1) (n + 1)(n + 2)
k= + (n + 1) = .
2 2
k=1

This completes the induction.

1.2.2. Exercises
1.
Pn
[10] Prove by induction that k=1 k 2 = n(n + 1)(2n + 1)/6.
4 CHAPTER 1. BASIC CONCEPTS

→ [20] Use induction to show that for all n ≥ 0, the number of subsets
of an n-element set is 2n .
2. [M30] Prove that the number of regions
into which n lines in general
position divide the plane is 1 + n + n2 .
 CHAPTER 2
LINEAR AND HIERARCHICAL STRUCTURES

The programmer, like the poet, works only slightly removed

from pure thought-stu. He builds his castles in the air, from
air, creating by exertion of the imagination.

 Frederick P. Brooks Jr., The Mythical Man-Month

(1975)

2.1. Introduction
Information must be organized in order to be useful. A data structure
is a systematic way of organizing and accessing data; the choice of data
structure profoundly aects the eciency of the algorithms that manipulate
it.
We begin with the simplest structural archetype: the linear list.

2.2. Linear Lists

A linear list is a sequence of zero or more nodes ⟨a1 , a2 , . . . , an ⟩. When
n = 0 the list is empty. The primary operations on linear lists are:
Insert(x, k). Insert item x at position k in the list (shifting subsequent
elements right).
Delete(k). Remove the item at position k.
Access(k). Return the item at position k.
Search(x). Find the rst occurrence of x in the list.
2.2.1. Stacks and Queues
Two especially important restricted forms of the linear list are the stack
(last-in, rst-out; LIFO) and the queue (rst-in, rst-out; FIFO).
A stack supports two core operations:

5
6 CHAPTER 2. LINEAR AND HIERARCHICAL STRUCTURES

ˆ Push(x): insert x at the top of the stack.

ˆ Pop: remove and return the top element.

The following algorithm evaluates a fully parenthesized arithmetic ex-

pression using two stacks:
Algorithm 2.1 (Dijkstra's Two-Stack Algorithm). Given a fully parenthe-
sized expression, compute its value.
D1. [Initialize.] Create an empty operand stack Sv and an empty op-
erator stack Sω .
D2. [Read token.] If no tokens remain, go to step D6.
D3. [Dispatch.] If the token is a number, push it onto Sv and go to D2.
If it is an operator (+, −, ×, ÷), push it onto Sω and go to D2. If it
is `)', go to D4. If it is `(' ignore it and go to D2.
D4. [Apply.] Pop operator ω from Sω ; pop values b then a from Sv ;
push a ω b onto Sv . Go to D2.
D5. [Result.] Pop and return the value from Sv . □

Theorem 2.2.1. Algorithm 2.1 correctly evaluates any fully parenthesized

arithmetic expression in O(n) time, where n is the number of tokens.

2.2.2. Exercises
1. [00] What is the dening property of a stack?
2. [10] Trace Algorithm D on the input ( ( 1 + 2 ) * ( 3 - 4 ) ).
→ [20] Implement a queue using two stacks. Show that each Enqueue
and Dequeue takes O(1) amortized time.
3. [M30] Prove that the number of distinct sequences obtainable by
passing n elements through a stack is the Catalan number Cn =
2n
1).


n /(n +
 ANSWERS TO EXERCISES

Chapter 1
1.1.1. gcd(12, 8) = 4.

1.1.2. With m = 35, n = 15:

m n r
35 15 5
15 5 0

The algorithm terminates with answer n = 5.

[Link] each step r < n, so the sequence n1 , n2 , . . . is strictly decreasing

and bounded below by 0; hence it must eventually reach 0.

1.2.1. Using the identity (n + 1)2 = n2 + 2n + 1:

n+1
X n(n + 1)(2n + 1) (n + 1)(n + 2)(2n + 3)
k2 = + (n + 1)2 = . □
6 6
k=1

Chapter 2
2.1.1. A stack is characterized by the LIFO (last-in, rst-out) discipline.

2.1.2. Trace of Algorithm D on ( ( 1 + 2 ) * ( 3 - 4 ) ):

7
8 Answers to Exercises

Token Sv (top →) Sω (top →)

(  
(  
1 1 
+ 1 +
2 2, 1 +
) 3 
* 3 ×
( 3 ×
3 3, 3 ×
- 3, 3 −, ×
4 4, 3, 3 −, ×
) −1, 3 ×
) −3 

Result: −3.
 TABLES OF NUMERICAL QUANTITIES

.1. Fundamental Constants (Decimal)

Constant Value
e 2.71828 18284 59045 23536 . . .
ln 2 0.69314 71805 59945 30941 . . .
log10 2 0.30102 99956 63981 19521 . . .
π
√ 3.14159 26535 89793 23846 . . .
2 √ 1.41421 35623 73095 04880 . . .
ϕ = (1 + 5)/2 1.61803 39887 49894 84820 . . .
Euler's γ 0.57721 56649 01532 86060 . . .

.2. Fibonacci Numbers

n Fn n Fn
0 0 10 55
1 1 11 89
2 1 12 144
3 2 13 233
4 3 14 377
5 5 15 610
6 8 16 987
7 13 17 1 597
8 21 18 2 584
9 34 19 4 181

9
10 APPENDIX . TABLES OF NUMERICAL QUANTITIES
 INDEX TO NOTATIONS

Notation Meaning
O(f (n)) At most a constant times f (n)
Ω(f (n)) At least a constant times f (n)
Θ(f (n)) Both O(f (n)) and Ω(f (n))
⌊x⌋ Floor of x (greatest integer ≤ x)
⌈x⌉ Ceiling of x (least integer ≥ x)
x mod y Remainder of x divided by y
m←n Assignment: set m equal to n
gcd(m, n) Greatest common divisor of m and n
n
Binomial coecient  n choose k 


k
Fn n-th Fibonacci number P
n
Hn n-th harmonic number k=1 1/k

11