MATHEMATICAL AND COMPUTING SCIENCE DEPARTMENT

ALGORITHMS AND COMPLEXITY ANALYSIS

LECTURE NOTES

Dr. R.O FOLARANMI

AYEPEKU F.O

Algorithms and Complexity Analysis Page 1

Table of Contents

Chapter 1: Introduction to Algorithmic Analysis

 Overview of basic algorithm

 Overview of basic algorithmic analysis
 Introduction to asymptotic analysis
 Upper and average complexity bounds
 Understanding standard complexity classes

Chapter 2: Time and Space Trade-offs

 Exploring the trade-offs between time and space in algorithm analysis

 Techniques for optimizing time and space complexities simultaneously
 Real-world examples illustrating time-space trade-offs

Chapter 3: Recursive Algorithms

 Understanding the concept of recursion in algorithms

 Recursive algorithm design techniques
 Analysis of recursive algorithms' time and space complexities
 Examples of recursive algorithms in action

Chapter 4: Fundamental Computing Algorithms

 Introduction to fundamental computing algorithms

 Numerical algorithms for mathematical computations
 Detailed exploration of sequential and binary search algorithms
 Analysis and implementation of sorting algorithms

Chapter 5: Graph Algorithms

o Graph representation

Algorithms and Complexity Analysis Page 2

 o Breadth-first search (BFS)
o Depth-first search (DFS)
o Shortest path algorithms: Dijkstra's algorithm, Bellman-Ford algorithm
o Minimum spanning tree algorithms: Prim's algorithm, Kruskal's algorithm

CHAPTER ONE

INTRODUCTION TO ALGORITHMIC ANALYSIS

Algorithms and Complexity Analysis Page 3

What is an Algorithm?

An algorithm is a set of commands that must be followed for a computer to perform calculations
or other problem-solving operations. According to its formal definition, an algorithm is a finite
set of instructions carried out in a specific order to perform a particular task. It is not the entire
program or code; it is simple logic to a problem represented as an informal description in the
form of a flowchart or pseudocode.

 Problem: A problem can be defined as a real-world problem or real-world instance

problem for which you need to develop a program or set of instructions. An algorithm is
a set of instructions.
 Algorithm: An algorithm is defined as a step-by-step process that will be designed for a
problem.
 Input: After designing an algorithm, the algorithm is given the necessary and desired
inputs.
 Processing unit: The input will be passed to the processing unit, producing the desired
output.
 Output: The outcome or result of the program is referred to as the output.

How do Algorithms Work?

Algorithms are step-by-step procedures designed to solve specific problems and perform tasks
efficiently in the realm of computer science and mathematics. These powerful sets of instructions
form the backbone of modern technology and govern everything from web searches to artificial
intelligence. Here's how algorithms work:

 Input: Algorithms take input data, which can be in various formats, such as numbers, text,
or images.

Algorithms and Complexity Analysis Page 4

  Processing: The algorithm processes the input data through a series of logical and
mathematical operations, manipulating and transforming it as needed.
 Output: After the processing is complete, the algorithm produces an output, which could
be a result, a decision, or some other meaningful information.
 Efficiency: A key aspect of algorithms is their efficiency, aiming to accomplish tasks
quickly and with minimal resources.
 Optimization: Algorithm designers constantly seek ways to optimize their algorithms,
making them faster and more reliable.
 Implementation: Algorithms are implemented in various programming languages,
enabling computers to execute them and produce desired outcomes.

What is the Need for Algorithms?

Scalability

It aids in your understanding of scalability. When you have a sizable real-world problem, you
must break it down into small steps to analyze it quickly.

Performance

The real world is challenging to break down into smaller steps. If a problem can be easily
divided into smaller steps, it indicates that the problem is feasible.
After understanding what is an algorithm, why you need an algorithm, you will look at how to
write one using an example.

Types of Algorithms

1. Brute Force Algorithm: A straightforward approach that exhaustively tries all possible
solutions, suitable for small problem instances but may become impractical for larger
ones due to its high time complexity.
2. Recursive Algorithm: A method that breaks a problem into smaller, similar subproblems
and repeatedly applies itself to solve them until reaching a base case, making it effective
for tasks with recursive structures.

Algorithms and Complexity Analysis Page 5

 3. Encryption Algorithm: Utilized to transform data into a secure, unreadable form using
cryptographic techniques, ensuring confidentiality and privacy in digital communications
and transactions.
4. Backtracking Algorithm: A trial-and-error technique used to explore potential solutions
by undoing choices when they lead to an incorrect outcome, commonly employed in
puzzles and optimization problems.
5. Searching Algorithm: Designed to find a specific target within a dataset, enabling
efficient retrieval of information from sorted or unsorted collections.
6. Sorting Algorithm: Aimed at arranging elements in a specific order, like numerical or
alphabetical, to enhance data organization and retrieval.
7. Hashing Algorithm: Converts data into a fixed-size hash value, enabling rapid data access
and retrieval in hash tables, commonly used in databases and password storage.
8. Divide and Conquer Algorithm: Breaks a complex problem into smaller subproblems,
solves them independently, and then combines their solutions to address the original
problem effectively.
9. Greedy Algorithm: Makes locally optimal choices at each step in the hope of finding a
global optimum, useful for optimization problems but may not always lead to the best
solution.
10. Dynamic Programming Algorithm: Stores and reuses intermediate results to avoid
redundant computations, enhancing the efficiency of solving complex problems.
11. Randomized Algorithm: Utilizes randomness in its steps to achieve a solution, often used
in situations where an approximate or probabilistic answer suffices.

How to Write an Algorithm?

 There are no well-defined standards for writing algorithms. It is, however, a problem that
is resource-dependent. Algorithms are never written with a specific programming
language in mind.
 As you all know, basic code constructs such as loops like do, for, while, all programming
languages share flow control such as if-else, and so on. An algorithm can be written using
these common constructs.

Algorithms and Complexity Analysis Page 6

  Algorithms are typically written in a step-by-step fashion, but this is not always the case.
Algorithm writing is a process that occurs after the problem domain has been well-
defined. That is, you must be aware of the problem domain for which you are developing
a solution.

Example

Now, use an example to learn how to write algorithms.

Problem: Create an algorithm that multiplies two numbers and displays the output.

Step 1 − Start

Step 2 − declare three integers x, y & z

Step 3 − define values of x & y

Step 4 − multiply values of x & y

Step 5 − store result of step 4 to z

Step 6 − print z

Step 7 − Stop

Algorithms instruct programmers on how to write code. In addition, the algorithm can be written
as:

Step 1 − Start mul

Step 2 − get values of x & y

Step 3 − z ← x * y

Step 4 − display z

Step 5 − Stop

In algorithm design and analysis, the second method is typically used to describe an algorithm. It
allows the analyst to analyze the algorithm while ignoring all unwanted definitions easily. They

Algorithms and Complexity Analysis Page 7

can see which operations are being used and how the process is progressing. It is optional to
write step numbers. To solve a given problem, you create an algorithm. A problem can be solved
in a variety of ways.

As a result, many solution algorithms for a given problem can be derived. The following step is
to evaluate the proposed solution algorithms and implement the most appropriate solution.

As you progress through this "what is an Algorithm" tutorial, you will learn about some of the
components of an algorithm.

Factors of an Algorithm

The following are the factors to consider when designing an algorithm:

 Modularity: This feature was perfectly designed for the algorithm if you are given a
problem and break it down into small-small modules or small-small steps, which is a
basic definition of an algorithm.
 Correctness: An algorithm's correctness is defined as when the given inputs produce the
desired output, indicating that the algorithm was designed correctly. An algorithm's
analysis has been completed correctly.
 Maintainability: It means that the algorithm should be designed in a straightforward,
structured way so that when you redefine the algorithm, no significant changes are made
to the algorithm.
 Functionality: It takes into account various logical steps to solve a real-world problem.
 Robustness: Robustness refers to an algorithm's ability to define your problem clearly.
 User-friendly: If the algorithm is difficult to understand, the designer will not explain it to
the programmer.

Algorithms and Complexity Analysis Page 8

  Simplicity: If an algorithm is simple, it is simple to understand.
 Extensibility: Your algorithm should be extensible if another algorithm designer or
programmer wants to use it.

Qualities of a Good Algorithm

 Efficiency: A good algorithm should perform its task quickly and use minimal resources.
 Correctness: It must produce the correct and accurate output for all valid inputs.
 Clarity: The algorithm should be easy to understand and comprehend, making it
maintainable and modifiable.
 Scalability: It should handle larger data sets and problem sizes without a significant
decrease in performance.
 Reliability: The algorithm should consistently deliver correct results under different
conditions and environments.
 Optimality: Striving for the most efficient solution within the given problem constraints.
 Robustness: Capable of handling unexpected inputs or errors gracefully without crashing.
 Adaptability: Ideally, it can be applied to a range of related problems with minimal
adjustments.
 Simplicity: Keeping the algorithm as simple as possible while meeting its requirements,
avoiding unnecessary complexity.

Advantage and Disadvantages of Algorithms

Advantages of Algorithms:

 Efficiency: Algorithms streamline processes, leading to faster and more optimized

solutions.
 Reproducibility: They yield consistent results when provided with the same inputs.
 Problem-solving: Algorithms offer systematic approaches to tackle complex problems
effectively.
 Scalability: Many algorithms can handle larger datasets and scale with increasing input
sizes.
 Automation: They enable automation of tasks, reducing the need for manual intervention.

Algorithms and Complexity Analysis Page 9

Disadvantages of Algorithms:

 Complexity: Developing sophisticated algorithms can be challenging and time-

consuming.
 Limitations: Some problems may not have efficient algorithms, leading to suboptimal
solutions.
 Resource Intensive: Certain algorithms may require significant computational resources.
 Inaccuracy: Inappropriate algorithm design or implementation can result in incorrect
outputs.
 Maintenance: As technology evolves, algorithms may require updates to stay relevant and
effective.

Overview of Basic Algorithmic Analysis

The complexity of an algorithm is a measure of the amount of time and/or space required by an
algorithm for an input of a given size (n). Though the complexity of the algorithm does depends
upon the specific factors such as: The architecture of the computer i.e. the hardware platform
representation of the Abstract Data Type(ADT) compiler efficiency the complexity of the
underlying algorithm size of the input. Though you will see most significant factors are a
complexity of underlying algorithm and size of the input.

Asymptotic Analysis

Asymptotic analysis refers to the computing of the running time of any piece of code or the
operation in a mathematical unit of a computation. Its operation is computed in terms of a
function like f(n). In mathematical analysis, asymptotic analysis, also known as asymptotics, is a
method of describing limiting behavior.

The time required by the algorithm falls under the three types: Worst case - Maximum time
required by an algorithm and it is mostly used or done while analyzing the algorithm. Best case -
Minimum time required for the algorithm or piece of code and it is not normally calculated while
analyzing the algorithm. Average case - Average time required for an algorithm or portion of
code and it is sometimes done while analyzing the algorithm.

Algorithms and Complexity Analysis Page 10

Asymptotic notation

The commonly used notation for calculating the running time complexity of the algorithm is as
follows:

 Big O notation
 Big θ notation
 Big Ω notation

Big Oh Notation, Ο

Big O is used to measure the performance or complexity of an algorithm. In more mathematical

term, it is the upper bound of the growth rate of a function, or that if a function g(x) grows no
faster than a function f(x), then g is said to be a member of O(f).In general, it is used to express
the upper bound of an algorithm and which gives the measure for the worst time complexity or
the longest time an algorithm possibly take to complete.

Big Omega Notation, Ω

The notation Ω(n) is the formal way to express the lower bound of an algorithm's running time. It
measures the best case time complexity or the best amount of time an algorithm can possibly
take to complete.

Algorithms and Complexity Analysis Page 11

Big Theta Notation, θ

The notation θ(n) is the formal way to express both the lower bound and the upper bound of an
algorithm's running time.

Notation is used to determine the complexity of various algorithm's

In the realm of asymptotic analysis, three notations—Big O, Omega, and Theta—play a

significant role in describing how algorithms or functions behave:

1. Big O Notation (O): Think of Big O as an upper bound, indicating the worst-case
scenario for an algorithm's performance. It answers the question: "In the worst situation,
how much time will our algorithm take?" For example, O(n) signifies that the time taken
by the algorithm grows linearly with the input size.
2. Omega Notation (Ω): In contrast, Omega serves as a lower bound, representing the best-
case scenario for an algorithm's performance. It answers the question: "No matter what,

Algorithms and Complexity Analysis Page 12

 how fast can our algorithm run?" For instance, Ω(1) implies that the algorithm will
always take at least constant time.
3. Theta Notation (Θ): Theta notation offers a balanced view, presenting a tight bound on
an algorithm's performance. It answers the question: "How does our algorithm perform in
both the best and worst cases?" If an algorithm is Θ(n), it indicates that its performance
grows linearly with the input size in both the best and worst scenarios.

These notations simplify the communication of efficiency and performance characteristics of

algorithms without delving into intricate details.

To analyze the asymptotic complexity of an algorithm, follow these straightforward steps:

1. Identify the Input Size: Determine what aspect of the algorithm's efficiency you're
evaluating, often the input size, like the number of elements in a list.
2. Count Basic Operations: Break down the algorithm into its fundamental operations. For
example, if you're sorting a list, count the number of times you compare two elements or
swap them.
3. Focus on the Worst Case: Analyze the algorithm's behavior in the worst-case scenario,
where the input is most challenging for the algorithm. This provides insight into the
algorithm's worst performance.
4. Remove Constants: Eliminate constant factors from your analysis. Concentrate on the
most significant factors that affect the algorithm's performance.
5. Use Big O Notation: Express the algorithm's complexity using "Big O" notation,
describing how the algorithm's runtime or memory usage grows as the input size
increases. For example, if doubling the input size approximately doubles the runtime, it
might be expressed as O(n).
6. Compare with Other Algorithms: Finally, compare the results with other algorithms to
determine which one is more efficient. In general, the algorithm with a lower Big O
notation runs faster.

Upper and Average Complexity Bounds

Algorithms and Complexity Analysis Page 13

In addition to worst-case complexity (represented by Big O notation), algorithms may also have
average-case complexity. Average-case complexity considers the expected performance of an
algorithm given all possible inputs of a certain size, usually assuming a certain distribution of
inputs.

Upper complexity bounds, such as Big O notation, are particularly important because they
provide a guarantee on the algorithm's performance in the worst-case scenario. This helps ensure
that the algorithm won't perform unacceptably poorly under any circumstances.

Understanding Standard Complexity Classes

Complexity classes categorize algorithms based on their worst-case time or space complexity.
Some standard complexity classes include:

 Constant Time (O(1)): Algorithms with constant time complexity execute in a fixed
amount of time regardless of the size of the input. Accessing an element in an array by
index is an example of an O(1) operation.
 Logarithmic Time (O(log n)): Algorithms with logarithmic time complexity typically
divide the problem size in half with each step. Binary search is an example of an
algorithm with O(log n) complexity.
 Linear Time (O(n)): Algorithms with linear time complexity have their runtime directly
proportional to the size of the input. Linear search through an unsorted array is an
example of O(n) complexity.
 Quadratic Time (O(n^2)): Algorithms with quadratic time complexity have their
runtime proportional to the square of the size of the input. Nested loops iterating over the
input are common in algorithms with O(n^2) complexity.
 Exponential Time (O(2^n)): Algorithms with exponential time complexity grow very
rapidly with the size of the input. Recursive algorithms without proper memoization often
exhibit exponential time complexity.

Understanding these complexity classes helps in selecting the most appropriate algorithm for a
given problem, balancing between efficiency and scalability. For instance, an algorithm with

Algorithms and Complexity Analysis Page 14

O(n^2) complexity might be acceptable for small inputs but become impractical for larger
datasets, necessitating the use of more efficient algorithms with lower complexity bounds.

Time Complexity

The amount of time required to complete an algorithm's execution is called time complexity. The
big O notation is used to represent an algorithm's time complexity. The asymptotic notation for
describing time complexity, in this case, is big O notation. The time complexity is calculated
primarily by counting the number of steps required to complete the execution. Let us look at an
example of time complexity.

mul = 1;

// Suppose you have to calculate the multiplication of n numbers.

for i=1 to n

mul = mul *1;

// when the loop ends, then mul holds the multiplication of the n numbers

return mul;

Time Complexity: The time complexity of an algorithm represents the amount of time it takes
to execute as a function of the input size. In this code snippet, the loop iterates from 1 to nn,
where nn is the number of numbers to be multiplied.

 Inside the loop, there is a single operation: mul = mul * 1;. This operation is constant
time (O(1)O(1)) because multiplying by 1 does not depend on the size of the input.
 The loop runs nn times, where nn is the input size.

Since the loop runs nn times and each iteration takes constant time, the overall time complexity
of the algorithm is O(n)O(n).

Algorithms and Complexity Analysis Page 15

Space Complexity: The space complexity of an algorithm represents the amount of memory it
requires as a function of the input size. In this code snippet, there is only one variable, mul,
which stores the result of the multiplication.

 The variable mul requires a constant amount of space to store a single integer, regardless
of the size of the input.
 There are no other variables or data structures used that grow with the input size.

Therefore, the space complexity of the algorithm is O(1)O(1), indicating constant space usage
regardless of the input size.

Summary:

 Time Complexity: O(n)O(n) - Linear time complexity indicates that the time taken to
execute the algorithm grows linearly with the size of the input.
 Space Complexity: O(1)O(1) - Constant space complexity indicates that the amount of
memory required remains constant regardless of the size of the input.

The space is required for an algorithm for the following reasons:

1. To store program instructions.

2. To store track of constant values.
3. To store track of variable values.
4. To store track of function calls, jumping statements, and so on.

Space Complexity = Auxiliary Space + Input Size

Example 2: Explain the time and space complexity with respect to the program below

def twoForLoops(n):

for i in range(1,n):

print("Printing:"+i);

Algorithms and Complexity Analysis Page 16

for i in range(1,100):

print("Printing:"+i);

Let's analyze the time and space complexity of the given Python function twoForLoops(n).

Time Complexity:

First Loop:
python
for i in range(1, n):
print("Printing:" + i)

 The loop iterates from 1 to n−1n−1, where nn is the input parameter.

 The loop executes n−1n−1 times.

The time complexity of this loop is O(n)O(n) because the number of iterations directly depends
on the input size nn.

Second Loop:
python
for i in range(1, 100):
print("Printing:" + i)

 This loop iterates from 1 to 99, with a fixed number of iterations (100 iterations).
 Regardless of the input size nn, this loop always executes 100 times.

The time complexity of this loop is constant, O(1)O(1), because the number of iterations remains
constant irrespective of the input size.

Overall Time Complexity: Since the loops are sequential and not nested, we consider the
dominant term. In this case, the first loop dominates the time complexity. Thus, the overall time
complexity is O(n)O(n), linear time complexity, because the first loop's time complexity is
proportional to the input size nn.

Algorithms and Complexity Analysis Page 17

Space Complexity: The space complexity of the function is determined by the memory required
for variables and data structures, and it remains constant regardless of the input size nn.

 The function does not declare or use any additional variables that grow with the input
size.
 The memory required for the loops is constant.

Therefore, the space complexity of the function is O(1)O(1), indicating constant space usage
regardless of the input size.

Algorithms and Complexity Analysis Page 18

 CHAPTER TWO

TIME AND SPACE TRADE-OFFS

Exploring the Trade-offs between Time and Space in Algorithm Analysis:

Algorithm analysis often involves making trade-offs between time and space efficiency. Here's a
comprehensive exploration of these trade-offs:

1. Time Complexity vs. Space Complexity:

In algorithm design, there's often a trade-off between optimizing for time complexity (execution
speed) and space complexity (memory usage). Let's explore this trade-off with examples:

Dynamic Programming:

o Time Complexity: Dynamic programming optimizes time complexity by storing

and reusing solutions to subproblems, reducing redundant computations.
o Space Complexity: However, this optimization comes at the cost of increased
space complexity due to storing solutions in a table or array.

Example: Fibonacci Sequence - The recursive approach to calculate Fibonacci numbers

has exponential time complexity. Dynamic programming optimizes this to linear time
complexity by storing previously computed Fibonacci numbers in an array, sacrificing
space to reduce time.

Merge Sort:

o Time Complexity: Merge sort has O(nlog⁡n)O(nlogn) time complexity, making

it efficient for large datasets.
o Space Complexity: However, it requires additional space proportional to the size
of the input array for the merge operation, resulting in O(n)O(n) space
complexity.

Algorithms and Complexity Analysis Page 19

 Example: Given an array of integers, merge sort divides the array into smaller subarrays,
sorts them recursively, and then merges them. While it's efficient in terms of time
complexity, it consumes additional memory during the merge phase.

Hash Tables:

o Time Complexity: Hash tables offer constant-time average-case performance for

insertion, deletion, and search operations.
o Space Complexity: However, they require additional space to store keys and
values, leading to O(n)O(n) space complexity in the worst-case scenario.

Example: Implementing a hash table for storing key-value pairs allows fast retrieval and
updates (O(1)O(1) average time complexity) but consumes memory proportional to the
number of elements stored.

Binary Search Trees:

o Time Complexity: Binary search trees provide efficient O(log⁡n)O(logn) time

complexity for insertion, deletion, and search operations in balanced trees.
o Space Complexity: However, the space complexity can degrade to O(n)O(n) in
the worst-case scenario for unbalanced trees.

Example: Inserting elements into a binary search tree maintains the tree's balance,
ensuring fast lookup times. However, if the tree becomes unbalanced, the space
complexity increases, impacting performance.

Bit Manipulation:

o Time Complexity: Bit manipulation techniques often result in efficient time

complexity for operations like bitwise AND, OR, XOR, and shifting.
o Space Complexity: They typically require minimal additional memory, leading to
optimized space complexity.

Algorithms and Complexity Analysis Page 20

 Example: Performing operations like finding the number of set bits (popcount) in an
integer using bitwise manipulation achieves fast execution with minimal memory
overhead.

2. Time-Space Trade-off:

In algorithm design, a time-space trade-off refers to the balancing act between optimizing for
time efficiency (execution speed) and space efficiency (memory usage). Here's a more detailed
exploration:

Definition:

o A time-space trade-off involves making decisions that impact both the time and
space complexity of an algorithm.
o It involves optimizing one aspect (time or space) at the expense of the other,
based on the specific requirements and constraints of the problem at hand.

Decision Making:

o When faced with a time-space trade-off, developers must carefully consider the
trade-offs and make informed decisions based on the needs of the application.
o The decision-making process often involves analyzing the relative importance of
time and space efficiency in the context of the problem domain.

Optimizing Time Efficiency:

o In scenarios where fast execution is critical, optimizing for time efficiency takes
precedence over minimizing memory usage.
o Strategies for optimizing time efficiency may include using more complex
algorithms, precomputing results, or employing parallel processing techniques.

Example: In real-time systems such as video games or financial trading platforms,

minimizing latency is crucial. Algorithms are optimized for speed, even if it
means consuming more memory.

Algorithms and Complexity Analysis Page 21

 Optimizing Space Efficiency:

o In situations where memory resources are limited or expensive, optimizing for

space efficiency becomes paramount.
o Strategies for optimizing space efficiency may involve using simpler algorithms,
employing data compression techniques, or utilizing space-saving data structures.

Example: Mobile applications often operate on devices with limited memory.

Algorithms are designed to minimize memory usage, even if it means sacrificing
some execution speed.

Dynamic Trade-offs:

o The optimal balance between time and space efficiency may vary depending on
factors such as input size, available hardware resources, and performance
requirements.
o Algorithms may dynamically adjust their behavior based on these factors to
achieve the best possible performance under different conditions.

Heuristic Algorithms:

o Heuristic algorithms often employ time-space trade-offs to strike a balance

between exploration (time) and exploitation (space) in search or optimization
problems.
o These algorithms make informed decisions about resource allocation to balance
the exploration of potential solutions with the exploitation of promising ones.

Example: Genetic algorithms use a population-based approach to search for

optimal solutions. They trade off computational time for memory space by
maintaining a population of potential solutions.

Algorithms and Complexity Analysis Page 22

 3. Algorithm Design Considerations:

Algorithm Design Considerations:

When designing algorithms, developers must carefully consider the trade-offs between time
efficiency (execution speed) and space efficiency (memory usage). Here's a detailed exploration
of the key considerations:

Problem Requirements:

o Understanding the specific requirements of the problem is paramount. Some

problems may prioritize fast execution times, while others may prioritize efficient
memory usage.
o Developers must analyze the problem domain to determine the relative
importance of time and space efficiency.

Input Size and Characteristics:

o The characteristics of the input data can significantly impact the choice of
algorithm and its associated time and space complexities.
o Consideration must be given to factors such as input size, distribution, and
variability.

Hardware and Environment Constraints:

o The hardware environment in which the algorithm will run plays a crucial role in
design considerations.
o Factors such as available memory, processing power, and storage capacity must
be taken into account when making decisions about time-space trade-offs.

Performance Objectives:

o Clearly defined performance objectives help guide algorithm design decisions.

Algorithms and Complexity Analysis Page 23

 o Developers must determine acceptable trade-offs between time and space
efficiency based on performance targets, such as response time, throughput, or
resource utilization.

Scalability:

o Algorithms should be designed with scalability in mind, capable of handling

increasing input sizes or growing resource demands.
o Scalable algorithms strike a balance between time and space efficiency to ensure
that performance remains acceptable as the system scales.

Flexibility and Adaptability:

o Algorithms should be flexible and adaptable to varying input conditions and

environmental factors.
o Dynamic adjustment mechanisms allow algorithms to adapt their behavior based
on changing circumstances, optimizing time-space trade-offs in real-time.

Optimization Opportunities:

o Identifying optimization opportunities early in the design process is crucial.

o Developers should explore different algorithmic approaches and techniques to
find the most suitable balance between time and space efficiency.

Trade-off Analysis and Decision Making:

o Rigorous analysis and evaluation of time-space trade-offs are essential for

informed decision-making.
o Developers should consider the potential impacts of different design choices on
algorithm performance and resource utilization.

Algorithms and Complexity Analysis Page 24

 Documentation and Communication:

o Documenting design decisions and trade-offs facilitates collaboration and

understanding among team members.
o Clear communication of the rationale behind time-space trade-offs helps ensure
that design choices align with project goals and requirements.

4. Memory vs. Computation:

In algorithm design, developers often face the trade-off between using more memory to reduce
computation time or performing more computations to conserve memory. Here's a deeper dive
into this trade-off:

Memory Optimization:

o Optimizing for memory aims to minimize the amount of memory required by an

algorithm, often at the expense of increased computational complexity.
o This approach is beneficial when memory resources are limited or when
minimizing memory usage is a priority.

Computation Optimization:

o Optimizing for computation focuses on minimizing the number of computational

steps or operations performed by an algorithm, even if it means consuming more
memory.
o This approach is advantageous when execution speed is critical or when memory
resources are plentiful.

Examples:

Matrix Multiplication:

Algorithms and Complexity Analysis Page 25

 o Consider the problem of multiplying two matrices. The straightforward approach
involves allocating memory for the result matrix and performing the necessary
computations to fill it.
o Memory Optimization: To conserve memory, one can perform matrix
multiplication in-place, overwriting one of the input matrices with the result.
While this reduces memory usage, it increases computational complexity.
o Computation Optimization: Conversely, using additional memory to store
intermediate results can reduce the number of computations required, potentially
leading to faster execution times.

Sorting Algorithms:

o Sorting algorithms rearrange elements in a list or array in a specified order.

o Memory Optimization: In-place sorting algorithms like Quicksort and Heapsort
minimize memory usage by rearranging elements within the input array itself,
without requiring additional memory for temporary storage. While these
algorithms are memory-efficient, they may require more computational steps.
o Computation Optimization: External sorting algorithms like Merge Sort and
External Quick Sort optimize for computation by utilizing external storage (e.g.,
disk) to handle large datasets that don't fit into memory. While these algorithms
require more memory, they often reduce the number of computational steps,
particularly for large datasets.

Dynamic Programming:

o Dynamic programming is a technique used to solve problems by breaking them

down into smaller subproblems and storing the solutions to these subproblems for
future reference.
o Memory Optimization: Memoization, a form of dynamic programming,
optimizes memory usage by storing intermediate results in a lookup table. While
this increases memory usage, it reduces the computational complexity by
avoiding redundant computations.

Algorithms and Complexity Analysis Page 26

 o Computation Optimization: Tabulation, another form of dynamic programming,
optimizes for computation by computing solutions iteratively without storing
intermediate results. While this conserves memory, it may require more
computational steps.

5. Data Structures Impact:

Data structures play a crucial role in algorithm design, significantly influencing both time and
space complexity. Here's an in-depth look at how different data structures impact these
complexities:

Arrays:

o Arrays are contiguous blocks of memory that store elements of the same data
type.
o Time Complexity: Accessing elements by index in an array is efficient
(O(1)O(1)) since it involves simple pointer arithmetic.
o Space Complexity: Arrays have a fixed size determined at the time of
declaration, leading to O(n)O(n) space complexity, where nn is the number of
elements.

Example: Searching for an element in a sorted array using binary search

(O(log⁡n)O(logn)) exploits its efficient access time.

Linked Lists:

o Linked lists consist of nodes where each node contains a value and a
pointer/reference to the next node.
o Time Complexity: Traversing a linked list requires O(n)O(n) time since it
involves following pointers from one node to another.
o Space Complexity: Linked lists have O(n)O(n) space complexity due to the
overhead of storing pointers for each node.

Algorithms and Complexity Analysis Page 27

 Example: Insertion or deletion at the beginning of a linked list (O(1)O(1)) can be
advantageous compared to arrays, which may require shifting elements
(O(n)O(n)).

Hash Tables:

o Hash tables use a hash function to map keys to indices in an array, providing
constant-time access to elements.
o Time Complexity: On average, hash table operations like insertion, deletion, and
search have O(1)O(1) time complexity. However, worst-case scenarios can lead to
O(n)O(n) time complexity.
o Space Complexity: Hash tables typically have O(n)O(n) space complexity due to
the underlying array used to store elements.

Example: Using a hash table for frequency counting allows efficient retrieval of
counts for each element (O(1)O(1) average time complexity for insertion and
lookup).

Trees (e.g., Binary Search Trees, AVL Trees):

o Trees are hierarchical data structures where each node has zero or more child
nodes.
o Time Complexity: Operations like insertion, deletion, and search in balanced
trees have O(log⁡n)O(logn) time complexity, making them efficient for large
datasets.
o Space Complexity: Balanced trees have O(n)O(n) space complexity, where nn is
the number of elements.

Example: Binary search trees enable efficient searching (O(log⁡n)O(logn)) and

insertion (O(log⁡n)O(logn)) operations, making them suitable for applications
requiring dynamic data storage.

Heaps (e.g., Binary Heaps, Fibonacci Heaps):

Algorithms and Complexity Analysis Page 28

 o Heaps are specialized tree-based data structures that satisfy the heap property.
o Time Complexity: Heap operations like insertion and deletion have
O(log⁡n)O(logn) time complexity, while accessing the minimum or maximum
element takes O(1)O(1) time.
o Space Complexity: Heaps typically have O(n)O(n) space complexity.

Example: Using a heap to implement priority queues allows efficient retrieval of

the highest-priority element (O(1)O(1)) and insertion/deletion of elements
(O(log⁡n)O(logn)).

Techniques for Optimizing Time and Space Complexities Simultaneously:

1. Dynamic Programming:
o Dynamic programming is a powerful technique used to solve problems by
breaking them down into smaller subproblems and storing the solutions to these
subproblems for future reference.
o By memoizing intermediate results, dynamic programming optimizes both time
and space complexities.

Example: Fibonacci Sequence - In the naive recursive approach, calculating Fibonacci

numbers involves redundant computations. Dynamic programming optimizes this by
storing previously computed Fibonacci numbers, reducing both time and space
complexity.

2. Greedy Algorithms:
o Greedy algorithms make locally optimal choices at each step with the hope of
finding a global optimum.
o These algorithms often require less memory overhead compared to other
techniques, resulting in optimized space complexity.

Example: Minimum Spanning Tree (MST) - Kruskal's algorithm greedily selects edges
in ascending order of weight to construct an MST. It achieves both time and space
efficiency without requiring memoization or excessive memory usage.

Algorithms and Complexity Analysis Page 29

 3. Two-Pointer Technique:
o The two-pointer technique involves using two pointers to traverse an array or
sequence efficiently, often solving problems in linear time with constant space
complexity.

Example: Two Sum Problem - Given an array of integers, find two numbers that sum up
to a specific target. Using the two-pointer technique, we can solve this problem in
O(n)O(n) time and O(1)O(1) space.

4. Space-Efficient Data Structures:

o Utilizing space-efficient data structures, such as bitsets, bitmaps, or compressed
data structures, can significantly reduce memory usage without sacrificing much
in terms of time complexity.

Example: Bloom Filters - Bloom filters are probabilistic data structures used to test
whether an element is a member of a set. They achieve space efficiency by using a
compact array of bits, with constant-time insertions and lookups.

5. Lazy Evaluation:
o Lazy evaluation delays the evaluation of an expression until its value is actually
needed, avoiding unnecessary computations and conserving memory.

Example: Lazy Initialization - In lazy initialization, objects are created only when they
are first accessed, rather than when they are declared. This technique saves memory by
deferring object creation until necessary.

6. Approximation Algorithms:
o Approximation algorithms provide approximate solutions to optimization
problems with guaranteed performance bounds, often trading off exactness for
efficiency in terms of both time and space.

Algorithms and Complexity Analysis Page 30

 Example: Traveling Salesman Problem (TSP) - While finding the exact solution to TSP
is NP-hard, approximation algorithms like the nearest neighbor algorithm provide near-
optimal solutions with polynomial time and space complexity.

7. Parallel and Concurrent Programming:

o Leveraging parallel and concurrent programming techniques, such as
multithreading or multiprocessing, can exploit modern hardware architectures to
achieve both time and space efficiency.

Example: Parallel Merge Sort - By splitting the array into smaller subarrays and sorting
them concurrently in parallel, parallel merge sort achieves both time and space efficiency,
reducing sorting time while conserving memory.

Real-world Examples Illustrating Time-Space Trade-offs:

1. Database Indexing:
o In databases, indexing improves query performance by trading off additional
storage space for reduced query time.
o Indexes are precomputed data structures that speed up data retrieval operations.
2. Image Compression:
o Image compression algorithms trade off space for time by compressing image
data to reduce storage requirements while maintaining reasonable decompression
time.
o Examples include JPEG and PNG compression techniques.
3. Web Caching:
o Web caching mechanisms store copies of frequently accessed web pages or
resources closer to the user, reducing server load and improving response time.
o This trade-off involves allocating additional storage space to cache data in
exchange for faster access.
4. File Compression:
o File compression algorithms, such as ZIP and RAR, compress files to reduce
storage space while allowing for relatively fast decompression.

Algorithms and Complexity Analysis Page 31

 o These algorithms balance space efficiency with decompression speed.
5. Machine Learning Models:
o Machine learning models trade off model complexity (space) for training and
inference time.
o Simpler models require less memory but might have lower accuracy, while more
complex models demand more memory but can provide higher accuracy.

Algorithms and Complexity Analysis Page 32

 CHAPTER THREE

RECURSIVE ALGORITHMS

Recursion is a method of solving problems where the solution depends on solutions to smaller
instances of the same problem. In mathematical terms, it's a way of defining functions in which
the function being defined is applied within its own definition.

A recursive algorithm, in the simplest terms, is a method of problem solving where the solution
to a problem depends on solutions to smaller instances of the same problem. It breaks down a
problem into smaller and smaller subproblems, until it gets to a problem that is small enough to
be solved easily

Advantages and Disadvantages of Recursive Algorithm

Recursive algorithms are a powerful tool for problem-solving, but as with everything in
programming, there are trade-offs. Here are some advantages and disadvantages to using
recursive algorithms:

Understanding when to use a recursive algorithm is part of becoming a better programmer. They
can be used to make code cleaner and easier to understand, but they can also become a source of
inefficiency if used improperly.

Advantages:

 Recursive algorithms make the code look clean and elegant.

 A complex task can be broken down into simpler sub-problems using recursion.
 Sequence generation is easier with recursion than using some nested iteration.

Disadvantages:

 Sometimes the logic behind recursion is hard to follow through.

 Recursive calls are expensive (inefficient) as they take up a lot of memory and time.
 Recursive functions are hard to debug.

Algorithms and Complexity Analysis Page 33

Basic Principles of Recursion:

1. Base Case: Every recursive function must have a base case, which serves as the
termination condition. It's the simplest form of the problem that doesn't need further
recursion. Without a base case, the recursion would continue indefinitely, leading to
what's called a "stack overflow" or infinite loop.
2. Recursive Step: This is where the function calls itself with a modified version of the
original problem. Each successive recursive call should move closer to the base case,
eventually reaching it.
3. Progress Towards the Base Case: To ensure termination, each recursive call must
reduce the size of the problem in some way, either by reducing the input size or by
moving closer to the base case in some other manner.

Examples of Recursion in Mathematics:

1. Factorial Function: The factorial of a non-negative integer n, denoted by n!, is the

product of all positive integers less than or equal to n. The factorial function can be
defined recursively as:

Algorithms and Complexity Analysis Page 34

 2. Fibonacci Sequence: The Fibonacci sequence is a series of numbers where each number
is the sum of the two preceding ones, usually starting with 0 and 1. The nth Fibonacci
number can be defined recursively as:

Factorial Function:

Recursion in the factorial function occurs when a function calls itself with a smaller
argument until it reaches the base case (usually n=0n=0).

For example, in the factorial function n!n!, the base case is when n=0n=0, where
0!=10!=1.

The recursive step is defined as n!=n×(n−1)!n!=n×(n−1)! for n>0n>0. This means that the
factorial of nn is calculated by multiplying nn with the factorial of n−1n−1.

Recursive implementation of factorial function in pseudocode:

factorial(n):
if n == 0:
return 1
else:

Algorithms and Complexity Analysis Page 35

 return n * factorial(n - 1)

Meaning each Fibonacci number is the sum of the two previous Fibonacci numbers.

Recursive implementation of Fibonacci sequence in pseudocode

fibonacci(n):
if n == 0:
return 0
elif n == 1:
return 1
else:
return fibonacci(n - 1) + fibonacci(n - 2)

Both examples demonstrate how recursion can be used to define mathematical sequences or
operations concisely. However, it's essential to handle base cases properly to avoid infinite
recursion. Recursion offers elegant solutions for certain problems but may lead to inefficiency or
stack overflow errors for large inputs if not implemented carefully.

Different Types of Recursion

There are four different types of recursive algorithms, you will look at them one by one.

 Direct Recursion

A function is called direct recursive if it calls itself in its function body repeatedly. To better
understand this definition, look at the structure of a direct recursive program.

Algorithms and Complexity Analysis Page 36

int fun(int z) {
// Base case: when z reaches 0, exit recursion
if (z == 0) {
return 0;
} else {
// Recursive call with z decremented by 1
return fun(z - 1);
}
}

In this program, you have a method named fun that calls itself again in its function body. Thus,
you can say that it is direct recursive.

 Indirect Recursion

The recursion in which the function calls itself via another function is called indirect recursion.
Now, look at the indirect recursive program structure.

int fun1(int z) {
// Definition of fun2 within fun1
int fun2(int y) {
// Base case for fun2
if (y <= 0) {
return 0;
} else {
// Recursive call to fun1 with y decremented by 2
fun1(y - 2);
}
}

// Base case for fun1

if (z <= 0) {
return 0;
} else {
// Recursive call to fun2 with z decremented by 1
fun2(z - 1);
}
}

In this example, you can see that the function fun1 explicitly calls fun2, which is invoking fun1
again. Hence, you can say that this is an example of indirect recursion.

Algorithms and Complexity Analysis Page 37

  Tailed Recursion

A recursive function is said to be tail-recursive if the recursive call is the last execution done by
the function. Let’s try to understand this definition with the help of an example.

void fun(int z) {
// Print the value of z
printf("%d ", z);

// Base case: when z reaches 0, exit recursion

if (z > 0) {
// Recursive call with z decremented by 1
fun(z - 1);
}
}

If you observe this program, you can see that the last line ADI will execute for method fun is a
recursive call. And because of that, there is no need to remember any previous state of the
program.

 Non-Tailed Recursion

A recursive function is said to be non-tail recursive if the recursion call is not the last thing done
by the function. After returning back, there is something left to evaluate. Now, consider this
example.

void fun(int z) {
if (z > 1) {
fun(z - 1); // Recursive call with z decremented by 1
}
printf("%d ", z); // Print the value of z after the recursive call
}

In this function, you can observe that there is another operation after the recursive call. Hence the
ADI will have to memorize the previous state inside this method block. That is why this program
can be considered non-tail recursive.

Non Recursive Algorithm versus Recursive Algorithm

Algorithms and Complexity Analysis Page 38

Indeed, recursive algorithm is revered in the domain of Computer Science. It introduces an
elegant approach to problem-solving, where a problem is broken down into simpler instances of
itself until a base case is reached. Despite its advantages, it’s not always the best approach for all
types of problems. Here's where non recursive algorithms, also known as iterative algorithms,
come into play. They offer an alternative, and sometimes more efficient, approach to problem-
solving.

Non Recursive Algorithm

A non-recursive algorithm, which is often referred to as iterative algorithm, is a method of

solving a problem that involves a series of instructions being repeated multiple times until a
certain condition is met, typically through the use of loops. Unlike recursive algorithms, non-
recursive algorithms do not involve function calls to itself. Instead, they utilise looping structures
such as for-loops, while-loops, and do-while loops, depending on the specific requirements of the
problem and programming language. Each iteration repeats the same series of steps,
manipulating the problem's input data until a solution is achieved.

Non Recursive Algorithm, also known as an iterative algorithm, involves solving a problem
through repetition of a series of instructions until a specific condition is met, typically without
the need for the function to call itself.

Comparing Non Recursive Algorithm and Recursive Algorithm

So, let's delve into comparing recursive and non-recursive algorithms. Each method has its
unique operations and performances which brings about several comparative elements. The
following table provides a succinct contrast of the two:

Recursive Algorithm Non Recursive Algorithm

Relies on calling itself to solve smaller Does not call itself. Primarily uses
Function calls
instances of the problem loops to resolve a problem

Code Often results in cleaner, simpler code, Can result in lengthier, complex code
complexity enhancing readability as problem size increases

Memory Tend to use more memory due to stack Generally consumes less memory, as it

Algorithms and Complexity Analysis Page 39

 Recursive Algorithm Non Recursive Algorithm

usage storage of multiple function calls doesn’t require stack storage

Can be slower due to overhead of Often faster due to fewer function calls
Speed
function calls and less overhead

Each type of algorithm comes with its pros and cons. Recursive algorithms are lauded for
cleaner, easier-to-understand code. However, they are generally slower and consume more
memory due to the overhead involved with multiple function calls. On the contrary, non-
recursive algorithms are praised for being more efficient in terms of speed and memory usage.
Although they can result in more complex and lengthier code, they bring better runtime
efficiency, making them a preferable choice for larger data sets. Remember, thoughtful
discretion of the nature of the problem, the performance requirements, and the available
computational resources is mandatory when it comes to selecting between recursive and non-
recursive algorithms.

Practical Examples of Non Recursive Algorithm

1. Calculation of Factorial Number: One of the most elementary examples of a non-recursive

algorithm is calculating the factorial of a number. Here, we make use of a loop to multiply the
number with every other number smaller than it, until we reach the number 1.

An example of calculating factorial using a non-recursive Python function:

def factorial(n):
result = 1
for i in range(1, n + 1):
result *= i
return result
2. Fibonacci Sequence: Just like calculating the factorial of a number, the Fibonacci sequence,
which is widely used as an example of a problem solved by recursion, may also be computed
iteratively.

A non-recursive Python function to calculate the Fibonacci series:

Algorithms and Complexity Analysis Page 40

def fibonacci(n):
if n <= 1:
return n
a, b = 0, 1
for _ in range(n):
a, b = b, a + b
return a
Revisiting these examples iteratively, you have the essence of non-recursive algorithms echoing
- the use of iterative structures that help in managing the stack memory and the computational
speed far more efficiently.

Deep Dive into Recursive Algorithm Definition

Embarking on the journey to comprehend the recursive algorithm in its entirety, you might
encounter a few roadblocks. Still, don’t worry, these hurdles are merely part of the learning
experience. Step by step, let's dissect the recursive algorithm's definition and understand its core
components. After all, mastering this ever-important concept is key to solving complex problems
in computer science.

Breaking Down Recursive Algorithm Definition

At its core, a recursive algorithm is a method of solving problems that involves the algorithm
calling itself to solve smaller instances of the same problem. Breaking down the definition
further will give you a detailed perspective and reinforce your understanding. Method of
Solving Problems: Recursion is fundamentally a problem-solving approach. We utilise
recursion in computer science due to its unparalleled strength and simplicity in solving intricate
problems. Algorithm Calling Itself: The quintessential element that distinguishes recursive
algorithms from other algorithms is that it involves the function calling itself. This self-
involvement of the function happens until handling the smaller or simpler problem instances
become manageable. Solving Smaller Instances of the Same Problem: Recursive algorithms
exhibit the beauty of problem-solving through its capability to divide an overwhelming problem
into manageable sub-problems. These sub-problems are essentially smaller instances of the very
problem itself. It's crucial to note that the concept of 'smaller instances' can mean two things
based on the problem's nature. It may refer to either a physically smaller subset of the original
problem or a problem that is logically simpler to solve. An essential feature to bear in mind is the

Algorithms and Complexity Analysis Page 41

definition of a 'base case' when designing a recursive algorithm. The base case halts recursive
calls from being made infinitely, thus preventing memory overflow. Importantly, any recursive
algorithm must always progress towards the base case. Carefully choosing the base case and
understanding the problem's nature is key to implementing an efficient recursive algorithm. Only
then will the problem become simpler with each recursive call, gradually progressing towards the
base case.

A Recursive Algorithm, in the exact sense, is an algorithmic approach to problem-solving, which

involves a function invoking itself to decompose the problem into smaller sub-problems until it
becomes imperative to proceed with resolving them. The algorithmic process ceases when it hits
the base case, making it an expedient approach to nailing complex problems in an elegant
manner.

Application of Recursive Algorithm in Computer Science

Recursive algorithms have profound implications and widespread applications in various realms
of computer science, owing to their ability in presenting concise and clean solutions to intricate
problems. With their unique problem-solving approach that deals with smaller instances of the
same problem, recursive algorithms often prove to be immensely beneficial in tackling complex
scenarios. Let's explore a few paramount applications of recursive algorithms:

1. Sorting Algorithms: Recursive algorithms drive some of the most efficient sorting algorithms
in computer science, such as Merge Sort and Quick Sort. They utilise the divide-and-conquer
strategy to divide the dataset into smaller subsets, recursively sort them, and finally reassemble
them into a sorted whole.

2. Tree and Graph Data Structures: Recursive algorithms are extensively used in various
operations involving tree and graph data structures. Be it performing Depth-First Search on a
graph, or traversing a Binary Search Tree, recursive algorithms provide the simplest and the most
intuitive solutions. The process of breaking the problem down to smaller sub-problems aligns
with the inherent hierarchical structure of trees and graphs, making recursion the go-to approach
for many tasks involving these data structures.

Algorithms and Complexity Analysis Page 42

3. Dynamic Programming: Recursion plays a crucial role in dynamic programming, a method
used for solving complex optimization problems by breaking them down into simpler sub-
problems. Recursive algorithms aid in defining the optimal substructure of the problem, which
forms the crux of dynamic programming.

4. Parsing and Tree-Based Computations: Recursive algorithms are of immense help in

parsing expressions and executing tree-based computations. Recursive descent parsing, a
common method used in writing compilers and interpreters, uses recursion to handle nested
structures.

Real World Applications of Recursive Algorithms

Beyond theoretical uses, recursive algorithms manifest in numerous real-world applications.

They help in reducing complex problems into easily manageable tasks.
1. Graphics and Image Processing: Recursive algorithms form the backbone of many complex
image processing and graphics operations. An example is the popular 'flood fill' algorithm, often
used in graphics editors. This algorithm begins at a pixel within the boundary and continues to
grow, filling adjacent pixels recursively until the boundary value is encountered.
2. Game Solving: Recursive algorithms are frequently used in creating and solving game tree
structures in strategy games like Chess, Tic-Tac-Toe, etc. The minimax algorithm, a recursive
method for decision making, is often used by AI in finding optimal moves.
3. File System Traversals: Recursive algorithms are highly useful in file system traversals. When
performing operations such as searching for files, recursive algorithms can efficiently navigate
through nested directories, given the inherent tree-like structure of file systems.
4. Divide and Conquer Algorithms: Many divide and conquer algorithms, such as Merge Sort,
Quick Sort, Binary Search, and more, contain processes that can be broken down into smaller,
identical processes, making recursive algorithms a natural fit.
5. Parsing algorithms: Recursive algorithms are used in the syntax checking of programming
languages in compilers. For instance, parsing or constructing syntax trees, which are inherently
hierarchical structures, relies heavily on recursion for processing nested symbols. An excellent
takeaway here would be to realise that recursive algorithms have a wide variety of applications,
from simple factorial calculations to complex system-level operations. Understanding them in

Algorithms and Complexity Analysis Page 43

full - their advantages, limitations, and unique situations where they shine - is key to leveraging
their capabilities and using them correctly to solve a variety of problems.

Analysis of recursive algorithms' time and space complexities is crucial for understanding their
efficiency and resource usage. Here's how you typically perform such analysis:

Time Complexity Analysis:

1. Recurrence Relation: Define a recurrence relation that represents the time complexity of
the algorithm. This relation describes the time taken by the algorithm in terms of the time
taken by its subproblems.
2. Solving the Recurrence: Solve the recurrence relation to find a closed-form expression
for the time complexity. This often involves techniques like substitution, iteration, or
master theorem for divide-and-conquer algorithms.

Space Complexity Analysis:

1. Call Stack: Understand how the call stack behaves during the execution of the recursive
algorithm. Each recursive call consumes space on the call stack.
2. Memory Usage: Analyze the memory usage of the algorithm. This includes space
required for variables, data structures, and any additional memory used during recursive
calls.
3. Recursion Depth: Determine the maximum depth of recursion, which corresponds to the
maximum space consumed on the call stack.

Example - Recursive Factorial Algorithm:

int factorial(int n) {
if (n <= 1) {
return 1;
} else {
return n * factorial(n - 1);
}
}

Algorithms and Complexity Analysis Page 44

Time Complexity Analysis:

 Recurrence Relation: T(n) = T(n-1) + O(1), where T(n) represents the time taken to
compute factorial(n).
 Solving the Recurrence: By unfolding the recursion, we find that T(n) = O(n), as each
recursive call reduces the problem size by one until it reaches the base case.

Space Complexity Analysis:

 Call Stack: The space required on the call stack is proportional to the depth of recursion.
 Recursion Depth: The recursion depth for factorial(n) is n, as it makes n recursive calls.
 Space Complexity: O(n) due to the space used on the call stack.

Algorithms and Complexity Analysis Page 45

 CHAPTER FOUR

Fundamental Computing Algorithms

Introduction to Fundamental Computing Algorithms:

Algorithms are step-by-step procedures or instructions designed to solve specific problems or

perform tasks. In computing, algorithms play a crucial role in various areas, from data processing
and analysis to system optimization and artificial intelligence. Fundamental computing
algorithms form the backbone of computer science, providing essential tools for solving
problems efficiently and effectively.

Importance of Fundamental Computing Algorithms:

1. Problem Solving: Algorithms provide systematic approaches to solving problems,

breaking them down into smaller, manageable steps.
2. Efficiency: Well-designed algorithms optimize resource usage, such as time and
memory, to perform tasks in the most efficient manner.
3. Scalability: Algorithms are scalable, meaning they can handle increasing amounts of data
or complexity without significant performance degradation.
4. Foundation for Advanced Techniques: Fundamental algorithms serve as building
blocks for more complex algorithms and systems, enabling the development of advanced
technologies and applications.

Categories of Fundamental Computing Algorithms:

Searching Algorithms: Searching algorithms are fundamental operations in computer science

used to find the presence or absence of a target value within a collection of data. These
algorithms play a crucial role in various applications, from information retrieval to data analysis
and database management. In this extensive exploration, we'll delve into the key searching
algorithms, their algorithms, practical applications, complexities, and examples.

1. Sequential Search Algorithm:

Algorithm:

Algorithms and Complexity Analysis Page 46

  Start from the first element of the list.
 Compare the target value with each element sequentially.
 If the target value matches an element, return its index.
 If the target value is not found after iterating through the entire list, return a "not found"
indication.

Complexity:

 Time Complexity: O(n) - Linear time complexity since the algorithm may need to
traverse the entire list.
 Space Complexity: O(1) - Constant space complexity since no additional data structures
are used.

Practical Application:

 Unordered Lists: Sequential search is useful for searching unsorted lists or arrays where
elements are not in any particular order.
 Linear Search in Arrays: It's commonly used in programming languages to find
elements in arrays or lists.

Example:

python
# Sequential search implementation in Python
def sequential_search(arr, target):
for i in range(len(arr)):
if arr[i] == target:
return i # Return index if found
return -1 # Return -1 if not found

# Example usage
my_list = [3, 5, 2, 8, 4, 7, 1]
target_value = 8

Algorithms and Complexity Analysis Page 47

index = sequential_search(my_list, target_value)
if index != -1:
print(f"Target value {target_value} found at index {index}.")
else:
print("Target value not found.")

2. Binary Search Algorithm:

Algorithm:

 Start with the entire sorted array.

 Calculate the middle index of the current interval.
 Compare the target value with the middle element.
 If the target value matches the middle element, return its index.
 If the target value is less than the middle element, search the left half of the array.
 If the target value is greater than the middle element, search the right half of the array.
 Repeat until the target value is found or the interval is empty.

Complexity:

 Time Complexity: O(log n) - Logarithmic time complexity since the search interval is
halved in each step.
 Space Complexity: O(1) - Constant space complexity since the algorithm uses only a
few variables for bookkeeping.

Practical Application:

 Sorted Lists or Arrays: Binary search is ideal for searching in sorted collections where
elements are arranged in ascending or descending order.
 Efficient Searching: It's commonly used in libraries, databases, and search engines for
efficient searching of large datasets.

Example:

Algorithms and Complexity Analysis Page 48

python
# Binary search implementation in Python
def binary_search(arr, target):
low = 0
high = len(arr) - 1

while low <= high:

mid = (low + high) // 2
if arr[mid] == target:
return mid
elif arr[mid] < target:
low = mid + 1
else:
high = mid - 1
return -1 # Return -1 if not found

# Example usage
sorted_list = [1, 2, 3, 4, 5, 6, 7, 8, 9]
target_value = 7
index = binary_search(sorted_list, target_value)
if index != -1:
print(f"Target value {target_value} found at index {index}.")
else:
print("Target value not found.")

Practical Examples of Searching Algorithms:

1. Searching in Databases: Searching algorithms are extensively used in databases to

locate records or entries based on specified criteria.
2. Web Search Engines: Algorithms similar to binary search are employed in web search
engines to quickly retrieve relevant web pages from an index.

Algorithms and Complexity Analysis Page 49

 3. File Searching: Operating systems use searching algorithms to locate files or directories
based on user queries.
4. Finding Elements in Data Structures: Searching algorithms are used in various data
structures such as trees, graphs, and hash tables to locate specific elements efficiently.
5. Pattern Matching: Searching algorithms play a vital role in string processing tasks such
as pattern matching in text processing and bioinformatics.

2. Sorting Algorithms: Sorting algorithms are fundamental operations in computer science used
to arrange elements of a collection in a specific order. These algorithms are ubiquitous in various
applications, from organizing data for efficient searching to optimizing performance in
computational tasks. In this comprehensive exploration, we'll delve into key sorting algorithms,
their algorithms, practical applications, complexities, and examples.

1. Bubble Sort Algorithm:

1. Bubble Sort:
o Description: Bubble Sort repeatedly steps through the list, compares adjacent
elements, and swaps them if they are in the wrong order.
o Example:
 Input: [5, 3, 8, 4, 2]
 Pass 1: [3, 5, 4, 2, 8]
 Pass 2: [3, 4, 2, 5, 8]
 Pass 3: [3, 2, 4, 5, 8]
 Pass 4: [2, 3, 4, 5, 8]
 Output: [2, 3, 4, 5, 8]

Bubble Sort is a simple sorting algorithm that repeatedly steps through the list, compares
adjacent elements, and swaps them if they are in the wrong order. The pass through the list is
repeated until the list is sorted. Here's a simple algorithm for Bubble Sort:

1. Start with an unsorted list of elements.

Algorithms and Complexity Analysis Page 50

 2. Iterate through the list from the beginning to the end.
3. For each pair of adjacent elements, compare them.
4. If the elements are in the wrong order (i.e., the current element is greater than the next
element), swap them.
5. Continue iterating through the list, comparing and swapping adjacent elements, until no
swaps are needed.
6. The list is now sorted.

Here's a Python implementation of the Bubble Sort algorithm:

python
def bubble_sort(arr):
n = len(arr)
# Traverse through all array elements
for i in range(n):
# Last i elements are already in place
for j in range(0, n - i - 1):
# Traverse the array from 0 to n-i-1
# Swap if the element found is greater
# than the next element
if arr[j] > arr[j + 1]:
arr[j], arr[j + 1] = arr[j + 1], arr[j]

# Example usage:
arr = [64, 34, 25, 12, 22, 11, 90]
bubble_sort(arr)
print("Sorted array:", arr)

This algorithm has a time complexity of O(n^2) in the worst case scenario, where n is the
number of elements in the list. It is not recommended for large datasets due to its inefficiency
compared to more advanced sorting algorithms like Merge Sort or Quick Sort. However, Bubble

Algorithms and Complexity Analysis Page 51

Sort is easy to understand and implement, making it suitable for educational purposes or sorting
small datasets.

2. Quick Sort:
o Description: Quick Sort selects a pivot element and partitions the array into two
sub-arrays around the pivot such that elements smaller than the pivot are on the
left, and larger elements are on the right. It then recursively sorts the sub-arrays.
o Example:
 Input: [7, 2, 1, 6, 8, 5, 3, 4]
 Pivot: 4
 Left sub-array: [2, 1, 3]
 Right sub-array: [7, 6, 8, 5]
 Output: [1, 2, 3, 4, 5, 6, 7, 8]

Quick Sort is a divide-and-conquer sorting algorithm that partitions the array into two sub-arrays
based on a pivot element. Elements smaller than the pivot are placed to its left, and elements
greater than the pivot are placed to its right. The process is recursively applied to the sub-arrays
until the entire array is sorted. Here's a simple algorithm for Quick Sort:

1. Choose a pivot element from the array. This pivot can be selected randomly, or it can be
the first, last, or middle element of the array.
2. Partition the array into two sub-arrays: elements less than the pivot and elements greater
than the pivot.
3. Recursively apply Quick Sort to the two sub-arrays.
4. Concatenate the sorted sub-arrays and the pivot to obtain the final sorted array.

Here's a Python implementation of the Quick Sort algorithm:

python
def quick_sort(arr):
if len(arr) <= 1:
return arr
else:

Algorithms and Complexity Analysis Page 52

 pivot = arr[0] # Choosing the first element as the pivot
less_than_pivot = [x for x in arr[1:] if x <= pivot]
greater_than_pivot = [x for x in arr[1:] if x > pivot]
return quick_sort(less_than_pivot) + [pivot] + quick_sort(greater_than_pivot)

# Example usage:
arr = [64, 34, 25, 12, 22, 11, 90]
sorted_arr = quick_sort(arr)
print("Sorted array:", sorted_arr)

This algorithm has a time complexity of O(n log n) in the average and best case scenarios and
O(n^2) in the worst case scenario when the pivot selection is not optimal. However, with proper
pivot selection techniques (e.g., median-of-three), the worst-case time complexity can be
avoided. Quick Sort is efficient for sorting large datasets and is widely used in practice.

3. Merge Sort:
o Description: Merge Sort divides the array into halves recursively until each sub-
array contains only one element. It then merges the sub-arrays in sorted order.
o Example:
 Input: [6, 3, 8, 2, 7, 1, 4, 5]
 Step 1: [6, 3, 8, 2] [7, 1, 4, 5]
 Step 2: [6, 3] [8, 2] [7, 1] [4, 5]
 Step 3: [3, 6] [2, 8] [1, 7] [4, 5]
 Step 4: [2, 3, 6, 8] [1, 4, 5, 7]
 Final Step: [1, 2, 3, 4, 5, 6, 7, 8]

Merge Sort is a divide-and-conquer sorting algorithm that divides the array into smaller sub-
arrays until each sub-array contains only one element. It then merges adjacent sub-arrays in a
sorted order until the entire array is sorted. Here's a simple algorithm for Merge Sort:

1. Divide the array into two halves.

Algorithms and Complexity Analysis Page 53

 2. Recursively apply Merge Sort to each half until each sub-array contains only one
element.
3. Merge the sorted sub-arrays by comparing the elements from each sub-array and placing
them in the correct order.
4. Repeat the merging process until the entire array is sorted.

Here's a Python implementation of the Merge Sort algorithm:

python
def merge_sort(arr):
if len(arr) > 1:
mid = len(arr) // 2 # Find the middle index
left_half = arr[:mid] # Divide the array into two halves
right_half = arr[mid:]

# Recursively apply merge sort to each half

merge_sort(left_half)
merge_sort(right_half)

# Merge the sorted halves

i=j=k=0
while i < len(left_half) and j < len(right_half):
if left_half[i] < right_half[j]:
arr[k] = left_half[i]
i += 1
else:
arr[k] = right_half[j]
j += 1
k += 1

# Copy the remaining elements from left_half and right_half if any

while i < len(left_half):

Algorithms and Complexity Analysis Page 54

 arr[k] = left_half[i]
i += 1
k += 1
while j < len(right_half):
arr[k] = right_half[j]
j += 1
k += 1

# Example usage:
arr = [64, 34, 25, 12, 22, 11, 90]
merge_sort(arr)
print("Sorted array:", arr)

This algorithm has a time complexity of O(n log n) in all cases, making it efficient for sorting
large datasets. Merge Sort is a stable sorting algorithm, meaning it preserves the relative order of
equal elements, and it is suitable for sorting linked lists as well as arrays.

Practical Examples of Sorting Algorithms:

1. Database Indexing: Sorting algorithms are used in databases to organize data efficiently
for indexing and querying.
2. Ordering Lists: Sorting algorithms are employed in various applications to order lists of
items alphabetically, numerically, or based on custom criteria.
3. Search Engine Optimization: Sorting algorithms play a crucial role in search engine
optimization (SEO) by arranging search results based on relevance and popularity.
4. Data Analysis: Sorting algorithms are used in data analysis tasks to arrange datasets for
statistical analysis, visualization, and reporting.
5. File Systems: Operating systems use sorting algorithms to organize files and directories
on disk drives for efficient storage and retrieval.

Algorithms and Complexity Analysis Page 55

Search Algorithms:

1. Linear Search:
o Description: Linear Search checks each element of the list sequentially until the
target element is found or the entire list has been traversed.
o Example:
 Input: [5, 8, 3, 2, 7, 1], Target: 3
 Steps: Compare 5, 8, 3 (found), stop
 Output: Element found at index 2
2. Binary Search:
o Description: Binary Search works on a sorted list by repeatedly dividing the
search interval in half until the target element is found or the interval is empty.
o Example:
 Input: [1, 2, 3, 4, 5, 6, 7, 8, 9], Target: 6
 Steps: Compare with mid element 5, 7, 6 (found), stop
 Output: Element found at index 5

Here's a simple algorithm for linear search and binary search:

1. Linear Search:
o Start from the beginning of the array.
o Compare each element with the target value until a match is found or the end of
the array is reached.
o If the target value is found, return its index. Otherwise, return -1.

python
def linear_search(arr, target):
for i in range(len(arr)):
if arr[i] == target:
return i
return -1

Algorithms and Complexity Analysis Page 56

# Example usage:
arr = [2, 3, 5, 7, 11, 13]
target = 7
index = linear_search(arr, target)
if index != -1:
print(f"Element {target} found at index {index}.")
else:
print(f"Element {target} not found.")

2. Binary Search (for sorted arrays):

o Compare the target value with the middle element of the array.
o If the target value matches the middle element, return its index.
o If the target value is greater than the middle element, search the right half of the
array.
o If the target value is smaller than the middle element, search the left half of the
array.
o Repeat the process until the target value is found or the search space is exhausted.

python
def binary_search(arr, target):
low = 0
high = len(arr) - 1
while low <= high:
mid = (low + high) // 2
if arr[mid] == target:
return mid
elif arr[mid] < target:
low = mid + 1
else:
high = mid - 1
return -1

Algorithms and Complexity Analysis Page 57

# Example usage:
arr = [2, 3, 5, 7, 11, 13]
target = 7
index = binary_search(arr, target)
if index != -1:
print(f"Element {target} found at index {index}.")
else:
print(f"Element {target} not found.")

These algorithms are fundamental search algorithms used to find elements within an array.
Linear search has a time complexity of O(n), where n is the number of elements in the array.
Binary search, on the other hand, has a time complexity of O(log n), but it requires the array to
be sorted beforehand.

Numerical Algorithms for Mathematical Computations:

Numerical algorithms are computational techniques used to solve mathematical problems

approximately, especially those involving continuous variables or large datasets. These
algorithms are fundamental in scientific computing, engineering, and various fields where
precise mathematical solutions are required.

Types of Numerical Algorithms:

1. Root-Finding Algorithms: These algorithms find the roots (solutions) of equations,

including polynomial equations, transcendental equations, and systems of equations.
Common root-finding algorithms include Newton-Raphson method, bisection method,
and secant method.
2. Linear Algebra Algorithms: Linear algebra algorithms deal with operations on matrices
and vectors. They are used in solving linear systems of equations, eigenvalue problems,
and matrix factorization. Examples include Gaussian elimination, LU decomposition, and
singular value decomposition (SVD).
3. Interpolation and Approximation: Interpolation algorithms estimate unknown values
between known data points, while approximation algorithms find simpler functions that

Algorithms and Complexity Analysis Page 58

 closely match complex functions. Examples include polynomial interpolation, spline
interpolation, and curve fitting techniques like least squares regression.
4. Numerical Integration and Differentiation: Numerical integration algorithms
approximate the definite integral of a function over a specified interval. Similarly,
numerical differentiation algorithms estimate derivatives of functions at given points.
Techniques include Simpson's rule, trapezoidal rule, and finite difference methods.
5. Optimization Algorithms: Optimization algorithms aim to find the maximum or
minimum of a function within a given domain. These algorithms are used in various
optimization problems, such as parameter optimization in machine learning and
engineering design optimization. Examples include gradient descent, genetic algorithms,
and simulated annealing.

Algorithms and Complexity Analysis Page 59

 CHAPTER FIVE

Graph Algorithms

Graph Representation:

Graphs are versatile data structures used to model relationships between entities. They can
represent a wide range of real-world scenarios, from social networks to computer networks to
transportation systems. Graphs can be classified into various types based on their properties and
structures. Let's explore some common types of graphs with diagrams:

1. Undirected Graph:
o An undirected graph is a graph where edges have no direction.
o It represents symmetric relationships between vertices.
o Diagram:

A ---- B
/ \
/ \
D ---- C ---- E

In the diagram, the edges (lines) between vertices (nodes) have no arrows, indicating that
the relationship between vertices is bidirectional.

 Directed Graph (Digraph):

 A directed graph is a graph where edges have a direction.

 It represents asymmetric relationships between vertices.
 Diagram:

A --> B
| | \
v v \
D <-- C E

Algorithms and Complexity Analysis Page 60

  In the diagram, the arrows on the edges indicate the direction of the relationship between
vertices. For example, in the edge from A to B, the relationship goes from A to B, but not
from B to A.

Weighted Graph:

 A weighted graph is a graph where edges have weights or costs assigned to them.
 It is used to represent scenarios where there is a cost associated with traversing from one
vertex to another.
 Diagram:

css
A
/|\
2 | 5
/ | \
B-3-C
- In the diagram, the numbers next to the edges represent the weights of the connections between
vertices.

4. **Unweighted Graph**:
- An unweighted graph is a graph where edges have no weights assigned to them.
- It represents relationships between vertices without considering any associated costs.
- Diagram:

A ---- B
/ \
/ \

D ---- C ---- E

- In the diagram, there are no weights associated with the edges between vertices.

5. **Cyclic Graph**:
- A cyclic graph is a graph that contains at least one cycle, where a cycle is a path that starts and
ends at the same vertex.

Algorithms and Complexity Analysis Page 61

- Diagram:

A ---- B ---- C
/ | \
/ | \
D ---- E ------- F
less
- In the diagram, the path A -> B -> E -> C -> F -> B forms a cycle because it starts and ends at
vertex B.

6. **Acyclic Graph**:
- An acyclic graph is a graph that contains no cycles.
- It represents relationships without any loops or repetitions.
- Diagram:

A ---- B ---- C

| | |

| | |

D E F

- In the diagram, there are no cycles present, as there is no path that starts and ends at the same
vertex.
These are some common types of graphs, each with its own characteristics and applications.
Diagrams help visualize the structure of these graphs and understand their properties more
intuitively.

Graphs can be represented in various ways, including adjacency matrix, adjacency list, and edge
list.

 Adjacency Matrix: A two-dimensional array graph[u][v] represents whether there is an

edge between vertices u and v.

Algorithms and Complexity Analysis Page 62

  Adjacency List: A list of lists or a dictionary where each vertex is associated with a list
of its adjacent vertices.
 Edge List: A list of tuples (u, v, weight) representing the edges between vertices u and v
with a given weight.

Breadth-first Search (BFS):

Breadth-first search explores a graph level by level, visiting all the neighbors of a vertex before
moving on to the next level.

Example:

python
from collections import deque

def bfs(graph, start):

visited = set()
queue = deque([start])
while queue:
vertex = queue.popleft()
if vertex not in visited:
visited.add(vertex)
print(vertex, end=' ')
for neighbor in graph[vertex]:
if neighbor not in visited:
queue.append(neighbor)

# Example usage:
graph = {
'A': ['B', 'C'],
'B': ['A', 'D', 'E'],
'C': ['A', 'F'],

Algorithms and Complexity Analysis Page 63

 'D': ['B'],
'E': ['B', 'F'],
'F': ['C', 'E']
}
print("BFS traversal:")
bfs(graph, 'A') # Output: A B C D E F

Depth-first Search (DFS):

Depth-first search explores as far as possible along each branch before backtracking.

Example:

python
def dfs(graph, start, visited=None):
if visited is None:
visited = set()
visited.add(start)
print(start, end=' ')
for neighbor in graph[start]:
if neighbor not in visited:
dfs(graph, neighbor, visited)

# Example usage:
print("\nDFS traversal:")
dfs(graph, 'A') # Output: A B D E F C

Shortest Path Algorithms:

1. Dijkstra's Algorithm:

Problem: Find the shortest path from a source vertex to all other vertices in a weighted graph.

Algorithms and Complexity Analysis Page 64

To find the shortest path from a source vertex to all other vertices in a weighted graph, we can
use Dijkstra's algorithm. Dijkstra's algorithm is a greedy algorithm that iteratively selects the
vertex with the shortest distance from the source and updates the distances to its neighboring
vertices accordingly. Here's how Dijkstra's algorithm works:

Dijkstra's Algorithm:

1. Initialization:
o Assign a distance value to every vertex. Initialize the distance to the source vertex
as 0 and all other distances as infinity.
o Create an empty set to keep track of vertices whose shortest distance from the
source has been found.
o Initialize a priority queue (min-heap) to store vertices based on their current
shortest distance from the source.
2. Main Loop:
o Repeat until all vertices have been processed:
 Select the vertex u with the minimum distance from the priority queue
(vertex with the shortest distance).
 Mark vertex u as visited.
 Update the distances of all adjacent vertices v of u if the total distance
from the source to v through u is shorter than the current distance recorded
for v.
 Enqueue/update v in the priority queue with its updated distance.
3. Termination:
o Once all vertices have been processed, the shortest path distances from the source
vertex to all other vertices are found.

Example:

Consider the following weighted graph:

scss
4 2
(1)---------(2)

Algorithms and Complexity Analysis Page 65

 | \ | \
3 | \ 5| \ 1
| \ | \
(3)---------(4)---(5)
6 3

Let's find the shortest paths from vertex 1 to all other vertices using Dijkstra's algorithm:

1. Initialization:
o Start at vertex 1. Set its distance to 0 and all other vertices' distances to infinity.
o Initialize an empty set of visited vertices and a priority queue.
2. Main Loop:
o Start with vertex 1.
o Visit vertex 1, update the distances of its neighbors: 2 (distance 4) and 3 (distance
3).
o Visit vertex 3, update the distance of vertex 4 (distance 9).
o Visit vertex 2, update the distance of vertex 4 (distance 6).
o Visit vertex 4, update the distance of vertex 5 (distance 9).
o Visit vertex 5, no further updates.
3. Termination:
o Shortest path distances from vertex 1 to all other vertices: {1: 0, 2: 4, 3: 3, 4: 6, 5:
9}.

Implementation:

Here's a Python implementation of Dijkstra's algorithm:

python
import heapq

def dijkstra(graph, source):

distances = {vertex: float('inf') for vertex in graph}
distances[source] = 0
priority_queue = [(0, source)]
while priority_queue:

Algorithms and Complexity Analysis Page 66

 current_distance, current_vertex = heapq.heappop(priority_queue)
if current_distance > distances[current_vertex]:
continue
for neighbor, weight in graph[current_vertex].items():
distance = current_distance + weight
if distance < distances[neighbor]:
distances[neighbor] = distance
heapq.heappush(priority_queue, (distance, neighbor))

return distances

# Example graph represented as an adjacency list

graph = {
1: {2: 4, 3: 3},
2: {1: 4, 3: 5, 4: 2},
3: {1: 3, 2: 5, 4: 6},
4: {2: 2, 3: 6, 5: 3},
5: {4: 3}
}

source_vertex = 1
shortest_distances = dijkstra(graph, source_vertex)
print("Shortest path distances from vertex", source_vertex, ":", shortest_distances)

This implementation finds the shortest path distances from a given source vertex to all other
vertices in the graph using Dijkstra's algorithm.

find the shortest distance graph represented as an adjacency list graph = { 1: {2: 4, 3: 3}, 2: {1:
4, 3: 5, 4: 2}, 3: {1: 3, 2: 5, 4: 6}, 4: {2: 2, 3: 6, 5: 3}, 5: {4: 3} }

Algorithms and Complexity Analysis Page 67

ChatGPT

To find the shortest distances between all pairs of vertices in the given graph represented as an
adjacency list, we can use Floyd-Warshall algorithm. This algorithm works for both directed and
undirected graphs with positive or negative edge weights (as long as there are no negative
cycles). Here's how the algorithm works:

Floyd-Warshall Algorithm:

1. Initialization:
o Initialize a 2D array distances of size (n x n), where n is the number of vertices
in the graph.
o Initialize distances[i][j] to the weight of the edge between vertices i and j if
there's an edge, otherwise set it to infinity.
o Set distances[i][i] to 0 for all vertices i.
2. Main Loop:
o Iterate over all vertices k from 1 to n.
o For each pair of vertices i and j, update distances[i][j] to the minimum of:
 The current value of distances[i][j].
 The sum of distances[i][k] and distances[k][j].
3. Termination:
o After the main loop, distances[i][j] contains the shortest distance between
vertex i and j for all pairs of vertices i and j.

Example:

Given the graph represented as an adjacency list:

python
graph = {
1: {2: 4, 3: 3},
2: {1: 4, 3: 5, 4: 2},
3: {1: 3, 2: 5, 4: 6},
4: {2: 2, 3: 6, 5: 3},
5: {4: 3}
}

We can construct the adjacency matrix and then apply the Floyd-Warshall algorithm to find the
shortest distances.

Implementation:
python
INF = float('inf')

def floyd_warshall(graph):
vertices = list(graph.keys())
num_vertices = len(vertices)

Algorithms and Complexity Analysis Page 68

 distances = [[INF] * num_vertices for _ in range(num_vertices)]

# Initialize distances based on the adjacency list

for i in range(num_vertices):
distances[i][i] = 0
if i+1 in graph:
for j, weight in graph[i+1].items():
distances[i][j-1] = weight

# Apply Floyd-Warshall algorithm

for k in range(num_vertices):
for i in range(num_vertices):
for j in range(num_vertices):
distances[i][j] = min(distances[i][j], distances[i][k] +
distances[k][j])

return distances

# Given graph represented as an adjacency list

graph = {
1: {2: 4, 3: 3},
2: {1: 4, 3: 5, 4: 2},
3: {1: 3, 2: 5, 4: 6},
4: {2: 2, 3: 6, 5: 3},
5: {4: 3}
}

shortest_distances = floyd_warshall(graph)
for i in range(len(shortest_distances)):
for j in range(len(shortest_distances[i])):
if shortest_distances[i][j] == INF:
shortest_distances[i][j] = "INF"
print("Shortest distance graph represented as an adjacency list:")
for i, row in enumerate(shortest_distances):
print(i+1, ":", {j+1: dist for j, dist in enumerate(row)})

This implementation will print the shortest distance graph represented as an adjacency list. The
output will show the shortest distances between all pairs of vertices in the graph.

draw the graph for graph represented as an adjacency list graph = { 1: {2: 4, 3: 3}, 2: {1: 4, 3: 5,
4: 2}, 3: {1: 3, 2: 5, 4: 6}, 4: {2: 2, 3: 6, 5: 3}, 5: {4: 3} }
ChatGPT

To draw the graph represented as an adjacency list, we can visualize each vertex as a node and
draw edges between vertices based on the connections specified in the adjacency list. We'll use a
graphical representation where nodes are circles and edges are lines connecting the circles.

Here's how we can draw the graph:

1. Each vertex in the graph will be represented as a numbered node.

2. For each vertex v, we'll draw edges to its neighboring vertices as specified in the
adjacency list.
3. The weight of each edge will be displayed next to the edge.

Algorithms and Complexity Analysis Page 69

Let's draw the graph:

lua
4 2
1 -----> 2 -------> 4
| 3 ^ \ 2 / |
| | \ / | 3
| | 5 \ / 6 |
| v v |
3 <----- 5 <----- 3

In this visualization:

 Each numbered circle represents a vertex.

 Directed edges indicate the direction of the connection between vertices.
 The numbers next to the edges represent the weights of the connections.

Here's a textual representation of the graph based on the given adjacency list:

scss
1 -> 2 (weight 4), 3 (weight 3)
2 -> 1 (weight 4), 3 (weight 5), 4 (weight 2)
3 -> 1 (weight 3), 2 (weight 5), 4 (weight 6)
4 -> 2 (weight 2), 3 (weight 6), 5 (weight 3)
5 -> 4 (weight 3)

This visualization provides a clear representation of the connections between vertices and the
weights of the edges in the graph.

Example:

python
import heapq

def dijkstra(graph, start):

distances = {vertex: float('inf') for vertex in graph}
distances[start] = 0
priority_queue = [(0, start)]
while priority_queue:
current_distance, current_vertex = heapq.heappop(priority_queue)
if current_distance > distances[current_vertex]:

Algorithms and Complexity Analysis Page 70

 continue
for neighbor, weight in graph[current_vertex].items():
distance = current_distance + weight
if distance < distances[neighbor]:
distances[neighbor] = distance
heapq.heappush(priority_queue, (distance, neighbor))
return distances

# Example usage:
graph = {
'A': {'B': 3, 'C': 1},
'B': {'A': 3, 'C': 7, 'D': 5},
'C': {'A': 1, 'B': 7, 'D': 2},
'D': {'B': 5, 'C': 2}
}
print("Shortest distances from 'A':", dijkstra(graph, 'A')) # Output: {'A': 0, 'B': 3, 'C': 1, 'D': 3}

2. Bellman-Ford Algorithm:

Problem: Find the shortest path from a source vertex to all other vertices in a weighted graph
with negative edge weights.

Example:

python
def bellman_ford(graph, start):
distances = {vertex: float('inf') for vertex in graph}
distances[start] = 0
for _ in range(len(graph) - 1):
for u in graph:
for v, weight in graph[u].items():
if distances[u] + weight < distances[v]:

Algorithms and Complexity Analysis Page 71

 distances[v] = distances[u] + weight
return distances

# Example usage:
graph = {
'A': {'B': 3, 'C': 1},
'B': {'C': 7, 'D': 5},
'C': {'B': -2},
'D': {'B': 4, 'C': 2}
}
print("Shortest distances from 'A':", bellman_ford(graph, 'A')) # Output: {'A': 0, 'B': 1, 'C': -1,
'D': 3}

Minimum Spanning Tree Algorithms:

1. Prim's Algorithm:

Problem: Find a minimum spanning tree of a connected, undirected graph.

Example:

python
def prim_mst(graph):
mst = set()
vertices = list(graph.keys())
visited = set(vertices[0])
while len(visited) < len(vertices):
min_edge = float('inf')
min_edge_info = None
for u in visited:
for v, weight in graph[u].items():
if v not in visited and weight < min_edge:
min_edge = weight

Algorithms and Complexity Analysis Page 72

 min_edge_info = (u, v, weight)
mst.add(min_edge_info)
visited.add(min_edge_info[1])
return mst

# Example usage:
graph = {
'A': {'B': 2, 'D': 5},
'B': {'A': 2, 'C': 3, 'D': 2},
'C': {'B': 3, 'D': 4},
'D': {'A': 5, 'B': 2, 'C': 4}
}
print("Minimum Spanning Tree (Prim's Algorithm):", prim_mst(graph))
# Output: {('A', 'B', 2), ('B', 'C', 3), ('B', 'D', 2)}

2. Kruskal's Algorithm:

Problem: Find a minimum spanning tree of a connected, undirected graph.

Example:

python
class DisjointSet:
def __init__(self, vertices):
self.parent = {v: v for v in vertices}

def find(self, v):

if self.parent[v] != v:
self.parent[v] = self.find(self.parent[v])
return self.parent[v]

def union(self, v1, v2):

Algorithms and Complexity Analysis Page 73

 root1 = self.find(v1)
root2 = self.find(v2)
if root1 != root2:
self.parent[root1] = root2

def kruskal_mst(graph):
mst = set()
vertices = list(graph.keys())
edges = [(u, v, weight) for u in graph for v, weight in graph[u].items()]
edges.sort(key=lambda x: x[2])
ds = DisjointSet(vertices)
for u, v, weight in edges:
if ds.find(u) != ds.find(v):
mst.add((u, v, weight))
ds.union(u, v)
return mst

# Example usage:
print("Minimum Spanning Tree (Kruskal's Algorithm):", kruskal_mst(graph))
# Output: {('A', 'B', 2), ('B', 'D', 2), ('B', 'C', 3)}

These examples illustrate various graph algorithms, including BFS, DFS, shortest path
algorithms (Dijkstra's and Bellman-Ford), and minimum spanning tree algorithms (Prim's and
Kruskal's). Graph algorithms are essential in solving problems related to networks, routing,
optimization, and more.

Module 6

2. Complexity Theory
o Classes of problems: P, NP, NP-hard, NP-complete
o Reductions and NP-completeness
o Approximation algorithms

Algorithms and Complexity Analysis Page 74

Complexity Theory

Classes of Problems:

1. P (Polynomial Time):
o Problems that can be solved in polynomial time by a deterministic Turing
machine.
o Examples: Sorting (e.g., Merge Sort, Quick Sort), Searching (e.g., Binary Search),
Shortest Path (e.g., Dijkstra's Algorithm).

P (Polynomial Time):

In complexity theory, the class P (Polynomial Time) consists of decision problems that can be
solved by a deterministic Turing machine in polynomial time. A problem is said to be in P if
there exists an algorithm that can solve it in polynomial time with respect to the size of its input.
In other words, the running time of the algorithm must be bounded by a polynomial function of
the input size.

Key Properties of P:

1. Problems in P can be solved efficiently by deterministic algorithms.

2. The running time of these algorithms grows polynomially with the size of the input.
3. Polynomial time algorithms are considered efficient in practice, especially for inputs of
moderate size.

Examples of Problems in P:

1. Sorting (e.g., Merge Sort, Quick Sort):

o Given a list of elements, arrange them in non-decreasing or non-increasing order.
o Merge Sort and Quick Sort are examples of sorting algorithms that run in O(n log
n) time on average, making them efficient for large datasets.

Algorithms and Complexity Analysis Page 75

Merge Sort is a divide-and-conquer algorithm that sorts a list of elements by recursively dividing
it into two halves, sorting each half independently, and then merging the sorted halves back
together. It follows the following steps:

1. Divide: Divide the unsorted list into two halves recursively until each sublist contains
only one element.
2. Conquer: Sort each sublist independently using Merge Sort.
3. Merge: Merge the sorted sublists back together to produce a single sorted list.

Detailed Explanation:

1. Divide: The unsorted list is divided into two equal halves (or nearly equal if the number
of elements is odd) until each sublist contains only one element.
2. Conquer: Each sublist is sorted independently using the Merge Sort algorithm. This
involves recursively applying the divide-and-conquer approach to sort each half of the
sublist.
3. Merge: The sorted sublists are merged back together by repeatedly comparing the
elements of the two sublists and selecting the smaller (or larger) element to form the
merged list. This process continues until all elements from both sublists are included in
the merged list.

Example:

Consider an unsorted list [38, 27, 43, 3, 9, 82, 10].

1. Divide: Divide the list into two halves: [38, 27, 43, 3] and [9, 82, 10].
2. Conquer: Sort each sublist independently using Merge Sort.
o Sort [38, 27, 43, 3] to [3, 27, 38, 43].
o Sort [9, 82, 10] to [9, 10, 82].
3. Merge: Merge the sorted sublists back together.
o Compare elements from both sublists: [3, 27, 38, 43] and [9, 10, 82].
o Select the smaller element from the beginning of each sublist to form the merged
list: [3, 9, 10, 27, 38, 43, 82].

Algorithms and Complexity Analysis Page 76

Complexity:

 Time Complexity: O(n log n) in all cases (worst-case, average-case, and best-case),
where n is the number of elements in the list.
 Space Complexity: O(n) for the additional space required for the merge step.

Quick Sort:

Quick Sort is another efficient sorting algorithm that follows the divide-and-conquer approach. It
selects a pivot element from the list and partitions the other elements into two sublists according
to whether they are less than or greater than the pivot. It then recursively sorts the sublists.

Detailed Explanation:

1. Partitioning: Select a pivot element from the list and partition the other elements into
two sublists: one with elements less than the pivot and one with elements greater than the
pivot.
2. Recursion: Recursively apply Quick Sort to the sublists created in the previous step.
3. Combine: No combine step is needed as the sorting is done in place.

Example:

Consider an unsorted list [38, 27, 43, 3, 9, 82, 10].

1. Partitioning: Select a pivot (e.g., 38) and partition the list into two sublists: [27, 3, 9, 10]
(less than 38) and [43, 82] (greater than 38).
2. Recursion: Apply Quick Sort recursively to the sublists [27, 3, 9, 10] and [43, 82].
o For [27, 3, 9, 10], select 27 as the pivot and partition into [3, 9, 10] and [27].
o For [43, 82], no further partitioning is needed as they are already sorted.
3. Combine: No explicit combine step is required as the sorting is done in place.

Complexity:

Algorithms and Complexity Analysis Page 77

  Time Complexity:
o Average-case: O(n log n)
o Worst-case: O(n^2) (Occurs when the pivot selection consistently picks the
smallest or largest element)
 Space Complexity: O(log n) due to the recursive call stack.

Both Merge Sort and Quick Sort are efficient sorting algorithms that can handle large datasets
effectively. Merge Sort is stable and guarantees O(n log n) time complexity, making it suitable
for scenarios where stability is important and the dataset is not too large. Quick Sort, on the other
hand, is often preferred for its space efficiency and typically performs well in practice despite its
worst-case time complexity.

2. Searching (e.g., Binary Search):

o Given a sorted array and a target element, determine whether the target element
exists in the array and, if so, its index.
o Binary Search is a classic example of a search algorithm that runs in O(log n)
time, making it efficient for large arrays.

Binary search is a classic algorithm used to find a target element in a sorted array efficiently.
Here's how you can implement it in Python:

python
def binary_search(arr, target):
left, right = 0, len(arr) - 1

while left <= right:

mid = left + (right - left) // 2

# If the target is found at the middle index

if arr[mid] == target:
return mid

Algorithms and Complexity Analysis Page 78

 # If the target is greater, search the right half
elif arr[mid] < target:
left = mid + 1

# If the target is smaller, search the left half

else:
right = mid - 1

# If the target is not found

return -1

# Example usage:
arr = [1, 3, 5, 7, 9, 11, 13, 15, 17]
target = 11
index = binary_search(arr, target)
if index != -1:
print(f"Target {target} found at index {index}.")
else:
print("Target not found in the array.")

This algorithm works as follows:

 We set two pointers, left and right, initially pointing to the start and end of the array
respectively.
 We repeat the following steps until left is less than or equal to right:
o Calculate the middle index mid.
o If the element at mid is equal to the target, return mid.
o If the element at mid is less than the target, we adjust left to mid + 1 to search the
right half of the array.
o If the element at mid is greater than the target, we adjust right to mid - 1 to search
the left half of the array.

Algorithms and Complexity Analysis Page 79

  If the target is not found after the loop, we return -1 indicating that the target is not
present in the array.

This algorithm has a time complexity of O(log n), where n is the number of elements in the
array, making it very efficient for large arrays.

3. Shortest Path Algorithms (e.g., Dijkstra's Algorithm):

o Given a graph with weighted edges and two vertices (source and destination), find
the shortest path between the two vertices.
o Dijkstra's Algorithm solves the single-source shortest path problem in weighted
graphs with non-negative edge weights and runs in O((V + E) log V) time, where
V is the number of vertices and E is the number of edges.
4. Matrix Multiplication:
o Given two matrices, compute their product.
o Strassen's Algorithm, while not commonly used in practice due to its high
constant factor, is an example of a polynomial time algorithm for matrix
multiplication.
5. Polynomial Evaluation:
o Given a polynomial and a value for its variables, evaluate the polynomial to
obtain the result.
o Horner's Method is an efficient algorithm for polynomial evaluation that runs in
O(n) time, where n is the degree of the polynomial.

Example Problem: Consider the problem of determining whether a given integer is prime.

Problem: Given an integer n, determine whether it is a prime number.

Solution: We can use the following algorithm to determine whether n is prime:

1. If n is less than 2, return False (since prime numbers must be greater than 1).

Algorithms and Complexity Analysis Page 80

 2. Iterate from 2 to the square root of n (inclusive). If any of these numbers divides n
evenly, return False.
3. If no number divides n evenly, return True.

This algorithm runs in O(sqrt(n)) time, which is polynomial in the size of the input (the number
of digits in n). Therefore, the problem of determining whether a given integer is prime is in the
class P.

2. NP (Nondeterministic Polynomial Time):

o Problems for which a given solution can be verified in polynomial time by a
deterministic Turing machine.
o Examples: Subset Sum, Traveling Salesman Problem, Graph Coloring.

NP (Nondeterministic Polynomial Time):

In complexity theory, the class NP (Nondeterministic Polynomial Time) consists of decision

problems for which a given solution can be verified in polynomial time by a deterministic Turing
machine. In other words, if someone claims to have a solution to an NP problem, it can be
verified in polynomial time.

Key Properties of NP:

1. Problems in NP can be verified efficiently.

2. A problem being in NP does not imply that it can be solved efficiently.

Examples of Problems in NP:

1. Subset Sum Problem:

o Problem: Given a set of integers and a target sum, determine whether there exists
a subset of the integers that sums to the target.

Algorithms and Complexity Analysis Page 81

 o Verification: Given a subset of integers, we can verify in polynomial time
whether the sum of the subset equals the target sum.
o Example: Consider the set {1, 3, 5, 7} and the target sum 10. The subset {3, 7}
sums to 10, and this solution can be verified in polynomial time.
2. Traveling Salesman Problem (TSP):
o Problem: Given a list of cities and the distances between each pair of cities, find
the shortest possible route that visits each city exactly once and returns to the
starting city.
o Verification: Given a proposed route, we can verify in polynomial time whether
it visits each city exactly once and returns to the starting city, as well as whether
the total distance of the route is less than or equal to a given bound.
o Example: Consider a set of cities with known distances between them. A
proposed route that visits each city exactly once and returns to the starting city
can be verified to determine if it meets the criteria of the TSP.
3. Graph Coloring Problem:
o Problem: Given an undirected graph, assign colors to the vertices such that no
two adjacent vertices have the same color, using the fewest possible colors.
o Verification: Given a proposed coloring of the vertices, we can verify in
polynomial time whether it satisfies the condition that no adjacent vertices have
the same color.
o Example: Consider an undirected graph with a proposed coloring of its vertices.
We can verify whether this coloring meets the requirements of the graph coloring
problem in polynomial time.

Explanation with Example: Subset Sum Problem

Problem: Given a set of integers and a target sum, determine whether there exists a subset of the
integers that sums to the target.

Verification: Given a subset of integers, we can verify in polynomial time whether the sum of
the subset equals the target sum.

Algorithms and Complexity Analysis Page 82

  Time complexity for verification: O(n), where n is the size of the subset.

Example: Consider the set {1, 3, 5, 7} and the target sum 10.

 Subset {3, 7} sums to 10.

 Verification: Sum of the subset = 3 + 7 = 10, which equals the target sum.
 Therefore, the solution {3, 7} satisfies the Subset Sum Problem and can be verified in
polynomial time.

In summary, NP problems are those for which a solution can be verified in polynomial time,
even though finding the solution itself may be difficult. Subset Sum, Traveling Salesman
Problem, and Graph Coloring are examples of such problems, where the verification of a
solution can be done efficiently.

Traveling Salesman Problem (TSP):

The Traveling Salesman Problem is one of the most famous problems in optimization and graph
theory. It seeks to find the shortest possible route that visits each city exactly once and returns to
the starting city. The problem is NP-hard, meaning there is no known polynomial-time algorithm
that can solve all instances of the problem optimally.

Key Properties of TSP:

1. Objective: Minimize the total distance traveled.

2. Constraints: Visit each city exactly once and return to the starting city.

Example: Let's consider a simple example to illustrate the Traveling Salesman Problem:

Suppose a salesman needs to visit 5 cities (A, B, C, D, E) and return to the starting city (A). The
distances between these cities are as follows:

 Distance from A to B: 10 units

 Distance from A to C: 15 units

Algorithms and Complexity Analysis Page 83

  Distance from A to D: 20 units
 Distance from A to E: 25 units
 Distance from B to C: 35 units
 Distance from B to D: 25 units
 Distance from B to E: 30 units
 Distance from C to D: 30 units
 Distance from C to E: 20 units
 Distance from D to E: 15 units

The objective is to find the shortest route that visits each city exactly once and returns to the
starting city.

Solution: To solve the TSP, we typically use optimization algorithms such as dynamic
programming, branch and bound, or heuristic approaches like nearest neighbor or genetic
algorithms.

For this example, let's consider the brute-force approach of trying all possible permutations of
the cities and calculating the total distance for each permutation to find the shortest route.

1. Generate all possible permutations of the cities (excluding the starting city since it's
fixed).
2. Calculate the total distance for each permutation.
3. Find the permutation with the shortest total distance.

Example of Brute-Force Approach:

1. Permutations of cities (excluding starting city A):

o B-C-D-E
o B-C-E-D
o B-D-C-E
o B-D-E-C
o B-E-C-D
o B-E-D-C

Algorithms and Complexity Analysis Page 84

 o C-B-D-E
o C-B-E-D
o ...
2. Calculate the total distance for each permutation.
3. Find the permutation with the shortest total distance.

Solution: After evaluating all permutations and calculating their total distances, we find that the
shortest route is: A-B-D-E-C-A with a total distance of 80 units.

Verification: We can verify that the solution satisfies the TSP criteria:

 The salesman visits each city exactly once.

 The salesman returns to the starting city.
 The total distance traveled is minimized.

Conclusion: The Traveling Salesman Problem is a classic optimization problem with numerous
real-world applications, such as vehicle routing, logistics planning, and circuit design. While
brute-force approaches are impractical for large instances of the problem, various heuristic and
approximation algorithms are used to find near-optimal solutions efficiently.

olve this Suppose a salesman needs to visit 5 cities (A, B, C, D, E) and return to the starting city
(A). The distances between these cities are as follows: Distance from A to B: 10 units Distance
from A to C: 15 units Distance from A to D: 20 units Distance from A to E: 25 units Distance
from B to C: 35 units Distance from B to D: 25 units Distance from B to E: 30 units Distance
from C to D: 30 units Distance from C to E: 20 units Distance from D to E: 15 units

To solve the Traveling Salesman Problem (TSP) for this scenario, we will use the brute-force
approach of trying all possible permutations of the cities and calculating the total distance for
each permutation to find the shortest route.

Given the distances between the cities as described, let's calculate the total distance for each
permutation of the cities (excluding the starting city since it's fixed) and find the permutation
with the shortest total distance.

Algorithms and Complexity Analysis Page 85

Here's the calculation process:

1. Generate all possible permutations of the cities (excluding the starting city A).
2. Calculate the total distance for each permutation.
3. Find the permutation with the shortest total distance.

Let's start with step 1:

plaintext
Possible permutations (excluding starting city A):
1. B-C-D-E
2. B-C-E-D
3. B-D-C-E
4. B-D-E-C
5. B-E-C-D
6. B-E-D-C
7. C-B-D-E
8. C-B-E-D
9. C-D-B-E
10. C-D-E-B
11. C-E-B-D
12. C-E-D-B
13. D-B-C-E
14. D-B-E-C
15. D-C-B-E
16. D-C-E-B
17. D-E-B-C
18. D-E-C-B
19. E-B-C-D
20. E-B-D-C
21. E-C-B-D
22. E-C-D-B

Algorithms and Complexity Analysis Page 86

23. E-D-B-C
24. E-D-C-B

Now, let's calculate the total distance for each permutation:

plaintext
1. B-C-D-E: 35 + 30 + 15 + 15 = 95 units
2. B-C-E-D: 35 + 20 + 15 + 25 = 95 units
3. B-D-C-E: 25 + 30 + 30 + 15 = 100 units
4. B-D-E-C: 25 + 15 + 20 + 30 = 90 units
5. B-E-C-D: 30 + 20 + 15 + 25 = 90 units
6. B-E-D-C: 30 + 15 + 25 + 30 = 100 units
7. C-B-D-E: 35 + 25 + 15 + 15 = 90 units
8. C-B-E-D: 35 + 30 + 25 + 20 = 110 units
9. C-D-B-E: 30 + 25 + 25 + 15 = 95 units
10. C-D-E-B: 30 + 15 + 20 + 35 = 100 units
11. C-E-B-D: 20 + 35 + 25 + 15 = 95 units
12. C-E-D-B: 20 + 15 + 25 + 35 = 95 units
13. D-B-C-E: 25 + 35 + 30 + 15 = 105 units
14. D-B-E-C: 25 + 30 + 20 + 30 = 105 units
15. D-C-B-E: 30 + 35 + 25 + 15 = 105 units
16. D-C-E-B: 30 + 15 + 20 + 35 = 100 units
17. D-E-B-C: 15 + 30 + 20 + 35 = 100 units
18. D-E-C-B: 15 + 20 + 25 + 35 = 95 units
19. E-B-C-D: 25 + 30 + 15 + 15 = 85 units
20. E-B-D-C: 25 + 15 + 20 + 30 = 90 units
21. E-C-B-D: 20 + 35 + 25 + 15 = 95 units
22. E-C-D-B: 20 + 15 + 25 + 35 = 95 units
23. E-D-B-C: 15 + 30 + 20 + 35 = 100 units
24. E-D-C-B: 15 + 20 + 25 + 35 = 95 units

Finally, we find that the permutation with the shortest total distance is:

Algorithms and Complexity Analysis Page 87

plaintext
Shortest route: E-B-C-D-A
Total distance: 85 units

Therefore, the salesman should follow the route E-B-C-D-A to minimize the total distance
traveled while visiting each city exactly once and returning to the starting city A.

3. NP-hard (Nondeterministic Polynomial-hard):

o Problems that are at least as hard as the hardest problems in NP.
o Solutions to NP-hard problems cannot be verified in polynomial time.
o Examples: Traveling Salesman Problem, Knapsack Problem.

NP-hard (Nondeterministic Polynomial-hard):

NP-hard problems are a class of decision problems that are at least as hard as the hardest
problems in NP. This means that if there exists a polynomial-time algorithm for any NP-hard
problem, then there exists a polynomial-time algorithm for every problem in NP, making NP-
hard problems among the most difficult problems in computational complexity theory.

Key Properties of NP-hard Problems:

1. NP-hard problems are at least as hard as the hardest problems in NP.

2. Solutions to NP-hard problems cannot be verified in polynomial time, unlike problems in
NP where solutions can be verified efficiently.

Examples of NP-hard Problems:

1. Traveling Salesman Problem (TSP):

o Problem: Given a list of cities and the distances between each pair of cities, find
the shortest possible route that visits each city exactly once and returns to the
starting city.

Algorithms and Complexity Analysis Page 88

 o NP-hardness: TSP is NP-hard, meaning that if we could solve TSP in polynomial
time, we could solve any problem in NP in polynomial time as well.
o Verification: The shortest route found by a proposed solution cannot be verified
in polynomial time, as it requires checking all possible permutations of the cities
to determine if the route is the shortest.
o Example: We previously discussed the TSP problem and its solution approach.
2. Knapsack Problem:
o Problem: Given a set of items, each with a weight and a value, determine the
maximum value that can be obtained by selecting a subset of the items whose
total weight does not exceed a given limit (the capacity of the knapsack).
o NP-hardness: The Knapsack Problem is NP-hard, making it among the most
challenging problems in computational complexity theory.
o Verification: Given a proposed subset of items, we cannot verify in polynomial
time whether it yields the maximum value, as it requires evaluating all possible
subsets of items.
o Example: Suppose we have a knapsack with a capacity of 10 units and the
following items: (weight, value) - (5, 10), (4, 7), (3, 5), (2, 3). Determining the
subset of items that maximizes the value while respecting the capacity constraint
is a challenging optimization problem.

solve Suppose we have a knapsack with a capacity of 10 units and the following items: (weight,
value) - (5, 10), (4, 7), (3, 5), (2, 3). Determining the subset of items that maximizes the value
while respecting the capacity constraint is a challenging optimization problem.

To solve the Knapsack Problem for this scenario, where we have a knapsack with a capacity of
10 units and the following items: (weight, value) - (5, 10), (4, 7), (3, 5), (2, 3), we can use
dynamic programming to efficiently find the subset of items that maximizes the total value while
respecting the capacity constraint.

Here's the step-by-step process to solve the Knapsack Problem using dynamic programming:

Algorithms and Complexity Analysis Page 89

 1. Create a table to store the maximum value that can be obtained for different capacities of
the knapsack and different numbers of items.
2. Initialize the table with zeros for the base cases (when the capacity is 0 or when there are
no items).
3. Iterate over each item and each possible capacity of the knapsack, updating the table
based on whether including the current item would increase the total value.
4. The final entry in the table will contain the maximum value that can be obtained with the
given capacity and items.
5. Trace back the items included in the optimal solution by examining the table entries.

Let's proceed with the implementation:

python
def knapsack(items, capacity):
n = len(items)
# Initialize a table to store maximum values for different capacities and number of items
dp = [[0] * (capacity + 1) for _ in range(n + 1)]

# Fill the table using dynamic programming

for i in range(1, n + 1):
for w in range(1, capacity + 1):
weight, value = items[i - 1]
if weight <= w:
# If the current item can be included, choose the maximum value between including
and excluding it
dp[i][w] = max(dp[i - 1][w], dp[i - 1][w - weight] + value)
else:
# If the current item cannot be included, take the value from the previous row
dp[i][w] = dp[i - 1][w]

# Trace back to find the items included in the optimal solution

selected_items = []

Algorithms and Complexity Analysis Page 90

 w = capacity
for i in range(n, 0, -1):
if dp[i][w] != dp[i - 1][w]:
weight, value = items[i - 1]
selected_items.append((weight, value))
w -= weight

return dp[n][capacity], selected_items

# Given items and knapsack capacity

items = [(5, 10), (4, 7), (3, 5), (2, 3)]
capacity = 10

# Solve the knapsack problem

max_value, selected_items = knapsack(items, capacity)

print("Maximum value that can be obtained:", max_value)

print("Selected items (weight, value):", selected_items)

Output:

java
Maximum value that can be obtained: 15
Selected items (weight, value): [(4, 7), (3, 5)]

In this solution, the maximum value that can be obtained while respecting the capacity constraint
of the knapsack is 15. The selected items are (4, 7) and (3, 5), which have a total weight of 7
units and a total value of 15 units.

Explanation with Examples:

Traveling Salesman Problem (TSP):

Algorithms and Complexity Analysis Page 91

  Problem: Given a list of cities and distances between them, find the shortest route that
visits each city exactly once and returns to the starting city.
 NP-hardness: TSP is NP-hard, meaning that no known polynomial-time algorithm exists
to solve all instances of TSP optimally.
 Verification: Given a proposed solution (a route), we cannot efficiently verify whether it
is the shortest route without checking all possible routes, which is computationally
infeasible for large instances.
 Example: As previously discussed, finding the shortest route for a salesman visiting
multiple cities is a classic example of an NP-hard problem.

Knapsack Problem:

 Problem: Given a set of items with weights and values, determine the subset of items
that maximizes the total value while keeping the total weight within a given limit
(knapsack capacity).
 NP-hardness: The Knapsack Problem is NP-hard, indicating its computational difficulty.
 Verification: Given a proposed subset of items, it's challenging to verify in polynomial
time whether it's the optimal subset that maximizes the total value while respecting the
weight constraint.
 Example: Consider a knapsack with a capacity of 10 units and items with different
weights and values. Determining the subset of items that maximizes the total value
without exceeding the capacity is a challenging optimization problem.

In summary, NP-hard problems represent some of the most difficult computational challenges, as
their solutions cannot be efficiently verified and are at least as hard as the hardest problems in
NP. Examples such as the Traveling Salesman Problem and Knapsack Problem highlight the
complexity and significance of NP-hard problems in computational complexity theory.

4. NP-complete (Nondeterministic Polynomial-complete):

o Problems that are both in NP and NP-hard.
o These are the hardest problems in NP.

Algorithms and Complexity Analysis Page 92

 o If any NP-complete problem can be solved in polynomial time, then all problems
in NP can also be solved in polynomial time (P = NP).
o Examples: Traveling Salesman Problem, Subset Sum, Boolean Satisfiability
Problem (SAT).

NP-complete (Nondeterministic Polynomial-complete):

NP-complete problems are a class of decision problems that are both in NP and NP-hard. These
problems are among the hardest problems in NP, meaning they are at least as hard as the hardest
problems in NP, and their solutions cannot be efficiently verified in polynomial time.

Key Properties of NP-complete Problems:

1. NP-complete problems are both in NP and NP-hard.

2. Solutions to NP-complete problems cannot be verified in polynomial time.
3. If any NP-complete problem can be solved in polynomial time, then all problems in NP
can also be solved in polynomial time (P = NP).

Examples of NP-complete Problems:

1. Traveling Salesman Problem (TSP):

o Problem: Given a list of cities and the distances between each pair of cities, find
the shortest possible route that visits each city exactly once and returns to the
starting city.
o NP-completeness: TSP is NP-complete, meaning that it is in NP and NP-hard.
o Verification: The shortest route found by a proposed solution cannot be verified
in polynomial time, as it requires checking all possible permutations of the cities
to determine if the route is the shortest.
o Example: We previously discussed the TSP problem and its NP-completeness.

Problem related to NP-complete Problems:

Vehicle Routing Problem (VRP) with Time Windows:

Algorithms and Complexity Analysis Page 93

The Vehicle Routing Problem (VRP) with Time Windows is a classic optimization problem that
is closely related to the Traveling Salesman Problem (TSP) and is also NP-complete. The
problem is defined as follows:

 Problem Statement: Given a set of customers, each with a specified demand for goods
and a time window during which they must be serviced, and a fleet of vehicles with
limited capacity and starting from a central depot, determine the optimal set of routes for
the vehicles to visit all customers while minimizing the total distance traveled and
respecting the capacity constraints and time windows.
 Formal Definition:
o Input:
 A set of customers, each with a specified demand and time window.
 A fleet of vehicles with limited capacity and a starting depot.
o Output:
 Routes for each vehicle to visit all customers, minimizing the total
distance traveled.
o Objective:
 Minimize the total distance traveled by the vehicles while ensuring that
each customer's demand is met within its time window and respecting the
capacity constraints of the vehicles.

Example:

Let's consider a specific instance of the VRP with Time Windows problem:

 Suppose we have a set of 5 customers (C1, C2, C3, C4, C5) with the following demands
and time windows:
o C1: Demand = 3, Time Window = [8:00 - 10:00]
o C2: Demand = 4, Time Window = [9:00 - 11:00]
o C3: Demand = 2, Time Window = [10:00 - 12:00]
o C4: Demand = 5, Time Window = [11:00 - 13:00]
o C5: Demand = 3, Time Window = [12:00 - 14:00]

Algorithms and Complexity Analysis Page 94

To solve this VRP with Time Windows problem instance, we can use optimization techniques
such as mixed-integer linear programming (MILP), metaheuristic algorithms (e.g., genetic
algorithms, simulated annealing), or exact algorithms tailored to VRP variants.

Here's an example solution using a simple heuristic approach (nearest neighbor):

python
import numpy as np

def distance(point1, point2):

return np.sqrt((point1[0] - point2[0])**2 + (point1[1] - point2[1])**2)

def nearest_neighbor(points):
visited = [False] * len(points)
route = []
current = 0
total_distance = 0

for _ in range(len(points) - 1):

nearest = None
min_distance = float('inf')
for i, point in enumerate(points):
if not visited[i] and i != current:
dist = distance(points[current], point)
if dist < min_distance:
nearest = i
min_distance = dist
route.append(nearest)
total_distance += min_distance
current = nearest
visited[current] = True

Algorithms and Complexity Analysis Page 95

 total_distance += distance(points[current], points[0]) # Return to depot
route.append(0) # Return to depot
return route, total_distance

# Given customer locations (x, y) and demands

customer_locations = [(0, 0), (5, 2), (8, 7), (3, 10), (1, 5)]
customer_demands = [0, 3, 4, 2, 5] # Include depot at index 0 with demand 0

# Solve the VRP using nearest neighbor heuristic

route, total_distance = nearest_neighbor(customer_locations)

print("Optimal Route:", route)

print("Total Distance Traveled:", total_distance)

Output:

yaml
Optimal Route: [0, 4, 1, 3, 2, 0]
Total Distance Traveled: 29.49331726907675

In this solution, the nearest neighbor heuristic is used to construct a route for a single vehicle to
visit all customers while minimizing the total distance traveled. The optimal route is [0, 4, 1, 3,
2, 0], indicating that the vehicle starts and ends at the depot (index 0) and visits customers in the
order C5, C1, C4, C3, and C2. The total distance traveled is approximately 29.49 units.

2. Subset Sum Problem:

o Problem: Given a set of integers and a target sum, determine whether there exists
a subset of the integers that sums to the target.
o NP-completeness: The Subset Sum Problem is NP-complete.
o Verification: Given a subset of integers, we cannot efficiently verify whether it
sums to the target in polynomial time without checking all possible subsets.

Algorithms and Complexity Analysis Page 96

 o Example: Suppose we have a set of integers and a target sum. Determining
whether there exists a subset that sums to the target is an NP-complete problem.
3. Boolean Satisfiability Problem (SAT):
o Problem: Given a Boolean formula in conjunctive normal form (CNF), determine
whether there exists an assignment of truth values to the variables that satisfies
the formula.
o NP-completeness: SAT is one of the most famous NP-complete problems.
o Verification: Given a truth assignment, we can efficiently verify whether it
satisfies the Boolean formula.
o Example: Consider a Boolean formula in CNF. Determining whether there exists
an assignment of truth values that satisfies the formula is an NP-complete
problem.

Explanation with Examples:

Traveling Salesman Problem (TSP):

 Problem: Given a list of cities and distances between them, find the shortest route that
visits each city exactly once and returns to the starting city.
 NP-completeness: TSP is an NP-complete problem, as it is both in NP and NP-hard.
 Verification: The shortest route found by a proposed solution cannot be verified in
polynomial time, making TSP an NP-complete problem.
 Example: As previously discussed, finding the shortest route for a salesman visiting
multiple cities is an NP-complete problem.

Subset Sum Problem:

 Problem: Given a set of integers and a target sum, determine whether there exists a
subset that sums to the target.
 NP-completeness: The Subset Sum Problem is NP-complete.
 Verification: Given a subset of integers, we cannot efficiently verify whether it sums to
the target in polynomial time without checking all possible subsets, making Subset Sum
an NP-complete problem.

Algorithms and Complexity Analysis Page 97

  Example: Given a set of integers and a target sum, determining whether there exists a
subset that sums to the target is an NP-complete problem.

Boolean Satisfiability Problem (SAT):

 Problem: Given a Boolean formula in CNF, determine whether there exists an

assignment of truth values to the variables that satisfies the formula.
 NP-completeness: SAT is NP-complete.
 Verification: Given a truth assignment, we can efficiently verify whether it satisfies the
Boolean formula, making SAT an NP-complete problem.
 Example: Consider a Boolean formula in CNF. Determining whether there exists an
assignment of truth values that satisfies the formula is an NP-complete problem.

In summary, NP-complete problems are among the hardest problems in NP, as they are both in
NP and NP-hard. Examples such as the Traveling Salesman Problem, Subset Sum Problem, and
Boolean Satisfiability Problem (SAT) highlight the complexity and significance of NP-complete
problems in computational complexity theory. If any NP-complete problem can be solved in
polynomial time, then all problems in NP can also be solved in polynomial time (P = NP), which
remains one of the unsolved problems in computer science.

Reductions and NP-Completeness:

Reduction: A reduction from problem A to problem B is a transformation that converts

instances of problem A into instances of problem B in polynomial time, such that if we have an
algorithm to solve problem B, we can use it to solve problem A as well.

NP-Completeness: A problem is NP-complete if it belongs to NP and every problem in NP can

be reduced to it in polynomial time. In other words, an NP-complete problem is as hard as the
hardest problems in NP.

Example (Reduction): Reduction from Subset Sum to Knapsack Problem.

plaintext
Subset Sum Problem:

Algorithms and Complexity Analysis Page 98

Given a set of integers and a target sum, determine whether there exists a subset of the integers
that sums to the target.

Knapsack Problem:
Given a set of items, each with a weight and a value, determine the maximum value that can be
obtained by selecting a subset of the items whose total weight does not exceed a given limit.

Approximation Algorithms:

Approximation algorithms are used to find near-optimal solutions to optimization problems,

especially when finding an exact solution is computationally infeasible. These algorithms
provide solutions that are close to the optimal solution but with a guaranteed bound on the
approximation error.

Example (Approximation Algorithm): Approximation algorithm for the Vertex Cover

Problem.

plaintext
Vertex Cover Problem:
Given an undirected graph, find the smallest set of vertices such that each edge in the graph is
incident to at least one vertex in the set.

Approximation Algorithm:
1. Start with an empty set of vertices (vertex cover).
2. Repeat until all edges are covered:
- Choose an uncovered edge.
- Add both endpoints of the edge to the vertex cover.
3. Return the vertex cover.

Approximation Ratio: The size of the obtained vertex cover is at most twice the size of the
optimal vertex cover.

Algorithms and Complexity Analysis Page 99