Analysis of Algorithms

Lecture-01
Design and Analysis of Algorithm: Objectives
This subject focus on following points
 To Construct the algorithms for the problems to be solved
or already solved by other methods, the problems may be
from diverse fields and not restricted to Computer Science
only.
 To prove that a proposed algorithm solve the problem
correctly.
 To analyze the time and space requirements of an
algorithm in standard, asymptotical notations, that is
independent of particular machine.
 To prove that the proposed algorithm solve the problem
faster than other solutions.
Introduction
 What is an Algorithm
 Definition, characteristics, Writing an algorithm
 Algorithm and Basic Concept
 Algorithm Complexity
 Analysis of the algorithms
 Computing run time of algorithms
 Bounds
 Important Problem Type
 Searching and Sorting- linear and binary search
 String processing
 Fundamental Data Structure
 Liner data Structure :- stack and queue
 Non Liner Data structure :- tree and graph
Algorithm: Definition
An algorithm is a steps of operations to solve a problem
performing calculation, data processing, and automated
reasoning tasks. And it is an efficient method that can be
expressed within finite amount of time and space.
• It can have zero or more inputs.
• It must have at least one output.
• It should be efficient both in terms of memory and time.
• It should be finite.
• Every statement should be unambiguous.
Algorithm is independent from any programming
language.
Note: A program can run infinitely, for example a server
program it runs 24X7X365. But an algorithm must be
finite.
Characteristics of an Algorithm
Not all procedures can be called an algorithm.
An algorithm should have the following characteristics
 Unambiguous − it should be clear and unambiguous.

Each of its steps (or phases), and their inputs/outputs should be

clear and must lead to only one meaning.
 Input − An algorithm should have 0 or more well-defined
inputs.
 Output − An algorithm should have 1 or more well-defined
outputs, and should match the desired output.
 Finiteness − it must terminate after a finite number of steps.
 Feasibility − Should be feasible with the available resources.
 Independent − it should be independent of any programming
code.
Note: It must have a termination criteria
Writing an Algorithm
An algorithm can be written in following ways:
 Natural Language Representation: Use of Natural language
in writing and algorithm can be ambiguous and therefore
algorithm may lack the characteristic of being definite.
 Flowcharts: Graphical representation of algorithmic steps,
flowcharts are not suitable to write the solutions for complex
problems.
 Pseudocode: an informal language, combination of human
language and matimatical expression.
 it has an advantage of being easily converted into any
programming language. A Table of pseudocode conventions are
given in textbook. Mostly used.
Flowchart
 Flowcharts pictorially depict a process.
 They are easy to understand and are commonly used in case
of simple problems.
Pseudo Code
 The pseudo code has an advantage of being easily converted
into any programming language.
 This way of writing algorithm is most acceptable and most
widely used.
 In order to be able to write a pseudo code, one must be familiar
with the conventions of writing it.

[Link] an algorhim that find sum of N positive integer numbers.

2. An algorhim find the biggest among N numbers.
Algorithm design strategies
 Brute force
 it is an approach that finds all the possible solutions to find
a satisfactory solution to a given problem
 Divide and conquer
 Breaking down a problem into two or more sub-problems of
the same (or related) type, until these become simple enough
to be solved directly.
 The solutions to the sub-problems are then combined to give a
solution to the original problem.
 Decrease and conquer (instance ->An occurrence of something)
 Change an instance into one smaller instance of the problem.
 Solve the smaller instance.
 Convert the solution of the smaller instance into a solution
for the larger instance.
Algorithm design strategies
 Transform and conquer
 Transform problem to easiar/familar form or use another
represenation or another porblem instance then solve it.
 Greedy approach
 Used to solve optimization problem.(optimization:-problem
which required max or min result)
 we follow predifined procedure to get the potimal result
 The problem could be solved in iterations.
 Dynamic programming
 Used to solve optimization problem.
 find all posible feasible solution and pickup the best solution.
 Time consuming
 most of the time solved by recursion formual
Algorithm: Design Considerations
The five most essential things are to be considered while
writing an algorithm are as follows:
 Time taken
 Memory usage
 Input
 Process
 Output
Algorithm Complexity
Suppose X is an algorithm and n is the size of input data, the
time and space used by the algorithm X are the two main
factors, which decide the efficiency of X.

 Time Factor − Time is measured by counting the number of

key operations such as comparisons in the sorting algorithm.

 Space Factor − Space is measured by counting the maximum

memory space required by the algorithm.

The complexity of an algorithm f(n) gives the running time

and/or the storage space required by the algorithm in terms of
n as the size of input data.
Types of the Algorithm Analysis
Efficiency of an algorithm can be analyzed at two different
stages, before implementation and after implementation. They
are the following
 A Priori Analysis − This is a theoretical analysis of an
algorithm.
 Efficiency of an algorithm is measured by assuming that
all other factors, for example, processor speed, are
constant and have no effect on the implementation.
 A Posterior Analysis − This is an empirical analysis of an
algorithm.
 The selected algorithm is implemented using
programming language. This is then executed on target
computer machine.
 In this analysis, actual statistics like running time and
space required, are collected.
Why to perform analysis of the Algorithms
The analysis of an algorithm is done because:
 The analysis of algorithm can be more reliable than
experiments.
 Experiments can guarantee the behavior of algorithms
on certain test cases, while theoretical analysis and
establishment of run time bounds can guarantee the
behavior of algorithm on whole domain.
 Analysis helps us to chose from many possible
solutions.
 Performance of the program can be predicted before it
is implemented.
 By analysis we can idea of slow and faster parts of the
algorithm, and accordingly we can plan our
implementation strategies.
Analysis of algorithms
 Issues:
 Correctness
 Space efficiency
 Time efficiency
 Optimality
 Approaches:
 Theoretical analysis
 Empirical analysis

*Apriori analysis vs. posterior analysis

Worst-Case/ Best-Case/ Average-Case Analysis

 Worst-Case Analysis –The maximum amount of time that an

algorithm require to solve a problem of size n.
 Best-Case Analysis –The minimum amount of time that an
algorithm require to solve a problem of size n. The best
case behavior of an algorithm is NOT so useful.
 Average-Case Analysis –The average amount of time that an
algorithm require to solve a problem of size n.
 Worst-case analysis is more common than average-case
analysis.
Asymptotic Analysis
 Asymptotic analysis of an algorithm refers to defining the
mathematical boundation/framing of its run-time
performance.
 Using asymptotic analysis, we can very well conclude the
best case, average case, and worst case scenario of an
algorithm.
 Asymptotic analysis is input bound i.e., if there's no input
to the algorithm, it is concluded to work in a constant
time.
 Other than the "input" all other factors are considered
constant.
 Asymptotic analysis refers to computing the running time
of any operation in mathematical units of computation.
Asymptotic Analysis
Asymptotic Notations
 Following are the commonly used asymptotic
notations to calculate the running time complexity
of an algorithm.

Ο Notation
 Ω Notation
 θ Notation
Big-O Notation
Definition: f(n) is in O(g(n)) if order of growth of f(n) ≤ order
of growth of g(n) (within constant multiple),
Examples:
 10n is O(n2)
 5n+20 is O(n)

 f (n) : there exist positive constants c and n0 s.t.

O( g (n))  
 0  f (n) cg (n) for all n n0 
Omega Notation, Ω

The notation Ο(n) is the formal way to express the

upper bound of an algorithm's running time.

 f (n) : there exist positive constants c and n0 s.t.

( g (n))  
 0 cg (n)  f (n) for all n n0 
Theta Notation, θ

The notation θ(n) is the formal way to express both the

lower bound and the upper bound of an algorithm's
running time. It is represented as follows
 f (n) : there exist positive constants c1 , c2 , and n0 s.t.
( g (n))  
 0 c1 g ( n )  f ( n ) c 2 g ( n ) for all n n0 
Common Asymptotic Notations
A Comparison of Growth-Rate
Functions
How to Compute the run time of an Algorithm
How to Compute the run time of an Algorithm
Cost of basic operations
Cost of basic operations
Cost of Instructions
Growth-Rate Functions – Example1

Cost
Times
i = 1; c1 1
sum = 0; c2 1
while (i <= n) { c3 n+1
i = i + 1; c4 n

sum = sum + i; c5 n
}

T(n) = c1 + c2 + (n+1)*c3 + n*c4 + n*c5

= (c3+c4+c5)*n + (c1+c2+c3)
= a*n + b
 So, the growth-rate function for this algorithm is O(n)
 Growth-Rate Functions – Example2
Cost Times
i=1; c1 1
sum = 0; c2 1
while (i <= n) { c3 n+1
j=1; c4 n
while (j <= n) { c5 n*(n+1)
sum = sum + i; c6 n*n
j = j + 1; c7 n*n
}
i = i +1; c8 n
}
T(n) = c1 + c2 + (n+1)*c3 + n*c4 +
n*(n+1)*c5+n*n*c6+n*n*c7+n*c8
= (c5+c6+c7)*n2 + (c3+c4+c5+c8)*n + (c1+c2+c3)
= a*n2 + b*n + c
 So, the growth-rate function for this algorithm is O(n2)
Growth-Rate Functions – Example3
Cost Times
for (i=1; i<=n; i++) c1 n+1
n
for (j=1; j<=i; j++) c2  ( j 1)
j 1

n j
for (no=1; no<=j; no++) c3  (k 1)
j 1 k 1

n j

x=x+1; c4  k
j 1 k 1

n j n j
n
T(n) = c1*(n+1) + c2*(  ( j 1)
j 1
) + c3* (   (k 1) ) + c4*(  
j 1 k 1
k
)
j 1 k 1

= a*n3 + b*n2 + c*n + d

 So, the growth-rate function for this algorithm is O(n3)
Some Well-known Computational Problems

 Searching and Sorting

 Combinatorial problems
 Geometrical Problems
 Traveling salesman problem
 Knapsack problem
 Chess
 Towers of Hanoi
 Graph Problems
Algorithms Examples
Search Problems

 Statement of problem:
 Input: A sequence of n numbers <a , a , …, a >
1 2 n
 Key= item to be Searched
 Output: Index of the item to be searched
 Instance: The sequence <5, 3, 2, 8, 3>

 Algorithms:
 Sequential Search
 Binary Search(for sorted arrays)
Linear Search

Worst and average case time complexity is order of N. So we need a

better algorithm.
Key=20

average case time complexity for serial search is O(n).

The worst case time for serial search requires n array accesses: O(n)
Binary Search
Binary Search
Binary Search Implementation
 Analysis of Binary Search

 Binary search algorithm worst case O(log n) and

 best case O(1) when the central index would directly match the desired value.
Sorting Problems
 Statement of problem:
 Input: A sequence of n numbers <a , a , …, a >
1 2 n

 Output: A reordering of the input sequence <a´1, a´2, …, a´n>

so that a´i ≤ a´j whenever i < j

 Instance: The sequence <5, 3, 2, 8, 3>

 Algorithms:
 Selection sort
 Insertion sort
 Merge sort
 Quick Sort
 (many others)
 Selection sort
 Given an array of length n,
 Search elements 0 through n-1 and select the smallest
─ Swap it with the element in location 0
 Search elements 1 through n-1 and select the smallest
─ Swap it with the element in location 1
 Search elements 2 through n-1 and select the smallest
─ Swap it with the element in location 2
 Search elements 3 through n-1 and select the smallest
─ Swap it with the element in location 3
 Continue in this fashion until there’s nothing left to search
 Insertion sort
String Processing
 A string is a sequence of characters from an alphabet.
 Text strings: letters, numbers, and special characters.
 String matching: searching for a given word/pattern in a text.

Examples:
(i) searching for a word or phrase on WWW or in a
Word document
(ii) searching for a short read in the reference
genomic sequence
String Matching
 Given a text string T of length n and a pattern string P of
length m, the exact string matching problem is to find all
occurrences of P in T.
 Example: T=“AGCTTGA” P=“GCT”
 Applications:
 Searching keywords in a file
 Searching engines (like Google and Openfind)
 Database searching (GenBank)
 More string matching algorithms (with source codes):
[Link]
String Matching
 Brute Force algorithm
 The brute force algorithm consists in checking, at all positions in
the text between 0 and n-m, whether an occurrence of the pattern
starts there or not. Then, after each attempt, it shifts the pattern
by exactly one position to the right.

Time: O(mn) where m=|P| and n=|

T|.
Graph Problems
 Informal definition
 A graph is a collection of points called vertices, some of
which are connected by line segments called edges.
 Modeling Real-life Problems
 Modeling WWW
 Communication networks
 Project scheduling …
 Examples of Graph Algorithms
 Graph traversal algorithms
 Shortest-path algorithms
 ……..
Data Structures Review
….

 Data Structures:
 An implementation of an ADT is a translation into
statements of a programming language,
─ the declarations that define a variable to be of that
ADT type
─ the operations defined on the ADT (using procedures
of the programming language)
 Each data structure is built up from the basic data types of the
underlying programming language using the available data
structuring facilities, such as
 arrays, records (structures in C), pointers, files, sets, etc.
Fundamental data structures

 list
 array
  graph
linked list
  tree and binary tree
string
 stack
 queue
 priority queue
 Linear Data Structures
 Arrays  Arrays
 A sequence of n items of the same data  fixed length (need
type that are stored contiguously in preliminary reservation
computer memory and made accessible of memory)
by specifying a value of the array’s
 contiguous memory
index. locations

 direct access
Linked List  Insert/delete
 A sequence of zero or more nodes each
 Linked Lists
containing two kinds of information:
some data and one or more links called
 dynamic length
pointers to other nodes of the linked list.
 arbitrary memory
locations
 Singly linked list (next pointer)
 access by following links
 Doubly linked list (next + previous  Insert/delete
pointers)
a1 a2 an
Stacks and Queues
 Stacks
 A stack of plates
─ insertion/deletion can be done only at the top.
─ LIFO
 Two operations (push and pop)
 Queues
 A queue of customers waiting for services
─ Insertion/enqueue from the rear & deletion/dequeue
from the front.
─ FIFO
 Two operations (enqueue and dequeue)
Priority Queue and Heap
 Priority queues (implemented using heaps)
 A data structure for maintaining a set of elements,
each associated with a key/priority, with the
following operations:

Finding the element with the highest priority

Deleting the element with the highest priority

Inserting a new element
 Scheduling jobs on a shared computer
Graphs
 Formal definition

A graph G = <V, E> is defined by a pair of two sets: a
finite set V of items called vertices and a set E of vertex
pairs called edges.
 Undirected and directed graphs (digraphs).
Complete, dense, and sparse graphs
 A graph with every pair of its vertices connected by an
edge is called complete, K|V|
 In Dense graph, the number of edges is close to the
maximal number of edges.
 A sparse graph in which the number of edges is much
less than the possible number of edges.  1  2

 3  4

Complete Dense
Graph Representation
 Adjacency matrix O(n2) ----> better for dense graph
 n x n boolean matrix if |V| is n.
 The element on the ith row and jth column is 1 if there’s an edge from
ith vertex to the jth vertex; otherwise 0.
 The adjacency matrix of an undirected graph is symmetric.
 Adjacency linked lists O(n+2e)--->better for sparce graph
 A collection of linked lists, one for each vertex, that contain all the
vertices adjacent to the list’s vertex.
 Which data structure would you use if the graph is a 100-node star
shape?
2 3 4
0111 4
0001
0001 4
0000
Weighted Graphs
 Graphs or digraphs with numbers assigned to the edges.
it may be called a weighted graph.

 1
5  2

6 7
9
3  4
8
Graph Properties -- Paths and Connectivity

 Paths
 A path from vertex u to v of a graph G is defined as a sequence of
adjacent (connected by an edge) vertices that starts with u and ends
with v.
 Simple paths: a path in a graph which does not have repeating
vertices..
 Path lengths: the number of edges, or the number of vertices – 1.
 Connected graphs
 A graph is said to be connected if for every pair of its vertices u and v
there is a path from u to v.
 Connected component
 The maximum connected subgraph of a given graph.
Graph Properties – A cyclicity
 Cycle
 A simple path of a positive length that starts and ends a
the same vertex.

 Acyclic graph
 A graph without cycles
 DAG (Directed Acyclic Graph)  1  2

 3  4
Trees
 Trees
 A Tree (or free tree) is a connected acyclic graph.
 Forest: a graph that has no cycles but is not necessarily connected.
 Properties of trees
 For every two vertices in a tree there always exists exactly one
simple path from one of these vertices to the other.
─ Rooted trees: The above property makes it possible to select an
arbitrary vertex in a free tree and consider it as the root of the so
called rooted tree.
─ Levels in a rooted tree.
 rooted

 3

 1  3  5
 4  1  5

 2  4
 2
Forest
Rooted Trees (I)
 Ancestors/predecessors
 For any vertex v in a tree T, all the vertices on the simple path from the root
to that vertex are called ancestors.
 Descendants/children
 All the vertices for which a vertex v is an ancestor are said to be descendants
of v.
 Parent, child and siblings
 If (u, v) is the last edge of the simple path from the root to vertex v, u is said to
be the parent of v and v is called a child of u.
 Vertices that have the same parent are called siblings. rooted

 Leaves  3
 A vertex without children is called a leaf.
 Subtree  4  1  5
 A vertex v with all its descendants is called the subtree of T rooted at v.
 2
Rooted Trees (II)
 Depth of a vertex
 The length of the simple path from the root to the vertex.
 Height of a tree
 The length of the longest simple path from the root to a
leaf.
h=2

 3

 4  1  5

 2
Ordered Trees
 Ordered trees
 it is a rooted tree in which all the children of each vertex are ordered.
 Binary trees
 it is a tree data structure in which each node has at most two
children, which are referred to as the left child and the right child
 Binary search trees 9

6 8
 Each vertex is assigned a number.
5 2 3

 A number assigned to each parental vertex is larger than all the

numbers in its left subtree and smaller than all the numbers in its
right subtree.
 6

 3  9

 2  5  8
Topics To be covered in next class

 Divide and Conquer

 General method,
 Applications-Binary search, Quick sort, Merge sort
 Selection Sort
 Finding maximum and minimum.
 Solving recurrence relations using Masters
Theorem,
 Substitution method