UNIT IV - ALGORITHM DESIGN TECHNIQUES

Dynamic Programming: Matrix-Chain Multiplication – Elements of Dynamic Programming –

Longest Common Subsequence- Greedy Algorithms: – Elements of the Greedy Strategy- An
Activity-Selection Problem - Huffman Coding.

Dynamic programming
Dynamic programming approach is similar to divide and conquer in breaking down the
problem into smaller and yet smaller possible sub-problems. But unlike divide and conquer,
these sub-problems are not solved independently. Rather, results of these smaller sub-problems
are remembered and used for similar or overlapping sub-problems.
Mostly, dynamic programming algorithms are used for solving optimization problems. Before
solving the in-hand sub-problem, dynamic algorithm will try to examine the results of the
previously solved sub-problems. The solutions of sub-problems are combined in order to
achieve the best optimal final solution. This paradigm is thus said to be using Bottom-up
approach.
So we can conclude that −
 The problem should be able to be divided into smaller overlapping sub-problem.
 Final optimum solution can be achieved by using an optimum solution of smaller sub-
problems.
 Dynamic algorithms use memorization.
Steps of Dynamic Programming Approach
Dynamic Programming algorithm is designed using the following four steps −
 Characterize the structure of an optimal solution.
 Recursively define the value of an optimal solution.
 Compute the value of an optimal solution, typically in a bottom-up fashion.
 Construct an optimal solution from the computed information.

Examples
 Fibonacci number series
 Knapsack problem
 Tower of Hanoi
 All pair shortest path by Floyd-Warshall and Bellman Ford
 Shortest path by Dijkstra
 Project scheduling
 Matrix Chain Multiplication

Matrix Chain Multiplication

Matrix Chain Multiplication is an algorithm that is applied to determine the lowest cost way
for multiplying matrices. The actual multiplication is done using the standard way of
multiplying the matrices, i.e., it follows the basic rule that the number of rows in one matrix
must be equal to the number of columns in another matrix. Hence, multiple scalar
multiplications must be done to achieve the product.
To brief it further, consider matrices A, B, C, and D, to be multiplied; hence, the multiplication
is done using the standard matrix multiplication. There are multiple combinations of the
matrices found while using the standard approach since matrix multiplication is associative.
For instance, there are five ways to multiply the four matrices given above −
 (A(B(CD)))
 (A((BC)D))
 ((AB)(CD))
 ((A(BC))D)
 (((AB)C)D)
Input: arr[] = {40, 20, 30, 10, 30}
Output: 26000
Explanation:There are 4 matrices of dimensions 40×20, 20×30, 30×10, 10×30.
Let the input 4 matrices be A, B, C and D.
The minimum number of multiplications are obtained by
putting parenthesis in following way (A(BC))D.
The minimum is 20*30*10 + 40*20*10 + 40*10*30
Input: arr[] = {1, 2, 3, 4, 3}
Output: 30
Explanation: There are 4 matrices of dimensions 1×2, 2×3, 3×4, 4×3.
Let the input 4 matrices be A, B, C and D.
The minimum number of multiplications are obtained by
putting parenthesis in following way ((AB)C)D.
The minimum number is 1*2*3 + 1*3*4 + 1*4*3 = 30

Elements of Dynamic Programming

 Optimal Substructure
 Overlapping Sub-problems
 Variant: Memoization
Optimal Substructure: OS holds if optimal solution contains within it optimal solutions to sub
problems. In matrix-chain multiplication optimally doing A1, A2, A3, ...,An required
A 1...k and A k+1 ...n to be optimal. It is often easy to show the optimal sub problem
property as follows:
Split problem into sub-problems
Sub-problems must be optimal, otherwise the optimal splitting would not have been optimal.
There is usually a suitable "space" of sub-problems. Some spaces are more "natural" than
others.
For matrix-chain multiply we chose sub-problems as sub chains. We could have chosen all
arbitrary products, but that would have been much larger than necessary! DP based on that
would have to solve too many sub-problems.
A general way to investigate optimal substructure of a problem in DP is to look at optimal
sub-, sub-sub, etc. problems for structure. When we noticed that sub problems of A1, A2, A3,
...,An consisted of sub-chains, it made sense to use sub-chains of the form Ai, ..., Aj as the
"natural" space for sub-problems.
Overlapping Sub-problems: Space of sub-problems must be small: recursive solution re-
solves the same sub-problem many times. Usually there are polynomially many sub-
problems, and we revisit the same ones over and over again: overlapping sub-problems.
Dynamic Programming Solution for Matrix Chain Multiplication using Memoization:
Below is the recursion tree for the 2nd example of the above recursive approach:

If observed carefully you can find the following two properties:

1) Optimal Substructure: In the above case, we are breaking the bigger groups into
smaller subgroups and solving them to finally find the minimum number of multiplications.
Therefore, it can be said that the problem has optimal substructure property.
2) Overlapping Subproblems: We can see in the recursion tree that the same subproblems
are called again and again and this problem has the Overlapping Subproblems property.
So Matrix Chain Multiplication problem has both properties of a dynamic
programming problem. So recomputations of same subproblems can be avoided by
constructing a temporary array dp[][] in a bottom up manner.
Follow the below steps to solve the problem:
 Build a matrix dp[][] of size N*N for memoization purposes.
 Use the same recursive call as done in the above approach:
 When we find a range (i, j) for which the value is already calculated, return
the minimum value for that range (i.e., dp[i][j]).
 Otherwise, perform the recursive calls as mentioned earlier.
 The value stored at dp[0][N-1] is the required answer.

Longest Common Subsequence (LCS)

A longest common subsequence (LCS) is defined as the longest subsequence which is
common in all given input sequences.
Examples:
Input: S1 = “AGGTAB”, S2 = “GXTXAYB”
Output: 4
Explanation: The longest subsequence which is present in both strings is “GTAB”.
Input: S1 = “BD”, S2 = “ABCD”
Output: 2
Explanation: The longest subsequence which is present in both strings is “BD”.
Recursive Approach for LCS:
Generate all the possible subsequences and find the longest among them that is present in
both strings using recursion.
Follow the below steps to implement the idea:
 Create a recursive function [say lcs()].
 Check the relation between the First characters of the strings that are not yet processed.

Greedy Algorithms

A greedy algorithm, as the name suggests, always makes the choice that seems to be the best
at that moment. This means that it makes a locally-optimal choice in the hope that this choice
will lead to a globally-optimal solution.

Assume that you have an objective function that needs to be optimized (either maximized or
minimized) at a given point. A Greedy algorithm makes greedy choices at each step to ensure
that the objective function is optimized. The Greedy algorithm has only one shot to compute
the optimal solution so that it never goes back and reverses the decision.

Greedy algorithms have some advantages and disadvantages:

1. It is quite easy to come up with a greedy algorithm (or even multiple greedy
algorithms) for a problem.
2. Analyzing the run time for greedy algorithms will generally be much easier than
for other techniques (like Divide and conquer). For the Divide and conquer technique,
it is not clear whether the technique is fast or slow. This is because at each level of
recursion the size of gets smaller and the number of sub-problems increases.
 3. The difficult part is that for greedy algorithms you have to work much harder to
understand correctness issues. Even with the correct algorithm, it is hard to prove
why it is correct. Proving that a greedy algorithm is correct is more of an art than a
science. It involves a lot of creativity.

This is a simple Greedy-algorithm problem. In each iteration, you have to greedily select the
things which will take the minimum amount of time to complete while maintaining two
variables currentTime and numberOfThings. To complete the calculation, you must:

1. Sort the array A in a non-decreasing order.

2. Select each to-do item one-by-one.
3. Add the time that it will take to complete that to-do item into currentTime.
4. Add one to numberOfThings.

Repeat this as long as the currentTime is less than or equal to T.

Let A = {5, 3, 4, 2, 1} and T = 6

After sorting, A = {1, 2, 3, 4, 5}

After the 1st iteration:

 currentTime = 1
 numberOfThings = 1

After the 2nd iteration:

 currentTime is 1 + 2 = 3
 numberOfThings = 2

After the 3rd iteration:

 currentTime is 3 + 3 = 6
 numberOfThings = 3

After the 4th iteration, currentTime is 6 + 4 = 10, which is greater than T. Therefore, the
answer is 3.

Applications of Greedy Approach:

Greedy algorithms are used to find an optimal or near optimal solution to many real-life
problems. Few of them are listed below:
(1) Make a change problem
(2) Knapsack problem
(3) Minimum spanning tree
(4) Single source shortest path
(5) Activity selection problem
(6) Job sequencing problem
(7) Huffman code generation.
(8) Dijkstra’s algorithm
(9) Greedy coloring
(10) Minimum cost spanning tree
(11) Job scheduling
(12) Interval scheduling
(13) Greedy set cover
(14) Knapsack with fractions
Advantages of the Greedy Approach:
 The greedy approach is easy to implement.
 Typically have less time complexity.
 Greedy algorithms can be used for optimization purposes or finding close to
optimization in case of Hard problems.
 Greedy algorithms can produce efficient solutions in many cases, especially when the
problem has a substructure that exhibits the greedy choice property.
 Greedy algorithms are often faster than other optimization algorithms, such as dynamic
programming or branch and bound, because they require less computation and memory.
 The greedy approach is often used as a heuristic or approximation algorithm when an
exact solution is not feasible or when finding an exact solution would be too time-
consuming.
 The greedy approach can be applied to a wide range of problems, including problems in
computer science, operations research, economics, and other fields.
 The greedy approach can be used to solve problems in real-time, such as scheduling
problems or resource allocation problems, because it does not require the solution to be
computed in advance.
 Greedy algorithms are often used as a first step in solving optimization problems,
because they provide a good starting point for more complex optimization algorithms.
 Greedy algorithms can be used in conjunction with other optimization algorithms, such
as local search or simulated annealing, to improve the quality of the solution.
Disadvantages of the Greedy Approach:
 The local optimal solution may not always be globally optimal.
 Greedy algorithms do not always guarantee to find the optimal solution, and may
produce suboptimal solutions in some cases.
 The greedy approach relies heavily on the problem structure and the choice of criteria
used to make the local optimal choice. If the criteria are not chosen carefully, the
solution produced may be far from optimal.
 Greedy algorithms may require a lot of preprocessing to transform the problem into a
form that can be solved by the greedy approach.
 Greedy algorithms may not be applicable to problems where the optimal solution
depends on the order in which the inputs are processed.
 Greedy algorithms may not be suitable for problems where the optimal solution depends
on the size or composition of the input, such as the bin packing problem.
 Greedy algorithms may not be able to handle constraints on the solution space, such as
constraints on the total weight or capacity of the solution.
 Greedy algorithms may be sensitive to small changes in the input, which can result in
large changes in the output. This can make the algorithm unstable and unpredictable in
some cases.
Standard Greedy Algorithms :
 Prim’s Algorithm
 Kruskal’s Algorithm
 Dijkstra’s Algorithm

Elements of the Greedy Strategy

Optimal Substructure:
An optimal solution to the problem contains within it optimal solutions to sub-problems. A'
= A - {1} (greedy choice) A' can be solved again with the greedy algorithm. S' = { i � S,
si � fi }
When do you use DP versus a greedy approach? Which should be faster?
The 0 - 1 knapsack problem:
A thief has a knapsack that holds at most W pounds. Item i : ( vi, wi ) ( v = value, w = weight
) thief must choose items to maximize the value stolen and still fit into the knapsack. Each
item must be taken or left ( 0 - 1 ).
Fractional knapsack problem:
takes parts, as well as wholes
Both the 0 - 1 and fractional problems have the optimal substructure property: Fractional: vi /
wi is the value per pound. Clearly you take as much of the item with the greatest value per
pound. This continues until you fill the knapsack. Optimal (Greedy) algorithm takes O ( n lg
n ), as we must sort on vi / wi = di.
Consider the same strategy for the 0 - 1 problem:
W = 50 lbs. (maximum knapsack capacity)
w1 = 10 v1 = 60 d1.= 6
w2 = 20 v2 = 100 d2.= 5
w3 = 30 v3 = 120 d3 = 4
were d is the value density
Greedy approach: Take all of 1, and all of 2: v1+ v2 = 160, optimal solution is to take all of 2
and 3: v2 + v3= 220, other solution is to take all of 1 and 3 v1+ v3 = 180. All below 50 lbs.
When solving the 0 - 1 knapsack problem, empty space lowers the effective d of the load.
Thus each time an item is chosen for inclusion we must consider both
i included
i excluded
These are clearly overlapping sub-problems for different i's and so best solved by DP!
Huffman Coding.
Huffman coding is a lossless data compression algorithm. The idea is to assign variable-
length codes to input characters, lengths of the assigned codes are based on the frequencies
of corresponding characters.
The variable-length codes assigned to input characters are Prefix Codes, means the codes
(bit sequences) are assigned in such a way that the code assigned to one character is not the
prefix of code assigned to any other character. This is how Huffman Coding makes sure
that there is no ambiguity when decoding the generated bitstream.
Let us understand prefix codes with a counter example. Let there be four characters a, b, c
and d, and their corresponding variable length codes be 00, 01, 0 and 1. This coding leads
to ambiguity because code assigned to c is the prefix of codes assigned to a and b. If the
compressed bit stream is 0001, the de-compressed output may be “cccd” or “ccb” or “acd”
or “ab”.
See this for applications of Huffman Coding.
There are mainly two major parts in Huffman Coding
1. Build a Huffman Tree from input characters.
2. Traverse the Huffman Tree and assign codes to characters.

Algorithm:

The method which is used to construct optimal prefix code is called Huffman coding.
This algorithm builds a tree in bottom up manner. We can denote this tree by T
Let, |c| be number of leaves
|c| -1 are number of operations required to merge the nodes. Q be the priority queue which
can be used while constructing binary heap.
Algorithm Huffman (c)
{
n= |c|
 Q=c
for i<-1 to n-1

do
{

temp <- get node ()

left (temp] Get_min (Q) right [temp] Get Min (Q)

a = left [templ b = right [temp]

F [temp]<- f[a] + [b]

insert (Q, temp)

return Get_min (0)

}
Steps to build Huffman Tree
Input is an array of unique characters along with their frequency of occurrences and output
is Huffman Tree.
1. Create a leaf node for each unique character and build a min heap of all leaf nodes (Min
Heap is used as a priority queue. The value of frequency field is used to compare two
nodes in min heap. Initially, the least frequent character is at root)
2. Extract two nodes with the minimum frequency from the min heap.

3. Create a new internal node with a frequency equal to the sum of the two nodes
frequencies. Make the first extracted node as its left child and the other extracted node
as its right child. Add this node to the min heap.
4. Repeat steps#2 and #3 until the heap contains only one node. The remaining node is the
root node and the tree is complete.
Let us understand the algorithm with an example:
character Frequency
a 5
b 9
c 12
d 13
e 16
f 45
Step 1. Build a min heap that contains 6 nodes where each node represents root of a tree
with single node.
Step 2 Extract two minimum frequency nodes from min heap. Add a new internal node
with frequency 5 + 9 = 14.
 Illustration of step 2

Now min heap contains 5 nodes where 4 nodes are roots of trees with single element each,
and one heap node is root of tree with 3 elements
character Frequency
c 12
d 13
Internal Node 14
e 16
f 45
Step 3: Extract two minimum frequency nodes from heap. Add a new internal node with
frequency 12 + 13 = 25

Illustration of step 3

Now min heap contains 4 nodes where 2 nodes are roots of trees with single element each,
and two heap nodes are root of tree with more than one nodes
character Frequency
Internal Node 14
e 16
Internal Node 25
f 45
Step 4: Extract two minimum frequency nodes. Add a new internal node with frequency 14
+ 16 = 30
 Illustration of step 4

Now min heap contains 3 nodes.

character Frequency
Internal Node 25
Internal Node 30
f 45
Step 5: Extract two minimum frequency nodes. Add a new internal node with frequency 25
+ 30 = 55

Illustration of step 5

Now min heap contains 2 nodes.

character Frequency
f 45
Internal Node 55
Step 6: Extract two minimum frequency nodes. Add a new internal node with frequency 45
+ 55 = 100
 Illustration of step 6

Now min heap contains only one node.

character Frequency
Internal Node 100
Since the heap contains only one node, the algorithm stops here.
Steps to print codes from Huffman Tree:
Traverse the tree formed starting from the root. Maintain an auxiliary array. While moving
to the left child, write 0 to the array. While moving to the right child, write 1 to the array.
Print the array when a leaf node is encountered.

Steps to print code from HuffmanTree

The codes are as follows:

character code-word
f 0
c 100
d 101
a 1100
b 1101
e 111