Dynamic Programming

General method

• Works the same way as divide-and-conquer, by combining solutions to subproblems

– Divide-and-conquer partitions a problem into independent subproblems
– Greedy method only works with the local information
• Dynamic programming is required to take into account the fact that the problems may not be partitioned into independent
subproblems
– The subproblem is solved only once and the answer saved in a table
∗ Applicable when the subproblems overlap, or subproblems share subsubproblems
– Solution to a problem can be viewed as the result of a sequence of decisions
∗ Knapsack problem solution
· Decide the values of xi , 0 ≤ i < n
· Make a decision on x0 , then x1 , and so on
P P
· An optimal sequence maximizes the objective function pi xi under the constraints wi xi ≤ m and
0 ≤ xi ≤ 1
∗ Shortest path problem
· Determine the shortest path from vertex i to vertex j by finding the second vertex in the path, then the
third, and so on, until vertex j is reached
· Optimal sequence of decisions is one with a path of least length
– Optimal sequence of decisions can be found by making the decisions one at a time and never making an erroneous
decision
∗ This is the idea followed in greedy algorithms
∗ Can be guaranteed by trying all possible decision sequences but the time and space costs will be prohibitive
∗ Shortest path problem
· Find shortest path from vertex i to vertex j
· Let Ai be the set of vertices adjacent from i
· The selection of vertex from Ai cannot be finalized without looking further ahead
· If we have to find a shortest path from a single source to all vertices in G, then at each step, a correct
decision can be made
– Dynamic programming drastically reduces the amount of enumeration by avoiding the enumeration of some deci-
sion sequences that cannot possibly be optimal; an optimal sequence of decisions is obtained by using the principle
of optimality
∗ Applicable when the subproblems are not entirely independent, they may have common subsubproblems
∗ Each subsubproblem is solved only once and the results used repeatedly
• The word programming in dynamic programming does not refer to coding but refers to building tables of intermediate
results
• Typically used for optimization problems that may have many possible solutions
– An optimal solution vs the optimum solution

Definition 1 The principle of optimality states that an optimal sequence of decisions has the property that whatever the
initial state and decision are, the remaining states must constitute an optimal decision sequence with regard to the state
resulting from the first decision.
Dynamic Programming 2

• Greedy method vs Dynamic programming

– In greedy method, only one decision sequence is ever generated
– In dynamic programming, many decision sequences may be generated
– Sequences containing suboptimal sequences cannot be optimal because of principle of optimality, and so, will not
be generated
– Shortest path problem
∗ Assume that i, i1 , i2 , . . . , ik , j is a shortest path from i to j
∗ Starting with vertex i, a decision is made to go to i1
∗ Following this decision, the problem state is defined by vertex i1 , and we need to find a path from i1 to j
∗ The sequence i1 , i2 , . . . , ik , j must be a shortest i1 to j path
∗ If not, let i1 , r1 , r2 , . . . , rq , j be a shortest i1 to j path
∗ Then, i, i1 , r1 , r2 , . . . , rq , j is an i to j path that is shorter than the path i, i1 , i2 , . . . , ik , j
∗ Hence, principle of optimality is applicable
• Development of solution broken into four steps:
1. Characterize the structure of an optimal solution
2. Recursively define the value of an optimal solution
3. Compute the value of an optimal solution in bottom-up fashion
4. Construct an optimal solution from computed information
• Computing the nth Fibonacci number

int fib ( const int n )

{
if ( n <= 0 ) return ( 0 );
if ( n == 1 ) return ( 1 );
return ( fib ( n - 1 ) + fib ( n - 2 ) );
}

– This code is extremely inefficient; why?

• An efficient code using an array of size n

int fib ( const int n )

{
if ( n <= 0 ) return ( 0 );
if ( n == 1 ) return ( 1 );
int * a = new int[n+1];
a[0] = 0;
a[1] = 1;
for ( int i ( 2 ); i <= n; i++ )
a[i] = a[i-1] + a[i-2];
int tmp ( a[n] );
delete[] a;
return tmp;
}

• If we want to save space as well, the following code is more appropriate

Dynamic Programming 3

int fib ( const int n )

{
if ( n <= 0 ) return ( 0 );
if ( n == 1 ) return ( 1 );
int f0 ( 0 ), f1 ( 1 ), f;
for ( int i ( 2 ); i <= n; i++ )
{
f = f1 + f0;
f0 = f1;
f1 = f;
}
return f;
}

The above solution is known as bottom-up dynamic programming – we compute the smallest values first and build the
solution using the solution to smaller problems; most of the real dynamic programming situations refer to top-down
dynamic programming (also known as memoization) as you will see next in knapsack problem
• Knapsack problem
– Recursive solution
∗ Each time you choose an item, you assume that you can optimally find a solution to pack the rest of the
knapsack
struct item
{
int size;
int val;
};

int knapsack ( const int capacity )

{
int t;
// N is the number of item types
for ( int i ( 0 ), int max ( 0 ); i < N; i++ )
if ( ( int space = capacity - items[i].size ) >= 0 )
{
Remove items[i] from items;
if ( ( t = knapsack ( space ) + items[i].val ) > max )
max = t;
}
return ( max );
}
– Dynamic programming solution
∗ Decide the values of xi , 1 ≤ i ≤ n
∗ Make a decision on x1 , then on x2 , and so on
P P
∗ An optimal sequence of decisions maximizes the objective function pi xi , under the constraints wi xi ≤ m
and 0 ≤ xi ≤ 1
int knapsack ( const int capacity )
{
int maxi, t;
if ( maxknown[capacity] )
return ( maxknown[capacity] );
for ( int i ( 0 ), int max ( 0 ); i < N; i++ )
Dynamic Programming 4

if ( ( int space = capacity - items[i].size ) >= 0 )

if ( ( t = knapsack ( space ) + items[i].val ) > max )
{
max = t;
maxi = i;
}
maxknown[capacity] = max;
itemknown[capacity] = items[maxi];
return ( max );
}
• 0/1 knapsack
– Same as regular knapsack problem except that the xi ’s are restricted to a value of 0 or 1 (take nothing or everything
for an item)
– Formally, an instance of knapsack problem knap ( l, j, y ) is
P
Maximize px
Pl≤i≤j i i
subject to l≤i≤j i xi ≤ y
w
xi = 0 or 1, l ≤ i ≤ j
and the knapsack problem is represented by knap ( 1, n, m )
– Let y1 , y2 , . . . , yn be an optimal sequence of 0/1 values for x1 , x2 , . . . , xn , respectively
∗ If y1 = 0, then y2 , y3 , . . . , yn must constitute an optimal sequence for the problem knap ( 2, n, m )
∗ If it is not an optimal subsequence, then y1 , y2 , . . . , yn is not an optimal sequence for knap ( 1, n, m )
∗ If y1 = 1, then y2 , y3 , . . . , yn must constitute an optimal sequence for the problem knap ( 2, n, m - w1 )
∗ If it is not an optimal subsequence, then
Pn
· there is another 0/1 sequence z2 , z3 , . . . , zn such that i=2 wi zi ≤ m − w1 , and
Pn Pn
· i=2 pi zi > i=2 pi yi
∗ Hence, the sequence y1 , z2 , z3 , . . . , zn gives a sequence with greater value
∗ Not possible due to principle of optimality QED

• General decision problem

– Let S0 be initial problem state
– Assume that you have to make n decisions di , 1 ≤ i ≤ n
– Let D1 = {r1 , r2 , . . . , rj } be the set of possible decision values for d1
– Let Si be the problem state following the choice of decision ri , 1 ≤ i ≤ j
– Let Γi be an optimal sequence of decisions with respect to the problem state Si
– Then, by principle of optimality, an optimal sequence of decisions with respect to S0 is the best of decision se-
quences ri , Γi , 1 ≤ i ≤ j
• Shortest path
– Which of the vertices in Ai (where Ai ⊂ V is the set of vertices adjacent to i) should be the second vertex on the
path?
– Let Ai be the set of vertices adjacent to vertex i
– For each vertex k ∈ Ai , let Γk be a shortest path from k to j
– Then, a shortest i to j path is the shortest of the paths {i, Γk | k ∈ Ai }.
– Dynamic programming solution for shortest path
∗ Let k be an intermediate vertex on a shortest i to j path i, i1 , i2 , . . . , k, p1 , p2 , . . . , j
∗ The paths i, i1 , . . . , k and k, p1 , . . . , j must, respectively, be shortest i to k and k to j paths.
Dynamic Programming 5

∗ Let Pj be the set of vertices adjacent to vertex j; k ∈ Pj ⇔ hk, ji ∈ E(G)

∗ For each k ∈ Pj , let Γk be a shortest i to k path
∗ By principle of optimality, a shortest i to k path is the shortest of paths {Γk , j|k ∈ Pj
· Start at vertex j and look at last decision made
· Last decision was to use one of the edges hk, ji, k ∈ Pj
• Dynamic programming eliminates all recomputation in any recursive program, by saving intermediate values in variables
whose scope is designed to allow them to be visible in more than one local context

Property 1 Dynamic programming reduces the running time of a recursive function to be at most the time required to
evaluate the function for all arguments less than or equal to the given argument, treating the cost of a recursive call as
constant.

• Property 1 implies that the running time for the knapsack problem is O(N M )
• Dynamic programming becomes ineffective when the number of possible function values that may be needed is so high
that we cannot afford to save or precompute all of them
• Dynamic programming solution to 0/1 knapsack
– The intermediate knapsack problem knap(l,j,y) can be represented by
P
Maximize px
Pl≤i≤j i i
subject to l≤i≤j wi xi ≤ y
xi ∈ {0, 1}, l ≤ i ≤ j
– The original knapsack problem now is: knap(0,n-1,m)
– Let gj (y) be the value of an optimal solution to knap(j + 1, n, y).
∗ g0 (m) is the value of an optimal solution to knap( 1, n, m )
∗ The possible decisions for x1 are 0 and 1 (D1 = {0, 1})
∗ From the principle of optimality, it follows that

g0 (m) = max{g1 (m), g1 (m − w1 ) + p1 }

– Let y1 , y2 , . . . , yn be an optimal solution to knap( 1, n, m ).

∗ For each j, 1 ≤ j ≤ n, y1 , . . . , yj , and yj+1 , . . . , yn must be optimal solutions to the problems knap(1, j,
Pj Pj
i=1 wi yi ) and knap(j+1, n,, m − i=1 wi yi ) respectively
∗ This observation allows us to generalize the previous function g() to

gi (y) = max{gi+1 (y), gi+1 (y − wi+1 ) + pi+1 }

∗ The recursion is solved by using gn (y) = 0 for y ≥ 0 and gn (y) = −∞ for y < 0
∗ From gn (y), we can obtain gn−1 (y) using the above recurrence; then, using gn−1 (y), we can find gn−2 (y),
and so on; finally, we can determine g0 (m)
– Example
∗ n = 3, w = {2, 3, 4}, p = {1, 2, 5}, m = 6
∗ We have to compute g0 (6)

g0 (6) = max(g1 (6), g1 (4) + 1)

g1 (6) = max(g2 (6), g2 (3) + 2)
g2 (6) = max(g3 (6), g3 (2) + 5)
= max(0, g3 (2) + 5)
= max(0, 0 + 5)
Dynamic Programming 6

= 5
g2 (3) = max(g3 (3), g3 (3 − 4) + 5)
= max(0, −∞)
= 0
g1 (6) = max(5, 2)
= 5
g1 (4) = max(g2 (4), g2 (4 − 3) + 2)
g2 (4) = max(g3 (4), g3 (4 − 4) + 5)
= max(0, 0 + 5)
= 5
g2 (1) = max(g3 (1), g3 (1 − 4) + 5)
= max(0, −∞)
= 0
g1 (4) = max(5, 2)
= 5
g0 (6) = max(5, 5 + 1)
= 6

– The sequence of decisions x1 , x2 , . . . , xn leads to

gj (y) = max({gj−1 (y), gj−1 (y − wj ) + pj }

where gj (y) is the value of an optimal solution to knap(1,j,y)

• 0/1 knapsack
– Looking backwards at the sequence of decisions x1 , x2 , . . . , xn , we see that

fj (y) = max{fj−1 (y), fj−1 (y − wj ) + pj }

where fj (y) is the value of an optimal solution to knap(1,j,y)

– Value of an optimal solution to knap(1,n,m) is fn (m)
– Solve by beginning with f0 (y) = 0 for all y, y ≥ 0, and f0 (y) = −∞ for all y, y < 0
– Successively obtain f1 , f2 , . . . , fn
– The soluion method may indicate that one has to look at all possible decision sequences to get an optimal sequence
using dynamic programming
∗ Using principle of optimality, suboptimal decision sequences are discarded
∗ Although total number of decision sequences are exponential in the number of decisions, dynamic program-
ming algorithms often have polynomial complexity
∗ Also, optimal solutions to subproblems are retained to avoid recomputing their value

Traveling Salesperson Problem

• Given a directed graph G = (V, E) with edge costs cij

• cij is defined such that cij > 0 for all i and j and cij = ∞ if hi, ji 6∈ E.
• |V | = n and n > 1
• Tour
Dynamic Programming 7

– A tour of G is a directed cycle that includes every vertex in V , and no vertex occurs more than once except for the
starting vertex
– Cost of a tour is the sum of the cost of edges on the tour
– Traveling salesperson problem is to find a tour of minimum cost
• Comments
– 0/1 knapsack is a subset selection problem
– Traveling salesperson is a permutation problem
– Permutation problems are harder to solve
∗ n! different permutations of n objects
∗ 2n different subsets of n objects
∗ n! > 2n
• Greedy algorithm
– Start with vertex v1 ; call it vi
– Visit the vertex vj that is nearest to vi , or can be reached from vi with least cost
– Repeat the above starting at vertex vj (call it as new vi ) taking care never to visit a vertex already visited
• Dynamic programming algorithm
– Regard the tour to be a simple path that starts and ends at vertex 1
– Every tour consists of an edge h1, ki for some k ∈ V − {1} and a path from vertex k to vertex 1
– The path from vertex k to vertex 1 goes through each vertex in V − {1, k} exactly once
– If the tour is optimal, then the path from k to 1 must be a shortest k to 1 path going through all vertices in V − {1, k}
– Let g(i, S) be the length of a shortest path starting at vertex i, going through all vertices in S, and terminating at
vertex 1
– g(1, V − {1}) is the length of an optimal salesperson tour
– From the principal of optimality

g(1, V − {1}) = min {c1k + g(k, V − {1, k})} (1)

2≤k≤n

– Generalizing (for i 6∈ S)
g(i, S) = min{cij + g(j, S − {j})} (2)
j∈S

– Equation 1 may be solved for g(1, V − {1}) if we know g(k, V − {1, k}) for all values of k
– The g values may be obtained by using Equation 2
∗ g(i, φ) = ci,1 , 1 ≤ i ≤ n
∗ We can use Equation 2 to obtain g(i, S) for all S of size 1
∗ Then we can obtain g(i, S) for S with |S| = 2
∗ When |S| < n − 1, the values of i and S for which g(i, S) is needed are such that i 6= 1, 1 6∈ S, and i 6∈ S
• Solving traveling salesperson problem with dynamic programming – example
– Consider the directed graph presented below
0 10 15 20
5 0 9 10
6 13 0 12
8 8 9 0
Dynamic Programming 8

– Solving for 2, 3, 4
g(2, φ) = c21 = 5
g(3, φ) = c31 = 6
g(4, φ) = c41 = 8
– Using Equation 2, we get

g(2, {3}) = c23 + g(3, φ) = 15 g(2, {4}) = 18

g(3, {2}) = 18 g(3, {4}) = 20
g(4, {2}) = 13 g(4, {3}) = 15

– Next, we compute g(i, S) with |S| = 2, i 6= 1, 1 6∈ S, and i 6∈ S

g(2, {3, 4}) = min{c23 + g(3, {4}), c24 + g(4, {3})} = 25

g(3, {2, 4}) = min{c32 + g(2, {4}), c34 + g(4, {2})} = 25
g(4, {2, 3}) = min{c42 + g(2, {3}), c43 + g(3, {2})} = 23

– Finally, from Equation 1, we obtain

g(1, {2, 3, 4}) = min{c12 + g(2, {3, 4}), c13 + g(3, {2, 4}), c14 + g(4, {2, 3})}
= min{35, 40, 43}
= 35

– Optimal tour
∗ Has cost 35
∗ A tour of this length may be constructed if we retain with each g(i, S) the value of j that minimizes the right
hand side of Equation 2
∗ Let this value be called J(i, S)
∗ Then, J(1, {2, 3, 4}) = 2
∗ Thus the tour starts from 1 and goes to 2
∗ The remaining tour may be obtained from g(2, {3, 4})
∗ Now, J(2, {3, 4}) = 4
∗ Thus the next edge is h2, 4i
∗ The remaining tour is for g(4, {3})
∗ J(4, {3}) = 3
∗ The optimal tour is 1, 2, 4, 3, 1
• Analysis of traveling salesperson
– Let N be the number of g(i, S)s that have to be computed before Equation 1 may be used to compute g(1, V − {1})
– For each value of |S|, there are n − 1 choices of i
 
n−2
– The number of distinct sets S of size k not including 1 and i is
k
– Hence,
n−2  
X n−2
N= (n − k − 1) = (n − 1)2n−2
k
k=0

– An algorithm that finds an optimal tour using Equations 1 and 2 will require Θ(n2 2n ) time as the computation of
g(i, S) with |S| = k requires k − 1 comparisons when solving Equation 2
– Better than enumerating all n! different tours to find the best one
– The most serious drawback of the dynamic programming solution is the space needed (O(n2n ))
∗ This can be too large even for modest values of n.
Dynamic Programming 9

Assembly line scheduling

• Manufacturing problem with two assembly lines

– Chassis enters an assembly line
– Parts added to chassis at each station
– Finished product exits the assembly line
– Each assembly line i has exactly n stations – Si1 , . . . , Sin
∗ Corresponding stations perform exactly the same function but may take different amount of time
∗ Assembly time required at station Sij is denoted by aij
– Shown in following figure

✓
Station
✓✏ S1,1 Station
✓✏ S1,2 Station
✓✏ S1,3 Station
✓✏ S1,n−1 Station
✓✏ ✏
S1,n
Assembly line 1 a 1,1 ✲ a 1,2 ✲ a 1,3 ✲ ··· ✲ a 1,n−1 ✲ a 1,n
✒✒✑ ✒✑ ✒✑ ✒✑ ✒✑ ✑
✒ ❆ ✕ ❆
✁ ✁✕ ✁✕ ❆ ✁✕ ❅
✓✏ ❆ ✁ ❆ ✁ ✁ ❆ ✁ ❅❘✓✏
e1 ❯
❆
✓✏ ✁ ✓✏❆❯ ✁ ✁ ❆❯
✓✏ ✁ x1
✒✑ ✁ ✁ ✁ ✒✑
✒ t1,1
✁
t1,2
✁
t1,n−1
✁ ❅
✒✑ ✒✑ ✒✑ ❅❘
Chassis ❆ ✁ ❆ ✁ ❆ ✁ Completed
enters ✁
❆ ✁
❆ ··· ✁
❆
✓✏ ✁ ❆ ✓✏ ✁ ❆ ✓✏ ✁ ❆ auto exits
❅ ✒
t2,1 ❆ t2,2 ❆ t2,n−1 ❆
❅❅✓✏
❘ ✒✑ ❆ ✒✑ ❆ ✒✑ ❆
✓✏
e2 ✕
✁ ✁✕ ❆ ✁✕ x2
✒✑ ❆ ❆ ❆ ✒✑
❅ ✁ ❆ ✁ ❆ ❆ ✁ ❆ ✒
❅✓
❘✓✏ ✁ ❯ ✁
❆
✓✏ ✓✏❆❯ ❆❯ ✁
✓✏ ❆❯
✓✏ ✏
Assembly line 2 a2,1 ✲ a2,2 ✲ a2,3 ✲ · · · ✲a2,n−1 ✲ a2,n
✒✒✑ ✒✑ ✒✑ ✒✑ ✒✑ ✑
Station S2,1 Station S2,2 Station S2,3 Station S2,n−1 Station S2,n
– Entry time for assembly line i denoted by ei
– Exit time for assembly line i denoted by xi
• Chassis can go from one station to another
– Within the same assembly line in negligible time, or at no cost
– To the other assembly line at some cost
∗ Cost to go from one assembly line to another after having gone through station Sij is tij
• Problem is to schedule the assembly line such that the selection of stations from each assembly line minimizes the overall
assembly cost
– Need to determine the stations to choose from assembly line 1 and 2 to minimize the assembly time
– In the following example, choose stations 1, 3, 6 from line 1 and 2, 4, 5 from line 2
Dynamic Programming 10

• Brute force solution

– Enumerate all possibilities for stations
– Compute how long does each one take, and pick the best
– Problem hard because there are 2n ways to choose the stations
– Time required is given by Ω(2n ) which is infeasible for large
• Dynamic programming solution: Step 1: Structure of fastest way through factory, or structure of optimal solution
– Fastest possible way for a chassis to get from starting point through station S1,j
– Only one possible way to go from starting point through station S1,1
– Two ways to arrive at each station Si,j , j > 1
1. Station on same line, from station Si,j−1
2. Station on other line, from station Si′ ,j−1 , i′ 6= i, at a cost ti′ ,j−1
– Assume that fastest way through station S1,j is through station S1,j−1
∗ The chassis must have taken the fastest way from starting point to station S1,j−1
∗ If there were a faster way to get through station S1,j−1 , we could substitute this faster way to get through
station S1,j – contradiction
– Assume that fastest way through station S1,j is through station S2,j−1
∗ The chassis must have taken the fastest way from starting point to station S2,j−1
∗ If there were a faster way to get through station S2,j−1 , we could substitute this faster way to get through
station S1,j – contradiction
Dynamic Programming 11

– Optimal substructure
∗ Optimal solution to a problem contains the optimal solution to the subproblems within it
∗ Fastest way to a station requires that the chassis must have taken the fastest way to the previous station in the
line
∗ To find the fastest way to a station Si,j , solve the subproblem to compute the fastest way to the two previous
stations – Si,j−1 and Si′ ,j−1
• Step 2: A recursive solution
– Define the value of an optimal solution recursively in terms of optimal solution to subproblems
∗ Subproblems will be defined as the problem of finding the fastest way through station j on both lines, for
j = 1, 2, . . . , n
– Let fi [j] be the fastest possible time to get a chassis from starting point through station Si,j
– Let f ∗ be the fastest time to get the chassis all the way through the factory

f ∗ = min(f1 [n] + x1 , f2 [n] + x2 )

– Time to go through the first station in each line is given by

f1 [1] = e1 + a1,1
f2 [1] = e2 + a2,1

– Compute fi [j] for j = 2, 3, . . . , n and i = 1, 2

– Adding the recursive step, it is easy to see that

f1 [j] = min(f1 [j − 1] + a1,j , f2 [j − 1] + t2,j−1 + a1,j )

f2 [j] = min(f2 [j − 1] + a2,j , f1 [j − 1] + t1,j−1 + a2,j )

– Define li [j] to keep track of line number whose station j − 1 is used to get to station Si,j
∗ No need to define l1 [j] because no station precedes station 1 on either line
– Define l∗ to be the line whose station n is used as the last station to get through the assembly line
– Starting with l∗ = 1, use station S1,6
– l1 [6] = 2 ⇒ station S2,5
– l2 [5] = 2 ⇒ S2,4
– l2 [4] = 1 ⇒ S1,3
– l1 [3] = 2 ⇒ S2,2
– l2 [2] = 1 ⇒ S1,1
• Step 3: Computing the fastest times
– Simple to write recursive algorithm but its running time is exponential in n
– Compute fi [j] values in increasing order of station numbers
∗ Leads to Θ(n) time