Analysis and Design of Algorithms

Instructor: Prof. Dr. Abeer Mahmoud

Computer science department,

Faculty of computer and information sciences,
Ain shams university.

1
 Lecture 7

Additional Algorithms
Examples on both design
Techniques
1-Barut force and excusive search
2- Divide and conquer

2
 follow definition /
Brute Force & try all possibilities
Exhaustive
Search
break problem
Divide & into distinct
Iterative Conquer sub-problems
Improvement

repeatedly improve Algorithm

current solution
Design Transformation
convert
problem to
Techniques another one

Randomization

use Dynamic break problem

random Programming into overlapping
numbers sub-problems
repeatedly do
Greedy what is best now
3
 1-Barut force and excusive search

4
 Brute Force & Exhaustive Search

 Straightforward way to solve a problem, often involves

checking all possibilities

5
Example-1: Brute Force for Combinatorial
Problems
 Knapsack Problem:
 There are n different items in a store
 Item i weighs wi pounds and is worth $vi
 A thief breaks in
 He can carry up to W pounds in his knapsack
 What should he take to maximize his benefit?

 Solution: Consider every possible subset of items, calculate

total value and total weight and discard if more than W; then
choose remaining subset with maximum total value.
 Takes Ω(2n) time

6
 Knapsack problem
 item 1: 7 lbs, $42
 item 2: 3 lbs, $12 subset total weight total value
 item 3: 4 lbs, $40 Ø 0 $0
{1} 7 $42
 item 4: 5 lbs, $25
{2} 3 $12
 W = 10 {3} 4 $40
{4} 5 $25
 need to check 16 {1,2} 10 $54
possibilities {1,3} 11 infeasible
{1,4} 12 infeasible
{2,3} 7 $52
{2,4} 8 $37
etc.
7
 Your Turn
Trace the solution of Knapsack problem using Brute Force search

 item 1: 10 lbs, $400

 item 2: 4 lbs, $1200
 item 3: 8 lbs, $120
 item 4: 3 lbs, $290
 W = 16
 How many step to reach
solution

8
Example-2: Closest Pair
 Closest-Pair Problem:
 Given n points in d-dimensional space, find the two that are
closest

 Applications:
 which post offices should be closed
 which DNA sequences are most similar

9
Example-2: Closest Pair
 Brute-force Solution (for 2-D case):
 compute distances between all pairs of points
 sqrt((xi – xj)2 + (yi – yj)2)
Why??
 scan all distances to find smallest
 Running time: Θ(n2), assuming each numerical operation is
constant time (including square root?)

 Improvements:
 drop the square root
 don’t compute distance for same 2 points twice

10
Example-3: Traveling Salesman Problem

 Given n cities with known distances between each pair, find the
shortest tour that passes through all the cities exactly once
before returning to the starting city
 Alternatively: Find shortest Hamiltonian circuit in a weighted
connected graph
 Example:
2
a b
5 3
8 4

c 7 d
11
Example-3: Traveling Salesman Problem

Tour Cost
a→b→c→d→a 2+3+7+5 = 17
a→b→d→c→a 2+4+7+8 = 21
2
a→c→b→d→a 8+3+4+5 = 20 a b
a→c→d→b→a 8+7+4+2 = 21 5 3
8 4
a→d→b→c→a 5+4+3+8 = 20
a→d→c→b→a 5+7+3+2 = 17 c d
7
Running time: Θ(n2)
Calculate time
• compute distances from stat to start complexity
• scan all distances to find smallest

12
Example-4: Brute Force Searching

Sequential search
Compare successive elements of a given list with a search key until
1. either a match is encountered
2. or the list is exhausted without a match.
0 1 N-1 N

Algorithm:
extra position
SequentialSearch(A[0..N], key){ to store the key
A[N]  key;
i  0;
while (i<N and A[i] != k ) do { i = i + 1};
if i < n return i; Best case: O(1)
else return –1; Analysis: Worst case: O(N)
} Avg. case: O(N/2)

13
Example-5: Brute-Force String Matching

 pattern: a string of m characters to search for

 text: a (longer) string of n characters to search in
 problem: find a substring in the text that matches the pattern

Brute-force algorithm
Step 1 Align pattern at beginning of text
Step 2 Moving from left to right, compare each character of
pattern to the corresponding character in text until
 all characters are found to match (successful search); or
 a mismatch is detected
Step 3 While pattern is not found and the text is not yet
exhausted, realign pattern one position to the right and repeat Step 2
14
Example-5: Brute-Force String Matching

1. Pattern: 001011
Text: 10010101101001100101111010

2. Pattern: happy
Text: It is never too late to decide to be happy

15
Example-5: Brute-force String Matching
Pattern: p[0] p[1] … p[m-1] m characters (m <= n)
Text: t[0] t[1] … … … t [n-1] n characters

Purpose: to find a substring of the text that matches the pattern.

More precisely, to find position k such that
t[k] = p[0], t[k+1] = p[1], … t[k+j]= p[j], …, t[k+m-1] = p[m-1]

Brute-force algorithm:
for k = 0 to n-m do {
j 0; // tracer for pattern
while (j < m and p[j] = t[k+j]) do
j  j+1; // move one step over pattern
if j = m return k; // matched
Complexity (assuming m << n)
}
Worst case: m (n – m + 1) in Θ (nm)
return –1; // ( no match)
16
Example-6: Selection sort: Brute Force Sorting
 scan array to find smallest element
 scan array to find second smallest element
 etc.

 take Θ(n2) time in the worst case.

17
Selection Sort :Brute-Force Sorting

1. Scan the array to find its smallest element and swap it with the
first element.
2. Then, starting with the second element, scan the elements to its
right to find the smallest among them and swap it with the
second elements.
3. Generally, on pass i (0  i  n-2), find the smallest element in
A[i..n-1] and swap it with A[i]:

18
Selection Sort :Brute-Force Sorting

Example:

marker

min Moving index

Moving index

19
Selection Sort :Brute-Force Sorting

Moving index
min

Moving index

min
Moving index

20
Selection Sort :Brute-Force Sorting

Moving index
min

Moving index
min

min
Moving index

21
Selection Sort :Brute-Force Sorting

Example:

Swap

marker
Repeat
min

22
 Your turn
apply the selection sort on the following number then write your
time complexity analysis
7 8 5 2 4 6 3

Iteration 1: CN: SWn:

23
Selection Sort :Brute-Force Sorting

Algorithm
for i  0 to N-2 do  ∑N-2
min  i;
i=0 Why
for j  i+1 to N-1 do
N-1 N-2
{ if (A[j] < A[min]) ∑ c & N-1
min  j; j=i+1
}
swap A[i] and A[min] ;
}
Analysis:
T(N) = ∑ N-2∑ N-1
c = ∑ cN-2
(N-1-i) = c (N-1)N / 2 in O(N ) 2
i=0 j=i+1 i=0

24
Example-7: Bubble sort: Brute Force Sorting

 Bubble sort

 scan array, swapping out-of-order neighbors

 continue until no swaps are needed

 take Θ(n2) time in the worst case.

25
Example-7: Bubble Sort :Brute-Force Sorting

1. In each pass, we compare adjacent elements and swap them if

they are out of order until the end of the list. By doing so,
the 1st pass ends up “bubbling up” the largest element to the
last position on the list

2. The 2nd pass bubbles up the 2nd largest, and so on until, after
N-1 passes, the list is sorted.

26
Example-7: Bubble Sort :Brute-Force Sorting

Example:

Pass 1 89 | 45 68 90 29 34 17 Pass 2 45 | 68 89 29 34 17
45 89 | 68 90 29 34 17 45 68 | 89 29 34 17
68 89 | 90 29 34 17 68 89 | 29 34 17
89 90 | 29 34 17 29 89 | 34 17
29 90 | 34 17 34 89 | 17
34 90 | 17 17 89
17 90

45 68 89 29 34 17 90 45 68 29 34 17 89
largest 2nd
largest

27
 Bubble Sort
Algorithm N-2
for i  0 to N-2 do  ∑
i=0 N-2-i
for j  0 to N-2-i do  ∑ c
j=i+1
if (A[j+1] < A[j]) swap A[j] and A[j+1] ;
}
}
Analysis: N-2 N-2-i N-2 2
T(N) = ∑ ∑ c = ∑ c (N-1-i) = c (N-1)N / 2 in O(N)
i=0 j=0 i=0

28
 2-Divide & Conquer

29
General Idea of Divide & Conquer

1. Take your problem and divide it up into smaller pieces

2. Solve one or more of the smaller problems
3. Combine solutions to sub-problems

 Running time for these algorithms can often be stated as

a recurrence and solved with the master theorem

30
Example-7: Quicksort: Divide & Conquer

 Unlike merge sort, no combining step: two sub-

arrays form an already-sorted array

 How does Quicksort work?

 Pick a pivot element
 Move everything smaller than the pivot to the left
 And everything else to the right
 Sort each side (recursively) in place

31
Example-7: Quicksort: Divide & Conquer
Given an array of n elements (e.g., integers):
 If array only contains one element, return
 Else
 pick one element to use as pivot.
 Partition elements into two sub-arrays:
 Elements less than or equal to pivot
 Elements greater than pivot
 Quicksort two sub-arrays
 Return results

32
Example-7: Quicksort: Divide & Conquer

We are given array of n integers to sort:

40 20 10 80 60 50 7 30 100

33
Pick Pivot Element

There are a number of ways to pick the pivot element. In

this example, we will use the first element in the array:

40 20 10 80 60 50 7 30 100

34
Partitioning Array

Given a pivot, partition the elements of the array

such that the resulting array consists of:
1. One sub-array that contains elements >= pivot
2. Another sub-array that contains elements < pivot

The sub-arrays are stored in the original data array.

Partitioning loops through, swapping elements

below/above pivot.

35
pivot_index = 0 40 20 10 80 60 50 7 30 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

36
 1. While data[i] <= data[pivot]
++i

pivot_index = 0 40 20 10 80 60 50 7 30 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

37
 1. While data[i] <= data[pivot]
++i

pivot_index = 0 40 20 10 80 60 50 7 30 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

38
 1. While data[i] <= data[pivot]
++i

pivot_index = 0 40 20 10 80 60 50 7 30 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

39
 1. While data[i] <= data[pivot]
++i
2. While data[j] > data[pivot]
--j

pivot_index = 0 40 20 10 80 60 50 7 30 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

40
 1. While data[i] <= data[pivot]
++i
2. While data[j] > data[pivot]
--j

pivot_index = 0 40 20 10 80 60 50 7 30 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

41
 1. While data[i] <= data[pivot]
++i
2. While data[j] > data[pivot]
--j
3. If i < j
swap data[i] and data[j]

pivot_index = 0 40 20 10 80 60 50 7 30 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

42
 1. While data[i] <= data[pivot]
++i
2. While data[j] > data[pivot]
--j
3. If i < j
swap data[i] and data[j]

pivot_index = 0 40 20 10 30 60 50 7 80 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

43
 1. While data[i] <= data[pivot]
++i
2. While data[j] > data[pivot]
--j
3. If i < j
swap data[i] and data[j]
4. While j > i, go to 1.

pivot_index = 0 40 20 10 30 60 50 7 80 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

44
 1. While data[i] <= data[pivot]
++i
2. While data[j] > data[pivot]
--j
3. If i < j
swap data[i] and data[j]
4. While j> i, go to 1.

pivot_index = 0 40 20 10 30 60 50 7 80 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

45
 1. While data[i] <= data[pivot]
++i
2. While data[j] > data[pivot]
--j
3. If i < j
swap data[i] and data[j]
4. While j > i, go to 1.

pivot_index = 0 40 20 10 30 60 50 7 80 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

46
 1. While data[i] <= data[pivot]
++i
2. While data[j] > data[pivot]
--j
3. If i < j
swap data[i] and data[j]
4. While j > i, go to 1.

pivot_index = 0 40 20 10 30 60 50 7 80 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

47
 1. While data[i] <= data[pivot]
++i
2. While data[j] > data[pivot]
--j
3. If i < j
swap data[i] and data[j]
4. While j > i, go to 1.

pivot_index = 0 40 20 10 30 60 50 7 80 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

48
 1. While data[i] <= data[pivot]
++i
2. While data[j] > data[pivot]
--j
3. If i < j
swap data[i] and data[j]
4. While j > i, go to 1.

pivot_index = 0 40 20 10 30 60 50 7 80 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

49
 1. While data[i] <= data[pivot]
++i
2. While data[j] > data[pivot]
--j
3. If i< j
swap data[i] and data[j]
4. While j> i, go to 1.

pivot_index = 0 40 20 10 30 7 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

50
 1. While data[i] <= data[pivot]
++i
2. While data[j] > data[pivot]
--j
3. If i < j
swap data[i] and data[j]
4. While j > i, go to 1.

pivot_index = 0 40 20 10 30 7 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

51
 1. While data[i] <= data[pivot]
++i
2. While data[j] > data[pivot]
--j
3. If i < j
swap data[i] and data[j]
4. While j > i, go to 1.

pivot_index = 0 40 20 10 30 7 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

52
 1. While data[i] <= data[pivot]
++i
2. While data[j] > data[pivot]
--j
3. If i < j
swap data[i] and data[j]
4. While j > j, go to 1.

pivot_index = 0 40 20 10 30 7 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

53
 1. While data[i] <= data[pivot]
++i
2. While data[j] > data[pivot]
--j
3. If i < j
swap data[i] and data[j]
4. While j > i, go to 1.

pivot_index = 0 40 20 10 30 7 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

54
 1. While data[i] <= data[pivot]
++i
2. While data[j] > data[pivot]
--j
3. If i < j
swap data[i] and data[j]
4. While j > i, go to 1.

pivot_index = 0 40 20 10 30 7 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

55
 1. While data[i] <= data[pivot]
++i
2. While data[j] > data[pivot]
--j
3. If i < j
swap data[i] and data[j]
4. While j > i, go to 1.

pivot_index = 0 40 20 10 30 7 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]

i j

56
 1. While data[i] <= data[pivot]
++i
2. While data[j] > data[pivot]
--j
3. If i < j
swap data[i] and data[j]
4. While j > i, go to 1.

pivot_index = 0 40 20 10 30 7 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]

i j

57
 1. While data[i] <= data[pivot]
++i
2. While data[j] > data[pivot]
--j
3. If i < j
swap data[i] and data[j]
4. While i > j, go to 1.

pivot_index = 0 40 20 10 30 7 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

58
 1. While data[i] <= data[pivot]
++i
2. While data[j] > data[pivot]
--j
3. If i < j
swap data[i] and data[j]
4. While j > i, go to 1.
5. Swap data[j] and data[pivot_index]

pivot_index = 0 40 20 10 30 7 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

59
 1. While data[i] <= data[pivot]
++i
2. While data[j] > data[pivot]
--j
3. If i < j
swap data[i] and data[j]
4. While j > i, go to 1.
5. Swap data[j] and data[pivot_index]

pivot_index = 4 7 20 10 30 40 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

60
Partition Result

7 20 10 30 40 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]

<= data[pivot] > data[pivot]

61
Recursion: Quicksort Sub-arrays

7 20 10 30 40 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7]
[8]

<= data[pivot] > data[pivot]

62
 Recursion: Quicksort Sub-arrays

Next Pivot=7 7 20 10 30 40 50 60 80 100

[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i j

63
 Recursion: Quicksort Sub-arrays
Swap

Next Pivot=7 7 20 10 30 40 50 60 80 100

[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i J
Stop

64
 Recursion: Quicksort Sub-arrays

Next Pivot=7 7 10 20 30 40 50 60 80 100

[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i J

65
 Recursion: Quicksort Sub-arrays

Next Pivot=7 7 10 20 30 40 50 60 80 100

[0] [1] [2] [3] [4] [5] [6] [7]
[8]
i J

66
 Quicksort Analysis
o Each call to Partition splits the array into two equal parts
• What would the best case be?
Partition 𝑛

Splitting the input as evenly as possible results in sub-problems of size 𝑛/2.

The total number of levels then becomes log2𝑛, and the effort to partition
each level is Θ(𝑛), for a total complexity Θ(𝑛 log 𝑛)
67
Quicksort Analysis
• What would the worst case be?
o Each call to Partition splits the array into an empty array and n-1 array

Partition 𝑛
Partition 𝑛 − 1
Partition 𝑛 − 2

Partition 𝑛 − 3

Partition 𝑛 − 4

Choosing the largest (or smallest) element results in 𝑛 − 1 calls

(levels), each call incurring a partitioning cost of order Θ(𝑛),
for a total worst-case complexity of: n 1
n(n  1)
i 
i 1 2
 ( n 2 )
68
 Quicksort Analysis

• What would the worst case be?

 When does this happen?

 sorted
 reverse sorted
 near sorted/reverse sorted

69
 Choosing a Good Pivot

 Good choice may be

 The element in the middle position

 Median of first, last, and middle

 Median of k sampled elements

 A random element
 Avoids the worst case on a completely sorted list.

Partition
 All the work is done in the partition() function

70
 Test Your Self
 Apply quick sort on the following array , the pivot is the
first element

16 14 25 21 12 78 97 2 89 21

pivot_index = 0

71
 Sorting summary so far
 Insertion sort:  Merge sort:  Quick sort:
 Easy to code  Divide-and-conquer:  Divide-and-
 Fast on small inputs (less  Split array in half
conquer:
than ~50 elements)  Recursive sort
 Recursively sort
 Fast on nearly-sorted  Partition array into
subarrays two subarrays
inputs
 Merge step  All of first subarray
 O(n2) worst case complexity is O(n) < all of second
 O(n2) average case subarray
 O(n2) reverse-sorted  No combine step
case  O(n lg n) worst case needed
 O(n lg n) average case
 Fast in practice
 O(n2) worst case
Bubble Sort :Brute-Force Sorting
Selection sort:  Worst case on
sorted input
O(n2) worst case

72
73