DATA STRUCTURES AND
ALGORITHMS
DATA STRUCTURES AND ALGORITHMS
WEEK 1 : THE BASIC CONCEPTS
3
CONTENTS
1. Illustrative example
2. Basic concepts about algorithm
3. Asymptotic notations
4. Algorithmic analysis techniques
4
OBJECTIVES
After this lesson, students can:
1. Understand some basic concepts about algorithm
2. Know to use asymptotic notation to analyze the complexity of an algorithm
3. Know how to analyze the complexity of an algorithm
5
CONTENTS
1. Illustrative example
2. Basic concepts about algorithm
3. Asymptotic notations
4. Algorithmic analysis techniques
6
1. ILLUSTRATIVE EXAMPLE
• The maximum subarray problem:
• Given an array of n numbers: a1, a2, … , an
• The contiguous subarray ai, ai+1 , …, aj with 1 ≤ i ≤ j ≤ n is a subarray of the given array and
𝑗
𝑘=𝑖 𝑎𝑘 is called as the value of this subarray
• The task is to find the maximum value of all possible subarrays, in other words, find the
𝑗
maximum 𝑘=𝑖 𝑎𝑘 . The subarray with the maximum value is called as the maximum subarray.
Example: Given the array -2, 11, -4, 13, -5, 2 then the maximum subarray is 11, -4, 13 with the
value = 11+ (-4)+13 =20
7
1. ILLUSTRATIVE EXAMPLE
Method 1: Brute force
• Browse all possible subarray off the given array: ai, ai+1 , …, aj where
0 ≤ i ≤ j ≤ n-1, and calculate the sum of all elements in subarray to find the maximum sum.
int maxSum = a[0];
for (int i = 0; i <= n-1; i++) {
for (int j = i; j <= n-1; j++) {
int sum = 0;
for (int k = i; k <= j; k++) sum += a[k];
if (sum > maxSum) maxSum = sum;
}
}
8
1. ILLUSTRATIVE EXAMPLE
Method 1: Brute force
• Browse all possible subarray off the given array: ai, ai+1 , …, aj where
0 ≤ i ≤ j ≤ n-1, and calculate the sum of all elements in subarray to find the maximum sum.
• Analyze the algorithm: we count the number of time the statement sum += a[k] that the
algorithm need to perform. The number is:
n 1 n 1 n 1 n 1
(n i )(n i 1)
int maxSum = a[0];
( j i 1) (1 2 ... (n i))
i 0 j i i 0 i 0 2
for (int i = 0; i<=n-1; i++) {
1 n 1 n 2 n 1 n(n 1)(2n 1) n(n 1)
k (k 1) k k
for (int j = i; j<=n-1; j++) {
2
int sum = 0;
for (int k=i; k<=j; k++) sum += a[k]; 2 k 1 2 k 1 k 1 2 6
if (sum > maxSum) maxSum = sum;
n3 n 2 n
}
}
6 2 3
9
1. ILLUSTRATIVE EXAMPLE
Method 2: Brute force with better implementation
Index i 0 1 2 3 4 5
a[i] -2 11 -4 13 -5 2
i = 0: 18 + (-5)=13 13
9 + (-4)=5 5
(-2),(-2, 11), (-2,11, -4), (-2,11,-4,13), (-2,11,-4,13,-5), (-2,11,-4,13,-5,2)
-2+11 = 9 9 5+13=18 18
• We could see that, we can calculate the sum of the elements from position i to j from the
sum of the elements from i to j-1 with just one addition:
j j 1
a[k ] a[ j ] a[k ]
k i k i
The sum of elements from i to j The sum of elements from i to j-1
10
1. ILLUSTRATIVE EXAMPLE
Method 2: Brute force with better implementation
j j 1
a[k ] a[ j ] a[k ]
k i k i
The sum of elements from i to j The sum of elements from i to j-1
int maxSum = a[0]; int maxSum = a[0];
for (int i=0; i<=n-1; i++) { for (int i=0; i<=n-1; i++) {
for (int j=i; j<=n-1; j++) { int sum = 0;
int sum = 0; for (int j=i; j<=n-1; j++) {
for (int k=i; k<=j; k++) sum += a[k]; sum += a[j];
if (sum > maxSum) maxSum = sum; if (sum > maxSum) maxSum = sum;
} }
} }
11
1. ILLUSTRATIVE EXAMPLE
Method 2: Brute force with better implementation
• Analyze the algorithm: We count the number of summation operations that the algorithm
need to execute, it means count the number of times that the statement sum += a[j] is
executed. The number of summation operations is:
n 1
n2 n
i 0
(n i ) n (n 1) ... 1
2 2
int maxSum = a[0];
for (int i=0; i<=n-1; i++) {
int sum = 0;
for (int j=i; j<=n-1; j++) {
sum += a[j];
if (sum > maxSum) maxSum = sum;
}
}
12
1. ILLUSTRATIVE EXAMPLE
The number of times that the summation operations need to do is:
n3 n 2 n
• Method1. Brute force
6 2 3
n2 n
• Method 2. Brute force with better implementation
2 2
For the same problem, we have proposed two algorithms that require different numbers of
operations, and therefore will require different calculation times.
The table below shows the calculation time of the above two algorithms, with the
assumption: the computer can perform 108 additions per second.
Complexity n=10 Time (sec) n=100 Time (sec) n=104 Time n=106 Time
n3 103 10-5 106 10-2 sec 1012 2.7 hours 1018 115 days
n2 100 10-6 10000 10-4 sec 108 1 sec 1012 2.7 hours
13
CONTENTS
1. Illustrative example
2. Basic concepts about algorithm
3. Asymptotic notations
4. Algorithmic analysis techniques
14
2. BASIC CONCEPTS ABOUT ALGORITHM
An algorithm for solving a given problem is a defined procedure that includes a finite sequence of steps that
need to be performed to obtain an output from a given input (input) of the problem.
Input Algorithm Output
Some basic characteristics of the algorithm:
Precision
Finiteness
Uniqueness
Generality
15
2. BASIC CONCEPTS ABOUT ALGORITHM
• Given 2 or more algorithms to solve the same problem, how do we select the best one?
• Some criteria for selecting an algorithm:
1) Is it easy to implement, understand, modify?
2) How long does it take to run it to completion? TIME
3) How much of computer memory does it use? SPACE
16
2. BASIC CONCEPTS ABOUT ALGORITHM
Complexity of an algorithm:
• How to measure the computation time?
The computation time of an algorithm depends on the size of input data (size increases, then
computation time increases).
Thus, analyze running time as a function of input size. However, in some cases, even on inputs of the
same size, running time can be very different
• Example: In order to find the first prime number in an array: the algorithm scans the array from left
to right
• Array 1: 3 9 8 12 15 20 (algorithm stops when considering the first element)
• Array 2: 9 8 3 12 15 20 (algorithm stops when considering the 3rd element)
• Array 3: 9 8 12 15 20 3 (algorithm stops when considering the last element)
There are 3 types of computation time
17
2. BASIC CONCEPTS ABOUT ALGORITHM
Types of computation time:
Best-case:
T(n) = minimum time of algorithm on any input of size n.
Average-case:
T(n) = expected time of algorithm over all inputs of size n.
Worst-case:
T(n) = maximum time of algorithm on any input of size n.
18
2. BASIC CONCEPTS ABOUT ALGORITHM
There are two ways to evaluate the computation time:
• Experimental Evaluation of computation time:
• Write a program implementing the algorithm
• Run the program with inputs of varying size and composition
• Use a method like clock( ) to get an accurate measure of the actual running time
• Theory: use asymptotic notations
19
CONTENTS
1. Illustrative example
2. Basic concepts about algorithm
3. Asymptotic notations
4. Algorithmic analysis techniques
20
3. ASYMPTOTIC NOTATION
Các ký hiệu tiệm cận (asymptotic notation):
Q, W, O, w
• Được sử dụng để mô tả thời gian tính của thuật toán, mô tả tốc độ tăng của thời gian chạy
phụ thuộc vào kích thước dữ liệu đầu vào.
• Ví dụ, khi nói thời gian tính của thuật toán cỡ Q(n2), tức là, thời gian tính tỉ lệ thuận với n2
cộng thêm các đa thức bậc thấp hơn.
21
3. ASYMPTOTIC NOTATION
Asymptotic notation:
Q, W, O, w
• Used to describe the calculation time of an algorithm, describing the increase in runtime
depending on the input data size.
• For example, when saying the computation time isQ(n2), that is, the computation time is
proportional to n2 plus lower order terms.
22
3. ASYMPTOTIC NOTATION
3.1. Asymptotic notation theta Q
•For a given function g(n), we denote by 𝚯(g(n)) the set of functions:
Q(g(n)) = {f(n): there exists constants c1, c2 and n0 such that:
0 c1g(n) f(n) c2g(n), for all n n0 }
(Set of all functions that have the same rate of growth as g(n))
•When we say that one function is theta of another, we mean that neither function goes to infinity faster than
the other.
23
3. ASYMPTOTIC NOTATION
3.1. Asymptotic notation theta Q
Example: Prove that 10n2 - 3n = 𝚯(n2)
We need to show with which values of the constants n0, c1, c2 then the inequality in the definition of the theta
notation is correct:
𝑐1 𝑛2 ≤ 𝑓 𝑛 = 10𝑛2 − 3𝑛 ≤ 𝑐2 𝑛2 ∀n ≥ n0
Suggestion: Make c1 a little smaller than the leading (the highest) coefficient, and c2 a little bigger.
Select: c1 = 1, c2 = 11, n0 = 1 then we have
n2 ≤ 10n2 – 3n ≤ 11n2, for n ≥ 1
∀n ≥ 1: 10n2 - 3n = 𝚯(n2)
Note: For polynomial functions: To compare the growth rate, it is necessary to look at the term with the highest
coefficient
24
3. ASYMPTOTIC NOTATION
3.2. Asymptotic notation big Oh O
For a given function g(n), we denote by O(g(n)) the set of functions:
O(g(n)) = {f(n): there exists positive constants c and n0 such that:
f(n) cg(n) for all n n0 }
(Set of all functions whose rate of growth is the same as or lower than that of g(n))
• O(g(n)) is the set of functions that go to infinity no faster than g(n).
Example: Prove that 2n + 10 = O(n)
f(n) = 2n+10, g(n) = n
• Need to find constants c and n0 such that:
2n + 10 cn for all n n0
(c 2) n 10
n 10/(c 2)
Select c = 3 and n0 = 10
25
3. ASYMPTOTIC NOTATION
3.2. Asymptotic notation big Oh O
Note: f(n) = 50n3 + 20n + 4 is O(n3)
Would be correct to say is O(n3+n)
Not useful, as n3 exceeds by far n, for large values
Would be correct to say is O(n5)
OK, but g(n) should be as close as possible to f(n)
Simple Rule: Drop lower order terms and constant factors
• Example:
All these functions are O(n): n, 3n, 61n + 5, 22n – 5, …
All these functions are O(n2): n2, 9 n2, 18 n2+ 4n – 53, …
All these functions are O(n log n): n(log n), 5n(log 99n), 18 + (4n – 2)(log (5n + 3)),...
26
3. ASYMPTOTIC NOTATION
3.2. Asymptotic notation Omega Ω
For a given function g(n), we denote by Ω(g(n)) the set of functions:
W(g(n)) = {f(n): there exists positive constants c and n0 such that:
cg(n) f(n) for all n n0 }
(Set of all functions whose rate of growth is the same as or higher than that of g(n))
W(g(n)) is the set of functions that go to infinity no slower than g(n)
Example: Prove that 5n2 = W(n)
Need to find c and n0 such that cn 5n2 for n n0
Inequality is correct when c = 1 and n0 = 1
27
CONTENTS
1. Illustrative example
2. Basic concepts about algorithm
3. Asymptotic notations
4. Algorithmic analysis techniques
28
4. ALGORITHMIC ANALYSIS TECHNIQUES
1. Consecutive Statements: The sum of running time of each segment.
Running time of “P; Q”, where P is implemented first, then Q, is
Time(P; Q) = Time(P) + Time(Q) ,
or if using asymptotic Theta:
Time(P; Q) = Q(max(Time(P), Time(Q)).
2. FOR loop: The number of iterations times the time of the inside statements.
for i =1 to m do P(i);
Assume running time of P(i) is t(i), then the running time of for loop is 𝑚
𝑖=1 𝑡(𝑖)
3. Nested loops: The product of the number of iterations times the time of the inside
statements.
for i =1 to n do
for j =1 to m do P(j);
Assume the running time of P(j) is t(j), then the running time of this nested loops is:
29
4. ALGORITHMIC ANALYSIS TECHNIQUES
4. If/Else
if (condition)
S1;
else
S2;
The testing time plus the larger running time of the S1 and S2.
30
4. ALGORITHMIC ANALYSIS TECHNIQUES
Example
Case1: for (i=0; i<n; i++)
for (j=0; j<n; j++) O(n2)
k++;
Case 2: for (i=0; i<n; i++)
k++;
for (i=0; i<n; i++)
O(n2)
for (j=0; j<n; j++)
k++;
Case 3: for (int i=0; i<n-1; i++)
for (int j=0; j<i; j++) O(n2)
int k+=1;
31
4. ALGORITHMIC ANALYSIS TECHNIQUES
The characteristic statement is the statement being executed with frequency at least as well as
any statement in the algorithm.
If the execution time of each statement is bounded above by a constant, then the running time
of the algorithm will be the same size as the number of times the execution of the characteristic
statement. Thus, in order to evaluate the running time, one can count the number of times the
characteristic statement being executed
Example 1: Function to calculate Fibonacci numbers f0=0; f1=1; fn= fn-1 + fn-2
function Fibiter(n)
i=0;
j=1; The number of times this characteristic
for k=1 to n-1 do statement being executed is n
j = i + j;
The running time of Fibiter is O(n)
i = j – i;
return j;
32
4. ALGORITHMIC ANALYSIS TECHNIQUES
Example 2: The maximum subarray problem
• Method 1: Brute force
int maxSum = a[0];
for (int i=0; i<n; i++) {
for (int j=i; j<n; j++) {
int sum = 0; Select the statement sum+=a[k] as the
for (int k=i; k<=j; k++) characteristic statement
sum += a[k]; Computation of the algorithm: O(n3)
if (sum > maxSum)
maxSum = sum;
}
}
33
4. ALGORITHMIC ANALYSIS TECHNIQUES
Example 2: The maximum subarray problem
• Method 2: Brute force with better implementation
int maxSum = a[0];
for (int i=0; i<n; i++) {
int sum = 0; Select the statement sum+=a[k] as the
for (int j=i; j<n; j++) { characteristic statement
sum += a[j];
Computation of the algorithm: O(n2)
if (sum > maxSum)
maxSum = sum;
}
}
34
4. ALGORITHMIC ANALYSIS TECHNIQUES
Example 3: Give asymptotic big-Oh notation for the running time T(n) of the following statement
segment:
a)
int x = 0;
for (int i = 1; i <= n; i *= 2) x=x+1;
b)
int x = 0;
for (int i = n; i > 0; i /= 2) x=x+1;
35
SUMMARY AND SUGGESTIONS
1. The lesson presented basic concepts of
algorithms and algorithm complexity
2. Following this lesson, learners will learn about
recursion - general diagram and some
examples
36
THANK YOU !
37
