Numerical Analysis:

Theory and Practices

Jeong Phill Kim

phill@[Link]

Chung-Ang University
Seoul, Korea
2
Contents

1 What and Why Numerical Analysis ? 5

2 Linear System of Equations 7

2.1 Inverse matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2 Backward-Substitution Process . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3 Root Finding 13
3.1 Bisection Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.2 Newton-Rapson Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

4 Interpolation 27
4.1 Polynomial Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.2 Spline Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.3 Least Squre-Error Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.4 Richardson Extrapolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

5 Numerical Integrations 53
5.1 Definite Integral . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
5.2 Infinite-Region Integral . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
5.3 Improper Integral . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

6 Numerical Solution of Differential and Integral Equations 65

6.1 Ordinary Differential Equations . . . . . . . . . . . . . . . . . . . . . . . . . . 65
6.2 Partial Differential and Integral Equations . . . . . . . . . . . . . . . . . . . . 77

7 Random Number 83

A Matlab - Getting Started 107

3
4
Chapter 1

What and Why Numerical Analysis ?

Many problems can not be handled with only paper and pencil !!!

• Multi-Dimensional Problems
• Complex Initial or Boundary-Value Problems
• Simple Iteration calculation

Now a days, Nowadays, the computing power of PCs has increased dramatically:

• Increasing clocks RAM memory

• Innovative computing architecture (GPU)
• Novel and updated algorithms

However, there is a dilemma in studying (Viewpoint of Electrical Engineering Student):

• Algorithm or Practical Use

• Conceptual understanding of algorithm
• More attention on What Algorithms and How to Use Them !!!

5
6
Chapter 2

Linear System of Equations

A number of problems in numerical analysis can be reduced to, or approximated by, a sys-
tem of linear equations. Our goal is the numerical solution of systems of linear eqs:

a11 x1 + a12 x2 + · · · + a1N xN = b1

a21 x1 + a22 x2 + · · · + a2N xN = b2
.. (2.1)
.
aN 1 x1 + aN 2 x2 + · · · + aN N xN = bN

We also write this in a matrix form as

Ax = b (2.2)
     
 a11 a12 · · · a1N   x1   b1 
 a21 a22 · · · a2N   x2   b2 
     
A =  .
. ..  ,
 x =  .  , b =  .  (2.3)
 . . . . .   ..   .. 
    
aN 1 aN 2 aN N xN bN


For a unique solution x, the matrix A should be invertible, which is verified by non-zero
determinant of A:

Det(A) , 0 (2.4)

In this case, x can be found as

x = A−1 b

7
As an example, let us consider the following linear equations:

3 x1 + 3 x2 − 3 x3 = 6
−3 x1 − 4 x2 − 4 x3 = −7 (2.5)
2 x1 + 1 x2 + 1 x3 = 7
   
 3 3 −3   6 
A =  −3 −4 4  b =  −7  (2.6)
   
2 1 1 7
   

Before entering into the problem, let us study some important properties of linear sysme
of equations. It is found that the solution of the system does not affected by the following
operations:

1) Change of order of rows · · · ex) row 1 ↔ row 2:

−3 x1 − 4 x2 − 4 x3 = −7
3 x1 + 3 x2 − 3 x3 = 6
2 x1 + 1 x2 + 1 x3 = 7

2) Scaling of the row · · · ex) (row 1)/3:

x1 + x2 − x3 = 2
−3 x1 − 4 x2 − 4 x3 = −7
2 x1 + 1 x2 + 1 x3 = 7

3) Row by row adding and subtracting · · · ex) row 2 + row 1 → row 2

−3 x1 − 4 x2 − 4 x3 = −7
− x2 − 7 x3 = −1
2 x1 + 1 x2 + 1 x3 = 7

2.1 Inverse matrix

Let us remind the well-known Gaussian Elimination Method for finding the inverse matrix
with the following matrix A:
 
 3 3 −3 
A =  −3 −4 4 
 
2 1 1
 

8
1) Make a matrix B by adding the identity matrix I whose size is same as A:
h i
B= A I
 
 3 3 −3 1 0 0 
=  −3 −4 4 0 1 0 
 
2 1 1 0 0 1
 

2) Dividing B the 1st row by B(1, 1) = 3 yields new B

 
 1 1 −1 1/3 0 0 
B =  −3 −4 4 0 1 0 
 
2 1 1 0 0 1
 

3) Operation on the 2nd row and 3rd row to rebuild B as

i) Row 2 − B(2, 1) × Row 1
ii) Row 3 − B(3, 1) × Row 1
 
 1 1 −1 1/3 0 0 
B =  0 −1 1 1 1 0 
 
0 −1 3 −2/3 0
 

4) Dividing the 2nd row by B(2, 2) = −1 yields

 
 1 1 −1 1/3 0 0 
B =  0 1 −1 −1 −1 0
 


0 −1 3 −2/3 0 1


5) Operation on the 1st row and 3rd row to rebuild B as

i) Row 1 − B(1, 2) × Row 2
ii) Row 3 − B(3, 2) × Row 2
 
 1 0 0 4/3 1 0 
B =  0 1 −1 −1 −1 0
 


0 0 2 −5/3 −1 1


6) Dividing the 3rd row by B(3, 3) = 2 yields

 
 1 0 0 4/3 1 0 
B =  0 1 −1 0
 
−1 −1 

0 0 1 −5/6 −1/2 1/2


9
 7) Operation on the 1st row and 2nd row to rebuild B as

i) Row 1 − B(1, 3) × Row 3

ii) Row 3 − B(2, 3) × Row 3
 
 1 0 0 4/3 1 0 
B =  0 1 0 −11/6 −3/2 1/2
 
 

0 0 1 −5/6 −1/2 1/2


8) The right sub-matrix is the inverse matrix of A:

 
 4/3 1 0 
−1
A =  −11/6 −3/2 1/2
 


−5/6 −1/2 1/2


The following is Matlab script for calculating the inverse matrix.

1 clear all, close all, clc

2

3 A = [1 2 3; 4 5 6; 7 8 1];
4 Ainv = func_Minv(A);
5

6 C = Ainv*A;
7

8 Ainv
9 C
10

11 %%
12 function Ainv = func_Minv(A)
13 [M,N] = size(A);
14 B = [A, eye(size(A))];
15

16 for k = 1:M
17 B(k,k:2*N) = B(k,k:2*N)/B(k,k);
18 B([1:k-1,k+1:M],k:2*N) = ...
19 B([1:k-1,k+1:M],k:2*N) - B([1:k-1,k+1:M],k)*B(k,k:2*N);
20 end
21 Ainv = B(:,N+1:2*N);
22 end

10
1 Ainv =
2 1.3333 1.0000 0
3 -1.8333 -1.5000 0.5000
4 -0.8333 -0.5000 0.5000
5

6 C =
7 1 0 0
8 0 1 0
9 0 0 1

This can be verified by the calculation with Malab built-in function “inv”. Then you will get
the same inverse matrix. Now the given linear equation can be solved as
 
 1 
−1
x = A b =  3 
 
2
 

This can be verified by the Matlab programming. Even we have already verified the our
inverse programm, we will use Malab built-in function “inv” for calculating the inverse ma-
trix from now. The following is the Matlab script for solving our linear system of equation:

1 clear all, close all, clc

2

3 A = [3 3 -3; -3 -4 4; 2 1 1];
4 b = [ 6; -7; 7];
5

6 x = inv(A)*b;
7 x

1 x =
2

3 1.0000
4 3.0000
5 2.0000

11
2.2 Backward-Substitution Process
We may have a question that we always have to use inverse matrix approach to solve the lin-
ear system of equations. Starting with this question, the method called Backward-Substitution
Process has been suggested. This method follows the same approach of finding inverse ma-
trix (but we will not find its inverse matrix).

With the following matrix B:

 
 3 3 −3 6 
B =  −3 −4 4 −7 
 
2 1 1 7
 

1) With the following operation:

2nd row - B(2,1)/B(1,1) × 1st row

3rd row - B(3,1)/B(1,1) × 1st row
 
 3 3 −3 6 
B =  0 −1 1 −1 
 
0 −1 3 3
 

2) With the following operations:

3rd row - B(3,2)/B(2,2) × 2nd row

 
 3 3 −3 6 
B =  0 −1 1 −1
 


0 0 2 4


3) The matrix is now in upper triangular form: there are no nonzero entries below the
diagonal. From the resulting set of equations in reverse order, the same x can be found:

2 x3 = 4 

 → x3 = 2 → x2 = 3 → x1 = 1
−x2 + x3 = −1 


3x1 + 3x2 − 3x3 = 6 
 
 1 
x =  3 
 
2
 

12
Chapter 3

Root Finding

We are now concerned with the problem of finding a zero for the function f (x).

f (x) = 0 → x=p ???

Three methods which have been widely:

• Bisection Method
• Newton-Rapson Method
• Secant Method, · · ·

3.1 Bisection Method

The bisection method is simple for root-finding applications. For a continuous function f (x)
between a and b (> a) and f (a)f (b) < 0

a+b
1) p =
 2
a = p, f (a)f (p) < 0


2) 
b = p, otherwise


3) Check the stop condition

4) Repeat or Stop

13
 ( )

( 1)

( 2)
( )

2 2 2

Figure 3.1: Concept of root finding with the bisection method.

Some candidates of the stop condition are

1) PN < ϵ,
2) PN − PN −1 < ϵ
P − PN −1
3) N <ϵ
PN

Ex. 1 For f (x) = 3x3 + 5x − 5 with

∆p
ϵ=
max(x) − min(x)

find the approximate zero of f (x) in x ∈ [−1, 2] with the bisection method. The stop condition
is ϵ <= 0.0005.

1 clear all, close all, clc

2

3 a = -1;
4 b = 2;
5 cond = 0.0005;

14
 6

7 xx = a:0.01:b;
8 fxx = func_fx(xx);
9 plot(xx,fxx)
10

11 intv = b-a;
12 ii = 0;
13 cond1 = 1;
14 while cond1 > cond
15 ii = ii+1;
16 xcond(ii) = cond1;
17 pii = (a+b)/2;
18 fpii = func_fx(pii);
19

20 if func_fx(a)*fpii < 0
21 b = pii;
22 else
23 a = pii;
24 end
25

26 p(ii) = pii;
27 fp(ii) = fpii;
28

29 if ii>=2
30 cond1 = abs(pii-p(ii-1))/intv;
31 end
32 idx(ii) = ii;
33 end
34

35 disp('[ii p f(p) xcond]')

36 [idx.' p.' fp.' xcond.’]
37

38 %% Function
39 function fx = func_fx(x)
40

41 fx = 3*x.^3+5*x-5;
42

43 end

15
 Figure 3.2: Example of root finding with the bisection method.

You can see the following result in the command window:

1 [ii p f(p) xcond]

2

3 ans =
4

5 1.0000 0.5000 -2.1250 1.0000

6 2.0000 1.2500 7.1094 1.0000
7 3.0000 0.8750 1.3848 0.2500
8 4.0000 0.6875 -0.5876 0.1250
9 5.0000 0.7813 0.3368 0.0625
10 6.0000 0.7344 -0.1400 0.0313
11 7.0000 0.7578 0.0947 0.0156
12 8.0000 0.7461 -0.0236 0.0078
13 9.0000 0.7520 0.0353 0.0039
14 10.0000 0.7490 0.0058 0.0020
15 11.0000 0.7476 -0.0089 0.0010

The bisection method, though conceptually clear, has significant drawbacks of slow convergence.
However, the method has the important property that it always converges to a solution, and

16
 1)

( 2)

Figure 3.3: Root finding with the Newton-Rapson method.

for that reason it is often used as an initial value for the more efficient methods (hybrid
methods). The following property is satisfied:
b−a
|Pn − p| < , n≥1
2n

3.2 Newton-Rapson Method

• This method is an iterative method for root finding with starting from some guess of
root x0 , one iteration produces a number x1 , which is supposed to be closer to a exact
root; guesses x2 , x3 , · · · xN to satisfy the given criterion.
• This can be clearly understood from Fig. XXX:

– At first, a function value f (x1 ) and f ′ (x1 ) at x1 are calculated.

f (x1 + ∆x) − f (x)

f ′ (x1 ) ←
∆x

17
 – Let us consider a line with a slope f ′ (x1 ) passing through (x1 , f (x1 )).

y − f (x1 ) = f ′ (x1 )(x − x1 )

Then x2 , the crossing point of this line and y = 0, is found to be much closer to
the solution than x1 .
f (x1 )
x2 = x1 −
f ′ (x1 )

– Repeating this process leads us to the destination.

Features and Issues

• This method shows a rapid convergence.

• To apply the Newton-Rapson method, the analytic or numerical derivative of the func-
tion must be known.
• Moreover, the success of this method depends on the initial guess x0 .
• Therefore, a hybrid approach combining the Newton-Rapson method and the bisec-
tion one becomes a promissing method.

– At first, some iterations are made with the bisection method.

– With the final x obtained by the bisection method as an initial value, the newton-
Rapson method can be invoked.
– Therefore, there is no need to set the initial values.

Ex Find the zero of the function given in the previous example with the initial value xini =
−0.9.

1 clear all, close all, clc

2

3 a = -1;
4 b = 2;
5 cond = 0.001;
6

7 %%
8 xx = a:0.01:b;
9 fxx = func_fx(xx);

18
10 plot(xx,fxx), hold on
11 plot([-1 2],[0 0])
12 p(1) = input('Initial value = ');
13 f(1) = func_fx(p(1));
14 plot(p(1),0,'o')
15 pause
16

17 ii = 1;
18 idx(1) = 1;
19 intv = b-a;
20 dx = (b-a)/1000;
21 cond1 = 1;
22 xcond(1) = cond1;
23

24 while cond1 > cond

25 ii = ii+1;
26 [f(ii),p(ii)] = func_newton_rapson(p(ii-1),dx);
27 plot([p(ii-1) p(ii-1) p(ii)],[0 f(ii-1) 0],'k')
28

29 cond1 = abs(p(ii)-p(ii-1))/intv;
30 xcond(ii) = cond1;
31 idx(ii) = ii;
32 pause
33 end
34 disp('[ii p f(p) xcond]')
35 [idx.' p.' f.' xcond.']

You can see the following result:

1 Initial value = -0.9

2 [ii p f(p) xcond]
3

4 ans =
5

6 1.0000 -0.9000 -11.6870 1.0000

7 2.0000 0.0528 -4.7355 0.3176
8 3.0000 0.9949 2.9289 0.3140
9 4.0000 0.7847 0.3733 0.0701

19
 Figure 3.4: Root finding example with Newton-Rapson method.

10 5.0000 0.7494 0.0094 0.0118

11 6.0000 0.7484 0.0000 0.0003

Ex Find the zero of the function given in the previous example with initial value obtained
after applying the bisection method three times.

1 clear all, close all, clc

2

3 a = -1;
4 b = 2;
5 cond = 0.001;
6 Nb = 3;
7 dx = 0.001;
8

9 intv = b-a;
10 xx = a:0.01:b;
11 fxx = func_fx(xx);
12 plot(xx,fxx)
13

20
14 for ii=1:Nb
15 [fpii,pii,a,b] = func_bisection(a,b);
16 p(ii) = pii;
17 fp(ii) = fpii;
18 end
19

20 ii = Nb;
21 xcond(1:Nb) = 1;
22 idx(1:Nb) = 1:3;
23

24 cond1 = 1;
25 while cond1 > cond
26 ii = ii+1;
27 [fp(ii),p(ii)] = func_newton_rapson(pii,dx);
28 if ii>=2
29 cond1 = abs(p(ii)-p(ii-1))/intv;
30 end
31 xcond(ii) = cond1;
32 idx(ii) = ii;
33 end
34 disp('[ii p f(p) xcond]')
35 [idx.' p.' fp.' xcond.']

1 [ii p f(p) xcond]

2

3 ans =
4

5 1.0000 0.5000 -2.1250 1.0000

6 2.0000 1.2500 7.1094 1.0000
7 3.0000 0.8750 1.3848 1.0000
8 4.0000 0.7586 0.1029 0.0388
9 5.0000 0.7586 0.1029 0

2-Dimensional Newton-Rapson Method

• Let us consider find the root in the domain of plane. The problem is to find (x, y)

21
 satisfying the following:

f (x, y) = 0 and g(x, y) = 0

This is a problem finding two unknowns from the two constraints.

• The initial staring point is assumed to be (x0 , y0 ). For estimating the next point for
solution, let us consider the Taylor series expansions for f (x, y) and g(x, y) with the
first-order term only as
df df
f (x, y) = f (x0 , y0 ) + (x0 , y0 )(x − x0 ) + (x , y )(y − y0 ) = 0
dx dy 0 0
dg dg
g(x, y) = g(x0 , y0 ) + (x0 , y0 )(x − x0 ) + (x , y )(y − y0 ) = 0
dx dy 0 0

• Expression in a matrix form and its solition

 df (x , y ) df (x , y ) "
 
# " #
0 0
 dx dy 0 0  x − x0 = −f (x0 , y0 )
 
 dg  y − y −g(x0 , y0
dg 0
(x0 , y0 ) dy (x0 , y0 ) 
 
dx
−1
#  df (x , y ) df (x , y ) "

" #
x − x0  dx 0 0 dy 0 0  −f (x0 , y0 )
=  
y − y0  dg dg  −g(x0 , y0
(x0 , y0 ) dy (x0 , y0 ) 
dx
−1
" # " #  df (x , y ) df (x , y ) "

#
x x0  dx 0 0 dy 0 0  f (x0 , y0 )
= − 
y y0  dg dg  g(x0 , y0
(x0 , y0 ) dy (x0 , y0 ) 
dx

Generaliztion to N -Dimensional Newton-Rapson Method

• Resulting expression
 df −1
 1 (x ) df1 (x ) ···
df1
(x ) 
 dx 0 dx 0 dxN 0   f1 (x0 ) 

 1 2
   
 df
 2 (x ) df2 (x ) df2   
0 ··· (x0 )  f1 (x0 ) 
 
x = x0 −  dx1 dx2 0 dxN

  
   
 .. .. .. ..   .. 


 . . . .   . 
 
 dfN 
dfN dfN 
f N (x0 )
 (x ) (x ) ··· (x ) 
dx1 0 dx2 0 dxN 0


22
    
 x1   x10 
 x   x 
 2  20 
x =  .  , x0 =  . 
 ..   .. 
   
xN xN 0

Roots and Polynomial Coefficients

• Sometimes we have to find the roots of the given polynomial, and vice versa, the poly-
nomial coefficients of the given roots.
• For example, the following is the given polynomial:

f (x) = 2x5 + 8x4 − 4x2 + 6x + 8

There will be roots, generally five the complex roots.

• This can be solved with some algorithms, however it may better to know how to find
them with the Matlab built-in function from the viewpoint of engineer. It can be found
as

1 >> P = [2 8 0 -4 6 8];
2 >> x0 = roots(P)

This operation is unique.

• The polynomial coefficients for the given roots can be found as

1 >> P1 = poly(x0)

• Contrary to finding roots from polynomial, the corresponding polynomial coefficients

to the given roots are not unique. If one set of coefficients is a solution, then its any
scaled version of coefficients becomes a solution, too. Therefore it is reasonable to set
the highest coefficient be “1”, which is adopted in Matlab built-in function “poly”.

1 clear all, close all, clc

2

3 %% f(x) = 2 x^3+5x^2-6x+8

23
 -3.63
Figure 3.5: Root finding and polynomial coefficients.

4 P = [2 5 -6 8];
5 R = roots(P)
6

7 x = -5:0.01:2;
8 y = polyval(P,x);
9 plot(x,y), grid on
10 xlabel('x'), ylabel('f(x)')
11 text(-3,-40,'f(x) = 2x^3 + 5 x^2-6x+8’)

1 >>
2 R =
3

4 -3.6300 + 0.0000i
5 0.5650 + 0.8847i
6 0.5650 - 0.8847i
7

8 >> P1 = poly(R)
9 P1 =
10

24
11 1.0000 2.5000 -3.0000 4.0000

25
26
Chapter 4

Interpolation

• We consider the problem of finding a function that interpolates a given set of values:

n 0 1 ··· n ··· N
x x0 x1 · · · xn · · · xN
y y0 y1 · · · yn · · · yN

• What kind of function is efficient ???

Common choice is a polynomial because of
– Easy evaluation
– Eeasy differentiation
– Easy integration

4.1 Polynomial Interpolation

• Let us consider f (x), a polynomial function with a coefficient vector P, which can
approximate the y:

f (xn ) = yn for n = 0, 1, · · · N .

In this case, the function is said to interpolate these data.

• Widely used approaches for constructing the polynomials:

27
 2

1
0

0 1 2

Figure 4.1: Concept of interpolation.

– Lagrange polynomial method

– Newton’s divided difference polynomial method

Linear System of Equations

• Since there are three data points, the polynomial passing through these points with
the order 2 is unique. (Of course, there are many polynomials whose order is larger
than 2).
• 2nd-order polynomial passing through the data points is set as:

y = ax2 + bx + c

Then the related conditions are





y0 = a x02 + b x0 + c

y1 = a x12 + b x1 + c




y2 = a x2 + b x2 + c


2

and this can be written in a matrix form as

XP = y

28
  2 
x0 x0 1  
a
 
y0 
 
X= x2 x1 1 , p = b ,
 
y = y1 
 
 1
c y2
 2     
x2 x2 1
| {z }
Vandermonde matrix

The coefficient vector p can be found as

p = X−1 y

Ex For the given data set, Find the polynomial passing through the data points given in the
following table:

n 0 1 N =2
x 1 1/2 3
y 3 −10 2
Solution
   
 1 1 1   3 
X =  1/4 1/2 1  , y =  −10 
   
9 3 1 2
   

 
 −10.6 
p = X−1 y =  41.9 
 
−28.3
 

Therefore the final Lagrange polynomial P3 (x) becomes

P3 (x) = −10.6 x2 + 41.9 x − 28.3

This is found to be same as the result obtained in the previous example !!!

Lagrange Interpolation
Joseph-Louis Lagrange (Jan. 25, 1736∼Apr. 10, 1813)

• He is a French mathematician and astronomer.

• He made significant contributions to the fields of analysis, number theory, and reformulation
of classical mechanics based on Lagrange multiplier (1788).

29
 1 1

0 0

0 1 2
0 1

Figure 4.2: Concept of Lagrange interpolation.

Linear Interpolation

• Lagrange suggested the following formula representing the function of curve passing
through the given data points (x0 , f (x0 )), (x1 , f (x1 )), · · · , and (xn , f (xN )):

fL (x) = f (x0 ) LN 0 (x) + f (x1 ) LN 1 (x) + · · · + x(xN ) LN N (x)


1, x = xn


LN n (x) = 
0, x = x (m = 0, 1, · · · , N (m , n))

m

• Such a polynomial LN n (x) can be found using the property of zero-crossing as

LN n (x) = Cn (x − x0)(x − x1 ) · · · (x − xn−1 )(x − xn+1 ) · · · (x − xN )

The coefficient Cn can be found from LN n (xn ) = 1 as

1
Cn =
(xn − x0)(xn − x1 ) · · · (xn − xn−1 )(xn − xn+1 ) · · · (xn − xN )
Therefote LN n (x) in a product form becomes
N
Y x − xm
LN n (x) =
xn − xm
m=0

• Final formula of the Lagrange interpolation becomes

N N
X Y x − xm
fL (x) = f (xn )
xn − xm
n=0 m=0

30
 Ex. Construct the polynomial interpolating the data by using Lagrange Polynomials.

n x y
0 1 3
1 1/2 −10
2 3 2

Solution

We construct the Lagrange Polynomials:

(x − 1/2) (x − 3) 1
 
L20 (x) = = − x − (x − 3)
(1 − 1/2) (1 − 3) 2
(x − 1)(x − 3) 4
L21 (x) = = (x − 1)(x − 3)
(1/2 − 1) (1/2 − 3) 5
(x − 1/2) (x − 1) 1 1
 
L22 (x) = = (x − 1) x −
(3 − 1/2) (3 − 1) 5 2

Interpolating polynomial, in Lagrange Form:

1
 
P2L (x) = −3 x − (x − 3)
2
− 8(x − 1)(x − 3)
2 1
 
+ (x − 1) x −
5 2
7 3
 
2
= −3 x − x +
2 2
2
− 8(x − 4x + 3)
2 3 1
 
+ x2 − x +
5 2 2
2
= −10.6 x + 41.9 x + −28.3

1 %% Lagrange Interpolation
2 clear all, close all, clc
3

4 x = [ 1 3 5 7];
5 y = [-5 1 -2 0];
6

31
 7 %%
8 N = length(x);
9 Norder = N-1;
10

11 for ii = 1:N
12 idx = [1:ii-1 ii+1:N];
13 % Indices except i-th index
14 Denii = prod(x(ii)-x(idx));
15 % Prod of Den. Elements (!=i-th)
16 Numii = poly(x(idx));
17 % Poly Coef of Num. (order=Norder)
18 C(ii,:) = y(ii)*Numii/Denii;
19 % Coef Vector of ii-th Polynomial
20 end
21 CL = sum(C)
22 % Coef Vector of Resulting Lagrange Polynomial
23

24 %% Check by polyfit and polyval

25 P = polyfit(x,y,3)
26 xx = 1:0.1:7;
27 yy = polyval(P,xx);
28 plot(x,y,'+',xx,yy)

1 CL =
2 0.2917 -3.7500 14.2083 -15.7500
3

4 P =
5 0.2917 -3.7500 14.2083 -15.7500

Newton’s Forward Divided Difference Method

• For a given data set, the Lagrange interpolation can provides the appropriate polyno-
mial solution.
• If a new data point (xN +1 , yN +1 ) is added to the original one, the resulting polynomial
solution can also be found. However, each Lagrange Polynomial would have to be
updated, resulting in complicated work for a large N .

32
 Figure 4.3: Example of Lagrange interpolation.

• Therefore there is a demand for alternative methods constructing the polynomials it-
eratively.
• With a proper PN (x) crossing the N + 1 data points, the following construction PN +1 (x)
seems to be reasonable:
PN +1 (x) = PN (x) + CN +1 (x)
CN +1 (x) = c (x − x0 )(x − x1 ) · · · (x − xN ), c: constant

• Note that
CN +1 (xn ) = 0, for n , N + 1
implying that PN +1 (x) will interpolate the data at x0 , x1 , · · · , xN .
• To get the value of the constant we calculate c such that
yN +1 = PN +1 (xN +1 )
= PN (xN +1 ) + c(xN +1 − x0 )(xN +1 − x1 ) · · · (xN +1 − xN )
yN +1 − PN (xN +1 )
c=
(xN +1 − x0 )(xN +1 − x1 ) · · · (xN +1 − xN )

yN +1 − PN (xN +1 )
→ cN +1 =
(xN +1 − x0 )(xN +1 − x1 ) · · · (xN +1 − xN )

33
 • This construction is known as Newton’s Algorithm.
Ex

Find a polynomial fitting the data by using Newton’s Algorithm.

n x y
0 1 3
1 1/2 −10
2 3 2

Solution

We construct the polynomial iteratively.

Step 1: P0 (x) = 3
1
 
Step 2: P1 (x) = P0 (x) + c(x − 1) → −10 = 3 + c −
2
→ c = 26 → P1 (x) = 3 + 26(x − 1) = 26x − 23
1
 
Step 3: P2 (x) = P1 (x) + c(x − 1) x −
2
5
2 = 26 × 3 − 23 + c × 2 ×
2
53
→ c=− = −10.6
5
1
 
P2 (x) = P1 (x) − 10.6(x − 1) x −
2
2 3 1
 
= 26x − 23 − 10.6 x − x +
2 2
2
= −10.6 x + 41.9 x − 28.3

• It is found to be same as the solution of the Lagrange polynomial method. (Uniqueness

theorem)
• The Newton’s method gives an advantage of iterative calculation, resulting in simpler
calculation of polynomials.

34
 1 %% Newton's Forward Divide Difference
2 % for Interpolation
3 clear all, close all, clc
4

5 %%
6 x = [1 1/2 3]';
7 y = [3 -10 2]';
8

9 %%
10 N = length(x);
11

12 P = y(1);
13 xz = [];
14 for ii=2:N
15 xz = [xz x(ii-1)];
16 Den = prod((x(ii)-xz));
17 Num = y(ii)-polyval(P,x(ii));
18 c = Num/Den;
19 P = [0 P]+c*poly(xz);
20 end
21 P

1 P =
2

3 -10.6000 41.9000 -28.3000

4.2 Spline Interpolation

• Spline interpolation is

– used to approximate complex functions and shapes.

– a function consisting of simple functions glued together.
– In this way a spline is different from a polynomial interpolation, which consists
of a single well defined function that approximates a given shape; splines are

35
 normally piecewise polynomial.
• Splines make use of a set of a number of subintervals obtained by cutting an interval.

Linear Splines

• A first spline (linear spline) is a function which is linear on each subinterval defined
by a partition:
• A function S is a linear spline on [a, b] if

1) The domain of S is [a, b].

2) S is continuous on [a, b].
3) There is a partition {ti }N
i=0 of [a, b] such that on each [ti , ti+1 ], S is a linear polynomial.

• A linear spline is defined entirely by its value at the specific points as

x x0 x1 · · · xN
y y0 y1 · · · yN

• For a spline with this data, the linear polynomial on each subinterval is defined with
a slope ks as

Si (x) − yi = ks (x − xi )
y −y
Si (x) = yi + i+1 i (x − xi )
xi+1 − xi

Quadratic splines

• For a smooth interpolation, a quadratic spline can be considered, where the interpola-
tion function between successive data points are represented with 2nd-order polyno-
mials.
• The problem is to determine the related coefficients.

– Given (x0 , y0 ), (x1 , y1 ), · · · , (xN , yN ), fit quadratic splines through the data. The

36
 Figure 4.4: Quadratic spline.

splines are given by





a1 x2 + b1 x + c1 , x0 ≤ x ≤ x1
2

a2 x + b2 x + c2 , x1 ≤ x ≤ x2



f (x) = .


..


aN x2 + bN x + cN , xN −1 ≤ x ≤ xN



– The problem is to determine the following 3N unknowns:

an , bn , and cn , n = 1, 2, · · · , N .

Therefore 3N conditions we have to set.

– Each quadratic spline goes through two consecutive data points:


 f (x0 ) = a1 x02 + b1 x0 + c1
f (x1 ) = a1 x12 + b1 x1 + c1



 f (x1 ) = a2 x12 + b2 x1 + c2
f (x2 ) = a2 x22 + b2 x2 + c2


..
.

37
 Figure 4.5: Example of quadratic spline.

 2
 f (xN −1 ) = aN xN −1 + bN xN −1 + cN

 2
f (xN ) = aN xN + bN xN + cN
This condition gives 2N equations.
– Next the spline becomes smooth at the boundary if the first derivatives of two
quadratic splines are same at the boundary.
At x = x1 : 2 a1 x1 + b1 = 2 a2 x1 + b2
→ 2(a1 x1 − a2 x2 ) + b1 − b2 = 0
For all interior boundary points, N − 1 conditions are obtained:
2(an xn − an+1 xn+1 ) + bn − bn+1 = 0, n = 1, 2, · · · , N − 1

– The total number of equations is

2N + N − 1 = 3N − 1
which is 1 smaller than the number of unknowns.
– For convenient solution, one additional constraint a1 = 0 is assumed, the resulting
equal number of conditions and unknowns allows us to solve the problems.

Ex Table shows the upward velocity of a rocket as a function of time.

t (s) 0 10 15 20 22.5 30
v (m/s) 0 227.04 362.78 517.35 602.97 901.67
Using quadratic splines, find the followings:

38
a) Velocity and acceleration at t = 16 seconds
b) Distance covered between t = 11 and t = 16 seconds

Solution Table shows the upward velocity of a rocket as a function of time.





a1 t 2 + b1 t + c1 , 0 ≤ t ≤ 10

a2 t 2 + b2 t + c2 , 10 ≤ t ≤ 15






 2
v(t) = 
a3 t + b3 t + c3 , 15 ≤ t ≤ 20

a t 2 + b t + c , 20 ≤ t ≤ 22.5

4 4 4




a5 t 2 + b5 t + c5 , 22.5 ≤ t ≤ 30



• The 5 interval results in N = 5, and the number of unknowns 3N = 15 requires 15 equa-

tions. Let us set up the equations.

− Because each spline goes through two consecutive data points:

• The 5 interval results in N = 5, and the number of unknowns 3N = 15 requires 15 equa-
tions. Let us set up the equations.
• Because each spline goes through two consecutive data points:
a1 t02 + b1 t0 + c1 = v0
a1 t12 + b1 t1 + c1 = v1
a2 t12 + b2 t1 + c2 = v1
a2 t22 + b2 t2 + c2 = v2
a3 t22 + b3 t2 + c3 = v2
a3 t32 + b3 t3 + c3 = v3
a4 t32 + b4 t3 + c4 = v3
a4 t52 + b4 t4 + c4 = v4
a5 t52 + b5 t4 + c5 = v4
a5 t62 + b5 t5 + c5 = v5
− From the equal derivatives at the interior data points,
2 x1 (a1 − a2 ) + (b1 − b2 ) = 0
2 x2 (a2 − a3 ) + (b2 − b3 ) = 0
2 x3 (a3 − a4 ) + (b3 − b4 ) = 0
2 x4 (a4 − a5 ) + (b4 − b5 ) = 0

39
 − Additional assumption

a1 = 0

• Matrix formulation

A15×15 C15×1 = B15×1 → C = A−1 B

with
 h i


A(1, 1 : 3) = x02 x0 1

 h i    
A(2, 1 : 3) = x12 x1 1 a1   0 




 h i    

A(3, 4 : 6) = x 2
x 1
b1  227.04
h 1 1

    

 i  c  227.04
A(4, 4 : 6) = x22 x2 1  1 

  
    

 h
A(5, 7 : 9) = x2 x 1
 i a2  362.78
2
   
h 2

 b  362.78


A(6, 7 : 9) = x2 x3 1
 i  2   
 3  c  517.35

 h i  2   
2

A(7, 10 : 12) = x x 1 a3 
 
517.35
 
h 3 3



 i    
A(8, 10 : 12) = x42 x4 1 C = b3  B = 602.97


    
 h i  c3  602.97
A(9, 13 : 15) = x42 x4 1



    

 h i a  901.67


A(10, 10 : 12) = x52 x5 1  4   
   


 h i b4   0 
A(11, [1 4 2 5]) = 2x1 −2x1 1 −1

    

  c   0 

 h i  4   
A(12, [4 7 5 8]) = 2x2 −2x2 1 −1

 a5 
 
 0 
 

 h i    

A(13, [7 10 8 11]) = 2x3 −2x3 1 −1

 b5 
 
 0 
 

 h i
c5 0
   
A(14, [10 12 11 14]) = 2x4 −2x4 1 −1






A(15, 1) = 1


• Coefficients

n an bn cn
1 0.0000 22.70 0.00
2 0.8888 4.93 88.88
3 −0.1356 35.66 −141.61
4 1.6048 −33.96 554.55
5 0.2089 28.86 −152.13

40
 • The resulting velocity function becomes




 + 22.70 t, 0 ≤ t ≤ 10

+0.8888 t 2 + 4.93 t + 88.88,




 10 ≤ t ≤ 15

v(t) = −0.1356 t 2 + 35.66 t − 141.61, 15 ≤ t ≤ 20



+1.6048 t 2 − 33.96 t + 554.55,




 20 ≤ t ≤ 22.5

+0.2089 t 2 + 28.86 t − 152.13,

 22.5 ≤ t ≤ 30

a) Velocty and acceleration at t = 16 sec

v(16) = −0.1356 (16)2 + 35.66 (16) − 141.61

= 394.24 (m/s)

Acceleration at t = 16 sec
d 
a(t) = − 0.1356 t 2 + 35.66 t − 141.61 , 15 ≤ t ≤ 20
dt
= −0.2712 t + 35.66
a(16) = 31.321 (m/s2 )

1 %% Quadratic Spline Example

2 clear all, close all, clc
3

4 x = [0 10 15 20 22.5 30];
5 y = [0 227.04 362.78 517.35 602.97 901.67];
6

7 N1 = length(x); % No. of data points

8 N = N1-1; % No. of intervals
9 Nm = 3*N; % Matrix size (Nm x Nm)
10

11 %%
12 A = zeros(Nm,Nm);
13 Ax = [(x.').^2 x.' ones(N1,1)];
14 for ii=1:N1
15 if ii==1
16 A(1,1:3) = Ax(1,:);
17 elseif ii==N1
18 ii2 = 3*(N1-2)+(1:3);

41
19 A(2*N,ii2) = Ax(N1,:);
20 else
21 ii1 = 2*(ii-1);
22 ii2 = 3*(ii-1)+(1:3);
23 A(ii1,ii2-3) = Ax(ii,:);
24 A(ii1+1,ii2) = Ax(ii,:);
25 end
26 end
27 dii = [1 3+1 2 3+2];
28 for ii=2:N
29 ii1 = 2*N+(ii-1);
30 ii2 = 3*(ii-2)+dii;
31 A(ii1,ii2) = [2*x(ii)*[1 -1] 1 -1];
32 end
33 A(3*N,1) = 1;
34

35 %%
36 B = zeros(Nm,1);
37 for ii=1:N
38 B(2*(ii-1)+1,1) = y(ii);
39 B(2*(ii-1)+2,1) = y(ii+1);
40 end
41 %%
42 A
43 B
44 C = inv(A)*B

b) Find the distance covered by the rocket from t = 11 s to t = 16 s

Z 16 Z 15
l(t) = v(t) dt = (0.8888 t 2 + 4.928 t + 88.88) dt
11 11
Z 16
+ (−0.1356 t 2 + 35.66 t − 141.61) dt
15
= 1595.9 (m)

• For more smmooth interpolation

– Matching up the derivatives

– The higher degree polynomials, More smoother the result.

42
 – So, where should we stop?
– Universal consensus on optimal degree is a cubic one.

Cubic Splines

• The cubic spline for N + 1 data points, the spline S(x) is expressed as




C1 (x), x0 ≤ x ≤ x1

C2 (x), x1 ≤ x ≤ x2



S(x) =  ..
.






CN (x), xN −1 ≤ x ≤ xN


Cn (x) = an x3 + bn x2 + cn x + dn

• There are 4N unknown coeffcients to determine.

– First we require that the spline interpolate the given value at every data point:

Cn (xn−1 ) = yn−1



Cn (xn ) = yn


2 3
an + bn xn−1 + cn xn−1 + dn xn−1 = yn−1


→
an + bn xn + cn xn2 + dn xn3 = yn


– Notice that there are 2N of these conditions. Then to make S(x) as smooth as possible,
it is reasopnable to impose the following conditions for n = 1, 2, · · · , N − 1:

′ ′
Cn (xn ) = Cn+1 (xn )


Cn′′ (xn ) = C ′′ (xn )


n+1

2 2
bn + 2cn xn + 3dn xn = bn+1 + 2cn+1 xn + 3dn+1 xn


→
2cn + 6dn xn = 2cn+1 + 6dn+1 xn


– Considering these 2(N − 1) conditions, total number of conditions is 4N − 2, so we are 2

equations short of being able to determine all the coefficients.
– At this point we have to make a choice. The usual choice is to require

′′
C1 (x0 ) = 0


C ′′ (xN ) = 0


N

43
 Figure 4.6: Example of quadratic spline.

These are called natural or simple boundary conditions.

– The other common option is called clamped boundary conditions:

′
C1 (x0 ) = 0


C ′ (xN ) = 0


N

– Note that the equations above are all linear equations with respect to the unknowns
(coefficients). This feature makes splines easy to calculate since solving linear systems
is what computers do best.

Ex. For the following dataset, find the interpolated data for x = 0 : 0.05 : 2

x 0 0.10 0.50 0.60 1.00 1.50 2.00

y 0 0.06 0.19 0.21 0.26 0.29 0.31

Solution Even though we can developthe code for this example, it may be sufficient to use
Matlab built-in function. The following is the Matlab scrip:

1 %% Cubic Spline
2 clear all, close all, clc
3

44
 Figure 4.7: Concept of least-square error method.

4 %%
5 x = [0 0.1 0.5 0.6 1 1.5 2];
6 y = [0 0.06 0.19 0.21 0.26 0.29 0.31];
7

8 xx = 0:0.05:2;
9 yy = spline(x,y,xx);
10 plot(x,y,'+',xx,yy)

45
4.3 Least Squre-Error Method
• Let us consider a moving car with a uniform velocity. The following table shows the
measured distance as a function of time.

n tn sn
1 1.2 1.1
2 2.3 2.1
3 3.0 3.1
4 3.8 4.0
5 4.7 4.9
6 5.9 5.9

• Let us try to find the coefficients of following line equation optimally fitting the above
measured data set:

sn → s(t) = c1 t + c0

• It is reasonable that the error at each point should be added independent of the sign
of error. Therefore the Mean Squred Error (MSE), given below, is very good canidate:

N
1 Xh i2
MSE = sn − s(tn )
N
n=1

• Next work is to find the best (c1 , c0 ) giving the Least MSE, which can be found as

∂MSE ∂MSE
= 0, =0
∂c1 ∂c0

• Here it is known that it is sufficient to consider the the total Squared Error (SE) instead
of the MSE to find (c1 , c0 ). the SE for this example can be found as

6 h
X i2 6 h
X i2
SE = sn − s(tn ) = sn − (c1 tn + c0 )
n=1 n=1
X6
= (sn2 − 2c1 tn sn − 2c0 sn + c12 tn2 + 2c1 c0 tn + c02 )
n=1

46
 • Collecting this equation in terms of c1 and c0
 6   6   6 
X  X  X 
SE =  sn2  − 2c1  tn sn  − 2c0  sn 
n=1 n=1 n=1
 6   6 
X  X 
+ c12  tn2  + 2c1 c0 − 2c0  tn  + c02
n=1 n=1
SE(c1 , c0 ) = 14.51 c1 − 29.50 c1 + 6.97 c1 c0 + c02 − 7.03 c0 + 15.00
2

• Taking derivatives w.r.t. c1 and c0 , and letting them to be zero:

∂SE
= 29.02 c1 + 6.97 c0 − 29.50 = 0
∂c1
∂SE
= 6.97 c1 + 2.00 c0 − 7.03 = 0
∂c0

• Finally the best fitting line is found as

c1 = 0.97, c0 = 0.15 → s = 0.97 t + 0.15

Matrix Approach - Pseudo Inverse Method

• The method of least-squre error is solved systematically with the help of the Psudo-
inverse. The previous example can be written in a matrix form with the following X,
b, and y as
   
1.2 1 1.1
   
2.3 1 2.1
  " #  
3.0 1 b1 3.1
X =   , b= , y =  
   
3.8 1 b0 4.0
   
4.7 1 4.9
   
5.9 1 5.9
   

Xb → y

• In fact, Xb , y. Even we accept Xb = y to try to find b, but we can know b can not be
found because of non-invertible X (non-square matrix).
• Let us think of an approximate solution b which gives a least squared error. The squred
error is formed as
 
SE = Σ |y − Ab|2 : Scalar quantity

47
• This sacalr quantity SE can be expressed in a matrix form as
h i
SE = yT − (Xb)T [y − Xb]
= yT y − yT Xb − (Xb)T y + (Xb)T (Xb)
   
= yT y − yT X b − bT XT y + bT (XT X)b

• What is noteworthy is

– SE is a scalar.
– All elements of SE are also scalar.
– Especially yT y is a constant w.r.t. b.

• Let us take derivatives of SE with respect to b1 and b0 . In a matrix form, let us consider
the following:
 ∂ SE  " #
 

∂ SE  ∂b1  ∂ SE

 
0
≜   = , is called Gradient !!!
∂b  ∂ SE 
  0 ∂b
∂b0
• Our concern is
∂ h T  i

y X b =???






 ∂b

 ∂ h T  T i


 b X y =???



 ∂b


 ∂ h T T i
b (X X)b =???



∂b
All qunatities in [·] are scalar, and the size of b is 2 × 1 in this example. Therefore we
know
yT X → 1 × 2
XT y → 2 × 1
XT X → 2 × 2

• With P = yT X, we can obtain the following by direct calculation:

∂ ∂
(P b) = f (b)
∂b ∂b
f (b) = p1 b1 + p2 b0

48
 Therefore
" #
∂ p
(Pb) = 1 = P = yT X
∂b p2

• Next for bT (XT y) = bT PT , bT PT = (P b)T = bP (because of scalar)

∂  T T ∂
b P = (Pb) = P = yT X
∂b ∂b
| {z }
from the
previous result

• Finally let us consider the case for bT Qb with Q = XT X.

– Since Q is a symmetric matrix, it can be expressed as

 
q1 q2 
Q =  
q2 q3 

– bT Qb can be expressed as
  
h i q1 q2  b1 
bT Qb = b1 b0    
q 2 q 3  b 0 
 
h i b1 
= b1 q1 + b0 q2 b1 q2 + b0 q3  
b0
= b1 q12 + b0 b1 q2 + b0 b1 q2 + b02 q3

– Therefore we can obtain the following:


∂
(bT Qb) = 2b1 q1 + 2b0 q2 = 2(b1 q1 + b0 q2 )



∂b



 1


 ∂
(bT Qb) = 2b1 q2 + 2b0 q3 = 2(b1 q2 + b0 q3 )



 ∂b0
    
∂ T q1 b1 + q2 b0  q1 q2  b1 
→ (b Qb) = 2   = 2 
q q  b  = 2 Qb
 
∂b q2 b1 + q3 b0  2 3 0

49
 • In summary
∂
 h  i
yT X b = XT y






 ∂b

 ∂
 ∂ SE
= −2 XT y + 2 XT Xb
 h  i
bT XT y = XT y →
∂b

 ∂b





 ∂ h i
bT (XT X)b = 2 (XT X)b



∂b

• Setting the derivative to zero

∂ SE
=0 → XT Xb = XT y
∂b

• Now XT X is a squre matrix (therefore invertible), the approximate solution b for least
SE can be found as

b = (XT X)−1 XT y

Here the matrix (XT X)−1 XT is called pseudo inverse matrix.

• This pseudo inverse matrix can be found in Matlab with “pinv(.)”.
• For the given problem, the line equation approximating the data is found to be

y = 1.0507 x − 0.1433

1 %% Least-Squre Error Method

2 clear all, close all, clc
3

4 %%
5 x = [1.2 2.3 3.0 3.8 4.7 5.9].';
6 y = [1.1 2.1 3.1 4.0 4.9 5.9].';
7

8 N = length(x);
9 A = [x ones(N,1)];
10 pinvA = inv(transpose(A)*A)*transpose(A);
11 del_pinvA = pinvA-pinv(A); % Check by ``pinv(.)”
12 del_pinvA
13

14 %%

50
 Figure 4.8: Example of least-square error method.

15 c = pinv(A)*y
16

17 xx = 1:0.01:6;
18 yy = c(1)*xx+c(2);
19 plot(x,y,'+',xx,yy)

1 del_pinvA =
2

3 1.0e-14 *
4

5 0.0139 -0.0167 -0.0132 -0.0003 0.0083 0.0139

6 -0.0111 0.0833 0.0611 -0.0028 -0.0555 -0.1110
7

8 c =
9

10 1.0507
11 -0.1433

51
Orthogonal Least Squares

• There is a way to define SE not in the previous way but a new way called orthogonal
SE.
• The fig. shows well the related concept, and this method give more optimal from the
other viewpoint.

4.4 Richardson Extrapolation

52
Chapter 5

Numerical Integrations

5.1 Definite Integral

Consider the following definite integral of some integrand:

Z b
f (x) dx =???
a

If f (x) is continuous, and F(x) is an antiderivative of f (x)

Z b
f (x) dx = F(b) − F(a)
a

There may be a case that the closed form of the antiderivative does not exist or at least is
not known to humankind. Nevertheless, if you have to know such an integrand, you have
no way without relying on an approximation.

Upper and Lower Sums

Often We will review the definition of the Riemann integral of a function. A partition of an
interval [a, b] is a finite, ordered collection of nodes xi :

a = x0 < x1 < x2 < · · · < xN = b

53
 1 1

(a) The Lower Sum (b) The Upper Sum

Figure 5.1: Lower sum and upper sum of integral.

( )

( )

Figure 5.2: Integral with trazoidal rule.

Given such a partition, let Ln and Un be the lower and upper bounds on the n-th subinterval
[xn , xn+1 ]. Then the upper and lower sums are defined as

N
X −1
L(f ) = Ln (xn+1 − xn )
n=0
N
X −1
U (f ) = Un (xn+1 − xn )
n=0

Reasonable approximation of the integration becomes

Z b
1
f (x) dx = [L(f ) + U (f )]
a 2

54
Trapezoidal Rule

For the following integral

Z b
f (x) dx =???
a

Trapezoidal rule for approximation of the integral is

Z b
f (a) + f (b)
f (x) dx ≃ (b − a)
a 2

With equal N -subsection having width ∆x, the composite trapezoidal rule yields a simplied
form:

Z b N −1
∆x X h i
f (x) dx ≃ f (xn ) + fn+1
a 2
n=0
∆x
 h i
= f (x0 ) + f (xN ) + 2 f (x1 ) + f (x2 ) + · · · + f (xN −1 )
2

It is worthwhile to know that the resulting intrgral can be expressed as a linear expansion
with the function value at given points !!!

N
X
I≃ an f (xn )
n=0

Simpson’s Rule

If a given integrand is approximated with the following quadratic equation, the its integral
can be found as

f (x) ≃ f (x) ≃ c0 + c1 x + c2 x2
Zb
c   c  
f (x) dx = c0 (b − a) + 1 b2 − a2 + 2 b3 − a3
a 2 3

55
 Figure 5.3: Integral with Gaussian quadrature.

where the coefficients c0 , c1 and c2 can be found from the following three data points as
" !#
a+b a+b
[a, f (a)] , ,f , [b, f (b)]
2 2
!
a+b
a2 f (b) + b2 f
(a) + ab [f (a) + f (b)] − 4ab f
2
c0 = 2
(b − a)
!
h i a+b
a f (a) + bf (b) + 3 af (b) + bf (a) − 4(a + b) f
2
c1 =
(b − a)2
" !#
a+b
2 f (a) + f (b) − 2f
2
c2 =
(b − a)2

Substituting the above c0 , c1 , c2

" ! #
b−a a+b
I≃ f (a) + 4f + f (b)
6 2

With h = (b − a)/2, the well-known Simpson’s 1/3 rule is obtained:

" ! #
h a+b
I≃ f (a) + 4f + f (b)
3 2

This is also known to be expressed as a linear expansion similar as the Trapzoidal rule.

56
Gaussian Quadrature
Meaning of Quadrature in Dictionary (Oxford Dictionary):

• Mathematics: The process of constructing a square with an area equal to that of a

circle, or of another figure bounded by a curve.
(cf) Mensuration (Qudrature) by parts
• Astronomy: The position of the moon or planet when it is 90◦ from the Sun as seen
from the earth.
• Electronics: A phase difference of 90◦ between two waves of the same frequency, as in
the color difference signals of a television screen.

Quarature in integral stands for means a approximate integral of a function as the linear
combination of the function at certain points:
Z b
f (x) dx ≃ a0 f (x0 ) + a1 f (x1 ) + · · · +aN f (xN )
a

for some collection of nodes {xn }, and weights {an }.

Let us consider the case in Fig. 5.4 (a). Because the trapezoidal rule must pass through the
end points, there are cases where the formula results in a large error. Now, suppose that
the constraint of fixed base points was removed and we were free to evaluate the area under
a straight line joining any two points on the curve. By positioning these points wisely, we
could define a straight line that would balance the positive and negative errors as shown in
Fig. 5.4(b). Therefore, we can get an improved estimate of the integral.

Gaussian quadrature is the name for one class of techniques for implementing such a strat-
egy. The particular Gaussian quadrature formulas described in this section are called Gauss-
Legendre formulas, and will be described later in detail.

Determination of Parameters of Gaussian Quadrature

Let us consider the following trapzoidal approximation:
b − ah i
I= f (a) + f (b)
2
Here we can rewrite this result in terms of new coefficients c0 and c1 as

I = c0 f (a) + c1 f (b)

57
 Figure 5.4: Algorithm of Gaussian quadrature.

b−a
c0 = c1 =
2

For the Gaussian quadrature, c0 , and c1 are determined to provide more accurate integration
results.

1) 2-Point Formula

• Just as was the case for the above derivation of the trapezoidal rule, the object of Gauss
quadrature is to determine the 4 unknowns (2 appropriate points and 2 coefficients) in
the following equation for much more accurate interal:

Z b
I= f (x) dx ≃ c0 f (x0 ) + c1 f (x1 ) · · · (8.3.2)
a

Thus we require 4 conditions to determine them exactly.

• Let us consider the integral of the function given in [−1, 1] for the simple derivation of the
parameters of the Gaussian quadrature.
• In this case, we have to determine two ponts x0 and x1 between −1 and 1, and the associ-
ated coefficients c0 and c1 .

58
• Without the loss of generality (reasonable selection), we can choose

x1 = −x0
c1 = c0

and the resulting expression of the integral becomes

Z b
I= f (x) dx ≃ 2 c1 f (x1 )
a

This implies there are only two unknowns to be determined, and therefore two indepen-
dent conditions are sufficient for this problem.
• For this purpose, consider the following two meaningful integrands:

1) f (x) = 1
(5.1)
2) f (x) = x2

We have to know that the integrand x becomes meaningless !!!

• The two equations to be solved are


 Z 1




 1) 2c1 1 = 1 dx = 2

 −1
1
 Z
2


2c1 x12 x2 dx =

2) =


3

−1

c = c0 = 1
 1



→ 1
x1 = −x0 = √



3

• Therefore the 2-point Gaussian quarature formula becomes

! !
−1 1
I = f √ +f √
3 3
√ √
This is interesting. Simple addition of the function values at x0 = −1/ 3 and x1 = 1/ 3
yields an integral estimate that is 3rd-order accurate.

Transform of integration region from [a, b] to [−1, 1]

59
 g(y)
y
b
a b y

f(x) a

-1 1 x -1 0 1 x
Figure 5.5: Transform for Gaussian quadrature.

• A simple change of variable can be used to translate other limits of integration into
this form.
Zb Z1
g(y) dy = f (x) dx → f (x) =???
a −1

b−a a+b b−a

y= x+ dy = dx
2 2 2
Zb Z1
b−a
g(y) dy = g(y) dx
a −1 2
| {z }
f (x)

b−a
→ f (x) = g(y)
2

Ex. Two-Point Gauss-Legendre Formula:

• Evaluate the following integral

Z 0.8
I= g(y) dy
0
g(y) = 0.2 + 25y − 200y 2 + 675y 3 − 900y 4 + 400y 5

Iexact = 1.640533

Solution

60
 • The intgeral can be transformed as

Z 1
I= f (x) dx
−1
b−a
f (x) = g(y)
 2 
= 0.2 + 25y − 200y 2 + 675y 3 − 900y 4 + 400y 5 × 0.4

b−a a+b
y= x+ → y = 0.4x + 0.4
2 2

• The integral by the Gaussian Quadrature is

! !
−1 1
IGQ = f √ + f √
3 3
!
−1

f √ = 0.516741





 3
→ IGQ = 1.822578

 !
1


f √ = 1.305837



3

which represents a percent relative error of 11.1 %.

• This result is comparable in magnitude to a four-segment application of the trape-
zoidal rule (Table 21.1) or a single application of Simpsons 1/3 and 3/8 rules (Exam-
ples 21.4 and 21.6).
• This latter result is to be expected because Simpson’s rules are also third-order accu-
rate. However, because of the clever choice of base points, Gauss quadrature attains
this accuracy on the basis of only two function evaluations.

2) 3-Point Formula

• Similar way, the 3-point formula can be found with setting x0 = −x2 and a0 = a2 . In
addition, it is reasonable to let x1 = 0, resulting in a problem with only 3 unknowns as

h i
IGQ = c0 f (x0 ) + c1 f (x1 ) + c2 f (x2 ) = c1 f (0) + c2 f (x2 ) + f (−x2 )

61
 Now applying the constraints:



 f (x) = 1 → c1 + 2c2 = 2

 f (x) = x2 → 2 c x2 = 2




2 2


 3

 2
 f (x) = x4 → 2 c2 x24 =



5
r
2/5 3 3
→ x22 = = → x2 = = 0.7746
2/3 5 5
5 8
→ c2 = = 0.5556 → c1 = = 0.8889
9 9

Ex Use the 3-point formula from Table 22.1 to estimate the integral for
the same function as in previous Example.
Solution

• According to Table 22.1, the three-point formula is

IGQ = 0.5555556 f (20.7745967) + 0.8888889 f (0)
+ 0.5555556 f (0.7745967)
= 0.2813013 + 0.8732444 + 0.4859876 = 1.640533
which is found to extremely accurate.

3) 4-Point Formula

• Equation of 4-point formula (4 unknows c−1 , c2 , x1 , x2 )

h i h i
IGQ = c1 f (x1 ) + f (−x1 ) + c2 f (x2 ) + f (−x2 )
Now applying the constraints:


 f (x) = 1 → c1 + 2c2 = 1
1


f (x) = x2 → c1 x12 + c2 x22 =



3







 1
4
 f (x) = x →
 c1 x14 + c2 x24 =
5






 1
f (x) = x6 → c1 x16 + c2 x26 =





 7

62
Figure 5.6: Coefficients for Gaussian quadrature.

 r √

 3 2 30
x1 = − ≃ 0.3400






 7 35
√

 r

3 2 30



x2 = 7 + 35 ≃ 0.8611



→



 √

 30
c1 = 1/2 + ≃ 0.6521





 36
√



30



c2 = 1/2 − ≃ 0.3479


36

63
4) Higher-Point Formulas
• Values for c′ s and x′ s for up to 6-
point formula are summarized
in Table 22.1

5.2 Infinite-Region Integral

• Let us consider the following integral:
Z ∞
I1 = f (x) dx
−∞

This can be rewritten as

Z0 Z ∞ Z ∞h i
I1 = f (x) dx + f (x) dx = f (−x) + f (x) dx
−∞ 0 0

With the following change of variables:

1−t
x= = (1 − t) t −1 → x ∈ [0, ∞] → t ∈ [1, 0]
t
−1
dx = −t −1 − (1 − t)t −2 = −2
t
Z 0 
1−t 1−t −1
    
I1 = f +f dt
1 t t t2
Z 1 
1−t 1−t 1
    
= f +f dt
0 t t t2

Now this integral can be calculated numerically !!!

• The following semi-infinite integral can be handled with the change of variables x =
a + (1 − t)/t
Z∞
I2 = f (x) dx
a

5.3 Improper Integral

64
Chapter 6

Numerical Solution of Differential and

Integral Equations

6.1 Ordinary Differential Equations

• Ordinary Differential Equations (ODEs) are used in approximating some physical sys-
tems. Some classical uses include simulation of the growth of populations and trajectory
of a particle. They are usually easier to solve or approximate than the more general Partial
Differential Equations (PDEs) which can contain partial derivatives.
• Much of the background material of this chapter should be familiar to the student of cal-
culus. We will focus more on the approximation of solutions, than on analytical solutions.
For more background on ODEs, see any of the standard texts.
• Let us consider the solution x(t) satisfying the following 1-D ODE:

 dx(t)

 = f (t, x(t))
dt



x(a) = c


• For example with f (t, x(t)) = t 2 − t, the solution is known to be

1 3 1 2
x(t) = t − t +K
3 2
where K is chosen such that x(a) = c.

65
 Figure 6.1: Example of Euler method.

1) Euler’s Method (1st-Order Taylor’s Series Method)

• We attempt to solve the above ODE by integrating both sides. That is

dx(t)
= f (t, x(t)) → dx = f (t, x) dt
dt
yields
Z t+∆t Z t+∆t
dx = f (t, x(t)) dt
t t

thus
Z t+∆t
x(t + ∆t) = x(t) + f (t, x(t)) dt
t

• If we can approximate the integral then we have a way of ‘stepping’ from t to t + ∆t, i.e.,
if we have a good approximate of x(t) we can approximate x(t + ∆t). Note that this means
that all the work we have put into approximating integrals can be used to approximate
the solution of ODEs.

• Using stepping schemes we can approximate x(tfinal ) given x(tinitial ) by taking a number of
steps. For simplicity we usually use the same step size, ∆t, for each step, though nothing
precludes us from varying the step size.

66
• If we apply the left-hand rectangle rule for approximating integrals to Eq. 10.2, we get

x(t + ∆t) ≃ x(t) + ∆t f (t, x(t))

| {z }
Area of rectangle

• This is called Euler’s Method (1st-order Taylor series method). Note this is essentially the
same as the forward difference approximation we made to the derivative, Eq. 7.1.

Ex. 10.1 Consider the ODE

dx(t)

 dt = x(t)





x(0) = 1


Solution The actual solution is x(t) = et .

• Euler’s Method will underestimate x(t) because the curvature of the actual x(t) is positive,
and thus the function is always above its linearization. In Fig. 10.1, we see that eventually
the Euler’s Method approximation is very poor.
• For expecting better accuracy of integration f (t, x) from t to t + ∆t, the Trapezoid rule can
be applied as:
∆t h i
x(t + ∆t) ≃ x(t) + f (t, x(t)) + f (t + ∆t, x(t + ∆t))
2
However we cannot evaluate this exactly, since x(t +∆t) appears on both sides of the equa-
tion.
• Nevertheless it can be caculated with the help of appropriate approximation, and this is
the idea behind the Runge-Kutta Method, which we will study later.

Application to Dynamic Electric Circuit Analysis

A Series RC Circuit

• Applying the Kirchoff Voltage Law (KVL) to the given RC circuit with vc (0) = 0,

dvc (t)

v s (t) = i(t) R + vc (t)
vs (t) = Tc dt + vc (t)

 

 
→ 


 dvc (t) 

i(t) = C v (0) = 0

c
dt


where Tc = RC is called a Time Constant of series RC circuit.

67
 + vs(t)
vs(t) vc(t) 1
-

0 t

Figure 6.2: Numerical solution of RC circuit.

• Applying the Lapalce transform to find the exact unit-step (US) response of vc (t),

Vs (s) = Tc s Vc (s) + Vc (s)

!
US 1 1
Vc (s) = V (s), Vs (S) =
s Tc + 1 s s
" #
US 1 1 1
Vc (s) =
Tc s + 1/Tc s

With a partial fraction,

" #
1 1 a b
= + , a = Tc , b = −Tc
s + 1/Tc s s s + 1/Tc
1 1
VcUS (s) = −
s s + 1/Tc

Therefore unit step response of vcUS (t) can be found as

 
vcUS (t) = 1 − e−t/Tc u(t)

• For a pulse input as shown in Fig. XXX, the response of vc (t) therefore becomes

vc (t) = vcUS (t) u(t) − vcUS (t − τ) u(t − τ)

• Let us express the governing equation in the form of general DE:

dvc (t) 1 1
+ vc (t) = vs (t)
dt Tc Tc

68
 • By the Euler method with approximate difference formula of the derivative,

dvc (t) v (t) − vc (t − ∆t)

→ c
dt ∆t
∆t ∆t
vc (t) − vc (t − ∆t) + δvc (t) = δ vs (t), δ= =
Tc RC
(1 + δ) vc (t) = vc (t − ∆t) + δ vs (t)

• Reaaranging this equation yields the following final update equation:

1
vc (t) = a vc (t − ∆t) + b vs (t), with a = , b = 1−a
1+δ

Ex For the given series RC circuit with

R = 50 Ω, C = 1 µF, V0 = 1, τ = 50 µs

calculate vc (t) with Euler’s method and plot the output waveform for −Tc ≤ t ≤ 5 Tc . Please
try the various time steps ∆t, and compare the results with the exact one.

B Series RLC Circuit

• Applying the Kirchoff Voltage Law (KVL) to the given RLC circuit with vc (0) = 0 and
iL = 0,
di(t)

vs (t) = L dt + R i(t) + vc (t)




dv (t)

i(t) = C c



dt
dvc2 (t) dv (t)

v (t) = LC + RC c + vc (t)


 s


 dt dt
v (0) = 0,

i (0) = 0
c L

Rearranging

d 2 vc (t) R dvc (t) 1 1

+ + vc (t) = v (t)
dt 2 L dt LC LC s

• We have already studied an analytic solution.

• Here, we will find a numerical solution.

69
 • Let us express the derivative with the Euler method:

dvc (t) v (t) − vc (t − ∆t)

→ c
dt ∆t
d 2 vc (t) dvc (t)/dt − dvc (t − ∆t)/dt
→
dt 2 ∆t
v (t) − vc (t − ∆t) − vc (t − ∆t) + vc (t − 2∆t)
= c
∆t 2
With some algebraic manipulations

avc (t) − bvc (t − ∆t) − cvc (t − 2∆t) = vs (t)

b c 1
vc (t) = vc (t − ∆t) + vc (t − 2∆t) + vs (t)
a a a
RC LC RC 2LC −LC
a = 1+ + 2, b= + 2
, c=
∆t ∆t ∆t ∆t ∆t 2
→ a − (b + c) = 1 → a = 1 + (b + c)

Ex For the given series RLC circuit with

R = 50 Ω, C = 1 µF, V0 = 1

and L satisfying ζ = 0.25, 1, 5, calculate the unit-step response of vc (t) with Euler’s method
and plot the output waveform for −Tc ≤ t ≤ 20 Tc . Please try the various time steps ∆t, and
compare the results with the exact one.

1 clear all, close all, clc

2

3 R = 50;
4 C = 1e-6;
5 stype = 1; % Source Type, 0: Unit Step, 1: Pulse
6 Tw = 100e-6; % Pulse Width
7 %%
8 Tc = R*C; % Time constant
9 Ts = Tc/100; % Sample Time
10 t = -Tc:Ts:5*Tc;
11 N = length(t);
12 if stype == 0 % Unit Step
13 Tw = max(t);
14 end

70
 Figure 6.3: Result of the numerical solution of the RC circuit.

15 %% Euler Method
16 Vc = zeros(1,N); Vs = zeros(1,N);
17 ii1 = find((t>0)&(t<=Tw));
18 Vs(ii1) = 1;
19

20 a = Ts/Tc; b1 = 1/(1+a); b2 = a/(1+a);

21 for ii=2:N
22 Vc(ii) = b1*Vc(ii-1)+b2*Vs(ii);
23 end
24 plot(t,Vs,t,Vc), axis([-Tc max(t) -0.5 1.5])
25 grid on

1 %% Euler Method of Series Dynamic RLC Circuit

2 % with Unit Step Source
3 % Output : Vc(0) = 0, IL(0) = 0
4

5 clear all, close all, clc

6

7 R = 50;
8 C = 1e-6;
9 zeta = 5;
10

71
 vs(t)

0 t

Figure 6.4: Unit step input to series RLC circuit.

11 %%
12 L = C/((2*zeta/R)^2)
13 Tc = R*C/zeta;
14 Ts = Tc/20; % Sample Time
15 t = -Tc:Ts:20*Tc;
16 N = length(t);
17

18 %% Euler Method
19 a = 1+R*C/Ts+L*C/Ts^2; % Time constant
20 b = R*C/Ts+2*L*C/Ts^2;
21 c = -L*C/Ts^2;
22

23 Vc = zeros(1,N);
24 Vs = zeros(1,N);
25 ii1 = find(t>0);
26 Vs(ii1) = 1;
27

28 for ii=3:N
29 Vc(ii) = (b/a)*Vc(ii-1)+(c/a)*Vc(ii-2)+(1/a)*Vs(ii);
30 end
31

32 plot(t,Vs,t,Vc), axis([-Tc max(t) -0.5 1.5])

33 grid on

Higher-Order Taylor’s Series Methods

• We see that our more accurate integral approximations will be useless since they require
information we do not know, i.e., evaluations of f (t, x) for yet unknown x values. Thus

72
 =5 =1 = 0.25

Figure 6.5: Result of numerical solution of RLC circuit.

we fall back on Taylor’s Theorem (Theorem 1.1). We can also view this as using integral
approximations where all information comes from the left-hand endpoint.
• By Taylor’s theorem, if x has at least m + 1 continuous derivatives on the interval in ques-
tion, we can write
∆t 2 ′′ ∆t m (m)
x(t + ∆t) = x(t) + ∆t x′ (t) + x (t) + · · · + x (t)
2 m!
∆t m+1 (m+1)
+ x (τ)
(m + 1)!
where τ is between t and t + ∆t. Since τ is essentially unknown, our best calculable ap-
proximation to this
′ ∆t 2 ′′ ∆t m (m)
x(t + ∆t) ≃ x(t) + ∆t x (t) + x (t) + · · · + x (t)
2 m!
• This approximate solution to the ODE is called a Taylor’s series method of order m. We
also say that this approximation is a truncation of the Taylor’s series.
• The difference between the actual value of x(t + ∆t), and this approximation is called the
truncation error. The truncation error exists independently from any error in computing
the approximation, called round-off error.

Ex. Derive the Taylor’s series method for m = 3 for the following ODE.
dx(t)

x
 dt = x + e





x(0) = 1


73
Solution ] The update equation can be found with simple calculus:

x(1) (t) = x + ex
x(2) (t) = x(1) + ex x(1) = x(1) (1 + ex )
x(3) (t) = x(2) + ex (x′ )2 + ex x(2) = ex (x′ )2 + x(2) (1 + ex )

(1) ∆t 2 (2) ∆t 3 (3)

x(t + ∆t) ≃ x(t) + ∆t x (t) + x (t) + x (t)
2 6

Errors

• There are two types of errors: truncation and round-off.

– Truncation error is the error of truncating the Taylor’s series after the m-th term. You
can see that the truncation error is O(∆t m+1 ).
– Round-off error occurs because of the limited precision of computer arithmetic.
• While it is possible these two kinds of error could cancel each other, given our pessimistic
worldview, we usually assume they are additive.
• We also expect some tradeoff between these two errors as ∆t varies. As ∆t is made smaller,
the truncation error presumably decreases, while the roundoff error increases. This re-
sults in an optimal ∆t-value for a given method. See Fig. 10.2.
• Unfortunately, the optimal h-value will usually depend on the given ODE and the ap-
proximation method. For an ODE with unknown solution, it is generally impossible to
predict the optimal ∆t. We have already observed this kind of behavior, when analyzing
approximate derivatives (cf. Ex. 7.5.)
• Both of these kinds of error can accumulate. Since we step from x(t) to x(t + ∆t), an error
in the value of x(t) can cause a greater error in the value of x(t + ∆t). It turns out that for
some ODEs and some methods, this does not happen. This leads to the topic of stability.

2) Runge-Kutta Methods
Martin Kutta (1867-1944), engineer and mathematician, introduced the method in his PhD thesis, which was
developed further by Carl Runge (1856-1927), physicist and mathematician.

2nd-Order Runge-Kutta Methods

74
• Recall that the ODE problem is to find some x(t) such that

 dx(t)

 = f (t, x(t))
dt



x(a) = c


where f , a, c are given.

• Even though the Taylor’s series method can be used for this purpose, it has problems:

– The first-order method (Euler’s Method) suffers inaccuracy.

– Even a higher-order method can provide a more accurate result, but many evaluations
of required higher-order derivatives of x(t) makes it less useful.

• The Runge-Kutta Method is known to be powerful to resolve this problems. We sup-

pose that α1 , α2 , ω1 and ω2 are fixed constants. We then compute:
K1 ← f (t, x) ∆t
K2 ← f (t + ∆t, x + K1 ) ∆t
Then we approximate:
x(t + h) ← x(t) + ω1 K1 + ω2 K2 .
| {z }
integral concept

• We now examine the proper choices of the constants. Note that ω1 = 1, ω2 = 0 cor-
responds to Euler’s Method, and does not require computation of K2 . We will pick
another choice.
• Also notice that the definition of K2 should be related to Taylor’s theorem in two di-
mensions. Let’s look at it:
K2
= f (t + α∆t, x + βK1 )
∆t  
= f t + α∆t, x + β∆t f (t, x)
(1) (1)
= f + α∆t ft (t, x) + β∆t f fx (t, x) + O(∆t 2 )

• Now reconsider our step:

x(t + ∆t) = x(t) + ω1 K1 + ω2 K2
 
2 (1) (1)
= x(t) + w1 ∆tf + ω2 ∆t f + α∆t ft + β∆t f fx + O(∆t 3 )
2

 
(1) (1)
= x(t) + (ω1 + ω2 )∆t x + ω2 ∆t αft + β f fx + O(∆t 3 )
′ 2

75
 • The resulting expression of x(t + ∆t), choosing the following constants, becomes sim-
plified as
1
ω1 + ω2 = 1, αω2 = = βω2
2
1 2 (1)
 
(1)
x(t + ∆t) = x(t) + ∆t x + ∆t ft + f fx + O(∆t 3 )
′
2
1
= x(t) + ∆t x′ + ∆t 2 x′′ + O(∆t 3 )
2
i.e., our choice of the constants makes the approximate x(t + ∆t) good up to a O(∆t 3 )
term, because we end up with the Taylor’s series expansion up to that term.
• The usual choice of constants is α1 = α2 = 1, ω1 = ω2 = 1/2. This gives the 2nd-order
Runge-Kutta Method:

∆t ∆t  
x(t + ∆t) ← x(t) + f (t, x) + f t + ∆t, x + ∆tf (t, x)
2 2
• This can be written (and evaluated) as

K1 ← ∆t f (t, x)
K2 ← ∆t f (t + ∆t, x + K1 )
1
x(t + ∆t) ← x(t) + (K1 + K2 )
2

Ex ] Consider the ODE:

 2
′ = (tx)3 − x



x
t





x(1) = 1


Use ∆t = 0.1 to compute x(1.1) using both Taylor’s Series Methods and the 2nd-order Runge-Kutta
methods.

4-th order Runge-Kutta Method

• The Runge-Kutta Method of order two has error term O(∆t 3 ). Sometimes this is not enough
and higher-order Runge-Kutta Methods are used.

76
• Among them, the 4-th-order Runge-Kutta method is known to be very accurate and effi-
cient. It is however too complex to derive its formula (out of scope of our study), and it
may be sufficient to use the final formula to our practical example.
• The following is the final formula of the 4-th order Runge-Kutta Method:

K1 ← ∆t f (t, x)
!
∆t 1
K2 ← ∆t f t + , x + K1
2 2
!
∆t 1
K3 ← ∆t f t + , x + K2
2 2
K4 ← ∆t f (t + ∆t, x + K3 )

• With the obtained K’s, x(t + ∆t) can be found from x(t) as
1
x(t + ∆t) = x(t) + [K + 2(K2 + K3 ) + K4 ]
6 1
1 1 1 1
= x(t) + K1 + K2 + K3 + K4
6 3 3 6
| {z }
Simpson’s Rule

The added terms in the final formula is same as that of the Simpson’s Rule in numerical
integrals.
• This method has order O(∆t 5 ).

6.2 Partial Differential and Integral Equations

1) FDM (Finite Difference Method)

• 2D difference approximation
∂2 V ∂2 V
∇2 V = + =0
∂x2 ∂y 2
∂V V (x + ∆x/2) − V (x − ∆x/2)
= V ′ (x) →
∂x ∆x
2
∂ V ′ ′
V (x + ∆x/2) − V (x − ∆x/2)
2
→
∂x ∆x
V (x + ∆x) − 2V (x) + V (x − ∆x)
=
∆x2

77
 +

− +

Figure 6.6: FDM solution of differential equation.

If ∆x = ∆y = h,

∂2 V ∂2 V
∇2 V = + =0
∂x2 ∂y 2
[V (x + h, y) − 2V (x, y) + V (x − h, y)]
→
h2
[V (x, y + h) − 2V (x, y) + V (x, y − h)]
+ =0
h2
1h i
V (x, y) = V (x + h, y) + V (x − h, y) + V (x, y + h) + V (x, y − h)
4

HW FDM Matlab program

Matlab programming with a = 10 cm, b = 5 cm, and h = 1 cm

• For the node 1

5 + V2 + 0 + V5 − 4V1 = 0
→ −4V1 + V2 + 0 + V5 = −5

• For the node 6

V5 + V7 + V2 + V10 − 4V6 = 0
→ V2 + V5 − 4V6 + V7 + V10 = 0
78
 y
0V
b
1 2 3 4 2 V2

V0 5 6 7 8
0V V5 V6 V7
9 10 11 12 5 6 7
h
0 x V10 10
0 0V a h

Figure 6.7: FDM solution example of differential equation.

A V = Vs → V = A−1 Vs
 
−4 1 0 0 1 0 0 0 0 0 0 0 
 .. 
.
 
 
 
A =  0 1 0 0 1 −4 1
 1 0 0 
 .. 
.
 
 
 
0 0 0 0 0 0 0 1 0 0 1 −4

   
 V1  −5
 .   . 
 ..   .. 
   
V =  V6  , Vs =  0 
   
 .   . 
 ..   .. 
   
V12 0
   

2) Method of Moments

• Up to now we have assumed a uniform charge distribution in conducting plates of a ca-

pacitor, and then we can find electric field distribution and capacitance. However, the size

79
 Source points
z y
Field points

x
1 2 M

+Q

-Q 2 W
1
L
Figure 6.8: MOM solution example of differential equation.

of a capacitor is finite, and the real distrbution is different to the assumed. For this case,
some numerical approaches can give us a solution even though it is not a closed form.
• Here one numerical approach called moment method (or method of moments) is studied.
The followings are the steps for our purpose:

– The conducting plate is divided into small rectanglular section as

L W
dx = , dy =
M N
i, m = 1 : M, j, n = 1 : N
Source points Field points
xsm = (m − 0.5) dx xi = (i − 0.5) dx + δx
ysn = (n − 0.5) dy yj = (j − 0.5) dy + δy
±d d
zs± = z=
2 2

– Linear exapansion of the charge distribution on the upper plate with basis functions
and unknown coefficients:

80
 R
X
qs = qsl δ(x − xsm , y − ysn )
l=1
l = (m − 1)N + j
qsl : point charge at the l-th index

– The charge distribution in the lower plate has the same but different sign.
– With an assumed potential difference V0 , the potential on the upper plate becomes
V + = V0 /2.

– Calculate the potential Vk at the reasonably chosen points on the upper plate:

k = (i − 1)N + j, l = (m − 1)N + n
R !
+ 1 X 1 1
Vk = al + − −
4πϵ0 Rkl Rkl
l=1
q
R+kl = (xi − xm )2 + (yj − yn )2
q
R−kl = (xi − xm )2 + (yj − yn )2 + d 2

V+R×1 = FR×R QR×1

!
1 1 1
Fkl = + − −
4πϵ0 Rkl Rkl

Q = F−1 V+
X Qsum
Qsum = Q → C=
V0

3) Finite Element Method (FEM)

81
82
Chapter 7

Random Number

Basic Probability and Related Parameters

• Let us consider a trial of throwing one dice. There are definitely 6 events (Different faces).
• If a variable x is the number of dots on the top surface, the following table shows the
related event, variable x, and probability of each event:

Event S1 S2 S3 S4 S5 S6
Variable x 1 2 3 4 5 6
PX (x) 1/6 1/6 1/6 1/6 1/6 1/6

• The event takes place randomly, therefore, the variable x is called Random Variable
(RV). However, for the time being, we call it Random Number (RN).

Mean and Variance

• Now let us consider the expectation (mean, average) and variance of x for one-time
throwing.

1) Expectation (or mean, average) notated as E[X] (or X, µX ) becomes

1 1 1
X = 1× + 2 × + · · · + 6 × = 3.5
6 x2 6 6
PX (x2 )

From this, we can know that the expectation can be expressed as

83
 PX(x)

1/6

0 1 6 x

Figure 7.1: Dice throwing experiment.

6
X
X= xn PX (xn )
n=1

2) Variance, notated as Var[X], (σX2 ), is given as

2 2
σX2 ≜ (X − X)2 = X 2 − 2X X + X = X 2 − 2X X + X
2
= X2 − X

Therefore the variance of X can be calculated (in two ways) as

1 h i
σX2 = (1 − 3.5)2 + (2 − 3.5)2 + · · · + (6 − 3.5)2 = 2.9167
6
1 h i
σX2 = 12 + 22 + · · · + 62 − 3.52 = 2.9167
6

The formula of variance with PDF can be written as

6 6
2
X X
2 2
σX = (xn − x) PX (xn ) = xn2 PX (xn ) − X
n=1 n=1

cf) Weighted sample variance (unbiased variance) = ???

84
 fX(x) Area = Probability
for interval x

Total Area = 1

0 x
x x+ x
Figure 7.2: Histogram to PDF conversion.

Continuously Distributed RN
• Let us consider one continupusly
distributed random number X.
• The probability of event from the
lowest to x is called the Cumulative
Distribution Function (CDF) which
is denoted by FX (x):

FX (x) = P (X ≤ x)

Therefore the probability that X lies in the semi-closed interval (a, b] can be written as

P (a < X ≤ b) = FX (b) − FX (a)

• If we let P∆x (x) as the probability of event from x to x + ∆x with a very small interval ∆x:

P∆x (x) = P (x + ∆x) − PX (x)

then the Probability Density Function (PDF) at x, fX (x) is defined as

P∆x (x) P (x + ∆x) − PX (x)

fX (x) = lim = lim
∆x→0 ∆x ∆x→0 ∆x
P (x)
= X
dx

85
 (1/6) (x-3)

q
1/6

0 1 6 x 0 1 x

1
1

1/6
0 1 6 x 0 1 x
(a) (b)

Figure 7.3: Relation of PDF and CDF.

Therefore the CDF PX (x) can be expressed as

Zx
PX (x) = fX (ξ) dξ
−∞

• Since the total pabability of event becomes 1,

Z∞
PX (∞) = fX (x) dx = 1
−∞

• Let us return to our 1st example of throwing a dice, we know the CDF beomes as

Variable x < 1 [1, 2) [2, 3) [3, 4) [4, 5) [5, 6) >6

FX (x) 0 1/6 2/6 3/6 4/6 5/6 6/6 = 1

and the PDF can be expressed with the help of Dirac Delta Function as
6
X 1
fX (x) = δ(x − n)
6
n=1

86
• Let us consider a uniform distributed RN on x ∈ [0, 1] (Fig. XXX), whose PDF can be
written as

q, 0 ≤ x ≤ 1


fX (x) = 
0, elsewhere


The unknown q can be found from the fact that the total probability should be 1:
Z1 Z1
fX (x) dx = q dx = 1 → q = 1
0 0

Now the CDF can be calculated as





0, x ≤ 0

PX (x) = x, 0 ≤ x ≤ 1




1, x ≥ 1


• The mean and variance of x can be found as

Z∞ Z1
X= x fX (x) dx = x dx
−∞ 0
1
=
2
Z ∞ Z 1
2 1
σx2 = (X − X)2 = 2
x fX (x) dx − x = x2 dx −
−∞ 0 4
1
=
12
Therefore the standard deviation becomes
q
1
σx = σx2 = √
2 3
− Let us generate this uniform RN by Matlab programming:

Uniform Random Number

• Uniform Random number in [0,1]

– Matlab built-in function “rand(N)”

• One alogorithm - Linear Congruential Method

87
 – Inputs

Number of data: N



Seed (Initial Condition): Z0


– Parameters and Algorithm




m = 2Nb : Maximum Allowable Period





Nb : Number of bits

a = 2Int(Nb /2) + 3




Zn+1 = mod(aZn , m)




Z



ux =

m

Estimation of the Value of π

• Let us assume that we do not know the value of π. Then is there a way estimating its value
with the help of random number ???

– Let us consider a unit circle and the square perfectly touch this circle as shown in Fig.
XXX.
– We can write the areas of the unit circle and square

Auc = π, Asq = 4

– At first let us generate N -times trials of two uniform random numbers in [−1, 1], and
placing its pair into the xy-plane.
– Next calculate their distances from the origin, and counting the points Nin whose dis-
tance meets the following condition:
q
xi2 + yi2 ≤ 1

– Now the ratio and the value of π can be estimated as

Auc Nin Nin
 
= → π≃4
Asq N N

– We can estimate π with different N , and the result will be close to the announced π
with increasing N .

88
 – Let us check this with Matlab simulation:

1 clear all, close all, clc

2

3 NN = 2.^([Link]);
4 th = ([Link])*pi/180;
5 for ii=1:length(NN)
6 N = NN(ii);
7 xn = 2*rand(1,N)-1;
8 yn = 2*rand(1,N)-1;
9 rn = sqrt(xn.^2+yn.^2); % Distance from the origin
10 ii1 = find(rn<=1);
11 Nin = length(ii1); % No. of points inside unit circle
12 PI(ii) = Nin/N*4; % Estimation of
13 if ii==1
14 subplot(2,1,1)
15 plot(xn,yn,’b+’), hold on, plot(cos(th),sin(th),'r'),
16 axis([-1.5 1.5 -1.5 1.5]), axis equal
17 pause(1)
18 end
19 end
20 subplot(2,1,2)
21 plot(NN,PI)

One alogorithm - Linear Congruential Method

• Inputs

Number of data: N



Seed (Initial Condition): Z0


• Parameters and Algorithm




m = 2Nb : Maximum Allowable Period


Nb


 : Number of bits

a = 2Int(Nb /2) + 3




Zn+1 = mod(aZn , m)




Z



ux =

m

89
 Figure 7.4: Calculation of π-value with random number.

Histogram to PDF Conversion

• Measurement of signal level to find PDF

– Histogram
– Normalization for all probability being 1.
• In statistics, histograms are widely used. For an N -times trials, we can count the number
of events Mn in the n-th small cell (interval) of the variable x. Repeating this for all evenly
spaced cells, the histogram having the following property is constructed:

h = {M1 , M2, · · · , MN }
N
X
Mn = N
n=1

90
• The approximate probability in the specific cell can be expressed as
Mn
Pn = ≃ fX (x)∆x
N
Therefore the PDF can be calculated as
h
fX (x) ≃
N ∆x

Linear Transformation of RN

• For a known RN X, let us consider a new transformed RN Y given as

Y = aX + b

Mean and variance of Y are

Y = aX + b
 2 2
σY2 = Y − Y = Y 2 − Y
h 2 i
= a2 X 2 + 2ab X + b2 − a2 X + 2ab X + b2
 
2
= a2 X 2 − X = a2 σX2

Sum of RN

• Let us consider a new RN:

Y = X1 + X2

where X1 and X2 are Identically and Independently Distributed (IID) RN. With

X1 = X2 = X, σX21 = σX22 = σX2

the mean and variance of Y become

Y = X 1 + X 2 = 2X
2
σY2 = (X1 + X2 )2 − X1 + X2
2 2
= X12 + X22 + 2 X1 X2 − X1 − X2 − 2 X1 X2
h 2i
= 2 X12 − X1 = 2 σX2

91
• The variance of Y becomes not 4 times but 2 times of that of X, which is interesting. It
is noteworthy that the sum of two IID RNs (X1 + X2 ) is definitely different to the RN of 2
times of single RN (2X1 ).
• Now for the following RN:
N
X
Y= Xn , X1 , X2 , · · · , XN : IID RNs
n=1

its mean and variance become

Y =NX
√
σY2 = N σX2 → σY = σX

its mean and variance become

Y =NX
√
σY2 = N σX2 → σY = σX

• This result shows the effect of averaging if the determinstic signal is contaminated by
random signal.√N -times averaging makes the signal voltage N times, but the standard
deviation only N times. In other words, signal power becomes N 2 times, but the noise
power N times, resulting in an N -times increase in Signal to Noise Ratio (SNR):

SNRN = N × SNR1

Now we can observe the signal clearly.

• This phenomena will be checked by Matlab simulation after studying the Gaussian RNs.
• Now let us consider a trial of throwing two dices, and taking their sum:

Y = X1 + X2
y 2 3 4 5 6 7 8 9 10 11 12
fY (y) 1/36 1/6 1/36

fY (y) = fX (x) ∗ fX (x) (Can you guess this result ?)

• Throwing of 4, 8, 16, ... dices ???

Z = Y1 + Y2 → W = Z1 + Z2 → U = W1 + W2 · · ·

92
• New random variable X is given as the sum of IID random variables Xn having zero mean:
N
X
X = X1 + X2 + · · · + Xn + · · · = Xn
n=1

As N increases, X is known to close to the Gaussian distribution which will be studied

later in deep.

Gaussian Random Number and It’s PDF

• As mentioned before, the thermal noise signal is known to follow the zero-mean Gaussian
distribution. Then what is the PDF of Gaussian random variable X with variance σ 2 .

– Let us start with an assumtion that its formula follows exponetial function of x2 as:
2
fX (x) = a e−bx

– There are two unknowns a and b, and there should be two independent conditions to
determinthese unknowns:
Z∞
i) fX (x) dx = 1
−∞
Z∞
ii) x2 fX (x) dx = σ 2
−∞

– a and b can be found using the following integral formulus as

Z∞ Z∞
√ 2 1√
−x2
e dx = π, x2 e−x dx = π
−∞ −∞ 2
1 1
→ a= √ , b=
2πσ 2 2σ 2

1 2 /(2σ 2 )
→ fX (x) = √ e−x
2π σ 2

• Let us generate the zero-mean Gaussian random number having a variance 1, that is
N (0, 1) in a two ways in Matlab program:

1) Sum of uniform RNs

2) Matlab built-in function “randn(.)”
3) Inverse Transform Method (Separate Section)

93
 It is noteworthy to compare the obtained PDF with the exact formula.

Gaussian RN from Uniform RN

• As mentioned before, the summation of many Identically and Independently Distributed

(IID) random numbers is known to follow the Gauusian random number. Let us check
this by simulation. Because the unform random number on [−0.5, 0.5] can be generated
with
√ h i
x1 = 12 rand(1,N) − 0.5

Since the variance of uniform RV on [0, 1] is 1/12, the RV x1 is found to have x1 = 0 and
σ12 = 1.
• Now if a new random variable x is given as follows (the sum of IID uniform random
numbers x1 , x2 , · · · xN ):
N
1 X
x= √ xn
N n=1

then this RV will have zero mean and unit variance. In addition, this will follows the
Gaussian RV if we inicrease N .
• Let us check this with Matab programming.

Generation of Gaussian RN with Matlab Built-In Function

• We can use “randn(.)” command to generate a zero-mean unit variance Gaussian RN

(N (0, 1)).
• The following is a Matlab script for this exercise:

Effect of Averaging

• As mentioned before, when we measure a signal, we cannot avoid random-noise contam-

ination. Even though it is well known to adopt data averaging in order to see the signal
more clearly, there are still many engineers and students who can not understand it theo-
retically.
• N -times data averaging
√ makes the signal voltage be N times, but the standard deviation
2
of random noise only N times. In other words, signal power becomes N times, with the

94
 noise power with being N times only, resulting in an N -times increase in Signal to Noise
Ratio (SNR):

SNRN = N × SNR1

Now we can observe the signal clearly.

• Let us check this with Matlab simulation:

Random Variable

• What is a random variable ?

→ Just a function mapping the events to numbers

Magnitude and Square of Complex Gaussian Random Number

• In most electronic circuits and equipments, a complex gaussian RNs are needed to model
the signal, especially in determine the envelope of signal. We call this the IQ-processing,
and the related block diagram is shown in Fig. XXX.
• Conceptually this is same as the dart throwing experiment (2-D Gaussian RN).
• In this case, our concern is the distribution of the magnitude and square of the complex
Gaussian RN. Especially how will be the PDF of the magnitude around the origin. Defi-
nitely most of the points of dart exist around the origin, and become space away from the
origin. And we may think that the density becomes high around the origin.
• Let us examine this phenomena by simulation.

– Let us generate two zero-mean unit variance Gaussian RNs, and place one RN along
x-axis and the other along the y-axis, that is, on the xy-plane.
– Then calculate their distance R, and count the points within the concentric rings (their
centers are the origin) having the same width ∆r.

Random Number Generation for Embedded System

[Link]

• Here is an interesting number:

0.814723686393179

This is the first number produced by the Matlab RN generator with its default settings
(Start up matlab and set format long, type rand), and it’s the number you get.

95
• If all Matlab users, all around the world, all on different computers, keep getting this
same number, is it really random? No, it isn’t.
• Computers are (in principle) deterministic machines and should not exhibit random be-
havior.

Uniform distributed random number (UDRN)

• Uniform RN generator by which RN between 0 and 1 is generted uniformly.

U (0, 1)

• Other distributed RN can be generated from the UDRN with an appropriate transform.
• The state of the art for generating UDRN has advanced considerably in the last decade
and now begins to resemble a mature field.
• Issues are new mathematical algorithms and HW-based implementation.
• Caution: Many out-of-date and inferior methods remain in general use.

– Never use a generator principally based on a linear congruential generator (LCG) or a

multiplicative linear congruential generator (MLCG).
– Never use a generator with a period less than ∼ 264 ≃ 2 × 1019 , or any generator whose
period is undisclosed.
– Avoid using generators with period > 10100 . You really will never need it, and, above
some minimum bound, the period of a generator has little to do with its quality.
• An acceptable RN generator must combine at least two (ideally, unrelated) methods.
The methods combined should evolve independently and share no state. The combina-
tion should be by simple operations that do not produce results less random than their
operands.

• Lehmer invented the Linear Congruential Generator (LCG), which is the basis for many
of the RN generators in use today. Lehmer’s generators involve three integer parameters,
a, c, and m, and an initial value, x0 , called the seed.
xk+1 = mod(axk + c, m)
With a = 13, c = 0, m = 31, x0 = 1
x = {1, 13, 14, 27, 10, 6, · · · }
c = 0 → Multiplicative LCG (MLCG).

96
 • What’s the next value? Well, it looks pretty unpredictable, but you’ve been initiated. So
you can compute

mod(13 × 6, 31) = 16

• Scaling (dividing with m) yields UDRN in theinterval (0, 1).

xn = {0.0323, 0.4194, 0.4516, 0.8710, 0.3226, · · · }

• Many weakeness are observed, and there are increasing needs for advanced algorithems.
A Combined Generator may be a candidate !!!

1 function ux = func_rand_phill(N,Z0,Nb)
2

3 % N = Length of RAND generation

4 % Z0 = Seed (Odd number) --> Z(1)
5 % Nb = number of bits for Maximum allowable period
6

7 m = 2^Nb;
8 a = 2^(round(Nb/2))+3;
9 Z(1) = Z0;
10 for ii=2:N
11 Z(ii) = mod(a*Z(ii-1),m);
12 end
13 ux = Z/m;

1 clear all, close all, clc

2

3 Z01 = 301; % Seed (odd number)

4 Z02 = 503; % Seed (odd number)
5 Nb = 20; % Maximum period = 2^Nb
6 N = 1000; % Length of Random number
7 ux1 = func_rand_phill(N,Z01,Nb);
8 ux2 = func_rand_phill(N,Z02,Nb);
9

10 plot(ux1,ux2,’+’), axis([-0.5 1.5 -0.5 1.5]), axis equal

97
General case
x x

f(x)
0
t

f(x)

0 x

Figure 7.5: histogram to pdf conversion.

98
 x
Trial (n)
Simula
on
Exact
fX(x)

x
Figure 7.6: Example of histogram to pdf conversion.

1 clear all, close all, clc

2

3 N = 10000;
4 x = rand(1,N);
5 mx = mean(x)
6 sx2 = var(x,1)
7

8 dx = 0.01;
9 xd = -1:dx:2;
10 y = hist(x,xd);
11 fx = y/(N*dx);
12

13 fu = zeros(1,length(xd));
14 ii1 = find((xd>=0)&(xd<=1));
15 fu(ii1) = 1;
16

17 subplot(2,1,1), plot(x), ylim([-0.5 1.5])

18 subplot(2,1,2), plot(xd,fx,xd,fu), ylim([-0.5 1.5])

99
 x

n
fX(x)

x
Figure 7.7: Gaussian random number.

1 clear all, close all, clc

2 N = 10000;
3 x = randn(1,N); % Normal distributed random number N(0,1)
4 dx = 0.25;
5 xd = -5:0.25:5;
6 y = hist(x,xd);
7 fx = y/(N*dx);
8 fg = 1/sqrt(2*pi)*exp(-xd.^2/2); % PDF of N(0,1)
9 plot(xd,fx,xd,fg)

Inverse Transform Method

• How can a RN x be generated for a specified PDF fX (x) from UDRN ???
fX (x) → x = {x1 , x2 , · · · }

• It can be feasible from a UDRNs U (0, 1) with a range (0,1) by using transformation method.
• Uniform distributed u has the following PDF:
u = U (0, 1) → fU (u) = 1, 0<u<1

100
• Relation between u and x by the transformation rule
du
fX (x) = fU (u)
dx
du
fX (x) =
dx
• Since fX (x) is always positive,

u = FX (x)

a RN x for a specified PDF can be found from the inverse CDF with an argument u.

x = FX−1 (u)

No problem for the known and invertible FY (y).

• If FY (y) in not invertible, various numerical method can be used. In this case, y for an x
can be found as follows:

– Since FY (y) is just an area of fY (y) to the left of y, y for a u can be found from the data
of FY (y).

Exponential and Rayleigh Distributions

• PDF and CDF of an exponential distribution with a mean µw = 1

fW (w) = e−w
FW (w) = 1 − e−w

• Since the inverse function of FW becomes

FW−1 = w = − ln(1 − F)

the exponential RN can be found as

w = − ln(1 − u)

• Now the Rayleigh distributed RN r, which is given as below, can be found:

1 √ p
w = r2 → r = 2w = −2 ln(1 − u)
2
2 /2
fR (r) = r e−r
101
Gaussian RN

• Normal distribution RN can also be found from the Rayleigh one and UDRN. For two
independent normal distributed RNs, their combined magnitude and phase are known
to follow Rayleigth and uniform distributions.
• Box-Muller method

– Two UDRNs are generated:

u1 = U (0, 1), u2 = U (0, 1)

– Exponential and Rayleigh distributed RNs are generated from u1 :

p
w = − ln(1 − u1 ) → r = −2 ln(1 − u1 )

– Finally I- and Q-signals are generated as

p
xI = r cos(2π u2 ) = −2 ln(1 − u1 ) cos(2π u2 )
p
xQ = r sin(2π u2 ) = −2 ln(1 − u1 ) sin(2π u2 )

• Efficient approach without triginometric function evaluation

– From two uniformly distributed RNs u1 and u2

v1 = 2u1 − 1, v2 = 2u2 − 1
q
R = v12 + v22 : Square

– If v12 + v22 ≤ 1 (inside the unit circle)

v1 v2
cos φ → , sin φ →
R R
else, skip (rejection) and go to the 1st of the loop.
– Finally
 v  v
−2 ln(1 − u1 ) 1 −2 ln(R) 1
p p

xI = 
xI =
R R

 

or
 
 v  v
xQ = −2 ln(1 − u1 ) 2 xQ = −2 ln(R) 2

 p 
 p
 
R R

102
 1/6

0 1 6 x

1/6

6 1 0

Convolution

6/36

1/36
0 2 7 12 x

1+1
1+2 2+1
1+6 2+5 3+4 4+3 5+2 6+1

Figure 7.8: Throwing two dices.

• The Gaussian RN can also be found directly by inverse transform method.

– PDF, CDF, and inverse function of F of normal distribution:

" !#
1 −ξ 2 /2 1 ξ
f (ξ) = e , F(ξ) = 1 + erf √
2π 2 2
−1
√ −1
F (ξ) = 2 erf (2F − 1)

– Normal distributed RN
√
ξ = 2 erf−1 (2u − 1)

– Rayleigh and exponential distribution RNs can be generated from the normal
distribution.
• Case by case, depending on the available computing resource, one of the aboves can
be used !!!

103
 Generalization
Central-Limit
Noise Voltage across a Resister Theorem Gaussian PDF

Single electron
in resistor at T = 300 oK Vn
t=0
+ - + -
+ -
Vn1(t0 )
t=t1 Vn1(t)
+ - 0 t
Vn1(t1 )
t=t2 Vn2(t)
- + 0 t
Vn1(t2 )
Vn1(t)
0 t
0 t
0 t

0 t

Figure 7.9: Thermal noise voltage becomes a Gaussian random noise.

1 clear all, close all, clc

2

3 M = 10; % No. of summation

4 N = 1000;
5 n = 1:N;
6 var_urn = 1/12; % Variance of Uniform RN with interval 1
7 xn0 = rand(M,N)-0.5;
8 xn = xn0/sqrt(var_urn); % Uniform RN (m=0, s2 = 1)
9 x = sum(xn)/sqrt(M); % Sum of uniform RNs s2 = 1
10 var_x = var(x,1)
11 dx = 0.1;
12 xd = -5:dx:5;
13 y = hist(x,xd);
14 ff = y/(N*dx);
15 fg = 1/sqrt(2*pi)*exp(-xd.^2/2);

104
 Vn1(t)
⋮
Vn8(t)

Vn(t)

Figure 7.10: Sum of uniform random numbers to generate Gaussian random noise.

16 subplot(3,1,1), plot(n,xn0(1,:)), ylim([-1 1])

17 subplot(3,1,2), plot(n,x), ylim([-3 3])
18 subplot(3,1,3), plot(xd,ff,xd,fg), ylim([-3 3])

Chaotic Signal

• Tent-Map based Chaotic Random Number Generator


µxn ,

 xn < 0.5
xn+1 = 
µ(1 − xn ), 0.5 ≤ xn < 1


1 µ
xn+1 = −µ x − +
2 2
µh i
= 1 − |2xn − 1|
2

105
106
Appendix A

Matlab - Getting Started

MATLAB is an abbreviation for Matrix Laboratory.

• While other programming languages usually work with numbers one at a time, MATLAB
operates on whole matrices and arrays.
• It include basic operations, such as creating variables, array indexing, arithmetic, and
data types.
• The well-known language C is mainly used for control of HW and embedded system
because of powerful capability of memory access. On the other hand, the Matlab has
an advantage in scientific and engineering calculations and data visualization.

Figure A.1: Matlab symbol.

107
 After launching the Matlab, you can see the following main windows:

• Command window
• Editor

Compared to other lamnguage such as C, it is rather easy to learn the Matlab. From now,
we will study how to use the Matlab with some practical examples.

You can type the followings in the command window:

1 >> clear all, close all, clc

2

3 >> a = 4;
4 >> b = 2;

or in Editor window:

1 clear all, close all, clc

2

3 a = 4;
4 b = 2;

Now if you type the followings in the command widow, you can the following results:

1 >> c1 = a+b
2 >> c1 =
3 6
4

5 >> c2 = a-b
6 c2 =
7 2

Differences between the followings:

1) c = a+b and c = a+b; ???

Result is displayed or not !!!

108
 2) test.m or test.p
test.p can be only run (source code is not seen)

Vector and Matrix

1) Vector

a = [1 2 3], b = [6 7 10] (A.1)

1 >> a = [1 2 3]
2 a =
3 1 2 3
4

5 >> b = [6 7 10]
6 b =
7 6 7 10

2) Matrix
   
1 1 1 1 2 3
a = 1 2 3 , b = 6 7 9 (A.2)
   
5 4 2 2 4 5
   

In the Editor,

1 clear all, close all, clc

2

3 a=[1 1 1; 1 2 3; 5 4 2];
4 b=[1 2 3; 6 7 9; 2 4 5];
5 a
6 b

Then the follows are displayed:

109
 1 a =
2 1 1 1
3 1 2 3
4 5 4 2
5

6 b =
7 1 2 3
8 6 7 9
9 2 4 5
10

Two types of multiplication

• Matrix multiplication
• Element by element multiplication

1 clear all, close all, clc

2

3 a = [1 1 1;
4 1 2 3;
5 5 4 2];
6 b = [1 2 3;
7 6 7 9;
8 2 4 5];
9

10 c = a*b;
11 d = a.*b;
12

13 c
14 d

While the size of two matrices satisfy the muliplication rule in ∗ multiplication, same size of
two matrices can be multiplied element by element (.∗). You can see the different results.

1 c =
2 9 13 17

110
 3 19 28 36
4 33 46 61
5

6 d =
7 1 2 3
8 6 14 27
9 10 16 10

Round, Fix, and Remainder

1 clear all, close all, clc

2

3 a = 3;
4 b1 = 2;
5 b2 = -2;
6

7 c1 = a/b1;
8 c2 = a/b2;
9 [c1 c2]
10

11 d1 = floor(a/b1);
12 d2 = ceil(a/b1);
13 [d1 d2]
14

15 d3 = rem(a,b1);
16 d4 = mod(a,b1);
17 [d3 d4]
18

19 d5 = rem(a,b2);
20 d6 = mod(a,b2);
21 [d5 d6]

1 ans =
2 1.5000 -1.5000
3

4 ans =
5 1 2

111
 6

7 ans =
8 1 1
9

10 ans =
11 1 -1

Zeros, Ones vector, Identity matrix

1 clear all, close all, clc

2

3 N = 5;
4 a = zeros(1,N);
5 b = ones(1,N);
6 c = eye(3)
7

8 a
9 b
10 c

1 a =
2 0 0 0 0 0
3

4 b =
5 1 1 1 1 1
6

7 c =
8 a =
9 1 0 0
10 0 1 0
11 0 0 1

Determinant and Inverse matrix

112
 1 clear all, close all, clc
2

3 a = [1 1 0; 2 2 2; 3 4 5];
4 d = det(a);
5 b = inv(a);
6

7 d
8 b

1 d =
2 -2.0000
3

4 b =
5 -1.0000 2.5000 -1.0000
6 2.0000 -2.5000 1.0000
7 -1.0000 0.5000 0

Advantage of Vector or Matrix Operations

Operation with a single for loop can be efficiently performed with vector operation.
10
X
x= (n2 + 3n) (A.3)
n=1

1 clear all, close all, clc

2

3 x = 0;
4 for n=1:10
5 xn = n^2+3*n;
6 x = x+xn;
7 end
8 x
9

10 n = 1:10;
11 yn = n.^2+3*n;

113
12 y = sum(yn);
13 y

You can see the same result.

1 x =
2 550
3

4 y =
5 550

10
Y
x= n (A.4)
n=1

1 clear all, close all, clc

2

3 x = 1;
4 for n=1:10
5 x = x*n;
6 end
7 x
8

9 n = 1:10;
10 y = prod(n);
11 y

You cal also see the same result.

1 x =
2 3628800
3

4 y =
5 3628800

114
 Signal plot

8 y=(abs(x 2)-1)

y
4

0
-4 -3 -2 -1 0 1 2 3 4
x

Figure A.2: 1D plot example

1-D Plots and Title, Labels, Text inputs

Let us plot y = (|x| − 1)2 , x = −4 : 0.1 : 4

1 clear all, close all, clc

2

3 x=-4:0.1:4;
4 y=(abs(x)-1).^2;
5

6 plot(x,y)
7 title(‘Signal plot’)
8 xlabel(‘x’)
9 ylabel(‘y’)
10

11 text(0,8,’y=(abs(x)-1)^2’)

There is another widely used 1D plot called stem plot

Axis Setting Here we will study how to set the axis with the stem plot.

1 clear all, close all, clc

2

3 f = 1e3;
4 Ts = 0.05e-3;
5 t = 0:Ts:4e-3;

115
 Figure A.3: Stem plot with axis setting

6 x = sin(2*pi*f*t);
7

8 figure(‘name’,’Sampled signal’)
9 stem(t,x)
10 title(‘Stem plot’), grid on
11 axis([0 max(t) -1.5 1.5])

axis equal in plotting Unit Circle

1 clear all, close all, clc

2

3 thdeg=[Link];
4 th=thdeg*pi/180;
5 x=exp(j*th);
6

7 plot(real(x),imag(x))
8 axis([-1.5 1.5 -1.5 1.5])
9 axis equal

116
 1

-1

-1 0 1

Figure A.4: Unit circle with axis equal

2-D Plots

• pcolor : 2D function plot with color

• contour: equi-value line
• quiver: Stream of vector flow

Let us plot the following function on the xy-plane.

2 +y 2 )
z = e−(x , −3 ≤ x, y ≤ 3 (step ∆x = ∆y = 0.2)

1 clear all, close all, clc

2 dx = 0.2;
3 dy = 0.2;
4 x = -3:dx:3;
5 Nx = length(x);
6 y = -3:dy:3;
7 Ny = length(y);
8 for m=1:Nx
9 for n=1:Ny
10 arg = x(m)^2+y(n)^2;
11 z(m,n)=exp(-arg);
12 end
13 end
14 pcolor(x,y,z), colorbar

117
 where colorbar command shows the level of color displayed.

Figure A.5: 2D pcolor plot

Next let us study how to use meshgrid and use it for pcolor plot.

1 clear all, close all, clc

2 dx = 0.5;
3 dy = 1;
4 x = [0 0.5 1];
5 y = [1 2 3 4];
6 [xx,yy] = meshgrid(x,y);
7 xx
8 yy

1 xx =
2

3 0 0.5000 1.0000
4 0 0.5000 1.0000
5 0 0.5000 1.0000
6 0 0.5000 1.0000
7

9 yy =
10

118
11 1 1 1
12 2 2 2
13 3 3 3
14 4 4 4

All 12 points of (x, y) pair are formed. By using the meshgrid command, it will be easy to
calculate the related function and pcolor plot. Then the operation with a double loops can
be efficiently performed with matrix operation.

1 clear all, close all, clc

2

3 dx = 0.2;
4 dy = 0.2;
5

6 x = -3:dx:3;
7 Nx = length(x);
8 y = -3:dy:3;
9 Ny = length(y);
10

11 [xx,yy] = meshgrid(x,y);
12 z = exp(-xx.^2-yy.^2);
13

14 pcolor(x,y,z), colorbar

Then the same pcolor can be obtained. Furthermore, pcolor plot can be smoothed with
shading interp command.

1 clear all, close all, clc

2

3 dx = 0.2;
4 dy = 0.2;
5

6 x = -3:dx:3;
7 Nx = length(x);
8 y = -3:dy:3;
9 Ny = length(y);
10

11 [xx,yy] = meshgrid(x,y);

119
12 z = exp(-xx.^2-yy.^2);
13

14 pcolor(x,y,z), colorbar
15 shading interp

The following is the example code for the contour plot:

Figure A.6: 2D pcolor plot with shade interp

1 clear all, close all, clc

2

3 dx = 0.2; dy = 0.2;
4

5 x = -3:dx:3; Nx = length(x);
6 y = -3:dy:3; Ny = length(y);
7

8 [xx,yy] = meshgrid(x,y);
9 z = exp(-xx.^2-yy.^2);
10

11 zmin = floor(min(min(z))*10)/10;
12 zmax = ceil(max(max(z))*10)/10;
13 dz = (zmax-zmin)/10;
14

15 zlevels = zmin:dz:zmax;
16 [C,h] = contour(x,y,z,zlevels)
17 clabel(C,h)

120
18 shading interp, colorbar

In addition, quiver (meaning of arrow) plot is very useful for visualizing the vector quantity,

Figure A.7: 2D plot of contour

such as electric field vector and direction of wind blowing as well as the gradient of scalar
functions. One example is as follows:

1 clear all, close all, clc

2

3 dx = 0.2; dy = 0.2;
4 x = -3:dx:3; Nx = length(x);
5 y = -3:dy:3; Ny = length(y);
6

7 [xx,yy] = meshgrid(x,y);
8 z = xx.*yy.*exp(-xx.^2-yy.^2);
9

10 [zx,zy] = gradient(z,dx,dy);
11

12 contour(x,y,z)
13 shading interp
14 colorbar
15 hold on
16

17 quiver(x,y,zx,zy)

121
 Figure A.8: 2D plot of quiver

If you need more information about these plot functions, you can use help command in
command window as

1 >> help quiver

Other detailed programming skills will be learned through many examples of Numerical
Analysis.

122