Numerical Techniques (MA-201)

Lecture Notes-15-16

by
Prof. Santwana Mukhopadhyay

Department of Mathematical Sciences

IIT(BHU), Varanasi

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 1 / 29
 Contents

1 Preliminaries

2 Iteration Methods
Jacobi iteration method:
Gauss-Seidel iteration method:

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 2 / 29
 Gauss-Jordan elimination method

We discussed in previous class about the method of Gauss-Jordan to find

exact solution of a system of equations.
This method also helps in findinding the inverse of a matrix.

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 3 / 29
 Gauss-Jordan elimination method contd...

Steps for Gauss-Jordan elimination:

1 Swap the rows so that all rows with all zero entries are on the bottom.
2 Swap the rows so that the row with the largest, leftmost nonzero
entry is on top.
3 Multiply the top row by a scalar so that top row’s leading entry
becomes 1.
4 Add/subtract multiples of the top row to the other rows so that all
other entries in the column containing the top row’s leading entry are
all zero.
5 Repeat steps 2-4 for the next leftmost nonzero entry until all the
leading entries are 1.
6 Swap the rows so that the leading entry of each nonzero row is to the
right of the leading entry of the row above it.
Prof. S. Mukhopadhyay (Department of Mathematical
Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 4 / 29
 Gauss-Jordan elimination method contd...

To obtain the inverse of a n × n matrix A:

Create the partitioned matrix (A|I) , where I is the identity matrix.

Perform elementary row operations on the partitioned matrix with the
objective of converting the first part of the matrix to reduced-row
echelon form.(or identity form)
If
 done correctly, the resulting partitioned matrix will take the form
I|A−1 .

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 5 / 29
 Gauss-Jordan elimination method contd...

Example 1:
Find the inverse of the matrix
 
2 1 1
A= 3 2 1 
 
2 1 2

by Gauss-Jordan method.

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 6 / 29
 Gauss-Jordan elimination method contd...

Solution:

Applying Gauss-Jordan method, we get

2 1 1 1 0 0 1 1/2 1/2 1/2 0 0

! !
R1 → 21 R1
[A|I] = 3 2 1 0 1 0 −−−−−→ 3 2 1 0 1 0
2 1 2 0 0 1 2 1 2 0 0 1

1 1/2 1/2 1/2 0 0

R2 →R2 −3R1
!
R3 →R3 −2R1
−−−−−−−−→ 3 1/2 −1/2 −3/2 1 0
0 0 1 −1 0 1
1 1/2 1/2 1/2 0 0
!
R2 →2R2
−−−−−→ 0 1 −1 −3 2 0
0 0 1 −1 0 1
1 0 1 2 −1 0
!
R1 →R1 − 12 R2
−−−−−−−−− → 0 1 −1 −3 2 0
0 0 1 −1 0 1

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 7 / 29
 Gauss-Jordan elimination method contd...

Solution contd...

 
R1 →R1 −R3 1 0 0 3 −1 −1
R2 →R2 +R3
−−−−−−−→  0 1 0 −4 2 1 
0 0 1 −1 0 1
 
3 −1 −1
So, A−1 =  −4 2 1 .
−1 0 1

Verify that you get the same result with partial pivoting in the present
case.

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 8 / 29
 Gauss-Jordan elimination method contd...

Example 2:
Find the inverse of the matrix
 
1 2 1
A =  2 3 −1 
 
2 −1 3

by Gauss-Jordan method
 with partial 
pivoting.
−8 7 5
Answer: A−1 = 161 
 8 −1 −3 .

8 −5 1

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 9 / 29
 Iteration Methods

Iteration methods:

1. Jacobi method
2. Gauss-Seidel method

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 10 / 29
 Iteration methods

Iteration methods:
Whenever the algebraic equations contain large number of equations
at that time Direct Method become lengthy and time consuming
headache.
In this case the Iteration Methods play a vital role by being the
substitute of the method.
In iterative methods, we start from an initial approximation to actual
solution and obtain the better approximation after repeating the
iteration. This method can give solution with desired accuracy.

Here we will discuss two types of iteration methods:

1 Jacobi Iteration Method
2 Gauss-Seidel Iteration Method

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 11 / 29
 Iteration method Continue

Assumpsions on Iteration method:

The following assumptions made on iterative method:

1 The given system of equations has a unique solution.
2 The coefficient matrix A has no zeros on its main diagonal, namely,
a11 , a12 ,...,ann are nonzero.
3 If any of the diagonal entries a11 , a12 ,...,ann are zero, then rows or
columns must be interchanged to obtain a coefficient matrix that has
nonzero entries on the main diagonal.

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 12 / 29
 Iteration methods contd...

Consider the sytem

a11 x1 + a12 x2 + ....a1n xn =b1

a21 x1 + a22 x2 + ....a2n xn =b2 (2.1)
........................................
an1 x1 + an2 x2 + ....ann xn =bn

We can write the system as

1
x1 = (b1 − a12 x2 − a13 x3 − ....a1n xn )
a11
1
x2 = (b2 − a21 x1 − a23 x3 − ....a2n xn ) (2.2)
a22
..............................................................
1

xn = bn − an1 x1 − an2 x2 − ....an(n−1) xn−1
ann

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 13 / 29
 Jacobi iteration method contd...

Hence, we need to solve the 1st equation for x1 , the 2nd

equation for x2 and so on.
For this,
 make an initial guess of the solution
(0) (0) (0)
(0)
x = x1 , x2 , x3 , ..., xn(0) . Substitute these values into the
right hand side of the rewritten equations
 (2.5) to obtain the first
(1) (1) (1) (1)
approximation, x1 , x2 , x3 , ..., xn .
This accomplishes the first iteration.
In
 the same way, the second approximation
(2) (2) (2) (2)
x1 , x2 , x3 , ..., xn is computed by substituting the first
approximation into the right hand side of the rewritten equations.

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 14 / 29
 Jacobi method Continue

By repeated iterations, we form a sequence of approximations

(k) T
 
(k) (k) (k)
x (k) = x1 , x2 , x3 , ..., xn , k = 1, 2, ...

For eack k ≥ 1, generate the Jacobi Iteration scheme to determine

(k)
the components xi of x (k) from x (k−1) from equations (2.5) by
 
n
(k) 1  X  
(k−1) 
xi = bi − aij xj , n = 1, 2, .., n (2.3)
aii j=1,j6=i

Stop when the max absolute error in solution reaches the desired accuracy.

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 15 / 29
 Matrix form of Jacobi iteration method

Explannation of Jacobi method in matrix form:

Consider system of linear equations Ax = b with
     
a11 a12 ... a1n b1 x1
 a21 a22 ... a2n  b2  x2 
A= ,b =  ,x = 
 
: : :  :  : 
an1 an2 ... ann bn xn

A can be written as
   
a11 0 ... 0 0 ... 0 0
 0 a22 ... 0   a21 ... 0 0 
A = +
: : :   : : 
0 0 ... ann an1 ... an(n−1) 0
 
0 ... a12 a1n
 0 ... 0 :
+

: a(n−1)n 
0 0 ... 0
=D + L + U

where D is diagnal matrix, L is lower triangular matrix, and U is upper triangular matrix.
Prof. S. Mukhopadhyay (Department of Mathematical
Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 16 / 29
 contd...

Ax = b is transformed to

(D + L + U) x = b ⇒ Dx = − (L + U) x + b

Assume D −1 exists, then

x = −D −1 (L + U) x + D −1 b

The matrix form of Jacobi iterative method is

x (k) = −D −1 (L + U) x (k−1) + D −1 b, k = 1, 2, ...

Define H = −D −1 (L + U) and c = D −1 b, Jacobi iteration method can

also be written as

x (k) = Hx (k−1) + c, k = 1, 2, ... (2.4)

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 17 / 29
 contd...

Implementation of Jacobi method:

 
1/a11 0 ... 0
 0 1/a22 ... 0 
D −1 =
: : : 
0 0 ... 1/ann
Substituting this in relation
x = −D −1 (L + U) x + D −1 b
yields

1
x1 = (b1 − a12 x2 − a13 x3 − ....a1n xn )
a11
1
x2 = (b2 − a21 x1 − a23 x3 − ....a2n xn ) (2.5)
a22
..............................................................
1

xn = bn − an1 x1 − an2 x2 − ....an(n−1) xn−1
ann
which is same as obtained earlier.
Prof. S. Mukhopadhyay (Department of Mathematical
Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 18 / 29
 Examples on Jacobi iteration method contd...

Example 1:
Apply the Jacobi method to solve

5x1 − 2x2 + 3x3 = − 1

−3x1 + 9x2 + x3 =2
2x1 − x2 − 7x3 =3

Continue iterations until two successive approximations are identical when

rounded to three significant
 digits.
 
5 0 0 0 0 0
Here, D =  0 9 0  , L =  −3 0 0  , U =
   
0 0 −7 2 −1 0
   
0 −2 3 −1
 0 0 1 ,b =  2 
   
0 0 0 3

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 19 / 29
 Jacobi iteration method contd...

Solution:

To begin, write the Jacobi scheme as

(k) −1 2 (k−1) 3 (k−1)
x1 = + x2 − x3
5 5 5
(k) 2 3 (k−1) 1 (k−1)
x2 = + x1 − x3
9 9 9
(k) −3 2 (k−1) 1 (k−1)
x3 = + x1 − x2
5 7 7
Choose the initial guess x1 = x2 = x3 = 0. The first approximation is
(1) −1 2 3
x1 = + (0) − (0) = −0.200
5 5 5
(1) 2 3 1
x2 = + (0) − (0) = 0.222
9 9 9
(1) −3 2 1
x3 = + (0) − (0) = −0.429
5 7 7

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 20 / 29
 Matrix form

Alternatively, in the matrix form, we can write the Jacobi scheme as

x (k) = −D −1 (L + U) x (k−1) + D −1 b, k = 1, 2, ...

      
1/5 0 0 0 −2 3 x1 1/5 0 0 −1
=− 0 1/9 0 −3 0 1 x2 + 0 1/9 0 2
0 0 −1/7 2 −1 0 x3 0 0 −1/7 3

# (k−1)

0 −2/5 3/5 x1 −1/5
" " #
(k−1)
=− −3/9 0 1/9  x2 + 2/9
−2/7 1/7 0 (k−1)
x3 −3/7
# (k−1)

0 −0.4 0.6 x1 −0.2
" " #
(k−1)
=− −0.333 0 0.111  x2 + 0.222
−0.286 0.143 0 (k−1)
x3 −0.429

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 21 / 29
 Example on Jacobi iteration method contd...

Solution contd...

Continuing iterations in both way (matrix form and system of equations

form), we obtain

n 0 1 2 3 4 5 6 7
x1 0.000 −0.200 0.146 0.192 0.181 0.185 0.186 0.186
x2 0.000 0.222 0.203 0.328 0.332 0.329 0.331 0.331
x3 0.000 −0.429 −0.517 −0.416 −0.421 −0.423 −0.423 -0.423

We stop in 7th iterations when the last two columns in above Table are
identical, and we can conclude that the solution correct to three significant
digits is
x1 = 0.186, x2 = 0.331, x3 = −0.423

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 22 / 29
 Gauss-Seidel method contd...

Matrix form of Gauss-Seidel method:W

We write Ax=b as
(D + L) x (k) = −Ux (k−1) + b
x (k) = − (D + L)−1 Ux (k−1) + (D + L)−1 b
Define Hg = − (D + L)−1 U and cg = (D + L)−1 b, Gauss-Seidel method
can be written as

x (k) = Hg x (k−1) + cg , k = 1, 2, ...

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 23 / 29
 Implementation

Implementation of Gauss-Seidel method:

We write (D + L) x (k) = −Ux (k−1) + b in the iterative scheme form.

Hence, for each k ≥ 1, the Gauss-Seidel iterative scheme to determine the
(k)
components xi is given by
 
i−1   n  
(k) X (k) X (k−1)
aii xi + aij xj = − aij xj + bi  , i = 1, 2, ...
j=1 j=i+1

i.e.,
 
i−1  n
(k) 1  X (k)
 X 
(k−1)

xi = − aij xj − aij xj + bi  , i = 1, 2, ...
aii j=1 j=i+1

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 24 / 29
 Gauss-Seidel Method contonue

(k+1)
With the Gauss-Seidel method, we use the new values xi as soon
as they are known.
(k+1)
For example, once we have computed x1 from the first equation,
(k+1)
its value is then used in the second equation to obtain the new x2
and so on.
(k)
Note: With the Jacobi method, the values of xi obtained in the
k-th iteration remain unchanged until the entire (k + 1)th iteration
has been calculated.

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 25 / 29
 Example: Gauss-Seidel method contd...

Example:
Use the Gauss-Seidel iteration method to approximate the solution to the
system of equations given in Example 1,i.e.,

5x1 − 2x2 + 3x3 = − 1

−3x1 + 9x2 + x3 =2
2x1 − x2 − 7x3 =3

Continue iterations until two successive approximations are identical when

rounded to three significant digits.

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 26 / 29
 Example Gauss-Seidel method contd...
Solution:

The first computation is identical to that given in Example 1. That is,

using (x1 , x2 , x3 ) = (0, 0, 0) as the initial approximation, you obtain the
following new value for x1 .
1 2 3
x1 = − + (0) − (0) = −0.200
5 5 5
Now that you have a new value for x1 , however, use it to compute a new
value for x2 . That is,
2 3 1
x2 = + (−0.200) − (0) = 0.156
9 9 9
Similarly, use x1 = −0.200 and x2 = 0.156 to compute a new value for x3 .
That is,
3 2 1
x3 = − + (−0.200) − (0.156) = −0.508
7 7 7
Prof. S. Mukhopadhyay (Department of Mathematical
Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 27 / 29
 Gauss-Seidel method contd...

Solution contd...

So the first approximation is x1 = −0.200, x2 = 0.156 and x3 = −0.508.

Continued iterations produce the sequence of approximations shown in the
following Table:

n 0 1 2 3 4 5
x1 0.000 −0.200 0.167 0.191 0.186 0.186
x2 0.000 0.156 0.334 0.333 0.331 0.331
x3 0.000 −0.508 −0.429 −0.422 −0.423 −0.423

Note that after only five iterations of the Gauss-Seidel method, you
achieved the same accuracy as was obtained with seven iterations of the
Jacobi method in Example 1.

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 28 / 29
 Gauss-Seidel method contd...

Task:
Solve the following system of equations by performing 3 iterations of (i)
Jacobi method (ii) Gauss-Siedel method:

4x1 + x2 + x3 =2
x1 + 5x2 + 2x3 = − 6
x1 + 2x2 + 3x3 = − 4

****************

Prof. S. Mukhopadhyay (Department of Mathematical

Numerical
SciencesTechniques
IIT(BHU),(MA-201)
Varanasi) 29 / 29