Chapter 3 The Iterative Solving Method for

Linear System of Equations

§1 Introduction

Assume the matrix A is nonsingular in the system

Ax=b
we know that there exists a unique solution of the system.
Construct system
x  Bx  f
We shall get the iterative formula:
x ( k 1)  Bx ( k )  f k  0,1, 2,

24/6/19 1
 For any initial value x(0) ， the iterative formula creates a series
{x(k)} . If 
lim x (k )
 x
k 
 
We have x  Bx  f
Thus, x* is the solution of system Ax=b.

Definition 3.1 If for any initial vector x （ 0 ） , any series {x （ k

）
} created by the iterative formula x(k+1)=Bx(k)+f
converge. We say that the iterative formula is
convergent. Otherwise, it is divergent.

24/6/19 2
 Note. When the matrix A of linear system is sparse, we often
use iterative methods to solve the linear system. Because you only
need compute those nonzero entries on this case, the computation
complexity can be reduced largely. On the contrary, using direct
method, for example, GEM, the sparse matrix may be become a dense
matrix and the computation will be increased largely.

Note.Because lim x ( k )  x
k 

 lim || x ( k )  x || 0
k 

 lim || x ( k 1)  x ( k ) || 0

k 
(k )
We think the iterative series {x } is convergent if

|| x ( k 1)
x (k )
|| 

24/6/19 3
 §2 Several Iterative Methods
Eg. 3.1 Using the iterative method, Solve the system
 8 x1  3 x2  2 x3  20

 4 x1  11x2  x3  33
6 x  3 x  12 x  36
 1 2 3

Solution. The system can be rewritten equivalently as

 1
 1 8（3x2  2 x3  20）
x 

 1
 x2  （  4 x1  x3  33）
 11
 1
 x3  12（  6 x1  3x2  36）

24/6/19 4
And we get the iterative formula
 ( k 1) 1
 1 x  （3 x2
(k )
 2 x3  20）
(k )

8

 ( k 1) 1
 2x  （  4 x 1
(k )
 x 3  33）
(k )
k  0,1, 2,
 11
 ( k 1) 1
x
 3  （  6 x 1
(k )
 3 x 2  36）
(k )

 12
Let x(0)=(0,0,0)T ， we obtain the series {x(k)}
k 0 1 2 3 4 5 6 7 8 9
x1 0 2.5000 2.8750 3.1364 3.0240 3.0003 2.9938 2.9990 3.0002 3.0001

x2 0 3.0000 2.3636 2.0455 1.9478 1.9840 2.0000 2.0026 2.0006 1.9999

x3 0 3.0000 1.0000 0.97160 0.92040 1.0010 1.0039 1.0031 0.99985 0.99988

In fact, the exact solution of the system x = (3,2,1)T,

and we can find the iterative series {x （ k ） } converge the
exact24/6/19
solution. 5
2.1 Jacobi’s Method

Consider the system

n

a x
j 1
ij j  bi i  1, 2,  , n

Suppose aii  0 , by a simple transformation, and we have

i 1 n
1
xi  (bi   aij x j   aij x j )
aii j 1 j  i 1
n
1
 (bi   aij x j ) i  1, 2,  , n
aii j 1
j i

24/6/19 6
 Further we obtain the iterative formula
i 1 n
1
x ( k 1)
i  (bi   aij x j   aij x j )
(k ) (k )

aii j 1 j i 1

i  1, 2,  , n; k  0, 1, 2, 

We call the iterative formula as Jacobi’s methods.

Denote
0   a11   0 a12 a13  a1n 
     
a 0 a  a2n 
 21   22   0 a23
L   a31 a32 0 , D   a33 , U   0   
     
          an,n1 
a a  a 0   a   0 
 n1 n 2 n 1, n   nn  

24/6/19 7
 A  L  D  U and Ax  b , namely,
(L  D  U )x  b  Dx  ( L  U ) x  b
Becaus e aii  0，t he mat r i x D i s nons i ngul ar . Thus
x   D 1 ( L  U ) x  D 1b
We can write Jacobi’formula in matrix form as
（k 1)
x  BJ x (k )
 fJ
 0  aa1211
a
 a1311  
a1 n1
a11  a111n 
a

 
here   aa2221 0  aa2223   aa222n1  a22 
a2 n

1
 
BJ   D ( L  U )       , f J  D 1b
   
 
  aan1 a
 annn2  
ann1
0 
 nn ann

24/6/19 8
 2.2 Gauss-Seidel’s Formula
( k 1)
x
In the Eg. 1, when we compute 2 , we need use the
value x1 . At the time, we have got the value x1( k 1), if we use
(k )

( k 1)
x
the value 1 to compute x2( k 1), we will get a more accurate
result. Similarly, when we want to compute xi( k 1) , we have
(k )
gotten new x p (i >p≥1). Using the way, we can get a new
algorithm, that is, Gauss-Seidel’s method.
Eg. 2 Solve the Eg.1 using Gauss-Seidel’s iterative formula
and then compare its result with that of Jacobi’s formula.
Solution.
 1 ( k 1)
 x 1  （3 x (k )
2  2 x3  20）
(k )

8

 ( k 1) 1 ( k 1)
 x2  （  4 x1  x3  33） k  0,1, 2, ,
(k )

 11
 ( k 1) 1 ( k 1) ( k 1)
24/6/19 x
 3  （  6 x1  3 x 2  36） 9
 12
Using Gauss-Seidel’s method, we get the series {x(k)} as follows
k 0 1 2 3 4 5 6
x1 0 2.5000 2.9773 3.0098 2.9998 2.9999 3.0000
x2 0 2.0909 2.0289 1.9968 1.9997 2.0001 2.0000
x3 0 1.2273 1.0041 0.9959 1.0002 1.0001 1.0000

Obviously, Gauss-Seidel’s method is more efficient than

Jacobi’s method in this example.
The Gauss-Seidel’s iterative formula is as follows:
i 1 n
1
( k 1)
x
i  (bi   aij x (jk 1)   aij x (jk ) )
aii j 1 j i 1

i  1, 2, , n; k  0,1, 2, 

24/6/19 10
 A  L  D  U and Ax  b, so
（L  D  U）x=b  Dx   Lx  Ux  b
Dx ( k 1)   Lx ( k 1)  Ux ( k )  b
( D  L) x ( k 1)  Ux ( k )  b
x ( k 1)  ( D  L) 1Ux ( k )  ( D  L) 1 b
We can give the Gauss-Seidel’s iterative formula in matrix form
x ( k 1)  Bs x ( k )  f s
1 1
wher e Bs  ( D  L) U , f s  ( D  L) b

Note. Normally, Gauss-Seidel’s method is more efficient than

Jacobi’s method.
24/6/19 11
 Algorithm （ Gauss-Seidel’s method ）
1. FOR i  1 TO n
xi  0
2. DO
3. p0
4. FOR i  1 TO n
5. t  xi
n
bi   aij x j
j 1
j i
xi 
aii
6. IF | xi  t | p THEN p | xi  t |
7. WHILE ( p   )
8. OUTPUT x  ( x1 , x2 , , xn )T
24/6/19 12
 2.3 SOR Method

Successive Over Relaxation Method can be viewed as a

modification or extension of Gauss-Seidel’s method.
The Gauss-Seidel’s formula
i 1 n
1
xi( k 1)  (bi   aij x (jk 1)   aij x (jk ) )
aii j 1 j  i 1

can be written as
1  i 1 n 
x ( k 1)
x (k )
  bi   aij x j   aij x (jk ) 
( k 1)
i i
aii  
 j 1 j i 

1  i 1 n
(k ) 
denote x  bi   aij x j   aij x j 
( k 1) ( k 1)
(k )
i x i x (k )
i 
aii  j 1 j i 

24/6/19 13
then
1  i 1 n
(k ) 
xi( k 1)  xi( k )  xi( k )  bi   aij x j   aij x j 
( k 1)
 xi 
(k )

aii  j 1 j i 
We add a parameter  , we have
xi( k 1)  xi( k )  xi( k )
 i 1 n 
 bi   aij x   aij x
( k 1)
x (k )
i  j
(k )
j 
aii  j 1 j i 
 i 1 n 
（1  ）x  bi   aij x a x
( k 1)
(k )
i  j  ij
(k )
j 
aii  j 1 j i 1 
We call the formula SOR method. In particular, when  =1,
the SOR becomes the Gauss-Seidel’ method.

24/6/19 14
SOR in matrix form
Dx ( k 1)  (1   ) Dx ( k )   (b  Lx ( k 1)  Ux ( k ) )
( D   L) x ( k 1)  [(1   ) D  U ]x ( k )   b

So, we have
x ( k 1)  B x ( k )  f
wher e B  ( D   L) 1[(1   ) D  U ]
f   ( D   L) 1 b
Eg. 3 Solve the follwing system using Gauss-Seidel’s method
and SOR with ω=1.45
 4 x1  2 x2  x3  0

 2 x1  4 x2  2 x3  2
  x  2 x  3x  3
24/6/19
 1 2 3
15
Solution. Let x(0)=(1,1,1)T and precision
|| x ( k 1)  x ( k ) ||  max | xi( k 1)  xi( k ) | 107
1i  n

The exact solution x=(1,1,2)T

Using Gauss-Seidel’s method
x (71)  (0.9999943, 0.9999936,1.999994)T
x (72)  (0.9999952, 0.9999945,1.999995)T
Using SOR with ω=1.45
x (24)  (0.9999987, 0.9999992, 2.000000)T
x (25)  (0.9999994, 0.9999998, 2.000000)T

24/6/19 16
§3 Convergence Analysis for Iterative Methods

（1） ( A) || A ||,

（2）lim Ak  0   ( A)  1
k 

Let lim x ( k )  x ， and denote

k 
 (k )
x (k )
x 

 ( k 1)  x ( k 1)  x  ( Bx ( k )  f )  ( Bx  f )

 B ( x ( k )  x )  B ( k )
We have  ( k )  B ( k 1)    B k  (0)

24/6/19 17
Theorem. Assume the iterative formula x ( k 1)  Bx ( k )  f
is convergent, then

lim x ( k )  x  lim  ( k )  0  lim B k  0

k  k  k 

Theorem 3.1 For any initial vector x(0) and vector f ， the
1)
x ( kby
series {x(k)} created  Bx ( k )  f are convergent iff
ρ(B)<1.
Proof. It1) is
Sufficient
obviouscondition.
by the previous Theorem.
2)Necessary condition.

Suppos e  ( B)  1, t hen 1 i s n' t an ei genval ue of B, and | I  B | 0，s ys t em

( I  B) x  f
has a uni que s ol ut i on x* , s uch t hat
x*  Bx*  f

24/6/19 18
 ( k 1)
Thus x  x  B( x
* (k )
x )
*

 ( k 1)  B k 1 (0)

Acor di ng t o r es ul t （2）, we have lim B k  0
k 

as a cons equence, lim  ( k )  0  lim x ( k )  x*

k  k 

Theorem 3.1 gives a necessary and sufficient condition about

convergence analysis of iterative methods. We can use it to
decide whether a iterative method is convergent or not.
Normally, computingρ(B) is much more difficult than
computing ||B||. On the other hand, because ρ(B) ≤||B|| , we can
decide ρ (B) <1 if ||B||<1.

24/6/19 19
Theorem 3.2 If ||B||<1 ， the series {x(k)} created by x(k+1)=Bx(k)+f
converge to the exact x* and the following inequations hold
 || B ||
(1) || x (k )
 x || || x ( k )  x ( k 1) ||
1 || B ||
 || B ||k
(2) || x (k )
 x || || x (1)  x ( 0) ||
1 || B ||

24/6/19 20
Eg. 3.4 Decide whether Jacobi’s method and Gauss-Seidel’s
method in the Eg 3.1 and 3.2 are convergent or not.
8 3 2
 
A  4 11 1
6 3 12   3 2
  0 
8 8
 
4 1 
Solution. 1 BJ   D 1 ( L  U )    0 .
 11 11 
 
  6 
3
0 
 12 12 

5 5 9 9
|| BJ ||  max{ , , }  1
8 11 12 12

Result: Jacobi’s method is convergent.

24/6/19 21
 Solution. 2  3

2
0 8 8
 
3 2 
BS  ( D  L ) 1U   0 
 22 11 
 
 27 7 
0 
 176 88 


5 7 41 5
|| BS ||  max{ , , }  1
8 22 176 8

Result. Gauss-Seidel’s method is convergent.

Eg. 3.5 Decide whether Jacobi’s method for the following

system is convergent or not.
24/6/19 22
 3 1 0   x1   3 
    
7 4 2   x2    5 
0 3 1  x   1
  3   

Solution.  0 0 
1
3
 
BJ   D 1 ( L  U )    74 0  12 
 0 3 0 

7 10 1 10
|| BJ ||1  max{ , , } 1
4 3 2 3
1 9
|| BJ ||  max{ , , 3}  3  1
3 4
11 11
|
Otherwise, JB   I |   3
    ( BJ )  1
12 12
So, Jacobi’s method is convergent.
24/6/19 23
Definition 3.2If A=(aij)n×n satisfies
n
| aii |  |aij | i  1, 2, , n
j 1
j i

We say A is 3.3If
Definition strictlyA=(a
diagonal dominant.
ij)n×n satisfies
n
| aii |  |aij | i  1, 2, , n
j 1
j i

We say A is 3.4
Definition diagonal dominant.
If A can be changed the form
 A11 A12 
 
 0 A22 
only by the exchanging between rows and between
columns, where A11 and A22 are square matrices. We say A
is reducible,
24/6/19 otherwise, irreducible. 24
 Theorem 3.3 （ 1 ） If A is strictly diagonal
dominant, then A is nonsingular.
（ 2 ） If A is irreducible and diagonal
dominant, then A is nonsingular.
Theorem 3.4 If A is strictly diagonal dominant or
irreducible and diagonal dominant, then Jacobi and
Gauss-Seidel’s method are convergent.
Eg. 3.6  8 x1  3 x2  2 x3  20

For system  4 x1  11x2  x3  33
6 x  3 x  12 x  36
 1 2 3

Show Jacobi and Gauss-Seidel’s method is convergent.

 8 3 2 
 
Because the matric A   4 11 1 is strictly diagonal
 6 3 12 
 
dominant, then methods converge.

24/6/19 25
 Theorem 3.5 If SOR method converges, then 0<ω<2.
Proof.Let  ， ，， ar e ei genval us of B , we have
1 2 n 

| det B || 12  n | [  ( B )]n

1
| det B |   ( B )  1
n

and det B  det( D   L) 1 det[(1   ) D  U ]

(1   ) n a11a22  ann
  (1   ) n
a11a22  ann
As a r es ul t |1   | 1  0    2
Theorem 3.6 If A is real symmetric positive definite and 0<ω<2
， then SOR method is convergent.
Theorem 3.7 If A is strictly diagonal dominant, if 0<ω≤1, SOR
method is convergent.
24/6/19 26
 Eg. 3.7 4 2 1
 
A  2 2 1
1 1 1
 
Show those methods above mentioned convergence.
Solution. 4 2 1
4 2
A' =A，and 4  0,  4  0, 2 2 1  2  0.
2 2
1 1 1
Then Seidel and SOR(0< ω <2) are convergent. To show the
convergence of Jacobi method, we compute ρ (BJ).
 0  0.5  0.25 
 
BJ    1 0  0.5 
 1 1 0 
 
 0.5 0.25
 I  BJ  1  0.5  3  1.25  0.5  0
1 1 
24/6/19 27