Numerical Solution of System of

Linear Equations

M. Humayun Kabir, PhD

Department of Mathematics
Jahangirnagar University
Dhaka, Bangladesh
5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 1
 Numerical methods
for solution of system
of linear equations

Direct Iterative
Methods Methods

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 2

 Chapter 2:
Direct Methods for solving linear System

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 3

Direct Methods for Linear System
We discuss direct methods for solving linear systems 𝐴𝑥 = 𝑏. The most basic method
for solving linear systems directly is Gaussian elimination which you will all have
practiced in linear algebra when you solved small linear systems. A formalized way of
describing Gaussian elimination via left-multiplication with elementary lower
triangular matrices leads us to the LU factorization of a matrix 𝐴 into 𝐴 = 𝐿𝑈, where 𝐿
is a normalized lower triangular matrix and 𝑈 an upper triangular matrix.

The methods to be discussed in this chapter are called direct methods because we
solve (in some clever way) the linear system directly. Direct methods are to be seen in
contrast to so-called iterative methods, where we compute a sequence of
approximations of the solution. Some well-known direct methods for solving linear
system are as follows:

1. LU Decomposition
2. QR Factorization
3. Cholesky Factorization

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 4

LU Decomposition

Elementary Lower Triangular Matrix:

For 1 ≤ 𝑖 ≤ 𝑛, let 𝒙 ∈ ℂ𝑛 such that 𝒆𝑗𝑇 𝒙 = 0 , for 𝑗 ≤ 𝑖, that is, 𝒙 is of the form
𝒙 = 0,0, ⋯ , 0, 𝑥𝑖+1 , ⋯ , 𝑥𝑛 𝑇 . The elementary lower triangular matrix 𝐿𝑖 𝑥 , is given by
1 0 0 ⋯ ⋯ ⋯ ⋯ ⋯ 0
0 1 0 ⋮
0 0 1 ⋱ ⋮
⋮ ⋱ ⋱ ⋱ ⋮
𝑇
𝐿𝑖 𝒙 ≔ 𝐼 − 𝒙𝒆𝑖 = ⋮ 0 1 0 ⋯ ⋯ 0
⋮ . . 0 −𝑥𝑖+1 1 0 ⋯ 0
⋮ ⋮ ⋮ 0 1 ⋱ ⋮
⋮ . . ⋮ ⋮ ⋮ ⋱ ⋱ 0
0 ⋯ ⋯ 0 −𝑥𝑛 0 ⋯ 0 1
5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 5
Elementary Lower Triangular Matrix
Note that in 𝐿𝑖 (𝒙) the vector 𝒙 has been subtracted from the ith column of the
identity matrix.

Properties of Elementary Lower Triangular Matrix

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 6

LU Decomposition
We consider a system of linear equations 𝐴𝒙 = 𝒃, where 𝐴 ∈ ℂ𝑛×𝑛 , 𝒙 ∈ ℂ𝑛 , 𝒃 ∈ ℂ𝑛 . We know
that each step in the Gaussian elimination by multiplication with an elementary lower
triangular matrix: Indeed with suitable lower triangular matrices
𝐿1 𝒎1 , 𝐿2 𝒎2 , ⋯ , 𝐿𝑛−1 𝒎𝑛−1 the result of Gaussian elimination for the linear system
𝐴𝒙 = 𝒃 can be written as 𝑈𝒙 = 𝒃′ .

𝑈 = 𝐿𝑛−1 𝒎𝑛−1 ⋯ 𝐿2 𝒎2 𝐿1 𝒎1 𝐴, 𝑏′ = 𝐿𝑛−1 𝒎𝑛−1 ⋯ 𝐿2 𝒎2 𝐿1 𝒎1 𝒃

and where 𝑈 is now upper triangular.

Therefore, we can construct a sequence of 𝑛 − 1 elementary lower triangular matrices

𝐿𝑗 = 𝐿𝑗 𝒎𝑗 , 𝑗 = 1,2, ⋯ , 𝑛 − 1, such that 𝐿𝑛−1 ⋯ 𝐿2 𝐿1 𝐴 = 𝑈. Since elementary lower triangular
matrices are invertible, we have
𝐴 = 𝐿1 −1 𝐿2 −1 ⋯ 𝐿𝑛−2 −1 𝐿𝑛−1 −1 𝑈.

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 7

LU Decomposition

Using 𝐿𝑖 −1 = 𝐿𝑖 𝒎𝑖 −1 = 𝐿𝑖 −𝒎𝑖 and 𝒆𝑗𝑇 𝒎𝑘 = 0 for 𝑗 ≤ 𝑘, we have

𝐴 = 𝐿1 −𝒎1 𝐿2 −𝒎2 ⋯ 𝐿𝑛−2 −𝒎𝑛−2 𝐿𝑛−1 −𝒎𝑛−1 𝑈

= 𝐼 + 𝒎1 𝒆1𝑇 𝐼 + 𝒎2 𝒆𝑇2 ⋯ 𝐼 + 𝒎𝑛−2 𝒆𝑇𝑛−2 𝐼 + 𝒎𝒏−1 𝒆𝑇𝒏−1 𝑈

= 𝐼 + 𝒎1 𝒆1𝑇 + 𝒎2 𝒆𝑇2 + ⋯ + 𝒎𝑛−2 𝒆𝑇𝑛−2 + 𝒎𝒏−1 𝒆𝑇𝒏−1 𝑈

= 𝐿𝑈
where the matrix 𝐿: = 𝐼 + 𝒎1 𝒆1𝑇 + 𝒎2 𝒆𝑇2 + ⋯ + 𝒎𝑛−2 𝒆𝑇𝑛−2 + 𝒎𝒏−1 𝒆𝑇𝒏−1
is lower triangular with all diagonal elements one.

Therefore, the system 𝐴𝒙 = 𝒃 ⇔ 𝐿𝑈𝒙 = 𝒃, and we proceed as follows:

1. Solve 𝐿𝒚 = 𝒃 by many forward substitutions (in parallel).
2. Solve 𝑈𝒙 = 𝒚 by many back substitutions (in parallel).

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 8

A Reasonable way of Computing the LU Decomposition
A possible way of deriving an algorithm for calculating the LU factorization starts with the
component-wise formulation of 𝐴 = 𝐿𝑈, that is,
𝑛

𝑎𝑖,𝑗 = 𝑙𝑖,𝑘 𝑢𝑘,𝑗 , 1 ≤ 𝑖, 𝑗 ≤ 𝑛. (1)

𝑘=1

Since 𝑈 = 𝑢𝑖,𝑗 is upper triangular and 𝐿 = 𝑙𝑖,𝑗 is lower triangular, the upper limit of
the sum is actually given by min 𝑖, 𝑗 . Taking the two possible cases separately gives
𝑗

𝑎𝑖,𝑗 = 𝑙𝑖,𝑘 𝑢𝑘,𝑗 , 1≤𝑗≤𝑖≤𝑛 (2)

𝑘=1

𝑎𝑖,𝑗 = 𝑙𝑖,𝑘 𝑢𝑘,𝑗 , 1 ≤ 𝑖 ≤ 𝑗 ≤ 𝑛. (3)

𝑘=1

For the convenience, it is useful to swap the indices 𝑖 and 𝑗 in (3), giving

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 9

A Reasonable way of Computing the LU Decomposition
𝑖

𝑎𝑗,𝑖 = 𝑙𝑗,𝑘 𝑢𝑘,𝑖 , 1 ≤ 𝑖 ≤ 𝑗 ≤ 𝑛. (4)

𝑘=1

Rearranging these equations, and using the fact that 𝑙𝑖,𝑖 = 1 for all 1 ≤ 𝑖 ≤ 𝑛, (3) and
(2) imply (3) and (4) below
𝑖−1
1
𝑙𝑗,𝑖 = 𝑎 − 𝑙𝑗,𝑘 𝑢𝑘,𝑖 , 1 ≤ 𝑖 ≤ 𝑛, 𝑖 ≤ 𝑗 ≤ 𝑛,
𝑢𝑖,𝑖 𝑗,𝑖
𝑘=1
𝑖−1

𝑢𝑖,𝑗 = 𝑎𝑖,𝑗 − 𝑙𝑖,𝑘 𝑢𝑘,𝑗 , 1 ≤ 𝑖 ≤ 𝑛, 𝑖 ≤ 𝑗 ≤ 𝑛,

𝑘=1
Example:
1 1 1 1 2 1
𝐴= 2 3 5 , 𝐴 = −1 1 2
4 6 8 2 2 4

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 10

LU Decomposition
Theorem 1:

Proof:
We know that Gaussian elimination, if it can be successfully applied, leads to an LU
factorization. Thus the proof is given by induction over the steps in the Gaussian
elimination.
Initial step: For 𝑘 = 1 we have det 𝐴1 = det 𝑎1,1 = 𝑎1,1 ≠ 0, hence the first step in the
Gaussian elimination is possible.
Induction step: For 𝑘 → 𝑘 + 1 assume our matrix 𝐴 ∈ ℂ𝑛×𝑛 . Then, we have with
suitable elementary lower triangular matrices 𝐿1 , 𝐿2 , ⋯ , 𝐿𝑘−1 , 𝐿𝑘
𝐴(𝑘) = 𝐿𝑘 𝐿𝑘−1 ⋯ 𝐿2 𝐿1 𝐴.

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 11

LU Decomposition

From the fact that multiplication with 𝐿𝑖 from the left subtracts multiples of the 𝑗th
row from other rows with row number > 𝑗 , and from the properties of the
determinant, the following is easily seen: multiplication of 𝐴 with lower triangular
matrices from the left, does not alter the value of the determinant of the 𝑝th principal
submatrix. Hence
(2) (𝑘)
0 ≠ det 𝐴𝑘 = det 𝐿𝑘 𝐿𝑘−1 ⋯ 𝐿2 𝐿1 𝐴 𝑘 = det 𝐴(𝑘) 𝑘
= 𝑎1,1 𝑎2,2 ⋯ 𝑎𝑘,𝑘 .
(𝑘)
So, particularly 𝑎𝑘,𝑘 ≠ 0. Hence, the next step in the
Gaussian elimination is possible as well, and after 𝑛 − 1 steps we have the LU
factorization of 𝐴.
LU decomposition was introduced by Polish
mathematician Tadeusz Banachiewicz in 1938.

Tadeusz Banachiewicz
Polish Mathematician
(1882 ~ 1954)
5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 12
LU Decomposition
Theorem 2:
If an invertible matrix 𝐴 has an LU factorization then its LU factorization is unique.

Proof:
Suppose 𝐴 = 𝐿1 𝑈1 and 𝐴 = 𝐿2 𝑈2 are two LU factorizations of a non-singular matrix 𝐴.
As 𝐴 is non-singular, we have 0 ≠ det 𝐴 = det 𝐿𝑖 det 𝑈𝑖 for 𝑖 = 1,2. Thus we know
that det 𝑈𝑖 ≠ 0 and det 𝐿𝑖 ≠ 0 for 𝑖 = 1,2 and hence 𝐿𝑖 and 𝑈𝑖 , 𝑖 = 1,2, are invertible.
Therefore,
𝐿1 𝑈1 = 𝐿2 𝑈2 = 𝐴 ⇒ 𝐿2 −1 𝐿1 = 𝑈2 𝑈1 −1
Furthermore, 𝐿2 −1 𝐿1 is a product of normalized lower triangular matrices. Hence, it is
a normalized lower triangular matrix. However, 𝑈2 𝑈1 −1 is upper triangular, since 𝑈2
and 𝑈1 −1 are upper triangular. The only possible answer to this is that both 𝐿2 −1 𝐿1
and 𝑈2 𝑈1 −1 must be diagonal matrices and equal to the identity matrix (since 𝐿2 −1 𝐿1
is normalized, that is, has entries one on the diagonal). Thus
𝐿2 −1 𝐿1 = 𝑈2 𝑈1 −1 = 𝐼 ⇒ 𝐿1 = 𝐿2 and 𝑈1 = 𝑈2 ,
d

and we see that the LU factorization is unique.

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 13

Flop count of LU Decomposition
Developing a flop count will tell how much work is actually involved in computing L and
U. We will count first for i = 1, then i = 2, and so forth until i = n − 1 and form the sum of
the counts. The annotated Figure 1 will aid in understanding the computation.

Flop Count:
i = 1:
𝑎21 𝑎31 𝑎
The n − 1 quotients , , ⋯ , 𝑛1 are the row multipliers.
𝑎11 𝑎11 𝑎11
The elements at indices (2, 1) , (3, 1) , . . . , (n, 1) are
replaced by the multipliers, so row elimination operations
occur in the (n − 1) × (n − 1) submatrix A (2 : n, 2 : n).
There are (𝑛 − 1)2 elements that must be modified. To
modify an element requires a multiplication and a Figure 1
subtraction, or 2 flops. The total flops required for row
elimination is 2(𝑛 − 1)2 . For i = 1, the flop count is
n − 1 + 2(𝑛 − 1)2 .

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 14

Flop count of LU Decomposition
i = 2:
𝑎32 𝑎42 𝑎
The n − 2 quotients , , ⋯ , 𝑛2 are the row multipliers. There are (𝑛 − 2)2 elements that
𝑎22 𝑎22 𝑎22
must be modified in rows 3 through n, and the total flops required for the modification is
2(𝑛 − 2)2 . For i = 2, the flop count is n − 2 + 2(𝑛 − 2)2 .

i = n − 1:
𝑎𝑛,𝑛−1
Compute 1 quotient and execute 1 multiplication and 1 subtraction, for a total of
𝑎𝑛−1,𝑛−1
1 + 2 (1) flops.
𝑛−1 𝑛−1 𝑛−1 𝑛−1
𝑛(𝑛 − 1) 𝑛 − 1 𝑛(2𝑛 − 1)
The total count is (𝑛 − 𝑖) + 2(𝑛 − 𝑖)2 = 𝑖+2 𝑖2 = +2 .
2 6
𝑖=1 𝑖=1 𝑖=1 𝑖=1

2 3 1 2 1
Hence, the flop count is 𝑛 − 𝑛 − 𝑛.
3 2 6
2 3 2
The dominant term is 𝑛 , so the flop count is 𝑛3 + 𝒪 𝑛2 , and LU decomposition is
3 3
a cubic algorithm.

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 15

Example of LU Decomposition
Solve the system of linear equations 𝐴𝒙 = 𝒃 using LU decomposition, where

1 6 2 1 𝑥1 5
2 2 8 9 𝑥 12 .
𝐴= 𝒙 = 𝑥2 𝒃=
12 5 1 9 3 1
−1 −7 1 5 𝑥4 −12

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 16

Cholesky Decomposition

Cholesky Factorization/Decomposition:
A factorization of an 𝑛 × 𝑛 matrix 𝐴 of the form 𝐴 = 𝐿𝐿∗ with an
𝑛 × 𝑛 lower triangular matrix 𝐿 is called a Cholesky factorization
of 𝐴.
It remains to verify for what kind of matrices a Cholesky André-Louis Cholesky
factorization exists. From 𝐿𝐿 = 𝐿 𝐿 = 𝐿𝐿 , we see that
∗ ∗ ∗ ∗ ∗ ∗
French Mathematician
only Hermitian matrices can have a Cholesky factorization. (1875 ~ 1918)

A matrix 𝐴 is Hermitian if 𝐴 = 𝐴∗ .

LU factorization gives 𝐴 = 𝐿𝑈 so that 𝐴 = 𝐴∗ = 𝐿𝑈 ∗ = 𝑈 ∗ 𝐿∗ (𝑈 ∗ is a lower triangular

matrix and 𝐿∗ is normalized upper triangular matrix). To ensure the uniqueness, we
must have 𝐿 = 𝑈 ∗ , 𝑈 = 𝐿∗ . Unfortunately, it‟s not possible because 𝑈 ∗ is not normalized!!!

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 17

Cholesky Decomposition
Theorem:
Suppose 𝐴 ∈ ℂ𝑛×𝑛 is Hermitian (that is 𝐴 = 𝐴∗ ) and positive definite. Then, 𝐴 possesses a
Cholesky factorization/decomposition.

Proof:
If a matrix 𝐴 is positive definite then the determinant of every upper principal submatrix
𝐴𝑝 is positive.
We know that LU factorization is unique and that gives 𝐴 = 𝐿𝑈 so that 𝐴 = 𝐴∗ = 𝐿𝑈 ∗ =
𝑈 ∗ 𝐿∗ (𝑈 ∗ is a lower triangular matrix and 𝐿∗ is normalized upper triangular matrix). To
ensure the uniqueness, we must have 𝐿 = 𝑈 ∗ , 𝑈 = 𝐿∗ . Unfortunately, it‟s not possible
because 𝑈 ∗ is not normalized.

But if 𝐴 is invertible, the LU factorization is unique and we can write 𝑈 = 𝐷𝑈 with a

diagonal matrix 𝐷 = diag 𝑑1,1 , 𝑑2,2 , ⋯ , 𝑑𝑛,𝑛 and a normalized upper triangular matrix 𝑈 .
∗
Then, we have 𝐴 = 𝐿𝐷𝑈 and hence 𝐴∗ = 𝐿𝐷𝑈 = 𝑈 ∗𝐷 ∗𝐿∗ = 𝐴 = 𝐿𝐷𝑈 = 𝐿𝑈 so that we can
apply the uniqueness result to derive 𝐿 = 𝑈 ∗ and 𝑈 = 𝐷𝑈 = 𝐷 ∗ 𝐿∗ which gives

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 18

Cholesky Decomposition
𝐴 = 𝐿𝐷 ∗ 𝐿∗
Applying now 𝐴∗ = 𝐴 and using 𝐿 is invertible, we find
𝐴∗ = 𝐿𝐷 ∗ 𝐿∗ ∗ = 𝐿𝐷𝐿∗ = 𝐴 = 𝐿𝐷 ∗ 𝐿∗ ⇒ 𝐿𝐷 ∗ 𝐿∗ = 𝐿𝐷𝐿∗ ⇒ 𝐷 ∗ = 𝐷

That is, the diagonal matrix has only real entries. Thus 𝐴 = 𝐿𝐷𝐿∗ , where 𝐿 is a
normalized lower triangular matrix, 𝐷 is a diagonal matrix with real entries.
Therefore, 𝐴 = 𝐿𝐷𝐿∗ ⇔ 𝐿−1 𝐴 𝐿−1 ∗ = 𝐷,
with a diagonal matrix 𝐷 = diag 𝑑1,1 , 𝑑2,2 , ⋯ , 𝑑𝑛,𝑛 , where 𝑑𝑖,𝑖 ∈ ℝ for all i = 1,2,...,n. Finally,
since 𝐴 is positive definite and 𝐿−1 non-singular,
∗
𝑑𝑖,𝑖 = 𝒆∗𝑖 𝐷𝒆𝑖 = 𝒆∗𝑖 𝐿−1 𝐴 𝐿−1 ∗ 𝒆𝑖 = 𝐿−1 ∗ 𝒆𝑖 𝐴 𝐿−1 ∗ 𝒆𝑖 > 0,

that is, the diagonal matrix 𝐷 has positive diagonal entries. As explained above, we can
draw the root 𝐷1/2 = diag 𝑑1,1 , 𝑑2,2 , ⋯ , 𝑑𝑛,𝑛 of and we obtain the Cholesky
factorization of 𝐴 in the following form. ∗
𝐴 = 𝐿𝐷1/2 𝐷1/2 𝐿∗ = 𝐿𝐷1/2 𝐿𝐷1/2 = 𝐿𝐿∗ with 𝐿 = 𝐿𝐷1/2 .

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 19

A Reasonable Way of Computing Cholesky Decomposition
The Cholesky factorization of a real 𝑛 × 𝑛 matrix 𝐴 reads 𝐴 = 𝐿𝐿𝑇 with a lower triangular
matrix 𝐿. For any 𝑖 ≥ 𝑗 we have
𝑗 𝑗−1

𝑎𝑖,𝑗 = 𝑙𝑖,𝑘 𝑙𝑗,𝑘 = 𝑙𝑖,𝑘 𝑙𝑗,𝑘 + 𝑙𝑖,𝑗 𝑙𝑗,𝑗

𝑘=1 𝑘=1
This can first be resolved for 𝑖 = 𝑗:
1/2
𝑗−1

𝑙𝑗,𝑗 = 𝑎𝑗,𝑗 − 𝑙𝑗,𝑘 2 .

𝑘=1
With this, we can successively compute the other coefficients. For 𝑖 > 𝑗 we have
𝑗−1
1
𝑙𝑖,𝑗 = 𝑎 − 𝑙𝑖,𝑘 𝑙𝑗,𝑘 .
𝑙𝑗,𝑗 𝑖,𝑗
𝑘=1

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 20

Example of Cholesky Decomposition
Example:
Find the Cholesky factorization of the following positive definite matrix 1 −1 2
𝐴 = −1 5 0
Algorithm of Cholesky Decomposition 2 0 6

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 21

Flop Counts of Cholesky Decomposition
The Cholesky factorization of a positive definite matrix gives the following steps
1/2 𝑗−1
𝑗−1
2
1
𝑙𝑗,𝑗 = 𝑎𝑗,𝑗 − 𝑙𝑗,𝑘 and 𝑙𝑖,𝑗 = 𝑎𝑖,𝑗 − 𝑙𝑖,𝑘 𝑙𝑗,𝑘 .
𝑙𝑗,𝑗
𝑘=1 𝑘=1

𝑗−1 2
The computation 𝑎𝑗,𝑗 − 𝑘=1 𝑙𝑗,𝑘 requires 1 + 2(𝑗 − 1) flops. Most of the work takes place
in the inner loop
𝑗−1
1
𝑙𝑖,𝑗 = 𝑎𝑖,𝑗 − 𝑙𝑖,𝑘 𝑙𝑗,𝑘 , 𝑖 = 𝑗 + 1, ⋯ , 𝑛
𝑙𝑗,𝑗
𝑘=1
It requires 𝑛 − 𝑗 2 + 2 𝑗 − 1 = 2𝑗(𝑛 − 𝑗) flops, so the total flop count for iteration 𝑖 is
1 + 2 𝑖 − 1 + 2𝑖 𝑛 − 𝑖 .
So, the algorithm flop count is
𝑛 𝑛 𝑛 𝑛

1 + 2 𝑖 − 1 + 2𝑖(𝑛 − 𝑖) = 1+2 (𝑖 − 1) + 2 𝑖(𝑛 − 𝑖)

𝑖=1 𝑖=1 𝑖=1 𝑖=1

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 22

Flop Counts of Cholesky Decomposition
𝑛 𝑛 𝑛 𝑛 𝑛 𝑛 𝑛 𝑛

1+2 (𝑖 − 1) + 2 𝑖 𝑛−𝑖 = 1+2 𝑖−2 1 + 2𝑛 𝑖−2 𝑖2

𝑖=1 𝑖=1 𝑖=1 𝑖=1 𝑖=1 𝑖=1 𝑖=1 𝑖=1
𝑛 𝑛 𝑛 𝑛

=− 1+2 𝑖 + 2𝑛 𝑖−2 𝑖2
𝑖=1 𝑖=1 𝑖=1 𝑖=1

𝑛3 5𝑛 𝑛 3
= − 𝑛2 + = + 𝒪 𝑛2
3 3 3
Like the LU decomposition, Cholesky decomposition is 𝒪 𝑛3 , but the leading term of
2𝑛3
the LU decomposition flop count is . Cholesky decomposition requires computing
3
n square roots, but the flop count for those computations is not significant compared
𝑛3
to flops. Thus, Cholesky decomposition is approximately twice as fast as the LU
3
decomposition for a positive definite matrix.

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 23

Solving 𝑨𝒙 = 𝒃 by Cholesky Decomposition
If a matrix 𝐴 is positive definite, it is very straightforward to solve 𝐴𝑥 = 𝑏.
a. Use the Cholesky decomposition to obtain 𝐴 = 𝐿𝐿𝑇 .
b. Solve the lower-triangular system 𝐿𝑦 = 𝑏.
c. Solve the upper-triangular system 𝐿𝑇 𝑥 = 𝑦.

Solve the system of linear equations 𝐴𝒙 = 𝒃 using Cholesky decomposition, where

1 1 4 −1 𝑥1 1
𝑥
𝐴= 1 5 0 −1 𝒙 = 𝑥2 𝒃= 2 .
4 0 21 −4 3 1
−1 −1 −4 10 𝑥4 4

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 24

QR Decomposition

QR Factorization:
A factorization of an 𝑛 × 𝑛 matrix 𝐴 of the form 𝐴 = 𝑄𝑅 with a unitary matrix 𝑄
and an 𝑛 × 𝑛 upper triangular matrix 𝑅 is called a QR
factorization/Decomposition of 𝐴.

A typical step in the Gaussian elimination process multiplies the given matrix 𝐴
with a lower triangular matrix 𝐿 from the left. For the condition number of the
resulting matrix, this means

𝜅 𝐿𝐴 = 𝐿𝐴 𝐿𝐴 −1 ≤ 𝐿 𝐴 𝐴−1 = 𝜅 𝐴 𝜅 𝐿

so that we expect to end up with a worse condition number.

From this point of view it would be good to have a process which transforms the
matrix 𝐴 into an upper triangular matrix without altering the condition number.
Such a process is given by the QR factorization.

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 25

QR Decomposition
Theorem:
Suppose 𝐴 ∈ ℂ𝑛×𝑛 possesses a QR factorization 𝐴 = 𝑄𝑅 with a unitary matrix 𝑄 and an
upper triangular matrix 𝑅.

Proof: We know that a Householder matrix is a matrix of the form 𝐻 = 𝐻 𝒘 = 𝐼 − 2𝒘𝒘∗ ,

where 𝒘𝒘∗ = 1, and that the vector 𝒘 ∈ ℂ𝑛 can be chosen such that 𝐻𝒙 = 𝑐𝒆1 for a given
vector 𝒙 ∈ ℂ𝑛 . The coefficient 𝑐 ∈ ℂ satisfies 𝑐 = 𝒙 2 . Also remember that 𝐻(𝒘) is
Hermitian and unitary and that det(𝐻 𝒘 ) = −1 for 𝒘 ≠ 0.
Hence, if we let 𝒙 be the first column of 𝐴, we find a first Householder matrix 𝐻1 ∈ ℂ𝑛×𝑛
such that the first column vector of 𝐴 is mapped by 𝐻1 onto 𝛼1 𝒆1 , that is,
𝛼1 ∗ ⋯ ∗
0
𝐻1 𝐴 = ,
⋮ 𝐴1
0
with 𝛼1 = 𝒙 2 and (𝑛 − 1) × (𝑛 − 1) an matrix 𝐴1 . Considering the first column
𝒙 ∈ ℂ𝑛−1 of 𝐴1 , we can find a second Householder matrix 𝐻2 ∈ ℂ(𝑛−1)×(𝑛−1) such that
𝐻2 𝒙 = 𝛼2 𝒆1 where now 𝒆1 ∈ ℂ𝑛−1 and 𝛼2 = 𝒙 2 .
5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 26
QR Decomposition
The matrix 1 0 ⋯ 0
0
𝐻2 = ,
⋮ 𝐻2
0 𝛼1 ∗ ∗ ⋯ ∗
0 𝛼2 ∗ ⋯ ∗
is easily seen to be unitary and we get 𝐻2 𝐻1 𝐴 = 0 0 ,
⋮ ⋮ 𝐴2
0 0

with an (𝑛 − 2) × (𝑛 − 2) an matrix 𝐴2 . We can proceed in this fashion to derive

𝐻𝑛−1 𝐻𝑛−2 ⋯ 𝐻2 𝐻1 𝐴 = 𝑅,
where 𝑅 is an upper triangular matrix. Since all Householder matrices 𝐻𝑗 are unitary,
so is their product and also their inverses. Therefore
𝐴 = 𝑄𝑅, with the unitary matrix 𝑄: = 𝐻1 −1 𝐻2 −1 ⋯ 𝐻𝑛−2 −1 𝐻𝑛−1 −1 .
From the construction of the unitary matrices 𝐻𝑗 , it is easily seen that 𝐻𝑗 = 𝐻𝑗 ∗ = 𝐻𝑗 −1 ,
and thus 𝑄 can be written as
𝑄 = 𝐻1 𝐻2 ⋯ 𝐻𝑛−2 𝐻𝑛−1 .

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 27

Example of QR Decomposition
Example:
2 −3 3
Find the QR factorization of the following matrix 𝐴 = −2 6 6
1 0 3

There are three different ways to compute QR decomposition a matrix:

1. Using Householder matrices, developed by Alston S. Householder
2. Using Givens rotations, also known as Jacobi rotations, used by W. Givens and
originally invented by Jacobi for use with in solving the symmetric eigenvalue
problem in 1846.
3. A third, less frequently used approach is the Gram-Schmidt orthogonalization.

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 28

Flops Count of QR Decomposition
The QR factorization of a matrix gives the following steps

Suppose we have a 𝑚 × 𝑛 matrix 𝐴, and we want to perform QR factorization using

Householder transformations.
We will have to go through 𝐴 as follows:
1. Start with the whole matrix
2. Proceed with the modified matrix of original size without it's first row and first
column.
3. Proceed with second modified matrix without first two rows and columns.
4. And so on...

In a more compact way, this is stated as 𝐴𝑘:𝑚,𝑘:𝑛 for every iteration step 𝑘. Now the
(part of the) algorithm (which does most of the work) can be denoted similarly, i.e. per
iteration we have
𝐴𝑘:𝑚,𝑘:𝑛 = 1 − 2𝑣𝑘 𝑣𝑘𝑇 𝐴𝑘:𝑚,𝑘:𝑛 = 𝐴𝑘:𝑚,𝑘:𝑛 − 2𝑣𝑘 𝑣𝑘𝑇 𝐴𝑘:𝑚,𝑘:𝑛

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 29

Flops Count of QR Decomposition
Now the operation we want to count are in these iterations. For every iteration we
have:
1. 2(𝑚 − 𝑘)(𝑛 − 𝑘) for the products 𝑣𝑘𝑇 𝐴𝑘:𝑚,𝑘:𝑛
2. (𝑚 − 𝑘)(𝑛 − 𝑘) for the outer product 2𝑣𝑘 𝑣𝑘𝑇 𝐴𝑘:𝑚,𝑘:𝑛
3. (𝑚 − 𝑘)(𝑛 − 𝑘) for the subtraction 𝐴𝑘:𝑚,𝑘:𝑛 − 2𝑣𝑘 𝑣𝑘𝑇 𝐴𝑘:𝑚,𝑘:𝑛

So that is 4(𝑚 − 𝑘)(𝑛 − 𝑘) in total. Which we do for every iteration (!), so finally we get
𝑛 𝑛

4 𝑚−𝑘 𝑛−𝑘 =4 𝑚𝑛 − 𝑘 𝑚 + 𝑛 + 𝑘 2
𝑘=1 𝑘=1

4 𝑚+𝑛 𝑛 𝑛+1 4𝑛 𝑛 + 1 2𝑛 + 1
= 4𝑚𝑛2 − +
2 6
2𝑛 3
2
≈ 2𝑚𝑛 −
3

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 30

Solving 𝑨𝒙 = 𝒃 by QR Factorization
For a full rank 𝑛 × 𝑛 matrix 𝐴, it is very straightforward to solve 𝐴𝒙 = 𝒃.
a. Use the QR Factorization to obtain 𝐴 = 𝑄𝑅.
b. Solve the upper-triangular system 𝑅𝒙 = 𝑄 ∗𝒃.

Solve the system of linear equations 𝐴𝒙 = 𝒃 using QR decomposition, where

1 1 4 −1 𝑥1 1
𝑥
𝐴= 1 5 0 −1 𝒙 = 𝑥2 𝒃= 2 .
4 0 21 −4 3 1
−1 −1 −4 10 𝑥4 4

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 31

Questions on Chapter 2:
1. Define LU factorization of a matrix. Derive LU decomposition to solve a system of
linear equations.
2. If an invertible matrix A has an LU factorization then prove that its LU
factorization is unique.
3. State algorithm of LU decomposition. Discuss flops count of LU decomposition.
4. Examples related to solve a linear system using LU decomposition.
5. Define Cholesky factorization of a matrix. Derive Cholesky decomposition to solve
a system of linear equations.
6. State algorithm of Cholesky decomposition. Discuss flops count of Cholesky
decomposition.
7. Examples related to solve a linear system using Cholesky decomposition.
8. Define QR factorization of a matrix. Derive QR decomposition to solve a system of
linear equations.
9. Mention an algorithm of the QR decomposition. Discuss flops count of QR
decomposition.
10. Examples related to solve a linear system using QR decomposition.
11. Make a comparative study of LU decomposition, Cholesky decomposition and QR
decomposition for solving a linear system in terms of their flop counts.
5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 32
Chapter 3:
Iterative Methods for solving linear System

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 33

Iterative Methods for Solving Linear System
The general idea behind iterative methods is the following. Suppose we want to
solve a linear system 𝐴𝒙 = 𝑏, where 𝐴 is an invertible square 𝑛 × 𝑛 matrix 𝐴. If 𝐴 is
very large, say 𝑛 = 106 , then a direct solver as discussed in the previous chapter is
no longer feasible for solving the linear system 𝐴𝒙 = 𝑏 (since the number of
elementary operations is 𝒪 𝑛3 ). Instead we want to use a so-called iterative method
which constructs a sequence 𝒙(𝑗) of solutions in ℝ𝑛 that approximate the true
solution 𝒙 of 𝐴𝒙 = 𝑏. For 𝑗 → ∞ , we should have that
lim 𝒙(𝑗) = 𝒙
𝑗→∞
although in reality this will not always be the case, due to rounding errors and
stability issues.
How do the ‘approximations’ 𝒙(𝒋) look like?
The most common form is
𝑥 (𝑗+1) = 𝑥 (𝑗) + 𝐵−1 𝑏 − 𝐴𝑥 (𝑗) (1)
where 𝐵 is an invertible matrix such that 𝐵−1 is an approximation of the inverse
𝐴−1 . Usually, 𝐵 will be a „simplified version‟ of 𝐴 whose inverse can be easily
computed.
5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 34
Iterative Methods for Solving Linear System
To understand (1), replace 𝐵−1 by 𝐴−1 ; we obtain (multiplying from the left by 𝐴 in the
second step)
𝑥 (𝑗+1) = 𝑥 (𝑗) + 𝐵 −1 𝑏 − 𝐴𝑥 (𝑗) ⟺ 𝐴𝑥 𝑗+1 = 𝐴𝑥 𝑗 + 𝑏 − 𝐴𝑥 𝑗 = 𝑏
that is, 𝑥 (𝑗+1) would then solve 𝐴𝑥 = 𝑏 . In (1), we have instead of 𝐴−1 , the
approximation 𝐵 −1 of 𝐴−1 , and we can interpret this as follows: Our previous iteration
step yielded the approximation 𝑥 (𝑗) of 𝑥 , and the term
𝑟 (𝑗) = 𝑏 − 𝐴𝑥 𝑗 = 𝐴 𝑥 − 𝑥 (𝑗)
is the residual of this approximation (which measures the discrepancy, mapped into
the range of 𝐴, between 𝑥 and 𝑥 (𝑗) ). We solve the the equation
𝐴𝑦 = 𝐴 𝑥 − 𝑥 (𝑗) = 𝑏 − 𝐴𝑥 (𝑗) = 𝑟 (𝑗)
approximately with the help of the approximation 𝐵 −1 of 𝐴−1 , giving 𝑦 = 𝑥 − 𝑥 𝑗 ≈
𝐵−1 𝑏 − 𝐴𝑥 and add this correction to the previous approximation; thus
𝑥 (𝑗+1) = 𝑥 (𝑗) + 𝐵 −1 𝑏 − 𝐴𝑥 (𝑗)

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 35

Iterative Methods for Solving Linear System

Sometimes it is not obvious that a complicated iterative method can be interpreted in

this sense, but we will study some suitable iterative methods in this chapter.

 Fixed Point Iteration

 Successive Over Relaxation (SOR) Method
 Steepest-Descent Method
 Conjugate Gradient Method

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 36

Fixed Point Iteration Method
Fixed Point:

Let us write the matrix 𝐴 for the linear system 𝐴𝒙 = 𝑏 as 𝐴 = 𝐴 − 𝐵 + 𝐵 with an

invertible matrix 𝐵 which can and will be chosen suitably. Then, the equation
𝐴𝒙 = 𝑏 can be reformulated as
𝑏 = 𝐴𝑥 = 𝐴 − 𝐵 𝒙 + 𝐵𝒙 ⟺ 𝐵𝒙 = 𝑏 − 𝐴 − 𝐵 𝒙 = 𝐵 − 𝐴 𝒙 + 𝑏
and hence as the fixed point equation
𝒙 = 𝐵−1 𝐵 − 𝐴 𝒙 + 𝐵−1 𝑏 ≔ 𝐶𝒙 + 𝑐

where 𝐶: = 𝐵 −1 (𝐵 − 𝐴) and 𝑐: = 𝐵 −1 𝑏. Thus the solution 𝒙 to 𝐴𝒙 = 𝑏 is a fixed point

of the mapping
𝐹 𝒙 = 𝐶𝒙 + 𝑐

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 37

Fixed Point Iteration Method
To calculate this fixed point, we can use the following simple iterative process. We first
pick a starting point 𝒙(0) and then form
𝒙(𝑗+1) ≔ 𝐹 𝒙 𝑗 , 𝑗 = 1,2,3, ⋯
From Banach‟s fixed point theorem below, we see that under certain assumptions on 𝐹
the sequence 𝒙(𝑗) converges to a fixed point of 𝐹.

Contraction Mapping:

Note that a contraction mapping is Lipschitz-continuous with Lipschitz-constant q < 1.

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 38

Fixed Point Iteration Method
Banach’s Fixed Point Theorem:

Applying this theorem to the mapping, 𝐹 𝑥 = 𝐶𝑥 + 𝑐 , where 𝐶 = 𝐵 −1 (𝐵 − 𝐴) is the

iteration matrix and 𝑐 = 𝐵 −1 𝑏, we see that
𝐹 𝑥 − 𝐹(𝑦) = 𝐶(𝑥 − 𝑦) ≤ 𝐶 𝑥 − 𝑦 , with 𝐶 < 1.

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 39

Convergence of Fixed Point Iteration
Theorem

Proof:
Assume first that 𝜌 𝐶 < 1. Then, we can pick an 𝜀 > 0 such that 𝜌 𝐶 + 𝜀 < 1. We know
that a norm ∙ for ℂ𝑛 (or ℝ𝑛 ) such that in the corresponding induced matrix norm
(which we also denote by ∙ )
𝐶 ≤ 𝜌 𝐶 + 𝜀 < 1.
Then we have
𝐹 𝑥 − 𝐹(𝑦) ≤ 𝐶 𝑥 − 𝑦 , 𝑤𝑖𝑡ℎ 𝐶 < 1,

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 40

Convergence of Fixed Point Iteration
and thus 𝐹 is a contraction. From Banach‟s fixed point theorem, the sequence 𝑥 (𝑗)
converges for every starting point 𝑥 (0) to the unique fixed point 𝑥 of 𝐹.
Assume now that, for every starting point 𝑥 (0) , the iteration 𝑥 (𝑗) converges to the
same vector 𝑥 . Since 𝐹 is continuous, we find from
𝑥 = lim 𝑥 (𝑗+1) = lim 𝐹 𝑥 (𝑗) = 𝐹 𝑥 ,
𝑗→∞ 𝑗→∞
and we see that the limit 𝑥 is a fixed point of 𝐹. If we pick the starting point 𝑥 (0) )
such that 𝑥 = 𝑥 (0) − 𝑥 is an eigenvector of 𝐶 with eigenvalue 𝜆 (that is, given the
eigenvector 𝑥, we choose 𝑥 (0) = 𝑥 + 𝑥 ), then
𝑥 (𝑗) − 𝑥 = 𝐹 𝑥 𝑗−1 − 𝐹 𝑥 = 𝐶 𝑥 𝑗−1 − 𝑥 = ⋯ = 𝐶 𝑗 𝑥 0 − 𝑥 = 𝜆𝑗 𝑥 0 − 𝑥 .
Since the expression on the left-hand side tends to the zero vector for 𝑗 → ∞, so
does the expression on the right-hand side. This, however, is only possible if 𝜆 <
1. Since 𝜆 was an arbitrary eigenvalue of 𝐶, this shows that all eigenvalues of 𝐶
satisfy 𝜆 < 1, and hence 𝜌 𝐶 < 1.

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 41

Successive Over Relaxation (SOR) Method
Successive OverRelaxation (SOR) Method, is a generalization of and improvement
on the Gauss-Seidel Method for solving a system of linear equations 𝐴𝑥 = 𝑏. SOR
method was jointly developed by David M. Young and Stanley P. Frankel. It has
faster convergence than Gauss-Jacobi and Gauss-Seidel iterative methods which
can significantly reduce the computation (CPU) time.

David M. Young Jr. Stanley Phillips Frankel

(1923 ~ 2008) (1919 ~1978)
American Mathematician American Computer Scientist

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 42

Successive Over Relaxation (SOR) Method
The SOR method was developed to accelerate the convergence of the Gauss-Seidel
method. The idea is to successively form a weighted average between the previously
computed value 𝑥 (𝑘−1) and the new value
𝑖−1 𝑛
(𝑘) 1 (𝑘) (𝑘−1)
𝑥𝑖 = 𝑏𝑖 − 𝑎𝑖𝑗 𝑥𝑗 + 𝑎𝑖𝑗 𝑥𝑗 , 𝑖 = 1,2, ⋯ , 𝑛.
𝑎𝑖𝑖
𝑗=1 𝑗=𝑖+1
Weight the newly computed value by 𝜔 and the previous value by 1 − 𝜔 . By assuming
(𝑘) (𝑘−1)
that 𝑖−1 𝑎 𝑥
𝑗=1 𝑖𝑗 𝑗 and 𝑛
𝑎 𝑥
𝑗=𝑖+1 𝑖𝑗 𝑗 are ignored when 𝑖 = 1 and 𝑛, respectively, we have
𝑖−1 𝑛
(𝑘) 𝜔 (𝑘) (𝑘−1) (𝑘−1)
𝑥𝑖 = 𝑏 − 𝑎𝑖𝑗 𝑥𝑗 + 𝑎𝑖𝑗 𝑥𝑗 + (1 − 𝜔)𝑥𝑖 , 𝑖 = 1,2, ⋯ , 𝑛.
𝑎𝑖𝑖 𝑖
𝑗=1 𝑗=𝑖+1
For this method to provide an improvement over the Gauss-Seidel method, 𝜔 must be
chosen carefully. It is called the relaxation parameter. If 𝜔 = 1, the SOR method and the
Gauss-Seidel method are identical. If 𝜔 > 1, we are said to be using overrelaxation and if
𝜔 < 1 we are using underrelaxation.

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 43

Matrix Form of Successive Over Relaxation (SOR) Method
The SOR method is given by
𝑖−1 𝑛
(𝑘) 𝜔 (𝑘) (𝑘−1) (𝑘−1)
𝑥𝑖 = 𝑏 − 𝑎𝑖𝑗 𝑥𝑗 + 𝑎𝑖𝑗 𝑥𝑗 + (1 − 𝜔)𝑥𝑖 , 𝑖 = 1,2, ⋯ , 𝑛.
𝑎𝑖𝑖 𝑖
𝑗=1 𝑗=𝑖+1

We Rearrange the equation by multiplying both sides by 𝑎𝑖𝑖 and placing all the 𝑘 terms
on the left-hand side to obtain
𝑖−1 𝑛
(𝑘) (𝑘) (𝑘−1) (𝑘−1)
𝑎𝑖𝑖 𝑥𝑖 +𝜔 𝑎𝑖𝑗 𝑥𝑗 = 𝜔𝑏𝑖 − 𝜔 𝑎𝑖𝑗 𝑥𝑗 + (1 − 𝜔)𝑎𝑖𝑖 𝑥𝑖
𝑗=1 𝑗=𝑖+1
We can now express the equation in the following matrix form
𝐷 + 𝜔𝐿 𝑥 (𝑘) = 1 − 𝜔 𝐷 − 𝜔𝑈 𝑥 (𝑘−1) + 𝜔𝑏
which leads to the specification of the SOR matrix
𝑥 (𝑘) = 𝐵𝑆𝑂𝑅 𝑥 (𝑘−1) + 𝑐𝑆𝑂𝑅
where, 𝐵𝑆𝑂𝑅 = 𝐷 + 𝜔𝐿 −1 1 − 𝜔 𝐷 − 𝜔𝑈 , 𝑐𝑆𝑂𝑅 = 𝐷 + 𝜔𝐿 −1 𝑏𝜔 and
𝑎11 0 0 0 0 0 0 𝑎12 ⋯ 𝑎1𝑛
𝑎22 𝑎 0 0 0 𝑎2𝑛
𝐷= , 𝐿 = 21 ⋱ , 𝑈 =
⋱ ⋮ ⋮ ⋱
0 𝑎𝑛𝑛 𝑎𝑛1 ⋯ 𝑎𝑛𝑛−1 0 0 ⋯ 0 0
5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 44
The Spectral Radius and Rate of Convergence
Intuitively, there should be a link between the spectral radius of the iteration matrix 𝐵𝑆𝑂𝑅 and the
rate of convergence. Suppose that 𝐵𝑆𝑂𝑅 has 𝑛 linearly independent eigenvectors, 𝑣1 , 𝑣2 , ⋯ , 𝑣𝑛 and
associated eigenvalues 𝜆1 , 𝜆2 , ⋯ , 𝜆𝑛 . Using the notation for the error 𝑒 (𝑘) = 𝑥 − 𝑥 (𝑘) at the kth step
of the iteration. Since the eigenvectors are a basis, we write
𝑛

𝑒 (0) = 𝑐𝑖 𝑣𝑖 .
𝑖=1
It follows that
𝑛 𝑛

𝑒 (1) = 𝐵𝑆𝑂𝑅 𝑒 (0) = 𝑐𝑖 𝐵𝑆𝑂𝑅 𝑣𝑖 = 𝑐𝑖 𝜆𝑖 𝑣𝑖

𝑖=1 𝑖=1
𝑛 𝑛

𝑒 (2) = 𝐵𝑆𝑂𝑅 𝑒 (1) = 𝑐𝑖 𝜆𝑖 𝐵𝑆𝑂𝑅 𝑣𝑖 = 𝑐𝑖 𝜆𝑖 2 𝑣𝑖 .

𝑖=1 𝑖=1
By continuing this fashion, there results
𝑛

𝑒 (𝑘) = 𝑐𝑖 𝜆𝑘𝑖 𝑣𝑖 .
𝑖=1

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 45

The Spectral Radius and Rate of Convergence
Let 𝜌 𝐵𝑆𝑂𝑅 = 𝜆1 and suppose that 𝜆1 > 𝜆2 ≥ 𝜆3 ≥ ⋯ ≥ 𝜆𝑛 so that
𝑛 𝑛 𝑘
𝜆𝑖
𝑒 (𝑘) = 𝑐1 𝜆𝑘1 𝑣1 + 𝑐𝑖 𝜆𝑘𝑖 𝑣𝑖 = 𝜆𝑘1 𝑐1 𝑣1 + 𝑐𝑖 𝑣𝑖 .
𝜆1
𝑖=2 𝑖=2
𝜆𝑖 𝑘
As 𝑘 becomes large, , 2 ≤ 𝑖 ≤ 𝑛 becomes small and we have 𝑒 (𝑘) ≈ 𝜆𝑘1 𝑐1 𝑣1 . This says
𝜆1
that the error varies with the 𝑘th power of the spectral radius is a good indicator for the
rate of convergence.

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 46

Convergence of Successive Over Relaxation (SOR) Method
It has seen that poor choice of the relaxation parameter 𝜔 can lead to poor convergence
rates. On the other hand, a good choice of 𝜔 can lead to very fast convergence compared
to the Jacobi or Gauss-Seidel methods. The spectral radius of 𝐵𝑆𝑂𝑅 is the eigenvalue of
𝐵𝑆𝑂𝑅 with maximum magnitude, and we want to choose 𝜔 so that 𝜌 𝐵𝑆𝑂𝑅 (𝜔) is a
minimum. Finding the optimal value of 𝜔 is very difficult in general, and the optimal
value is known only for special types of matrices. It is known, however, that 𝜔 must
satisfy 0 < 𝜔 < 2.

Necessary and Sufficient Condition for the convergence of SOR Method

Theorem:
If 𝐴 is a Hermitian (symmetric) positive definite matrix and 0 < 𝜔 < 2, the SOR iterations
converge for any choice of 𝑥 (0) .
Proof:
We write the iteration matrix for SOR method as follows:
𝐵𝑆𝑂𝑅 = 𝐷 + 𝜔𝐿 −1 1 − 𝜔 𝐷 − 𝜔𝑈 = 𝐷 + 𝜔𝐿 −1 𝐷𝐷 −1 1 − 𝜔 𝐷 − 𝜔𝑈
𝐵𝑆𝑂𝑅 = 𝐷 −1 𝐷 + 𝜔𝐿 −1 1 − 𝜔 𝐼 − 𝜔𝐷 −1 𝑈 = 𝐼 + 𝐷 −1 𝜔𝐿 −1 1 − 𝜔 𝐼 − 𝜔𝐷 −1 𝑈 . (1)

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 47

Convergence of Successive Over Relaxation (SOR) Method
Consider the representation of 𝐵𝑆𝑂𝑅 (𝜔) in the last line of (1). The first matrix in this
product is a normalized lower triangular matrix and the second matrix is an upper
triangular matrix with diagonal entries all equal to 1 − 𝜔 . Thus the determinant of
𝐵𝑆𝑂𝑅 (𝜔) is given by
𝑑𝑒𝑡 𝐵𝑆𝑂𝑅 (𝜔) = 𝑑𝑒𝑡 𝐼 + 𝐷 −1 𝜔𝐿 −1 𝑑𝑒𝑡 1 − 𝜔 𝐼 − 𝜔𝐷 −1 𝑈
Since the determinant of a matrix equals the product of its eigenvalues, we have the
following: Denote by 𝜆1 , 𝜆2 , ⋯ , 𝜆𝑛 the 𝑛 eigenvalues of 𝐵𝑆𝑂𝑅 (𝜔), then
𝑑𝑒𝑡 𝐵𝑆𝑂𝑅 (𝜔) = 𝜆1 𝜆2 ⋯ 𝜆𝑛 = 1 − 𝜔 𝑛 . (2)
It can easily be shown by proof by contradiction that (2) implies that there exists at least
one eigenvalue 𝜆𝑗 of 𝐵𝑆𝑂𝑅 (𝜔) with
𝜆𝑗 ≥ 1 − 𝜔 .
Thus, from the definition of the spectral radius,
𝜌 𝐵𝑆𝑂𝑅 (𝜔) ≥ 1 − 𝜔 . (3)
We know that the method converges if and only if 𝜌 𝐵𝑆𝑂𝑅 (𝜔) < 1. Thus from (3) convergence is only
possible if 0 < 𝜔 < 2.

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 48

Convergence of Successive Over Relaxation (SOR) Method
We now assume that the coefficient matrix 𝐴 is Hermitian and we will prove that
𝜌 𝐵𝑆𝑂𝑅 (𝜔) < 1. The iteration matrix for SOR method as follows:
𝐵𝑆𝑂𝑅 = 𝐷 + 𝜔𝐿 −1 1 − 𝜔 𝐷 − 𝜔𝑈 = 𝐷𝜔 + 𝐿 −1 𝐷 + 𝜔𝐿 − 𝜔(𝐿 + 𝐷 + 𝑈)
−1
−1
1
𝐵𝑆𝑂𝑅 = 𝐼 − 𝜔 𝐷 + 𝜔𝐿 𝐴 = 𝐼 − 𝐷+𝐿 𝐴 = 𝐼 − 𝐵 −1 𝐴,
𝜔
1
With 𝐵 = 𝐷 + 𝐿. Let 𝜆 ∈ ℂ be an eigenvalue of 𝐵𝑆𝑂𝑅 (𝜔) with corresponding eigenvector 𝒙 ∈ ℂ𝑛 ,
𝜔
which we assume to be normalized, such that 𝒙 2 = 1. Then, we have

𝐵𝑆𝑂𝑅 𝜔 𝒙 = 𝐼 − 𝐵 −1 𝐴 𝒙 = 𝜆 𝒙 ⟺ 1 − 𝜆 𝒙 = 𝐵 −1 𝐴𝒙 ⟺ 𝐴𝒙 = 1 − 𝜆 𝐵𝒙.
Since 𝐴 is positive definite, and hence invertible, we must have 𝜆 ≠ 1. (Otherwise we would have
𝐴𝑥 = 0 for the non-zero vector 𝒙 which would be in contradiction to the fact that 𝐴 is invertible). Since
𝐴 is positive definite, we have
0 < 𝒙∗𝐴𝒙 = 1 − 𝜆 𝒙∗ 𝐵𝒙,
and since 𝜆 ≠ 1,
1 𝒙∗ 𝐵𝒙
= .
1 − 𝜆 𝒙∗ 𝐴𝒙
Since 𝐴 is Hermitian, that is, 𝐴∗ = 𝐴 , we have 𝐿∗ + 𝐷 ∗ + 𝑈 ∗ = 𝐿 + 𝐷 + 𝑈 ⇒ 𝐿∗ = 𝑈, 𝐷 ∗ =
𝐷, 𝑈 ∗ = 𝐿.
5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 49
Convergence of Successive Over Relaxation (SOR) Method
1 1 ∗ 1 1 2 2
We find, 𝐵 + 𝐵∗ = 𝐷 +𝐿+ 𝐷 +𝐿 = 𝐷 +𝐿+ 𝐷 +𝑈 = 𝐷 −𝐷+𝐴= − 1 𝐷 + 𝐴.
𝜔 𝜔 𝜔 𝜔 𝜔 𝜔
1
Since 𝐴 is positive definite, for 𝑥 ∈ ℂ𝑛 \*0+, 𝒙∗ 𝐴𝒙 is real and positive. The real part of
1−𝜆
satisfies
1 𝒙∗ 𝐵𝒙 𝑅𝑒 𝒙∗ 𝐵𝒙 1 1 ∗ ∗
1 1 ∗ 𝐵𝒙 + 𝒙∗ 𝐵 ∗ 𝒙
𝑅𝑒 = 𝑅𝑒 ∗ = = 𝒙 𝐵𝒙 + 𝒙 𝐵𝒙 = 𝒙
1−𝜆 𝒙 𝐴𝒙 𝒙∗ 𝐴𝒙 2 𝒙∗ 𝐴𝒙 2 𝒙∗ 𝐴𝒙
1 1 1 1 2 1 2 𝒙 ∗ 𝐷𝒙
= ∗
𝒙∗ 𝐵 + 𝐵 ∗ 𝒙 = ∗
𝒙∗ −1 𝐷+𝐴 𝒙= −1 ∗ +1
2 𝒙 𝐴𝒙 2 𝒙 𝐴𝒙 𝜔 2 𝜔 𝒙 𝐴𝒙
2
Since 0 < 𝜔 < 2, the expression − 1 is positive. We have 0 < 𝑒𝑗∗ 𝐴𝑒𝑗 = 𝑎𝑗𝑗 , 𝑗 = 1,2, ⋯ , 𝑛
𝜔
𝒙∗ 𝐷𝒙
because 𝐴 is positive definite. Therefore is positive for all 𝑥 ∈ ℂ𝑛 \*0+, because 𝐴 and
𝒙∗ 𝐴𝒙
𝐷 are positive definite. Thus
1 1 2 𝒙∗ 𝐷𝒙 1
𝑅𝑒 = −1 ∗ +1 > .
1−𝜆 2 𝜔 𝒙 𝐴𝒙 2

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 50

Convergence of Successive Over Relaxation (SOR) Method
If we write 𝜆 = 𝑢 + 𝑖𝑣 then we can conclude that

1 1 1 1 − 𝑢 + 𝑖𝑣 1−𝑢
< 𝑅𝑒 = 𝑅𝑒 = 𝑅𝑒 = ,
2 1−𝜆 1 − 𝑢 − 𝑖𝑣 (1 − 𝑢)2 +𝑣 2 (1 − 𝑢)2 +𝑣 2
and hence
1 1−𝑢
< 2 2
⇔ (1 − 𝑢)2 +𝑣 2 < 2 1 − 𝑢 ⟺ 𝑢2 + 𝑣 2 < 1,
2 (1 − 𝑢) +𝑣
that is, 𝜆 2 = 𝑢2 + 𝑣 2 < 1. Since the eigenvalue 𝜆 of the iteration matrix 𝐵𝑆𝑂𝑅 (𝜔) was
arbitrary, we have shown that for 0 < 𝜔 < 2 𝜌 𝐵𝑆𝑂𝑅 (𝜔) < 1 which confirms that SOR
iteration method converges.

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 51

Algorithm of SOR Method for solving the system 𝑨𝒙 = 𝒃

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 52

Example: 1 1 1 −1
Solve the linear system 𝐴𝑥 = 𝑏 using SOR method, where 𝐴 = 1 2 1 , 𝑏 = 5 . Here
1 1 3 7
−11
the exact solution is 𝑥 = 6 .
4
Solution:
The iterative formula of SOR method for the system 𝐴𝑥 = 𝑏 is given by
𝑖−1 𝑛
(𝑘) 𝜔 (𝑘) (𝑘−1) (𝑘−1)
𝑥𝑖 = 𝑏𝑖 − 𝑎𝑖𝑗 𝑥𝑗 + 𝑎𝑖𝑗 𝑥𝑗 + (1 − 𝜔)𝑥𝑖 , 𝑖 = 1,2, ⋯ , 𝑛.
𝑎𝑖𝑖
𝑗=1 𝑗=𝑖+1
We choose an initial guess, 𝒙 = 0,0,0 𝑇

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 53

Steepest-Descent Method for solving the system 𝑨𝒙 = 𝒃
Steepest Descent (SD) is an elegant iterative method for solving large systems of linear
equations. SD is effective for systems
𝐴𝒙 = 𝑏 (1)
where 𝒙 is an unknown vector, 𝒃 is a known vector, and 𝐴 is a known, 𝑛 × 𝑛 square,
symmetric, positive-definite matrix.

The scalar product of two vectors is written 𝒙𝑇 𝒚, and represents the following sum:
𝒙𝑇 𝒚 = 𝑛𝑖=1 𝑥𝑖 𝑦𝑖 .

A quadratic form is a scalar quadratic function of a vector,

1
𝑓 𝒙 = 𝒙𝑇 𝐴𝑥 − 𝒃𝑇 𝒙 + 𝑐,
2
Where 𝐴 is a matrix, 𝒙 and 𝒃 are vectors, and 𝑐 is a scalar constant. It can be shown
that if 𝐴 is symmetric and positive-definite, 𝑓 𝒙 is minimized by the solution to 𝐴𝒙 = 𝒃.
The gradient of a quadratic form is defined by,
𝑇
𝜕 𝜕 𝜕
𝑓’ 𝒙 = 𝑓 𝒙 𝑓 𝒙 ⋯ 𝑓 𝒙
𝜕𝑥1 𝜕𝑥2 𝜕𝑥𝑛

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 54

Steepest-Descent Method
The gradient is a vector that, for a given point x, points in the direction of greatest
increase of 𝑓(𝒙). At the bottom of the paraboloid bowl, the gradient is zero. One can
minimize 𝑓(𝒙) by setting 𝑓’(𝒙) equal to zero, that is,
𝑓 ’ 𝒙 = 𝐴𝒙 − 𝑏 = 0.

Steepest-Descent Method: The steepest descent method was first published by

Peter Debye in 1909.

Peter Joseph William Debye

(1884 ~1966)
Dutch Physicist & Physical Chemist
Nobel Laurate in Chemistry in 1936

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 55

Steepest-Descent Method
We eager to solve the following large systems of linear equations using Steeepst Descent
method
𝐴𝒙 = 𝒃 (0)
where 𝒙 is an unknown vector, 𝒃 is a known vector, and 𝐴 is a known, 𝑛 × 𝑛 square,
symmetric, positive-definite matrix. We then consider a quadratic form of the vector 𝒙 of
the following form
1
𝑓 𝒙 = 𝒙𝑇 𝐴𝒙 − 𝒃𝑇 𝒙.
2

In the method of Steepest Descent, we start at an arbitrary point 𝒙(0) and slide down to
the bottom of the paraboloid. We take a series of steps 𝒙(1) , 𝒙(2) , ⋯ until we are satisfied
that we are close enough to the solution 𝒙.
When we take a step, we choose the direction in which 𝑓 decreases most quickly, which
is the direction opposite 𝑓 ’ 𝒙(𝑖) . According to the system 𝐴𝒙 = 𝒃,
we find −𝑓 ’ 𝒙 𝑖 = 𝒃 − 𝐴𝒙 𝑖 .

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 56

Steepest-Descent Method
The residual 𝒓(𝑖) = 𝒃 − 𝐴𝒙(𝑖) indicates how far we are from the correct value of 𝑏. It is
easy to see that 𝒓(𝑖) = 𝐴𝒙 − 𝐴𝒙 𝑖 = −𝐴 𝒙 𝒊 − 𝒙 = −𝐴𝒆 𝑖 , where 𝒆𝑖 = 𝒙(𝑖) − 𝒙. The residual
can be treated as the error transformed by 𝐴 into the same space as 𝒃 . More
importantly,
𝒓(𝑖) = −𝑓 ‘ 𝒙 𝑖 .

a. Starting at (−2, −2)𝑇

b. Intersection of two surfaces ( 𝑓 and search line)
c. Parabola (intersection of two surfaces)
d. The Gradient at the bottonmost point is
orthogonal to the gradient of the previous step.

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 57

Steepest-Descent Method
The residual can be considered as the direction of steepest descent (whenever we read
“residual”, think “direction of steepest descent”).
Suppose we start at 𝒙(0) . Our first step, along the direction of steepest descent. In other
words, we will choose a point
𝒙(1) = 𝒙(0) + 𝛼𝒓(0) (1)
A line search is a procedure that chooses 𝛼 to minimize 𝑓 along a line 𝒙(0) + 𝛼𝒓(0) .
𝑑
𝛼 minimizes 𝑓 when the derivative 𝑓 𝒙(1) is equal to zero. By the chain rule,
𝑑𝛼
𝑑𝑓 𝒙 (1) (1)
𝑑 𝑑𝒙
𝑓 𝒙(1) = = −𝑓 ‘ 𝒙 1 𝒓 0 (2)
𝑑𝛼 𝑑𝒙(1) 𝑑𝛼

Solid line: gradient 𝑓 ’ Dotted line: Projection of 𝑓 ’

𝑓 is minimized where 𝑓 ’ is orthogonal to the search line.

Line search direction:

𝑓 𝑥𝑘 + 𝛼 = 𝑓 𝑥𝑘 + 𝛼𝑝𝑇 ∇𝑓 𝑥𝑘 + ⋯
The rate of change in 𝑓 along the direction 𝑝 at 𝑥𝑘 is the coefficient
−∇𝑓
of 𝛼, 𝑝 =
∇𝑓

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 58

Steepest-Descent Method
Since, the Steepest Descent method approaches the minimum, where the new search
direction is orthogonal to the previous. To determine 𝛼, we note that 𝑓 ‘ 𝒙 𝑖 = −𝑟 (𝑖) , and
we have
𝑇 𝑇 𝑇
𝒓(𝟏) 𝒓(0) =0⇒ 𝒃− 𝐴𝒙(𝟏) 𝒓(0) =0⇒ 𝒃−𝐴 𝒙(𝟎) + 𝛼𝒓(𝟎) 𝒓(0) = 0
𝑇
𝑇 (0) 𝑇 (0) 𝒓(0) 𝒓(0)
𝒃 − 𝐴𝒙(0) 𝒓 = 𝛼 𝐴𝒓(0) 𝒓 ⇒ 𝛼 = (0) 𝑇 (0) .
𝒓 𝐴𝒓

Putting it all together, the method of Steepest Descent is

𝒓(𝑖) = 𝒃 − 𝐴𝒙(𝑖)
𝑇
𝒓(𝑖) 𝒓(𝑖)
𝛼 = (𝑖) 𝑇 (𝑖) for 𝑖 = 0,1,2,3, ⋯
𝒓 𝐴𝒓
𝒙(𝑖+1) = 𝒙(𝑖) + 𝛼𝑖 𝒓(𝑖)

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 59

Algorithm of Steepest-Descent Method

The Algorithm of method of Steepest Descent is

Choose 𝒙(0) (initial guess). For 𝑖 = 0,1,2,3, ⋯, compute

1. 𝒓(𝑖) = 𝒃 − 𝐴𝒙(𝑖)
𝑇
𝒓(𝑖) 𝒓(𝑖)
2. 𝛼 = (𝑖) 𝑇 (𝑖)
𝒓 𝐴𝒓
3. 𝒙(𝑖+1) = 𝒙(𝑖) + 𝛼𝑖 𝒓(𝑖)
untill 𝒓(𝑖) < 𝜀 for a small 𝜀 of your choice.

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 60

Example of Steepest-Descent Method
Solve the following system 𝐴𝒙 = 𝒃 by using Steepest Descent method, where
3 −1 1 −1
𝐴 = −1 3 −1 , 𝒃= 7
1 −1 3 −7
Solution:
Let us verify the necessary and sufficient condition for using the steepest descent
method. We have to check whether 𝐴 is a symmetric positive definite matrix,
3 −1 1
3 −1
3 = 3 > 0, = 9 − 1 = 8 > 0, −1 3 −1 = 20 > 0
−1 3
1 −1 3
All leading principal minors are positive and so the matrix 𝐴 is positive definite. By
observation, 𝐴 is symmetric and the steepest descent method can be used to solve
the system.
We therefore start with an initial guess 𝒙(0) = 0 0 0 𝑇 .
For 𝒊 = 𝟎:
−1 3 −1 1 0 −1
(0) 0
𝒓 = 𝒃 − 𝐴𝒙 = 7 − −1 3 −1 0 = 7
−7 1 −1 3 0 −7

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 61

Example of Steepest-Descent Method
−1
𝑇 −1 7 −7 7
𝒓(0) 𝒓(0) −7 99
𝛼0 = = = = 0.2340
𝒓(0) 𝑇 𝐴𝒓(0) 3 −1 1 −1 423
−1 7 −7 −1 3 −1 7
1 −1 3 −7
0 −1 −0.2340
𝑥 (1) = 𝑥 (0) + 𝛼0 𝑟 (0) = 0 + 0.2340 7 = 1.6383
0 −7 −1.6383

1
Exact solution of the system is 𝑥 = 2
−2

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 62

Drawback of Steepest Descent Method

The convergence of steepest descent is slow.

Question:
How can we accelerate it!

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 63

Conjugate Gradient Method for solving the system 𝐴𝒙 = 𝒃
The conjugate gradient method is an algorithm to solve a particular system of linear
equations, namely those whose matrix is symmetric and positive definite. The conjugate
gradient method is often implemented as an iterative method, applicable to
sparse systems that are too large to be handled by a direct implementation or other
direct methods. It was mainly developed by Magnus Hestenes and Eduard Stiefel.

Eduard L. Stiefel Magnus Rudolph Hestenes

(1909 ~1978) (1906 ~1991)
Swiss Mathematician American Mathematician

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 64

Conjugate Gradient Method
Conjugate-gradient method uses so-called A-conjugate search directions instead of
orthogonal search directions.
A-Conjugacy:
If A is an n × n positive definite matrix and x, y ∈ Rn , then 𝑥, 𝑦 = 𝑥 𝑇 𝐴𝑦 is an inner
product, and if 𝑥, 𝑦 𝐴 = 0, x and y are said to be A-conjugate. The corresponding
norm, 𝑥 𝐴 = 𝑥 𝑇 𝐴𝑥 is called the A-norm or the energy norm.

We eager to solve the following large systems of linear equations using Conjugate-
Gradient method
𝐴𝒙 = 𝒃 (0)
where 𝒙 is an unknown vector, 𝒃 is a known vector, and 𝐴 is a known, 𝑛 × 𝑛 square,
symmetric, positive-definite matrix. We then consider a quadratic form of the vector 𝒙 of
the following form
1
𝑓 𝒙 = 𝒙𝑇 𝐴𝒙 − 𝒃𝑇 𝒙.
2

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 65

Conjugate Gradient Method
In the conjugate-gradient method, we start at an arbitrary point 𝒙(0) and slide down to
the bottom of the paraboloid. We take a series of steps 𝒙(1) , 𝒙(2) , ⋯ until we are satisfied
that we are close enough to the solution 𝒙.

We recall that the Gram-Schmidt process takes a set of linearly independent vectors and
produces a set of mutually orthogonal unit vectors relative to the 2-norm that span the
same subspace as the original set. To find 𝑃𝑖 , we apply the Gram-Schmidt process to
the residual vectors 𝒓𝑖 but using the energy inner product and norm. Start with
𝒓0 = 𝒃 − 𝐴𝒙0 . The set of vectors 𝑃𝑖 must be A-conjugate 𝑃𝑖𝑇 𝐴𝑃𝑗 = 0, 𝑖 ≠ 𝑗 but it is not
necessary that they be unit vectors. If we skip the normalization step of Gram-Schmidt at
step 𝑖 we have
𝑖−1
𝑟𝑖𝑇 𝐴𝑃𝑗
𝑃𝑖 = 𝑟𝑖 − 𝑐𝑗,𝑖 𝑃𝑗 , where 𝑐𝑗𝑖 = (1)
𝑃𝑗𝑇 𝐴𝑃𝑗
𝑗=1

The purpose of Equation (1) is to determine the next search direction from the current
residual and the previous search directions.

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 66

Conjugate Gradient Method
Let 𝒙 be the true solution to 𝐴𝒙 = 𝒃, and the error at step 𝑖 defined by 𝒆𝑖 = 𝒙 − 𝒙𝑖 . Then
we find that
𝑇
𝑟𝑖+1 𝑃𝑖 = 𝑟𝑖𝑇 𝑃𝑖 − 𝛼𝑖 𝐴𝑃𝑖 𝑇 𝑃𝑖 = 𝑟𝑖𝑇 𝑃𝑖 − 𝛼𝑖 𝑃𝑖𝑇 𝐴𝑃𝑖 = 0.
We also have
𝑒𝑖+1 , 𝑃𝑖 𝐴 = 𝑃𝑖𝑇 𝐴𝑒𝑖+1 = 𝑃𝑖𝑇 𝐴 𝑥 − 𝑥𝑖+1 = 𝑃𝑖𝑇 𝑏 − 𝐴𝑥𝑖+1 = 𝑃𝑖𝑇 𝑟𝑖+1 = 0.

We are now in a position to simplify the values of 𝛼𝑖 . The algorithm begins with the
assignment 𝑃0 = 𝑟0 = 𝑏 − 𝐴𝑥0 and the computation of 𝑥1 . We find that 𝑒1 , 𝑃0 𝐴 = 0. In the
nest step, the Gram-Schmidt process creates 𝑃1 so that 𝑃1 , 𝑃0 𝐴 = 0, determining 𝑥2 .
Again, we have 𝑒2 , 𝑃1 𝐴 = 0. At this point we have 𝑒1 , 𝑃0 𝐴 = 0 and 𝑒2 , 𝑃1 𝐴 = 0. Noting
that 𝑃1 , 𝑃0 𝐴 = 0 to obtain
𝑒2 , 𝑃0 𝐴 = 𝑒1 − 𝛼1 𝑃1 , 𝑃0 𝐴 = 𝑒1 , 𝑃0 𝐴 − 𝛼1 𝑃1 , 𝑃0 𝐴 = 0.
Thus, 𝑒2 is A-orthogonal to both 𝑃0 and 𝑃1 . Continuing in this fashion, we see that 𝑒𝑖 is
A-orthogonal to 𝑃1 , 𝑃2 , ⋯ , 𝑃𝑖−1 or 𝑒𝑖 , 𝑃𝑗 = 0, 𝑗 < 𝑖 which gives
𝐴
𝑒𝑖 , 𝑃𝑗 = 𝑃𝑗𝑇 𝐴𝑒𝑖 = 𝑃𝑗𝑇 𝐴 𝑥 − 𝑥𝑖 = 𝑃𝑗𝑇 𝑏 − 𝐴𝑥𝑖 = 𝑟𝑖 , 𝑃𝑗 = 0, 𝑗 < 𝑖.
𝐴

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 67

Conjugate Gradient Method
We take the inner product of 𝑃𝑗 and 𝑟𝑖 , 𝑗 < 𝑖,
𝑗−1

𝑟𝑖𝑇 𝑃𝑗 = 𝑟𝑖𝑇 𝑟𝑗 − 𝑐𝑘𝑗 𝑟𝑖𝑇 𝑝𝑘 = 𝑟𝑖𝑇 𝑟𝑗 = 0 (2)

𝑘=1
and all residuals are orthogonal to the previous ones. Similarly
𝑖−1
2 2
𝑃𝑖 , 𝑟𝑖 = 𝑟𝑖 2 − 𝑐𝑗𝑖 𝑟𝑖 , 𝑃𝑗 = 𝑟𝑖 2
𝑗=1
Finally, we obtain
2
𝑟𝑖 2
𝛼𝑖 = .
𝐴𝑃𝑖 , 𝑃𝑖
The evaluation of formula (1) seems expensive, and it appears that we must retain all
the previous search directions. Here is where the CG algorithm is remarkable. We get
the benefit of a combination of search directions by computing only 𝑐𝑖−1,𝑖 , the
coefficient of the last direction 𝑃𝑖−1 ! Take the inner product of 𝑟𝑖 to obtain 𝑟𝑖𝑇 𝑟𝑗+1 =
𝑟𝑖𝑇 𝑟𝑗 − 𝛼𝑗 𝑟𝑖𝑇 𝐴𝑃𝑗 , and so
𝛼𝑗 𝑟𝑖𝑇 𝐴𝑃𝑗 = 𝑟𝑖𝑇 𝑟𝑗 − 𝑟𝑖𝑇 𝑟𝑗+1 . (3)

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 68

Conjugate Gradient Method
Equation (2) and (3) give the following results
1 𝑇
𝑟𝑖 𝑟𝑖 , 𝑗=𝑖
𝛼
𝑟𝑖𝑇 𝐴𝑃𝑗 = 1 𝑇 (4)
− 𝑟 𝑟, 𝑗 =𝑖−1
𝛼𝑖−1 𝑖 𝑖
0, 𝑗<𝑖−1
𝑟𝑖𝑇 𝐴𝑃𝑗
Recall that the Gram-Schmidt constants are 𝑐𝑗𝑖 = and we find
𝑃𝑗𝑇 𝐴𝑃𝑗
2
𝑟𝑖𝑇 𝐴𝑃𝑖−1 1 𝑟𝑖𝑇 𝑟𝑖 𝐴𝑃𝑖−1 , 𝑃𝑖−1 𝑟𝑖𝑇 𝑟𝑖 𝑟𝑖 2
𝑐𝑖−1,𝑖 = 𝑇 =− 𝑇 = − 2 = − 2 .
𝑃𝑖−1 𝐴𝑃𝑖−1 𝛼𝑖−1 𝑃𝑖−1 𝐴𝑃𝑖−1 𝑟𝑖−1 2 𝐴𝑃𝑖−1 , 𝑃𝑖−1 𝑟𝑖−1 2
From (4), 𝑐𝑗,𝑖 = 0, 𝑗 < 𝑖 − 1. Thus, all the coefficients 𝑐𝑗𝑖 in Equation (1) are zero except
𝑐𝑖−1,𝑖 . By
defining
2
𝑟𝑖 2
𝛽𝑖 = 2,
𝑟𝑖−1 2
We see that
𝑃𝑖 = 𝑟𝑖 + 𝛽𝑖 𝑃𝑖−1 .
5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 69
Algorithm of Conjugate Gradient Method

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 70

Example of Conjugate Gradient Method
Solve the following system 𝐴𝒙 = 𝒃 by using Conjugate gradient method, where
3/2 0 1/2 1
𝐴= 0 3 0 , 𝒃= 1
Solution: 1/2 0 3/2 −1
Let us verify the necessary and sufficient condition for using the conjugate gradient
method. We have to check whether 𝐴 is a symmetric positive definite matrix,
3 3 3/2 0 1/2
= 1.5 > 0, 0 = 4.5 > 0, 0 3 0 =6>0
2 2
0 3 1/2 0 3/2
All leading principal minors are positive and so the matrix 𝐴 is positive definite. By
observation, 𝐴 is symmetric and the conjugate gradient method can be used to solve
the system.
We therefore start with an initial guess 𝒙(0) = 0 0 0 𝑇 .
For 𝒊 = 𝟎:
1 3/2 0 1/2 0 1
𝒓(0) = 𝒃 − 𝐴𝒙 0 = 1 − 0 3 0 0 = 1
−1 1/2 0 3/2 0 −1
5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 71
Example of Conjugate Gradient Method

2
Since 𝑟0 2 = 𝑟0𝑇 𝑟0 = 3 and

3/2 0 1/2 1 𝑇
1 1 1
𝑇
𝐴𝑃0 = 0 3 0 1 = 3 and 𝑃0 𝐴𝑃0 = 1 3 =5
1/2 0 3/2 −1 −1 −1 −1
2
𝑟03 2
𝛼0 = 𝑇 =
𝑃0 𝐴𝑃0 5
0 3 1 3 1
𝑥 (1) = 𝑥 (0) + 𝛼0 𝑃0 = 0 + 1 = 1 .
5 5
0 −1 −1

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 72

Convergence of Conjugate Gradient Method
The convergence of conjugate gradient (CG) method can be investigated in the A-norm
or energy norm, defined by
𝒙 𝐴 = 𝒙, 𝒙 𝐴 = 𝒙𝑇 𝐴𝒙.
ith Krylov Space:
Let 𝐴 ∈ ℝ𝑛×𝑛 and 𝒑 ∈ ℝ𝑛 be given. For 𝑖 ∈ ℕ we define the ith Krylov space to 𝐴 and 𝒑 as
𝛫𝑖 (𝒑, 𝐴) = span 𝒑, 𝐴𝒑, 𝐴2 𝒑, ⋯ , 𝐴𝑖−1 𝒑 .
Affine Krylov space 𝒙𝟎 + 𝛫𝑖 (𝒑, 𝐴) defined as the collection of vectors 𝒙 such that 𝒙 − 𝒙𝟎
belongs to the Krylov space 𝛫𝑖 𝒑, 𝐴 .

Theorem:
Let the CG method be applied to a symmetric positive definite
system 𝐴𝒙 = 𝒃, where the condition number of 𝐴 is 𝑘. Let 𝑥 be the
exact solution to the system. Let be the sequence of of approximate
solutions produced by the CG method. We have Then the 𝐴-norms
of the errors satisfy
𝑘 Aleksey N. Krylov
𝑥𝑘 − 𝑥 𝐴 𝑘−1
≤2 . Russian Mathematician
𝑥0 − 𝑥 𝐴 𝑘+1 (1863 ~ 1945)

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 73

Convergence of Conjugate Gradient Method
Proof:
We know that 𝑥𝑘 can be derived as the minimum on the space 𝒙𝟎 + 𝛫𝑘 of the function
defined by
𝑛
1 1 2 1
𝑧 ∈ ℝ ⟼ 𝑓 𝑧 = 𝐴𝑧, 𝑧 − 𝑏, 𝑧 = 𝑥 − 𝑧 𝐴 − 𝐴𝑥, 𝑥 ,
2 2 2
2 2
where 𝑦 𝐴 = 𝐴𝑦, 𝑦 . Computing 𝑥𝑘 is thus equivalent to minimizing 𝑥 − 𝑧 𝐴 , that is, the
error in the norm ∙ 𝐴 , on the space 𝒙𝟎 + 𝛫𝑘 . Now we compute 𝑒𝑘 𝐴 . CG method gives
that
𝑘−1

𝑥𝑘 = 𝑥0 + 𝑐𝑗 𝑃𝑗 ,
𝑗=0
with each 𝑃𝑗 equal to a polynomial in 𝐴 (of degree ≤ 𝑗) applied to 𝑃0 , so there exists a
polynomial 𝑞𝑘−1 ∈ ℙ𝑘−1 such that
𝑥𝑘 = 𝑥0 + 𝑞𝑘−1 𝐴 𝑃0 .
Since 𝑃0 = 𝑟0 = 𝑏 − 𝐴𝑥0 = 𝐴(𝑥 − 𝑥0 ), the errors satisfy the relations
𝑒𝑘 = 𝑥𝑘 − 𝑥 = 𝑒0 + 𝑞𝑘−1 𝐴 𝑃0 = 𝑄𝑘 𝐴 𝑒0 ,
where 𝑄𝑘 is the polynomial defined, for all 𝑡 ∈ ℝ by 𝑄𝑘 𝑡 = 1 − 𝑞𝑘−1 𝑡 𝑡.

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 74

Convergence of Conjugate Gradient Method
We denote by 𝑢𝑗 an orthonormal basis of eigenvectors of 𝐴, i.e., 𝐴𝑢𝑗 = 𝜆𝑗 𝑢𝑗 , and by 𝑒0,𝑗 the
entries of the initial error 𝑒0 in this basis: 𝑒0 = 𝑛𝑗=1 𝑒0,𝑗 𝑢𝑗 . We have
𝑛
2 2
𝑒0 𝐴 = 𝑒0 , 𝐴𝑒0 = 𝜆𝑗 𝑒0,𝑗
𝑗=1
and
𝑛
2 2 2
𝑒𝑘 𝐴 = 𝑄𝑘 𝐴 𝑒0 𝐴 = 𝜆𝑗 𝑒0,𝑗 𝑄𝑘 𝜆𝑗
𝑗=1
Since the conjugate gradient method minimizes 𝑒𝑘 𝐴 , the polynomial 𝑄𝑘 must satisfy
𝑛 𝑛
2 2 2
𝑒𝑘 𝐴 = 𝜆𝑗 𝑒0,𝑗 𝑄𝑘 𝜆𝑗 = min0 𝜆𝑗 𝑒0,𝑗 𝑄𝑘 𝜆𝑗 (2)
𝑄∈ℙ𝑘
𝑗=1 𝑗=1
where minimization is carried in ℙ0𝑘 , the set of polynomials 𝑄 ∈ ℙ𝑘 that satisfy 𝑄 0 = 1
(indeed, we have 𝑄𝑘 ∈ ℙ0𝑘 ). From (2) we get an easy upper bound
2
𝑒𝑘 𝐴 2 2
2 ≤ min0 max 𝑄𝑘 𝜆𝑗 ≤ min0 max 𝑄𝑘 𝑥 (3)
𝑒0 𝐴 𝑄∈ℙ𝑘 1≤𝑗≤𝑛 𝑄∈ℙ𝑘 𝜆1 ≤𝑥≤𝜆𝑛

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 75

Convergence of Conjugate Gradient Method
The last min–max problem in the right-hand side of (3) is a classical and celebrated
polynomial approximation problem: Find a polynomial 𝑝 ∈ ℙ𝑘 minimizing the quantity
max 𝑝(𝑥) . To avoid the trivial zero solution, we impose an additional condition on 𝑝, for
𝑥∈,𝑎,𝑏-
instance, 𝑝 𝛽 = 1 for a number 𝛽 ∉ ,𝑎, 𝑏-. Its unique solution is the following Tchebyshev
polynomial
2𝑥 − (𝑎 + 𝑏)
𝑇𝑘
𝑏−𝑎
(4)
2𝛽 − (𝑎 + 𝑏)
𝑇𝑘
𝑏−𝑎
where 𝑇𝑘 is the kth-degree Tchebyshev polynomial defined for all 𝑡 ∈ ,−1,1- by 𝑇𝑘 (𝑡) =
cos 𝑘 arccos 𝑡 . The maximum value reached by (4) on the interval ,𝑎, 𝑏- is
1
.
2𝛽 − (𝑎 + 𝑏)
𝑇𝑘
𝑏−𝑎
In our case (2), we have 𝑎 = 𝜆1 , 𝑏 = 𝜆𝑛 , 𝛽 = 0. The matrix 𝐴 being positive definite, 𝛽 ∉ 𝑎, 𝑏 ,
we conclude that
1
min0 max 𝑄(𝑥) = (5)
𝜆
𝑄∈ℙ𝑘 1 ≤𝑥≤𝜆 𝜆 + 𝜆
𝑛
𝑇𝑘 𝑛 1
𝜆𝑛 − 𝜆1
5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 76
Convergence of Conjugate Gradient Method
We observe now that
𝜆𝑛 +𝜆1 𝑘+1 𝜆𝑛
• 𝐴 being symmetric, we have = , with 𝑘 = cond2 𝐴 = ;
𝜆𝑛 −𝜆1 𝑘−1 𝜆1
• Chebyshev polynomials satisfy
𝑘 𝑘
2𝑇𝑘 𝑥 = 𝑥 + −1 + 𝑥−𝑥2 −1 , 𝑥2
for all 𝑥 ∈ −∞, −1 ∪ ,1, +∞). Therefore, we infer the following lower bound:
1
𝑘 2
𝜆𝑛 + 𝜆1 𝑘+1 𝑘+1 1 𝑘+1
2𝑇𝑘 ≥ + −1≥ 𝑘+1 +2 𝑘 = .
𝜆𝑛 − 𝜆1 𝑘−1 𝑘−1 𝑘−1 𝑘−1
Combining these results, equality (5), and the upper bound (3), we obtain
2𝑘
2 𝑘+1 2
𝑒𝑘 𝐴 ≤4 𝑒0 𝐴 .
𝑘−1
Finally, we find
𝑘
𝑥𝑘 − 𝑥 𝐴 𝑘−1
≤2 .
𝑥0 − 𝑥 𝐴 𝑘+1

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 77

Questions on Chapter 3:
1. Discuss fixed point iteration method for solving a system of linear equations.
2. State and prove convergence of fixed point iteration method.
3. Explain successive over relaxation (SOR) method for solving a system of linear
equations.
4. State and prove convergence of successive over relaxation (SOR) method.
5. Mention an algorithm of successive over relaxation (SOR) method.
6. Examples related to successive over relaxation (SOR) method.
7. Describe steepest descent method for solving a system of linear equations.
8. State algorithm of steepest descent method.
9. Examples related to solve a linear system using steepest descent method.
10. Describe conjugate gradient method for solving a system of linear equations.
11. Mention an algorithm of the conjugate gradient method.
12. State and prove convergence of conjugate gradient method.
13. Examples related to solve a linear system using conjugate gradient method.
14. Make a comparative study of SOR method, steepest descent method and conjugate
gradient method for solving a linear system in terms of their convergence rate and
CPU time.

5/8/2024 Math407: Numerical Linear Algebra & ODEs (MHK) 78