Numerical Analysis II

Prof. Benabderrahmane Benyattou

2024/2025
Contents

1 Chapter 2 : Eigenvalue Problems 2

1.1 The Power Method . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.1 Basic Power Method . . . . . . . . . . . . . . . . . . . 5
1.1.2 Inverse Power Method . . . . . . . . . . . . . . . . . . . . 11
1.1.3 Comparison between the Power Method and the Inverse
Power Method . . . . . . . . . . . . . . . . . . . . . . . . 14
1.1.4 Shifted Inverse Power Method. . . . . . . . . . . . . . . . 14
1.1.5 De‡ation Method . . . . . . . . . . . . . . . . . . . . . . . 19
1.2 The QR Method for Eigenvalues . . . . . . . . . . . . . . . . . . 21

1
Chapter 1

Chapter 2 : Eigenvalue
Problems

Eigenvalue problems are a fundamental concept in linear algebra and have sig-
ni…cant applications in various …elds such as physics, engineering, computer
science, and …nance. At their core, eigenvalue problems involve studying linear
transformations represented by matrices, speci…cally how these transformations
a¤ect vectors in a vector space.
In this chapter, we provide a brief overview of two numerical methods for
solving the standard eigenvalue problem, which involves …nding the eigenvalues
and eigenvectors v of an n n matrix A. The …rst method is a simple iterative
technique for identifying the dominant eigenvalue of a matrix, known as the
power method. The second method, which is more computationally intensive,
aims to …nd all eigenvalues of a general matrix and is referred to as the QR
method.
Both methods can be enhanced through an appropriate (inverse) shift. Spe-
cial orthogonal transformations, speci…cally Householder re‡ections, are pivotal
in several widely used eigenvalue algorithms and Singular Value Decomposition
(SV D) methods.
The algebraic eigenvalue problem is de…ned as follows:
Given a matrix A 2 Rn n , …nd a nonzero vector x 2 Rn and the scalar
such that
Ax = x
Note that this says that the vector Ax is parallel to x, with being an
ampli…cation factor, or gain. Note also that the above implies that (A I) x =
0; showing that (A I) is singular matrix. Hence,

p ( ) = det jA Ij = 0;

which is a polynomial (of degree n) in , known as the characteristic poly-

nomial of A, so that the eigenvalues are the roots of a polynomial. Although

2
this is not a good way to compute the eigenvalues, it does give us some insight
into their properties.
Basic Eigenvalue Properties
Eigenvalue Count: A matrix A has exactly n eigenvalues, counted with
their algebraic multiplicities. Any complex eigenvalues will appear in conjugate
pairs.
Independence of Eigenvectors: Eigenvectors associated with distinct
eigenvalues of A are linearly independent.
Diagonalizability: If A has n linearly independent eigenvectors, there ex-
ists a nonsingular matrix P such that:
1
P AP = D
Here, D is a diagonal matrix with eigenvalues on its diagonal, and the
columns of P are the corresponding eigenvectors of A.
Properties of Symmetric Matrices: If A is symmetric, then all eigen-
values are real. We can also choose the eigenvectors to be real and orthogonal.
Orthogonal Diagonalization: For symmetric matrices, there exists an
orthogonal matrix Q (where QT = Q 1 ) such that
QT AQ = D
Here, D is a diagonal matrix containing the eigenvalues of A.
Eigenvalues of Triangular Matrices: For an upper or lower triangular
matrix A, the eigenvalues are simply the diagonal entries aii = i .
Example 1 Let 0 1
4 1 0
A=@ 1 4 1 A
0 1 4
The characteristic polynomial for this matrix is
3 2
P( )= 12 + 46 56
and the eigenvalues are

1 = 5:4142; 2 = 4; 3 = 2:5858:
Since the matrix is symmetric, we can …nd a set of orthonormal eigenvectors:
0 1 0 1 0 1
0:5 0:7071 0:5
x1 = @ 0:7071 A ; x2 = @ 0 A ; x1 = @ 0:7071 A :
0:5 0:7071 0:5
Since these vectors are orthonormal, when we arrange them as the columns of
a matrix, we get an orthogonal matrix:
0 1
0:5 0:7071 0:5
Q = @ 0:7071 0 0:7071 A :
0:5 0:7071 0:5

3
 1
Moreover, we have Q AQ = QT AQ = D; diagonal
0 1
5:4142 0 0
Q AQ = @
T
0 4 0 A
0 0 2:5858

De…nition 2 (Spectrum of the matrix) The collection of all eigenvalues of a

matrix is typically called the spectrum of the matrix, which we denote by

(A) = f 2 C; Ax = x; x 6= 0g

The vector space spanned by all the eigenvectors corresponding to a single

eigenvalue is called the eigenspace and is denoted by

EA ( ) = f0 6= x 2 Cn ; Ax = xg :

1.1 The Power Method

This method is a classical approach primarily used to …nd the dominant eigen-
value (the one with the largest absolute value) and its corresponding eigenvector
of a matrix. While it is not the most e¢ cient technique for computing all eigen-
values and eigenvectors, it is valuable for identifying certain eigenvalues, and
the foundational concepts from power methods underpin many more advanced
techniques.
We begin by presenting a formal theorem, which will serve as a basis for
developing the …rst algorithm.

Theorem 3 Suppose that A has a dominant eigenvalue:

j 1j >j 2j ::: j nj

and a complete set of eigenvector x1 ; x2 ; :::; xn associated with the eigenvalues

1 ; 2 ; :::; n , respectively. Then

Ak z
lim k
= cx1 ; for some c 6= 0; and (1)
k!1
1

z; Ak z
lim = 1; (2)
k!1 hz; Ak 1 zi

where
P
n
z= i xi ; 1 6= 0:
i=1

Proof. We have
n
X
k P
n
k k
A z= iA xi = i i xi :
i=1 i=1

4
 So, that
k
Ak z P
n
i
k
= 1 x1 + i xi ;
1 i=2 1

where i
1
< 1; for all k 2:
Thus
k
P
n
i
lim i xi = 0;
k!1 i=1 1

which implies (1):

On the other hand, we can write
P
n
k
k z; i i xi
z; A z i=1
= =
hz; Ak 1 zi P
n
k 1
z; i i xi
i=1

k P
n k
1 z; 1 x1 + i
i
1
xi
i=2
=
k 1 Pn k 1
1 z; 1 x1 + i
i
1
xi
i=2

k
Since each of the i
1
;i 2 terms goes to zero, then

z; Ak z hz; 1 x1 i
lim = 1 = 1:
k hz; Ak 1 zi hz; 1 x1 i

1.1.1 Basic Power Method

Algorithm 4 For k from 1 until k+1 k < "; do the following
1: Compute y (k) = Az (k 1) ;
(k) (k)
2: Set k = yi ; where y (k) 1 = yi ;
y (k)
3: Update z (k) = :
k

Convergence Result
Regarding this algorithm, we have the following result concerning its con-
vergence:

Theorem 5 Let A 2 Rn n be given, and assume the following conditions:

1) A has n linearly independent eigenvectors, xk ; for k = 1; ::; n:
2) The eigenvalues k ; k = 1; : : : ; n verify

j 1j >j 2j ::: j nj :

5
 3) The vector z (0) has a nonzero component in the direction of x1 . This
means that z (0) 2 Rn can be expressed as
P
n
z (0) = i xi ; with 1 6= 0:
i=1

Then, there exists a constant c 6= 0 such that

lim z (k) = cx1 ; and

k!1

lim k = 1;
k!1

and the errors can be expressed as:

!
k
(k) 2
z cx1 = O ; k ! 1;
1 1
!
k
2
j k 1j = O ; k ! 1:
1

Proof. An inductive argument can be used to deduce that

y (k) Az (k 1)
A y (k 1)
A Az (k 2)
z (k) = = = =
k k k k 1 k k 1
A2 z (k 2)
Ak z (0)
= = = = Ck Ak z (0) ;
k k 1 k k 1 ::: 1

1
where Ck = :
k k 1 ::: 1
We then expand in terms of the eigenvector basis, as we did before, to get
P
n P
n Pn
k
z (k) = Ck Ak i xi = iA
k
xi = Ck i i xi
i=1 i=1 i=1
!
k
k P
n
i
= Ck 1 1 x1 + i xi :
i=2 1

Since each z (k) veri…es z (k) 1

= 1 and i
1
< 1; it follows that
0 1
k
P
n
i
1 = lim Ck k
1
@ 1 x1 + i xi 0 A
k !1 i=2 1
! 1

= lim Ck k1 k 1 x1 k1 ;
k !1

which implies
k 1
lim Ck 1 = < 1:
k !1 j 1 j kx 1 k1

6
So that there is at least c = j
1
1 jkx1 k1
6= 0 such that

lim z (k) = cx1 :

k !1

Moreover, we clearly have that

k k
k P
n
i 2
z (k) cx1 = Ck 1 i xi C ;k ! 1
1 i=2 1 1
1

which establishes the error estimate for the eigenvector.

As for the eigenvalue, we have that
(k)
kz = y (k) = Az (k 1)
;

so that, taking dot products of both sides with z (k) we get

Az (k 1) ; z (k) k!1 hA (cx1 ) ; cx1 i h 1 cx1 ; cx1 i

k = ! = = 1:
z (k) ; z (k) hcx1 ; cx1 i hcx1 ; cx1 i

Example 6 Assume
0 1 !
1 3 2 T
3 1
A=@ 4 4 1 A has a dominant eigenpair 7; ; ;1
10 15
6 3 5

Determine the largest eigenvalue of A; 1 and its associated eigenvector by

T
the Power method. Start your iteration with z (0) = (1; 1; 1) and stop the
iteration when k+1 k < 0:05:
Using …ve-digits ‡oating-point decimal arithmetic with rounding.
We have
0 10 1 0 1 0 1
1 3 2 1 0 0
y (1) = Az (0) =@ 4 4 1 A @ 1 A = @ 7 A = 14 @ 0:5 A = 1z
(1)
;
6 3 5 1 14 1
0 1 0 1
0:5 0:07692
y (2) = Az (1) = @ 1 A = 6:5 @ 0:15385 A = 2z
(2)
;
6:5 1
Since j 2 1j
0:05; then
0 1
0:27273
y (3) = Az (2) = 5:9231 @ 0:01299 A = 3z
(3)
;
1
Since j 3 2j 0:05; then

7
 0 1 0 1
2:311 7 0:350 40
y (4) = Az (3) = @ 0:038 96 A = 6:5974 @ 0:00591 A = 4z
(4)
;
6:5974 1
Since j 4 3j
0:05; then
0 1 0 1
2:3327 0:32762
y (5) = Az (4) = @ 0:42524 A = 7:1201 @ 0:05972 A = 5 = z (5) ;
7:1201 1
Since j 5 4j 0:05; then
0 1 0 1
2:1485 0:3007
y (6) = Az (5) = @ 0:54936 A = 7:1449 @ 0:07689 A = 6z
(6)
:
7:1449 1
Since j 6 5j = 0:0248 < 0:05; then
T
1 ' 6 = 7:1449; and x1 ' z (6) = (0:3007; 0:07689; 1) :

Example 7 Use the Power method to approxmate the largest eigenvalue of A:

(
5 T
6 5 1 = 10; x1 = 4 ; 1
A= ; T
4 5 2 = 1; x2 = (1; 1)

T
z (0) = (1; 0) : Using …ve-digits ‡oating-point decimal arithmetic with rounding.
We have
6 5 1 6 1
y (1) = Az (0) = = =6 = 1 z (1) ;
4 5 0 4 0:66667
9:3334 1
y (2) = Az (1) = = 9:3334 = 2 z (2) ;
7:3334 0:78572
9:9286 1
y (3) = Az (2) = = 9:9286 = 3 z (3) ;
7:9286 0:79856
9:9928 1
y (4) = Az (3) = = 9:9928 = 4 z (4) ;
7:9928 0:799 86
9:9993 1
y (5) = Az (4) = = 9:9993 = 5 z (5) ;
7:9993 0:799 99
10 1
y (6) = Az (5) = = 10 = 6 z (6) ;
8 0:8
10 1
y (7) = Az (6) = = 10 = 6 z (6) :
8 0:8
Hence
T
1 = 10; and x1 = (1; 0:8) :

Remark 8 The power method relies on several hypotheses for the theory to
hold. It’s worth considering whether all of these assumptions are strictly neces-
sary. What occurs if some of these conditions are violated ?

8
 1: It is not necessary for the matrix A to have a complete set of eigenvectors
for the Power method to converge. Recall that if a matrix A does not pos-
sess n linearly independent eigenvectors, it is considered defective. Eigenvalue
computations for defective matrices are inevitably more challenging. However,
just as all singular matrices are close to nonsingular matrices, it follows that
all defective matrices can be made arbitrarily close to non-defective matrices.
Consequently, applying the power method to a defective matrix typically leads
to, at worst, slow convergence. In some cases, the convergence may be very
slow, particularly if the dependent eigenvectors correspond to the two largest
eigenvalues, 1 and 2 :
Example of a Defective Matrix : Consider the following 2 2 defective
matrix:
1 1 1
A= ; eigenvectors : $1
0 1 0
T
Let’s apply the power method starting with an initial vector z (0) = (1; 1) :
Iteration Steps:

1 1 1 2 1
y (1) = = =2 = 1z
(1)
;
0 1 1 1 0:5

1
y (2) = Az (1) = 1:5 = 2z
(2)
;
0:33333
1
y (3) = Az (2) = 1:3333 = 3z
(3)
;
0:25

1
y (4) = Az (3) = 1:25 = 4 z (4) ;
0:2
:::::::::::::::::::::::::::::::::::::::::

Conclusion. As we can see, despite starting with a vector that isn’t aligned
with the dominant eigenvector, the power method is converging slowly. Ulti-
mately, you can observe that the results will oscillate without converging to
a well-de…ned eigenvector direction because the matrix A is defective, lacking
su¢ cient independent eigenvectors.
In this case, slow convergence may arise, especially if we need to deal with
eigenvalues that are close in value, which could cause signi…cant delays in reach-
ing an accurate approximation of the dominant eigenvector.
2: If the two largest eigenvalues (in absolute value) are not well-separated,
meaning that j 1 j = j 2 j ; the iteration will cycle through a family of vectors
within the subspace spanned by the eigenvectors x1 and x2 associated with these
eigenvalues. This phenomenon can lead to oscillations instead of converging to
a single vector.
Example: Consider a matrix

2 0
A=
0 2

9
 For the matrix A, the eigenvalues and corresponding eigenvectors are as
follows:
T T
1 = 2; x1 = (1; 0) and 2 = 2; x2 = (0; 1) :
T
If we start with an initial vector z (0) = (1; 1) , the iterations will involve the
linear combination of x1 and x2 . Due to the equal magnitudes of the eigenvalues,
the iterations will move between vectors like
2 0 1 2 1
y (1) = = =2 = 1z
(1)
;
0 2 1 2 1

2 0 1 2 1
y (2) = = =2 = 2z
(2)
;
0 2 1 2 1
ultimately cycling between vectors in the subspace spanned by x1 and x2 without
converging to a …xed point.
3: The Power method works by repeatedly multiplying the initial vector
by the matrix A. If z (0) has no component in the direction of the dominant
eigenvector, the results will not converge to the eigenvector corresponding to the
dominant eigenvalue. Instead, they could remain in the span of the components
of the other eigenvectors.
In most situations, particularly those involving non-integer components,
rounding errors during computations will typically introduce some component
in the direction of x1 into the initial guess, allowing the iteration to begin con-
verging toward the correct eigenvalue and eigenvector. However, if the initial
guess contains absolutely no component in the direction of x1 , there is a risk of
converging to the wrong eigenpair. Therefore, using random non-integer initial
guesses is generally more e¤ective to ensure convergence.
Example: Suppose we have a matrix
4 1
A=
2 3
The eigenvalues and eigenvectors of this matrix are:
T T
1 = 5; x1 = (1; 1) and 2 = 2; x2 = (1; 2) :
Initial Guesses
Non-integer guess (e¤ective): Let’s use the initial guess z (0) = (0:1; 0:3)T :

4 1 0:1 0:7 0:63636

y (1) = = = 1:1 = 1z
(1)
;
2 3 0:3 1:1 1

4 1 0:63636 3:545 4 0:829 78

y (2) = = = 4:2727 = 2z
(2)
;
2 3 1 4:272 7 1:0

4 1 0:82978 4: 319 1 0:926 93

y (3) = = = 4:6596 = 2z
(2)
:
2 3 1 4: 659 6 1

10
Due to rounding errors and the in‡uence of the dominant eigenvector x1 ; z (0)
will still converge to the eigenpair associated with 1 :
Integer guess (ine¤ective): Now consider an initial guess that is entirely
aligned with x2 ; z (0) = ( 1; 2)T :
1
4 1 1 2
y (1) = = =4 2 = 1z
(1)
;
2 3 2 4 1
1 1
4 1 1
y (2) = 2 = =2 2 = 2z
(2)
;
2 3 1 2 1
This guess has no component in the direction of x1 . As a result, it may lead
to convergence towards the eigenpair associated with 2 , which is undesirable
if our goal is to …nd the dominant eigenvalue.
Conclusion. In this example, the initial guess z (0) introduces a component
in the direction of the dominant eigenvector x1 , promoting convergence toward
the desired eigenpair ( 1 ; x1 ) : In contrast, the guess z (0) fails to do so and risks
converging toward the smaller eigenvalue.

Advantages of the Power Method

Simplicity: The Power Method is straightforward to implement and under-
stand. You only need to repeatedly multiply the matrix by a vector and nor-
malize it.
Dominant Eigenvalue: It e¢ ciently …nds the largest eigenvalue (in mag-
nitude) and its corresponding eigenvector, making it particularly useful in many
applications where the dominant eigenvalue is of interest.
Monotonicity: The convergence is monotonous for matrices with a unique
dominant eigenvalue, as the iterates will consistently move closer to the domi-
nant eigenvector.
There are many variants to the basic Power method. We …rst observe that
if we use the power method on A 1 , then we get an algorithm that converges
to the smallest eigenvalue of A:

1.1.2 Inverse Power Method

The Inverse Power Method is an algorithm used in linear algebra to …nd the
eigenvalues and eigenvectors of a matrix, particularly the smallest eigenvalue. It
is especially e¤ective for …nding eigenvalues that are closest to a speci…ed value
(often zero).

Algorithm 9 For k from 1 until convergence, do the following

1: Solve Ay (k) = z (k 1) ;
(k) (k)
2: Set k = yi ; where y (k) 1 = yi ;
y (k)
3: Update z (k) = :
k

11
 Convergence Result
Concerning the Invesre Power Mathod Algorithm, we have the following
result of convergence.

Theorem 10 Let A 2 Rn n be a nonsingular matrix, and assume the following

three conditions:
1: A has n linearly independent eigenvectors, xk ; k = 1; ::; n:
2: The eigenvalues k verify

j 1j j 2j ::: j n 1j >j nj :

3: The vector z (0) has a nonzero component in the direction of xn :

P
n
z (0) = i xi ; and n 6= 0:
i=1

Then the inverse power method converges in the sense that there is c 6= 0
such that
lim z (k) = cxn ; and
k!1

1
lim k =
k!1 n
with the errors
!
k
n
z (k) cxn = O ; k ! 1;
1 n 1
!
k
1 n
k = O ; k ! 1:
n n 1

1
Proof. This is equivalent to applying the previous theorem to A , so every-
thing follows, with the largest (in absolute value) eigenvalue of A 1
being 1n .

Example 11 Do four iterations of the inverse Power method to …nd the ap-
proximate smallest eigenvalue and the corresponding eigenvector of the matrix

2 12 T
A= ; with z (0) = (1; 1) :
1 5
n o
T T
eigenpairs : 2; (3; 1) ; 1; (4; 1) :
U sin g …ve-digits ‡oating-poin t decim al arithm etic w ith roun din g.

We have
5
1 2 6
A = 1 :
2 1
Hence

12
 5
6 1 3:5 1
y (1) = A 1 (0)
z = 2
1 = = 3:5 =
2 1 1 0:5 0:14286
(1)
1z ;
5
6 1 1:6428 1
y (2) = A 1 (1)
z = 2
1 = = 1:6428 =
2 1 0:14286 0:35714 0:21740
(2)
2z ;
5
6 1 1:1956 1
y (3) = A 1 (2)
z = 2
1 = = 1:1956 =
2 1 0:21740 0:282 6 0:23637
(3)
3z ;
5
(4) 1 (3) 2 6 1 1:081 8 1
y =A z = 1 = = 1:0818 =
2 1 0:23637 0:26364 0:2437
(4)
4z ;
Thus
1 1 T
2 ' = = 0:92439; and x1 ' z (4) = (1; 0:2437) :
4 1:0818

Remark on the Inverse Power Method

It is important to emphasize that the Inverse Power Method does not involve
directly multiplying by the inverse of the matrix. Instead, at each iteration, the
method solves the linear system:

Ay (k) = z (k 1)

This is typically accomplished using LU decomposition, which allows for

more e¢ cient computation than calculating the inverse explicitly.
The Inverse Power Method is particularly advantageous because it enables
the computation of several eigenvalues by applying shifts. By selecting appro-
priate shifts, we can target convergence towards nearly any desired eigenvalue.
Additionally, this approach allows for an increased convergence rate, making it
a powerful tool in numerical linear algebra.

Advantages of the Inverse Power Method

Targeting Di¤erent Eigenvalues: Unlike the standard Power Method, the
Inverse Power Method can be used to …nd smaller eigenvalues (in magnitude).
By introducing shifts, it can be adjusted to converge towards any desired eigen-
value.
Accelerated Convergence: With appropriate shifts, the convergence can
be signi…cantly faster than the Power Method, especially when the sought eigen-
value is close to a speci…ed shift.
Applying LU Decomposition: The use of LU decomposition to solve the
linear system Ay (k) = z (k 1) prevents direct computation with the inverse of
the matrix, making it more numerically stable and e¢ cient. This avoids issues
associated with singular or nearly singular matrices.

13
1.1.3 Comparison between the Power Method and the In-
verse Power Method
Both the Power Method and the Inverse Power Method rely on iterative processes
to re…ne approximations of eigenvalues and eigenvectors. Each method begins
with an initial vector and performs successive iterations to converge towards an
eigenvector.
While both approaches are e¤ective in addressing eigenvalue problems, they
have distinct characteristics that in‡uence their performance:
Iteration Process: Both methods iteratively update a vector to approxi-
mate an eigenvector, with convergence in‡uenced by the properties of the matrix
and the choice of the initial vector.
Convergence Rates:The convergence rate can vary signi…cantly based on
the matrix’s spectral properties. For the Power Method, the choice of initial
vector is particularly crucial, as it can impact how quickly the method converges
to the dominant eigenvalue.
The Inverse Power Method, on the other hand, allows for accelerated con-
vergence toward smaller eigenvalues and can be adapted for various eigenvalues
by incorporating shifts.
Application Focus:The Power Method excels at identifying the dominant
eigenvalue and its corresponding eigenvector, making it suitable for applications
where the largest eigenvalue is of primary interest.
In contrast, the Inverse Power Method o¤ers greater ‡exibility by enabling
convergence toward almost any desired eigenvalue, which is bene…cial in situa-
tions involving multiple eigenvalues.
Conclusion
In conclusion, while both methods are valuable tools for solving eigenvalue
problems, the choice between them should be guided by the speci…c eigenval-
ues of interest and the characteristics of the matrix in question. The Power
Method is ideal for extracting the dominant eigenvalue, whereas the Inverse
Power Method provides enhanced ‡exibility and speed for targeting other eigen-
values e¤ectively.

1.1.4 Shifted Inverse Power Method.

The Shifted Inverse Method is an e¢ cient numerical technique for …nding eigen-
values and eigenvectors of a matrix, particularly those that are close to a spec-
i…ed shift . This method is an extension of the Inverse Power Method, and it
incorporates a shift to accelerate convergence toward the desired eigenvalue.

Algorithm 12 Given a matrix A and a shift :

1: Choose a Shift: Select a shift not an eigenvalue of A, which serves
as an estimate for the eigenvalue of interest.
2: Form the Shifted Matrix: Calculate the shifted matrix:

S=A I

14
 where I is the identity matrix of the same size as A. Therefore, if i are the
eigenvalues of A; then the eigenvalues of S are given by i for each i:
3: Apply the Inverse Power Method: Use the Inverse Power Method on
the shifted matrix S to …nd the eigenvalue and corresponding eigenvector closest
to the shift :

Concerning this Algorithm, we have the following result of convergence.

Theorem 13 Let A 2 Rn n and a scalar be given, with , not an eigenvalue

of A. De…ne S = A I, and assume the following three conditions:
1: A has n linearly independent eigenvectors, xk ; 1 k n.
2: There exists an index J such that J is the eigenvalue of A that is strictly
closest to :
j J j<j k j ; for all k 6= J:
3: The vector z (0) has a nonzero component in the direction of xJ :
P
n
z (0) = i xi ; and J 6= 0:
i=1

Then the shifted inverse power method converges in the sense that there is
c 6= 0 such that
lim z (k) = cxJ ; and
k!1
1
lim k =( J ) :
k!1

In addition, the error estimates are

m
J
z (m) cxJ = O ;m ! 1
K
m
1 J
m ( J ) = O ; km ! 1
K

where K is the index of the eigenvalue that is second closest to .

Proof. Given that i 6= 0; for all i = 1; : : : ; n; we know that the matrix S

is invertible, allowing us to apply the inverse power method.
Here’s how the inductive argument can be structured:

y (k) S 1 (k 1)
z S 1
y (k 1)
S 1
S 1 (k 2)
z
z (k) = = = =
k k k k 1 k k 1
2 (k 2) k (0)
S z S z k (0)
= = = = Ck S z ;
k k 1 k k 1 ::: 1

1
where Ck = :
k k 1 ::: 1

15
 We then expand in terms of the eigenvector basis, as we did before, to get
k
P
n P
n 1
z (k) = Ck S k
i xi = Ck i xi
i=1 i=1 i
!
k k
1 P
n
J
= Ck J xJ + i xi :
J i=1;i6=J i

Since each z (k) satis…es z (k) 1

= 1 and J
i
< 1 for all i 6= J, it follows that
!
k k
1 P
n
J
1 = lim Ck J xJ + i xi
k !1 J i=1;i6=J i
1
k
1
= lim Ck k J xJ k 1
k !1 J

which implies
k
1 1
lim Ck = < 1:
k !1 J j J j kxJ k1

So that there is at least c = j J jkxJ k1

J
6= 0 such that

lim z (k) = cxJ :

k !1

As for the eigenvalue, we have that

(k)
kz = y (k) = S 1 (k 1)
z ;

so that, taking dot products of both sides with z (k) we get

1 (k 1)
S z ; z (k) k!1 S 1
(cxJ ) ; cxJ 1 hcxJ ; cxJ i 1
k = ! = = :
z (k) ; z (k) hcxJ ; cxJ i J hcxJ ; cxJ i J

Moreover, we clearly, when m ! 1; have that

m m m
1 P
n
J J
z (m) cxJ = Cm i xi C ;
1 J i=1;i6=J i K
1

where K is the index of the eigenvalue that is second closest to .

Remark on Using Shifts in the Inverse Power Method

The introduction of shifts in the Inverse Power Method is signi…cant because it
provides control over which eigenvector we converge to, as well as the rate of
convergence. Speci…cally, the inverse power iteration typically converges to the
eigenvector associated with the smallest eigenvalue.

16
 By introducing a shift , we can target an eigenvalue more e¤ectively. If we
designate J as the eigenvalue such that J is the smallest, the method
will converge to the eigenvector corresponding to the eigenvalue closest to the
shift :
Moreover, the convergence rate is signi…cantly in‡uenced by the proximity
of the shift to an actual eigenvalue. When the shift is chosen to be close to
J , it generally results in much faster convergence than if the shift is far from
the eigenvalue. This capability makes the Inverse Power Method highly ‡exible
and e¢ cient in targeting various eigenvalues, enhancing its utility in practical
applications.

Advantages of the Shifted Inverse Power Method

Targeted Eigenvalue Convergence: By using a shift , the method allows
convergence toward a speci…c eigenvalue of interest, especially when that eigen-
value is not the smallest in magnitude. This means users can focus on the
eigenvalue closest to the selected shift.
Accelerated Convergence Rates: The convergence rate generally im-
proves when the shift is chosen close to the desired eigenvalue. This results in
faster convergence compared to the standard Inverse Power Method, making it
more e¢ cient in practice.
Flexibility in Eigenvalue Targeting:The method can be adapted easily
to target multiple eigenvalues simply by varying the shift. This ‡exibility makes
it useful in a variety of applications, including those requiring the computation
of speci…c eigenvalues.
Applicability to Complex Eigenvalues:The Shifted Inverse Power Method
can e¤ectively handle complex eigenvalues and is particularly useful for matrices
that might have multiple or close eigenvalues.
Ability to Handle Defective Matrices:This method can also be utilized
when dealing with defective matrices (where not all eigenvectors exist), as the
shifts can help isolate certain eigenvalues for convergence.
More Control Over the Eigenvector Output: By strategically selecting
shifts, users gain more control over which eigenvector is obtained, aiding in
applications where certain eigenvectors have greater signi…cance.
In summary, the Shifted Inverse Power Method is a highly e¤ective tech-
nique for eigenvalue computation. By leveraging shifts, it enables targeted con-
vergence, improved speed, and enhanced numerical stability. Its adaptability
makes it a valuable tool in both theoretical studies and practical applications
involving eigenvalue problems.

Exercise 14 (a) Do three iterations of Power method to …nd the approximate

largest eigenvalue and the corresponding eigenvector of the matrix.
(b) Pick = 2:5;use the shifted inverse Power method to …nd the approximate
smallest eigenvalue and the corresponding eigenvector of the matrix, stop the
itaration when k+1 k < 0:05.

17
 (c) Deduce an approximate eigenvalue 2 based on the previous results.
0 1 0 1
4 1 0 1
A = @ 1 4 1 A ; z (0) = @ 0:5 A
0 1 4 1
np p T
o n o n p p T
o
T
Knowing that the eigenvectors: 2 + 4; 1; 2; 1 ; 4; ( 1; 0; 1) ; 4 2; 1; 2; 1 :
D o al l calculation in …ve decim al places.

Solution 15 (a) We 0 have 10 1 0 1

4 1 0 1 1
y (1) = Az (0) = @ 1 4 1 A @ 0:5 A = 4:5 @ 89 A ;
0 1 4 1 1
0 10 1 0 1
4 1 0 1 0:879 99
y (2) = Az (1) = @ 1 4 1 A @ 89 A = 5:555 6 @ 1 A;
0 1 4 1 0:879 99
0 10 1 0 1
4 1 0 0:87999 0:78472
y (3) = Az (2) = @ 1 4 1 A @ 1 A = 5:7600 @ 1 A:
0 1 4 0:87999 0:78472
So, the approximate larget eigenvalue and the corresponding eigenvector are
T
' 5:760 0; and (0:784 72; 1:0; 0:784 72) :
We pick = 2:5; then
0 1 0 1
4 2:5 1 0 1
S=A I=@ 1 4 2:5 1 A ; z (0) = @ 0:5 A :
0 1 4 2:5 1
The inverse of S is
0 1
3:3333 4 2:6667
S 1
=@ 4 6 4 A:
2:6667 4 3:3333
Hence, 0 1 0 1
4 0:8
y (1) = S 1 (0)
z = @ 5 A = 5 @ 1 A = 1 z (1) ;
0 4 1 00:8 1
8:8 0:70968
y (2) = S 1 (1)
z = @ 12:4 A = 12:4 @ 1 A= 2z
(2)
;
8:8 0:70968
since j 2 1j 0:05; then
0 1
0:70721
y (3) = B 1 (2)
z = 11:677 @ 1 A= 3z
(3)
;
0:70721
since j 3 2j 0:05; then

18
 0 1
0:70709
y (4) = S 1 (3)
z = 11:658 @ 1 A= 4z
(4)
;
0:70709

since j 4 3j = j11:677 11:658j = 0:019 < 0:05; then

1
k =( 3 ) = 11:658:
Hence,
1 T
3 = + 2:5 = 2:585 8; and x3 = ( 0:70709; 1; 0:707 09) :
11:658
(c) We have
1 + 2 + 3 = T race (A) = 12:
Then
2 = 12 5:76 2:5858 = 3:654 2:

1.1.5 De‡ation Method

The goal of de‡ation is to remove the in‡uence of a known dominant eigenvalue,
making it easier to …nd the next dominant eigenvalue in the spectrum of the
matrix.
Let A be a be a symmetric matrix with eigenvalues 1 ; : : : ; n and the
corresponding eigenvectors x1 ; : : : ; xn : Once the dominant eigenpair ( 1 ; x1 ) of
A is computed, we may wish to compute 2 : (Recall that j 1 j > j 2 j > j 3 j
::: j n j): Then we have the following result

Theorem 16 Let A be a symmetric matrix, then we construct a new matrix B

de…ned as :
x1 xT1
B=A 1 2
kx1 k2

The, the eigenvalues of de‡ate matrix B are 0; 2 ; :::; n and the eigenvectors
of B are the eigenvectors of A: Here, 0 corresponds to the removed eigenvalue
1:
Proof. The proof of this theorem is left as an exercise in the worksheet02.
To …nd 2 , we can apply the Power Method to B. However, it is important
to note that since 1 is not exact, some error will be introduced in the Power
Method when applied to B. The use of the Power Method to …nd is referred to
as the method of de‡ation.

Example 17 Let
5 2
A=
2 8
(a) Use the Power Method to …nd the dominant eigenvalue 1, and its eigen-
T
vector v, with z (0) = (0; 1) :

19
 T
(b) Apply De‡ation Method to …nd 2; with z (0) = (0; 1) : Use ( 1 ; x1 )
otained in (a) :
Stop when k+1 k < 0:03:

Solution 18 (a) We have

5 2 0 2:0 0:25
y (1) = Az (0) = = =8 ; 1 = 8;
2 8 1 8:0 1
5 2 0:25 0:382 35
y (2) = Az (1) = = 8:5 ; 2 = 8:5;
2 8 1 1:0
with j 2 1j 0:03:
(3) (2) 5 2 0:382 35 0:446 31
y = Az = = 8:764 7 ;
2 8 1:0 1:0
3 = 8:7647;
with j 3 2j 0:03:
(4) (3) 5 2 0:446 31 0:475 86
y = Az = = 8:892 6 ; 4=
2 8 1:0 1:0
8:8926;
with j 4 3j 0:03:
5 2 0:475 86 0:489 21
y (5) = Az (4) = = 8:9517 ; 5=
2 8 1:0 1
8:9517;
with j 5 4j 0:03:
(6) (5) 5 2 0:489 21 0:495 20
y = Az = = 8:978 4 ; 6=
2 8 1 1:0
8: 978 4;
with j 6 5 j = 0:026 7 < 0:03; hence

1 = 8:9784; and x1 = ( 0:49520; 1) T :

(b) Consider the matrix B :

x1 xT1
B=A 8:9784 2;
kx1 k2
so 0 10 1T
@
0:49520 A@
0:49520 A
5 2 1 1 3:2319 1:5705
B= (8:9784) 0 1 2 = :
2 8 0:49520 1:5705 0:78973
@ A
1 2
The Power Method applied to B yields
3:2319 1:5705 0 1:0
y (1) = Az (0) = = 1:5705 ; 1 =
1:5705 0:78973 1 0:502 85
1:570 5;
3:231 9 1:5705 1 1:0
y (2) = Az (1) = = 4:0216 ; 2 =
1:5705 0:78973 0:502 85 0:489 26
4:0216;
with j 2 1j 0:03:

20
 3:2319 1:570 5 1 1:0
y (3) = Az (2) = = 4:0003 ; 3 =
1:570 5 0:789 73 0:48926 0:489 19
4:0003;
Since j 3 2j = 0:021 3 < 0:03; then
T
2 ' 4:0003; and v2 = (1; 0:489 19) :

1.2 The QR Method for Eigenvalues

The QR Algorithm is an iterative method used to …nd the eigenvalues of a
square matrix. It is widely used in numerical linear algebra due to its e¢ ciency
and stability. Below is a step-by-step explanation of the algorithm.
Overview
The QR Algorithm is based on repeated QR factorization and matrix re-
composition. Given a square matrix A, the method constructs a sequence of
matrices that converge to an upper triangular matrix (or Schur form), from
which eigenvalues can be easily extracted.
Algorithm 19 1: Initialize: Set A0 = A ( the initial matrix )
2: Compute QR factorization : Use Householder Re‡ections or Gram-
Schmidt Process to factor the matrix:
Ak = Qk Rk :
Here, Qk is an orthogonal matrix, and Rk is an upper triangular matrix.
3: Recompose the matrix: Update the matrix :
Ak+1 = Rk Qk = QTk Ak Qk :
4: Repeat until convergence: Continue iterating until convergence.
5: Extract eigenvalues:The eigenvalues of A are the diagonal entries of
the converged upper triangular matrix.
Why does it work?
1: Similarity Preservation:T he QR factorization maitains similarity :
Ak+1 is similar to Ak ; preserving the eigenvalues.
2: Convergence to Triangular Form: The process gradually drives Ak
toward an upper triangular form, where eigenvalues become visible.

Advantages
1. More stable than direct methods based on determinants.
2. E¢ cient for large matrices.
3. Can be parallelized for high-performance computing.

Disadvantages
1. Requires multiple matrix factorizations.
1. Computationally expensive for very large matrices.

21
Convergence
Theorem 20 If A be a symmetric matrix or has distinct eigenvalues, then QR
Algorithm converges to an upper triangular matrix whose diagonal elements are
the eigenvalues of A:

Proof. To show convergence, we rely on similarity transformations and Schur’s

theorem: "Any square matrix A can be written as A = QT Q , where Q is
unitary and T is upper triangular with the eigenvalues of A on the diagonal."
The QR Algorithm implicitly seeks to diagonalize A through similarity trans-
formations. If A is symmetric, it has an orthonormal basis of eigenvectors. The
QR process iteratively drives the o¤-diagonal elements toward zero, leading to
a diagonal (or nearly diagonal) form.

Example 21 Use the QR method to calculate the eigenvalues of

5 2
A= ; the true eigenvalues are f4; 9g :
2 8

Solution 22 We use the formulas for the 2 2 case each time we need a QR
factorization.

5 2
A1 = A = ;
2 8
0:928 0:371 5:385 4:828
Q = Q1 = ; R = R1 =
0:371 0:928 0 6:685

6:788 5 2:482 5
A2 = R1 Q1 = ;
2:480 1 6:203 7

0:939 24 0:343 27 7:232 7 4:462 5

Q2 = ; R2 = 5
0:343 27 0:939 24 4:330 8 10 4:977 4

8:325 1 1:708 6
A3 = R2 Q2 = ;
1:708 6 4:675 0

0:979 58 0:201 04 8:498 6 2:613 6

Q3 = ; R3 = 5 ;
0:201 04 0:979 58 3:228 4 10 4:236

8:850 0:852
A4 = R3 Q3 =
0:852 4:149
::::::::::::::::::::::
4
9 5:955 9 10
A13 = 4 :
5:955 6 10 4

Approximate eigenvalues are on the diagonal.

22