Introduction to Eigenvalues
Introduction to Eigenvalues
y1 .t/ D e t a
Look for In vector notation this is y.t/ D e t x (2)
y2 .t/ D e t b
That vector x D .a; b/ is called an eigenvector. The growth rate is an eigenvalue. This
section will show how to find x and . Here I will jump to x and for the matrix in (1).
a 1
First eigenvector x = D and first eigenvalue D 5 in y D e 5t x
b 1
y1 D e 5t y1 0 D 5 e 5t D 4y1 C y2
5t
has 0 5t
y2 D e y2 D 5 e D 3y1 C 2y2
a 1
Second eigenvector x = D and second eigenvalue D 1 in y D e t x
b 3
This y D e t x is a y1 D et y1 0 D e t D 4y1 C y2
has
second solution y2 D 3 et y2 0 D 3 e t D 3y1 C 2y2
325
326 Chapter 6. Eigenvalues and Eigenvectors
Those two x’s and ’s combine with any c1 , c2 to give the complete solution to y 0 D Ay :
5t
e et 5t 1 t 1
Complete solution y.t/ D c1 5t C c2 D c1 e C c2 e : (3)
e 3 et 1 3
This is exactly what we hope to achieve for other equations y 0 D Ay with constant A.
The solutions we want have the special form y.t/ D e t x. Substitute that solution
into y 0 D Ay, to see the equation Ax D x for an eigenvalue and its eigenvector x :
d t
.e x/ D A.e t x/ is e t x D Ae t x: Divide both sides by e t :
dt
Those eigenvalues (5 and 1 for this A) are a new way to see into the heart of a matrix.
This chapter enters a different part of linear algebra, based on Ax D x. The last page of
Chapter 6 has eigenvalue-eigenvector information about many different matrices.
Those were the vectors .a; b/ in our special solutions y D e t x. Both components of y
have the growth rate , so the differential equation was easily solved : y D e t x.
Two eigenvectors gave two solutions. Combinations c1 y 1 C c2 y 2 give all solutions.
:8 :3
Example 2 Find the eigenvalues and eigenvectors of the Markov matrix A D .
:2 :7
:8 :3 3 1
det.A I / D det D 2 C D . 1/ 1
2
:
:2 :7 2 2
I factored the quadratic into 1 times 12 , to see the two eigenvalues D 1 and 12 .
The eigenvectors x 1 and x 2 are in the nullspaces of A I and A 21 I .
" # "
#" #
:6 :6 :8 :3
x1 D and Ax 1 D D x 1 (Ax D x means that 1 D 1)
:4 :4 :2 :7
" # " #" # " #
1 :8 :3 1 :5
x2 D and Ax 2 D D (this is 12 x 2 so 2 D 21 ).
1 :2 :7 1 :5
Notice 2 . This pattern keeps going, because the eigenvectors stay in their own directions.
They never get mixed. The eigenvectors of A100 are the same x 1 and x 2 . The eigenvalues
of A100 are 1100 D 1 and . 21 /100 D very small number.
We mention that this particular A is a Markov matrix. Its entries are positive and
every column adds to 1. Those facts guarantee that the largest eigenvalue must be D 1.
328 Chapter 6. Eigenvalues and Eigenvectors
" #
:6 A2 x 1 D .1/2 x 1
D1 Ax 1 D x 1 D 2
D1
:4
2 D :25 " #
2 2
:25
" # A x 2 D .:5/ x 2 D
:5 :25
D :5 Ax 2 D 2 x 2 D
:5
:8 :3 Ax D x
" # AD
1 :2 :7 An x D n x
x2 D
1
Figure 6.1: The eigenvectors keep their directions. A2 has eigenvalues 12 and .:5/2 .
The eigenvector Ax1 D x 1 is the steady state—which all columns of Ak will approach.
Giant Markov matrices are the key to Google’s search algorithm. It ranks web pages.
Linear algebra has made Google one of the most valuable companies in the world.
Powers of a Matrix
When the eigenvalues of A are known, we immediately know the eigenvalues of all
powers Ak and shifts A C cI and all functions of A. Each eigenvector of A is also an
eigenvector of Ak and A 1 and A C cI :
1
If Ax D x then Ak x D k x and A 1
xD x and .A C cI /x D . C c/x: (6)
Start again with A2 x, which is A times Ax D x. Then Ax is the same as Ax for any
number , and Ax is 2 x. We have proved that A2 x D 2 x.
For higher powers Ak x, continue multiplying Ax D x by A. Step by step you reach
A x D k x. For the eigenvalues of A 1 , first multiply by A 1 and then divide by :
k
1 1 1 1 1
Eigenvalues of A are Ax D x x D A x A xD x (7)
1
We are assuming that A exists ! If A is invertible then will never be zero.
Invertible matrices have all ¤ 0. Singular matrices have the eigenvalue D 0.
The shift from A to A C cI just adds c to every eigenvalue (don’t change x) :
Shift of A If Ax D x then .A C cI /x D Ax C cx D . C c/x: (8)
As long as we keep the same eigenvector x, we can allow any function of A :
Functions of A .A2 C 2A C 5I /x D .2 C 2 C 5/x e A x D e x: (9)
6.1. Introduction to Eigenvalues 329
Let me show you the powers of the Markov matrix A in Example 2. That starting matrix
is unrecognizable after a few steps.
" # " # " # " #
:8 :3 :70 :45 :650 :525 :6000 :6000
:2 :7 :30 :55 :350 :475 :4000 :4000 (10)
A A2 A3 A100
A100 was found by using D 1 and its eigenvector Œ:6; :4, not by multiplying 100 matrices.
The eigenvalues of A are 1 and 12 , so the eigenvalues of A100 are 1 and . 12 /100 . That last
number is extremely small, and we can’t see it in the first 30 digits of A100 .
How could you multiply A99 times another vector like v D .:8; :2/ ? This is not an
eigenvector, but v is a combination of eigenvectors. This is a key idea, to express any
vector v by using the eigenvectors.
" # " # " #
Separate into eigenvectors :8 :6 :2
vD D C : (11)
v D x 1 C .:2/x 2 :2 :4 :2
Each eigenvector is multiplied by its eigenvalue, when we multiply the vector by A.
After 99 steps, x 1 is unchanged and x 2 is multiplied by . 12 /99 :
" # " # 2 very 3
:8 1 :6
A99 is A99 .x 1 C :2x 2 / D x 1 C .:2/. /99 x 2 D C 4 small 5 :
:2 2 :4 vector
This is the first column of A100 , because v D .:8; :2/ is the first column of A. The number
we originally wrote as :6000 was not exact. We left out .:2/. 21 /99 which wouldn’t show up
for 30 decimal places.
The eigenvector x 1 D .:6; :4/ is a “steady state” that doesn’t change (because 1 D 1/.
The eigenvector x 2 is a “decaying mode” that virtually disappears (because 2 D 1=2/.
The higher the power of A, the more closely its columns approach the steady state.
Determinants
The determinant is a single number with amazing properties. It is zero when the matrix has
no inverse. That leads to the eigenvalue equation det.A I / D 0. When A is invertible,
the determinant of A 1 is 1=.det A/. Every entry in A 1 is a ratio of two determinants.
I want to summarize the algebra, leaving the details for my companion textbook
Introduction to Linear Algebra. The difficulty with det.A I / D 0 is that an n by n
determinant involves n Š terms. For n D 5 this is 120 terms—generally impossible to use.
For n D 3 there are six terms, three with plus signs and three with minus. Each of
those six terms includes one number from every row and every column :
2 3
1 2 3 1 2 Determinant from nŠ D 6 terms
6 7
44 5 65 4 5 Three plus signs, three minus signs
7 8 9 7 8 C.1/.5/.9/ C.2/.6/.7/ C.3/.4/.8/
C C C .3/.5/.7/ .1/.6/.8/ .2/.4/.9/
That shows how to find the six terms. For this particular matrix the total must be det A D 0,
because the matrix happens to be singular : row 1 C row 3 equals 2.row 2/.
Let me start with five useful properties of determinants, for all square matrices.
1. Subtracting a multiple of one row from another row leaves det A unchanged.
By combining 1; 2; 3 you will see how the determinant comes from elimination :
Property 1 says that A and U have the same determinant, unless rows are exchanged.
Property 2 says that an odd number of exchanges would leave det A D det U .
Property 3 says that det U is the product of the pivots on its main diagonal.
When elimination takes A to U , we find det A D ˙ (product of the pivots). This is how
all numerical software (like MATLAB or Python or Julia ) would compute det A.
Plus and minus signs play a big part in determinants. Half of the n Š terms have plus
signs, and half come with minus signs. For n D 3, one row exchange puts 3 5 7
or 1 6 8 or 2 4 9 on the main diagonal. A minus sign from one row exchange.
6.1. Introduction to Eigenvalues 331
Two row exchanges (an even number) take you back to (2) (6) (7) and (3) (4) (8). This
indicates how the 24 terms would go for n D 4, twelve terms with plus and twelve with
minus.
Even permutation matrices have det P D 1 and odd permutations have det P D 1.
Inverse of A If det A ¤ 0, you can solve Av D b and find A 1 using determinants :
The matrix Bj replaces the j th column of A by the vector b. Cramer’s Rule is expensive !
To find the columns of A 1 , we solve AA 1 D I . That is the Gauss-Jordan idea : For
each column b in I , solve Av D b to find a column v of A 1 .
In this special case, when b is a column of I , the numbers det Bj in Cramer’s Rule are
called cofactors. They reduce to determinants of size n 1, because b has so many zeros.
Every entry of A 1 is a cofactor of A divided by the determinant of A.
I will close with three examples, to introduce the “trace” of a matrix and to show
that real matrices can have imaginary (or complex) eigenvalues and eigenvectors.
" #
2 1
Example 3 Find the eigenvalues and eigenvectors of S D .
1 2
Solution You can see that x D .1; 1/ will be in the same direction as S x D .3; 3/.
Then x is an eigenvector of S with D 3. We want the matrix S I to be singular.
" # " #
2 1 2 1
SD det .S I / D det D 2 4 C 3 D 0:
1 2 1 2
Notice that 3 is the determinant of S (without ). And 4 is the sum 2 C 2 down the central
diagonal of S . The diagonal sum 4 is the “trace” of A. It equals 1 C 2 D 3 C 1.
The eigenvalues 3 and 1 are real. The eigenvectors .1; 1/ and .1; 1/ are orthogonal.
Those properties always come together for symmetric matrices (Section 6.5).
Here is an antisymmetric matrix with AT D A. It rotates all real vectors by D 90ı .
Real vectors can’t be eigenvectors of a rotation matrix because it changes their direction.
332 Chapter 6. Eigenvalues and Eigenvectors
Example 4 This real matrix has imaginary eigenvalues i , i and complex eigenvectors :
0 1 T 1
AD D A det.A I / D det D 2 C 1 D 0:
1 0 1
That determinant 2 C 1 is zero for D i and i . The eigenvectors are .1; i / and .1; i / :
0 1 1 i 1 0 1 1 i 1
D Di D D i
1 0 i 1 i 1 0 i 1 i
Somehow those complex vectors x 1 and x 2 don’t get rotated (I don’t really know how).
Multiplying the eigenvalues .i /. i / gives det A D 1. Adding the eigenvalues gives
.i / C . i / D 0. This equals the sum 0 C 0 down the diagonal of A.
Those are true statements for all square matrices. The trace is the sum a11 C C ann
down the main diagonal of A. This sum and product are is especially valuable for 2
by 2 matrices, when the determinant 1 2 D ad bc and the trace 1 C 2 D a C d
completely determine 1 and 2 . Look now at rotation of a plane through any angle .
Example 5 Rotation comes from an orthogonal matrix Q. Then 1 D e i and 2 D e i
:
cos sin 1 D cos C i sin 1 C 2 D 2 cos D trace
QD
sin cos 2 D cos i sin 1 2 D 1 D determinant
5. The sum down the main diagonal of A (the trace) is the sum of the eigenvalues.
6. The determinant is the product of the ’s. It is also ˙ (product of the pivots).
Eigenvalues of A C B (are equal to) (might not be equal to) eigenvalues of A plus
eigenvalues of B.
334 Chapter 6. Eigenvalues and Eigenvectors
9 What do you do to the equation Ax D x, in order to prove (a), (b), and (c) ?
10 Find the eigenvalues and eigenvectors for both of these Markov matrices A and A1 .
Explain from those answers why A100 is close to A1 :
:6 :2 1=3 1=3
AD and A1 D :
:4 :8 2=3 2=3
12 Find three eigenvectors for this matrix P . Projection matrices only have D 1 and 0.
Eigenvectors are in or orthogonal to the subspace that P projects onto.
2 3
:2 :4 0
Projection matrix P 2 D P D P T P D 4 :4 :8 0 5 :
0 0 1
If two eigenvectors x and y share the same repeated eigenvalue , so do all their
combinations cx C d y. Find an eigenvector of P with no zero components.
13 From the unit vector u D 16 ; 61 ; 36 ; 65 construct the rank one projection matrix
P D uuT . This matrix has P 2 D P because uT u D 1.
15 Find three 2 by 2 matrices that have 1 D 2 D 0. The trace is zero and the
determinant is zero. A might not be the zero matrix but check that A2 is all zeros.
16 This matrix is singular with rank one. Find three ’s and three eigenvectors :
2 3 2 3
1 2 1 2
Rank one A D 425 2 1 2 D 44 2 45:
1 2 1 2
(a) Give a basis for the nullspace of A and a basis for the column space.
(b) Find a particular solution to Ax D v C w. Find all solutions.
(c) Ax D u has no solution. If it did then would be in the column space.
20 Choose the last row of A to produce .a/ eigenvalues 4 and 7 .b/ any 1 and 2 .
0 1
Companion matrix AD :
22 Construct any 3 by 3 Markov matrix M : positive entries down each column add
to 1. Show that M T .1; 1; 1/ D .1; 1; 1/. By Problem 21, D 1 is also an eigenvalue
of M . Challenge : A 3 by 3 singular Markov matrix with trace 12 has what ’s ?
23 Suppose A and B have the same eigenvalues 1 ; : : :; n with the same independent
eigenvectors x 1 ; : : :; x n . Then A D B. Reason : Any vector v is a combination
c1 x 1 C C cn x n . What is Av ? What is Bv ?
336 Chapter 6. Eigenvalues and Eigenvectors
24 The block B has eigenvalues 1; 2 and C has eigenvalues 3; 4 and D has eigenval-
ues 5; 7. Find the eigenvalues of the 4 by 4 matrix A :
2 3
0 1 3 0
B C 6 2 3 0 47
AD D6 4 0 0 6 15:
7
0 D
0 0 1 6
30 The sum of the diagonal entries (the trace) equals the sum of the eigenvalues :
a b
AD has det.A I / D 2 .a C d / C ad bc D 0:
c d
p
The quadratic formula gives the eigenvalues D .aCd C /=2 and D .
Their sum is . If A has 1 D 3 and 2 D 4 then det.A I/ D .