v
Introduction to Linear Algebra, Sixth Edition
Gilbert Strang Wellesley-Cambridge Press 2022
One goal of this Preface can be achieved right away. You need to know about the video
lectures for MIT’s Linear Algebra course Math 18.06. Those videos go with this book,
and they are part of MIT’s OpenCourseWare. The direct links to linear algebra are
https://ocw.mit.edu/courses/18-06-linear-algebra-spring-2010/
https://ocw.mit.edu/courses/18-06sc-linear-algebra-fall-2011/
On YouTube those lectures are at https://ocw.mit.edu/1806videos and /1806scvideos
The first link brings the original lectures from the dawn of OpenCourseWare. Problem
solutions by graduate students (really good) and also a short introduction to linear algebra
were added to the new 2011 lectures. And the course today has a new start—the crucial
ideas of linear independence and the column space of a matrix have moved near the front.
I would like to tell you about those ideas in this Preface.
Start with two column vectors a1 and a2 . They can have three components each,
so they correspond to points in 3-dimensional space. The picture needs a center point
which locates the zero vector :
2 1 0
a1 = 3 a2 = 4 zero vector = 0 .
1 2 0
The vectors are drawn on this 2-dimensional page. But we all have practice in visualizing
three-dimensional pictures. Here are a1 , a2 , 2a1 , and the vector sum a1 + a2 .
3
1 a1 + a2 = 7
3
a2 = 4 2a1 + a2
2
0
0= 0
2
4
0
a1 = 3 2a1 = 6
1 2
vi
That picture illustrated two basic operations—adding vectors a1 + a2 and multiplying
a vector by 2. Combining those operations produced a “linear combination” 2a1 + a2 :
Linear combination = ca1 + da2 for any numbers c and d
Those numbers c and d can be negative. In that case ca1 and da2 will reverse their direc-
tions : they go right to left. Also very important, c and d can involve fractions. Here is a
picture with a lot more linear combinations. Eventually we want all vectors ca1 + da2 .
2a1 + a2
a2 a1 + a2
−a1 + a2
1
2a1 + 21 a2
−a1 + 2 a2
a1 2a1
−a1
0
2a1 − 12 a2
−a1 − 21 a2
2a1 − a2
a1 − a2
−a2
−a1 − a2
Here is the key ! The combinations ca1 + da2 fill a whole plane. It is an infinite plane
in 3-dimensional space. By using more and more fractions and decimals c and d, we fill in
a complete plane. Every point on the plane is a combination of a1 and a2 .
Now comes a fundamental idea in linear algebra : a matrix. The matrix A holds
n column vectors a1 , a2 , . . . , an . At this point our matrix has two columns a1 and a2 , and
those are vectors in 3-dimensional space. So the matrix has three rows and two columns.
3 by 2 matrix 2 1
m = 3 rows A = a1 a2 = 3 4
n = 2 columns 1 2
The combinations of those two columns produced a plane in three-dimensional space.
There is a natural name for that plane. It is the column space of the matrix. For any A,
the column space of A contains all combinations of the columns.
Here are the four ideas introduced so far. You will see them all in Chapter 1.
1. Column vectors a1 and a2 in three dimensions
2. Linear combinations ca1 + da2 of those vectors
3. The matrix A contains the columns a1 and a2
4. Column space of the matrix = all linear combinations of the columns = plane
vii
Now we include 2 more columns in A 2 1 3 0
A= 3 4 7 0
The 4 columns are in 3-dimensional space 1 2 3 −1
Linear algebra aims for an understanding of every column space. Let me try this one.
Columns 1 and 2 produce the same plane as before (same a1 and a2 )
Column 3 contributes nothing new because a3 is on that plane : a3 = a1 + a2
Column 4 is not on the plane : Adding in c4 a4 raises or lowers the plane
The column space of this matrix A is the whole 3-dimensional space : all points !
You see how we go a column at a time, left to right. Each column can be independent
of the previous columns or it can be a combination of those columns. To produce every
point in 3-dimensional space, you need three independent columns.
Matrix Multiplication A = CR
Using the words “linear combination” and “independent columns” gives a good picture
of that 3 by 4 matrix A. Column 3 is a linear combination : column 1 + column 2.
Columns 1, 2, 4 are independent. The only way to produce the zero vector as a
combination of the independent columns 1, 2, 4 is to multiply all those columns by zero.
We are so close to a key idea of Chapter 1 that I have to go on. Matrix multiplication
is the perfect way to write down what we know. From the 4 columns of A we pick out the
independent columns a1 , a2 , a4 in the column matrix C. Every column of R tells us the
combination of a1 , a2 , a4 in C that produces a column of A. A equals C times R :
2 1 3 0 2 1 0 1 0 1 0
A = 3 4 7 0 = 3 4 0 0 1 1 0 = CR
1 2 3 −1 1 2 −1 0 0 0 1
Column 3 of A is dependent on columns 1 and 2 of A, and column 3 of R shows how.
Add the independent columns 1 and 2 of C to get column a3 = a1 + a2 = (3, 7, 3) of A.
Matrix multiplication : Each column j of CR is C times column j of R
Section 1.3 of the book will multiply a matrix times a vector (two ways). Then Section
1.4 will multiply a matrix times a matrix. This is the key operation of linear algebra.
It is important that there is more than one good way to do this multiplication.
I am going to stop here. The normal purpose of the Preface is to tell you about the
big picture. The next pages will give you two ways to organize this subject—especially
the first seven chapters that more than fill up most linear algebra courses. Then come
optional chapters, leading to the most active topic in applications today : deep learning.
viii
The Four Fundamental Subspaces
You have just seen how the course begins—with the columns of a matrix A. There were
two key steps. One step was to take all combinations ca1 + da2 + ea3 + f a4 of the
columns. This led to the column space of A. The other step was to factor the matrix into
C times R. That matrix C holds a full set of independent columns.
I fully recognize that this is only the Preface to the book. You have had zero practice
with the column space of a matrix (and even less practice with C and R). But the good
thing is : Those are the right directions to start. Eventually, every matrix will lead to four
fundamental spaces. Together with the column space of A comes the row space—all
combinations of the rows. When we take all combinations of the n columns and all
combinations of the m rows—those combinations fill up “spaces” of vectors.
The other two subspaces complete the picture. Suppose the row space is a plane in
three dimensions. Then there is one special direction in the 3D picture—that direction is
perpendicular to the row space. That perpendicular line is the nullspace of the matrix.
We will see that the vectors in the nullspace (perpendicular to all the rows) solve Ax = 0 :
the most basic of linear equations.
And if vectors perpendicular to all the rows are important, so are the vectors
perpendicular to all the columns. Here is the picture of the Four Fundamental Subspaces.
Row space Column space
Dimension r Dimension r
combinations combinations
of the rows of the columns
The Big Picture
n dimensional (m rows and n columns) m dimensional
space space
perpendicular
perpendicular to the columns
to the rows
Nullspace of A Nullspace of AT
dimension n − r dimension m − r
The Four Fundamental Subspaces : An m by n matrix with r independent columns.
This picture of four subspaces comes in Chapter 3. The idea of perpendicular spaces is
developed in Chapter 4. And special “basis vectors” for all four subspaces are discovered
in Chapter 7. That step is the final piece in the Fundamental Theorem of Linear Algebra.
The theorem includes an amazing fact about any matrix, square or rectangular :
The number of independent columns equals the number of independent rows.
ix
Five Factorizations of a Matrix
Here are the organizing principles of linear algebra. When our matrix has a special property,
these factorizations will show it. Chapter after chapter, they express the key idea in a direct
and useful way.
The usefulness increases as you go down the list. Orthogonal matrices are the win-
ners in the end, because their columns are perpendicular unit vectors. That is perfection.
cos θ − sin θ
2 by 2 Orthogonal Matrix = = Rotation by Angle θ
sin θ cos θ
Here are the five factorizations from Chapters 1, 2, 4, 6, 7 :
1 A = CR = R combines independent columns in C to give all columns of A
2 A = LU = Lower triangular L times Upper triangular U
4 A = QR = Orthogonal matrix Q times Upper triangular R
6 S = QΛQT = (Orthogonal Q) (Eigenvalues in Λ) (Orthogonal QT )
7 A = U ΣV T = (Orthogonal U ) (Singular values in Σ) (Orthogonal V T )
May I call your attention to the last one ? It is the Singular Value Decomposition (SVD).
It applies to every matrix A. Those factors U and V have perpendicular columns—all of
length one. Multiplying any vector by U or V leaves a vector of the same length—so
computations don’t blow up or down. And Σ is a positive diagonal matrix of “singular
values”. If you learn about eigenvalues and eigenvectors in Chapter 6, please continue
a few pages to singular values in Section 7.1.
Deep Learning
For a true picture of linear algebra, applications have to be included. Completeness is
totally impossible. At this moment, the dominating direction of applied mathematics
has one special requirement : It cannot be entirely linear !
One name for that direction is “deep learning”. It is an extremely successful approach to
a fundamental scientific problem : Learning from data. In many cases the data comes in a
matrix. Our goal is to look inside the matrix for the connections between variables. Instead
of solving matrix equations or differential equations that express known input-output rules,
we have to find those rules. The success of deep learning is to build a function F (x, v)
with inputs x and v of two kinds :
The vectors v describes the features of the training data.
The matrices x assign weights to those features.
The function F (x, v) is close to the correct output for that training data v.
When v changes to unseen test data, F (x, v) stays close to correct.
This success comes partly from the form of the learning function F , which allows it to
include vast amounts of data. In the end, a linear function F would be totally inadequate.
The favorite choice for F is piecewise linear. This combines simplicity with generality.
x
Applications in the Book and on the Website
I hope this book will be useful to you long after the linear algebra course is complete.
It is all the applications of linear algebra that make this possible. Matrices carry data,
and other matrices operate on that data. The goal is to “see into a matrix” by understand-
ing its eigenvalues and eigenvectors and singular values and singular vectors. And each
application has special matrices—here are four examples :
Markov matrices M Each column is a set of probabilities adding to 1.
Incidence matrices A Graphs and networks start with a set of nodes. The matrix
A tells the connections (edges) between those nodes.
Transform matrices F The Fourier matrix uncovers the frequencies in the data.
Covariance matrices C The variance is key information about a random variable.
The covariance explains dependence between variables.
We included those applications and more in this Sixth Edition. For the crucial computation
of matrix weights in deep learning, Chapter 9 presents the ideas of optimization. This is
where linear algebra meets calculus : derivative = zero becomes a matrix equation at the
minimum point because F (x) has many variables.
Several topics from the Fifth Edition gave up their places but not their importance.
Those sections simply moved onto the Web. The website for this new Sixth Edition is
math.mit.edu/linearalgebra
That website includes sample sections from this new edition and solutions to all Problem
Sets. These sections (and more) are saved online from the Fifth Edition :
Fourier Series Norms and Condition Numbers
Iterative Methods and Preconditioners Linear Algebra for Cryptography
Here is a small touch of linear algebra—three questions before this course gets serious :
1. Suppose you draw three straight line segments of lengths r and s and t on this page.
What are the conditions on those three lengths to allow you to make the segments
into a triangle ? In this question you can choose the directions of the three lines.
2. Now suppose the directions of three straight lines u, v, w are fixed and different.
But you could stretch those lines to au, bv, cw with any numbers a, b, c.
Can you always make a closed triangle out of the three vectors au, bv, cw ?
3. Linear algebra doesn’t stay in a plane ! Suppose you have four lines u, v, w, z
in different directions in 3-dimensional space. Can you always choose the numbers
a, b, c, d (zeros not allowed) so that au + bv + cw + dz = 0 ?
For typesetting this book, maintaining its website, offering quality textbooks to Indian fans,
I am grateful to Ashley C. Fernandes of Wellesley Publishers (www.wellesleypublishers.com)
