Linear Algebra Review

(Adapted from Punit Shah’s slides)

Introduction to Machine Learning (CSC 311)

Spring 2020

University of Toronto

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 1 / 28

Basics

A scalar is a number.

A vector is a 1-D array of numbers. The set of vectors of length n

with real elements is denoted by Rn .
Vectos can be multiplied by a scalar.
Vector can be added together if dimensions match.

A matrix is a 2-D array of numbers. The set of m × n matrices

with real elements is denoted by Rm×n .
Matrices can be added together or multiplied by a scalar.
We can multiply Matrices to a vector if dimensions match.

In the rest we denote scalars with lowercase letters like a, vectors

with bold lowercase v, and matrices with bold uppercase A.

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 2 / 28

Norms

Norms measure how “large” a vector is. They can be defined for
matrices too.

The `p -norm for a vector x:

" #1
X p

kxkp = |xi |p .
i

The `2 -norm is known as the Euclidean norm. P

The `1 -norm is known as the Manhattan norm, i.e., kxk1 = i |xi |.
The `∞ is the max (or supremum) norm, i.e., kxk∞ = maxi |xi |.

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 3 / 28

Dot Product

Dot product is defined as v · u = v> u =

P
i ui vi .
√
The `2 norm can be written in terms of dot product: kuk2 = u.u.

Dot product of two vectors can be written in terms of their `2

norms and the angle θ between them:

a> b = kak2 kbk2 cos(θ).

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 4 / 28

Cosine Similarity

Cosine between two vectors is a measure of their similarity:

a·b
cos(θ) = .
kak kbk

Orthogonal Vectors: Two vectors a and b are orthogonal to

each other if a · b = 0.

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 5 / 28

Vector Projection
b
Given two vectors a and b, let b̂ = kbk be the unit vector in the
direction of b.
Then a1 = a1 · b̂ is the orthogonal projection of a onto a straight
line parallel to b, where
b
a1 = kak cos(θ) = a · b̂ = a ·
kbk

Image taken from wikipedia.

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 6 / 28

Trace

Trace is the sum of all the diagonal elements of a matrix, i.e.,

X
Tr(A) = Ai,i .
i

Cyclic property:

Tr(ABC) = Tr(CAB) = Tr(BCA).

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 7 / 28

Multiplication
Matrix-vector multiplication is a linear transformation. In other
words,
X
M(v1 + av2 ) = Mv1 + aMv2 =⇒ (Mv)i = Mi,j vj .
j

Matrix-matrix multiplication is the composition of linear

transformations, i.e., P
(AB)v = A(Bv) =⇒ (AB)i,j = k Ai,k Bk,j .

Intro ML (UofT) Image

CSC311 taken
– Tut from wikipedia.
2 – Linear Algebra 8 / 28
Invertibality

I denotes the identity matrix which is a square matrix of zeros

with ones along the diagonal. It has the property IA = A
(BI = B) and Iv = v

A square matrix A is invertible if A−1 exists such that

A−1 A = AA−1 = I.

Not all non-zero matrices are invertible, e.g., the following matrix
is not invertible:  
1 1
1 1

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 9 / 28

Transposition

Transposition is an operation on matrices (and vectors) that

interchange rows with columns. (A> )i,j = Aj,i .

(AB)> = B> A> .

A is called symmetric when A = A> .

A is called orthogonal when AA> = A> A = I or A−1 = A> .

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 10 / 28

Diagonal Matrix

A diagonal matrix has all entries equal to zero except the diagonal
entries which might or might not be zero, e.g. identity matrix.

A square diagonal matrix with diagonal enteries given by entries of

vector v is denoted by diag(v).

Multiplying vector x by a diagonal matrix is efficient:

diag(v)x = v  x,

where  is the entrywise product.

Inverting a square diagonal matrix is efficient

 1 1 
diag(v)−1 = diag [ , . . . , ]> .
v1 vn

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 11 / 28

Determinant

Determinant of a square matrix is a mapping to scalars.

det(A) or |A|

Measures how much multiplication by the matrix expands or

contracts the space.

Determinant of product is the product of determinants:

det(AB) = det(A)det(B)

 
a b 
 c d = ad − bc
 

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 12 / 28

List of Equivalencies

Assuming that A is a square matrix, the following statements are

equivalent
Ax = b has a unique solution (for every b with correct
dimension).
Ax = 0 has a unique, trivial solution: x = 0.
Columns of A are linearly independent.
A is invertible, i.e. A−1 exists.
det(A) 6= 0

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 13 / 28

Zero Determinant

If det(A) = 0, then:

A is linearly dependent.

Ax = b has infinitely many solutions or no solution. These cases

correspond to when b is in the span of columns of A or out of it.

Ax = 0 has a non-zero solution. (since every scalar multiple of

one solution is a solution and there is a non-zero solution we get
infinitely many solutions.)

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 14 / 28

Matrix Decomposition

We can decompose an integer into its prime factors, e.g.,

12 = 2 × 2 × 3.

Similarly, matrices can be decomposed into product of other

matrices.
A = Vdiag(λ)V−1

Examples are Eigendecomposition, SVD, Schur decomposition, LU

decomposition, . . . .

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 15 / 28

Eigenvectors

An eigenvector of a square matrix A is a nonzero vector v such

that multiplication by A only changes the scale of v.

Av = λv

The scalar λ is known as the eigenvalue.

If v is an eigenvector of A, so is any rescaled vector sv. Moreover,

sv still has the same eigenvalue. Thus, we constrain the
eigenvector to be of unit length:

||v||2 = 1

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 16 / 28

Characteristic Polynomial(1)

Eigenvalue equation of matrix A.

Av = λv
λv − Av = 0
(λI − A)v = 0

If nonzero solution for v exists, then it must be the case that:

det(λI − A) = 0

Unpacking the determinant as a function of λ, we get:

PA (λ) = det(λI − A) = 1 × λn + cn−1 × λn−1 + . . . + c0

This is called the characterisitc polynomial of A.

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 17 / 28

Characteristic Polynomial(2)

If λ1 , λ2 , . . . , λn are roots of the characteristic

Qn polynomial, they are
eigenvalues of A and we have PA (λ) = i=1 (λ − λi ).

cn−1 = − ni=1 λi = −tr(A). This means that the sum of

P
eigenvalues equals to the trace of the matrix.

c0 = (−1)n ni=1 λi = (−1)n det(A). The determinant is equal to

Q
the product of eigenvalues.

Roots might be complex. If a root has multiplicity of rj > 1 (This

is called the algebraic dimension of eigenvalue), then the geometric
dimension of eigenspace for that eigenvalue might be less than rj
(or equal but never more). But for every eigenvalue, one
eigenvector is guaranteed.

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 18 / 28

Example

Consider the matrix:  

2 1
A =
1 2

The characteristic polynomial is:

 
λ − 2 −1
det(λI − A) = det = 3 − 4λ + λ2 = 0
−1 λ − 2

It has roots λ = 1 and λ = 3 which are the two eigenvalues of A.

We can then solve for eigenvectors using Av = λv:

vλ=1 = [1, −1]> and vλ=3 = [1, 1]>

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 19 / 28

Eigendecomposition

Suppose that n × n matrix A has n linearly independent

eigenvectors {v(1) , . . . , v(n) } with eigenvalues {λ1 , . . . , λn }.

Concatenate eigenvectors (as columns) to form matrix V.

Concatenate eigenvalues to form vector λ = [λ1 , . . . , λn ]> .

The eigendecomposition of A is given by:

AV = Vdiag(λ) =⇒ A = Vdiag(λ)V−1

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 20 / 28

Symmetric Matrices
Every symmetric (hermitian) matrix of dimension n has a set of
(not necessarily unique) n orthogonal eigenvectors. Furthermore,
all eigenvalues are real.
Every real symmetric matrix A can be decomposed into
real-valued eigenvectors and eigenvalues:
A = QΛQ>
Q is an orthogonal matrix of the eigenvectors of A, and Λ is a
diagonal matrix of eigenvalues.
We can think of A as scaling space by λi in direction v(i) .

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 21 / 28

Eigendecomposition is not Unique

Decomposition is not unique when two eigenvalues are the same.

By convention, order entries of Λ in descending order. Then,

eigendecomposition is unique if all eigenvalues have multiplicity
equal to one.

If any eigenvalue is zero, then the matrix is singular. Because if v

is the corresponding eigenvector we have: Av = 0v = 0.

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 22 / 28

Positive Definite Matrix

If a symmetric matrix A has the property:

x> Ax > 0 for any nonzero vector x

Then A is called positive definite.

If the above inequality is not strict then A is called positive

semidefinite.

For positive (semi)definite matrices all eigenvalues are positive(non

negative).

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 23 / 28

Singular Value Decomposition (SVD)

If A is not square, eigendecomposition is undefined.

SVD is a decomposition of the form A = UDV> .

SVD is more general than eigendecomposition.

Every real matrix has a SVD.

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 24 / 28

SVD Definition (1)

Write A as a product of three matrices: A = UDV> .

If A is m × n, then U is m × m, D is m × n, and V is n × n.

U and V are orthogonal matrices, and D is a diagonal matrix (not

necessarily square).

Diagonal entries of D are called singular values of A.

Columns of U are the left singular vectors, and columns of V

are the right singular vectors.

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 25 / 28

SVD Definition (2)

SVD can be interpreted in terms of eigendecompostion.

Left singular vectors of A are the eigenvectors of AA> .

Right singular vectors of A are the eigenvectors of A> A.

Nonzero singular values of A are square roots of eigenvalues of

A> A and AA> .

Numbers on the diagonal of D are sorted largest to smallest and

are non-negative (A> A and AA> are semipositive definite.).

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 26 / 28

Matrix norms

We may define norms for matrices too. We can either treat a

matrix as a vector, and define a norm based on an entrywise norm
(example: Frobenius norm). Or we may use a vector norm to
“induce” a norm on matrices.
Frobenius norm: sX
kAkF = a2i,j .
i,j

Vector-induced (or operator, or spectral) norm:

kAk2 = sup kAxk2 .

kxk2 =1

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 27 / 28

SVD Optimality

Given a matrix A, SVD allows us to find its “best” (to be defined)

rank-r approximation Ar .
We can write A = UDV> as A = ni=1 di ui vi> .
P

For r ≤ n, construct Ar = ri=1 di ui vi> .

P

The matrix Ar is a rank-r approximation of A. Moreover, it is the

best approximation of rank r by many norms:
When considering the operator (or spectral) norm, it is optimal.
This means that kA − Ar k2 ≤ kA − Bk2 for any rank r matrix B.
When considering Frobenius norm, it is optimal. This means that
kA − Ar kF ≤ kA − BkF for any rank r matrix B. One way to
interpret this inequality is that rows (or columns) of Ar are the
projection of rows (or columns) of A on the best r dimensional
subspace, in the sense that this projection minimizes the sum of
squared distances.

Intro ML (UofT) CSC311 – Tut 2 – Linear Algebra 28 / 28