MA571 Numerical Linear Algebra

Linear Least-Squares Problem(LSP)

Lecture 13: Singular Value Decomposition (SVD)

Rafikul Alam
Department of Mathematics
IIT Guwahati

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Outline

• Singular Value Decomposition (SVD)

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Spectral Theorem
Definition. Let A ∈ Cn×n . Then λ ∈ C is said to be an eigenvalue of A if there exists a nonzero
vector v ∈ Cn such that
Av = λv .
The vector v is called an eigenvector of A corresponding to the eigenvalue λ.

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Spectral Theorem
Definition. Let A ∈ Cn×n . Then λ ∈ C is said to be an eigenvalue of A if there exists a nonzero
vector v ∈ Cn such that
Av = λv .
The vector v is called an eigenvector of A corresponding to the eigenvalue λ.

Spectral theorem: Let A ∈ Cn×n be Hermitian. Then there exists a unitary matrix U ∈ Cn×n
and real numbers λ1 , . . . , λn such that

A = U diag(λ1 , . . . , λn ) U ∗ .

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Spectral Theorem
Definition. Let A ∈ Cn×n . Then λ ∈ C is said to be an eigenvalue of A if there exists a nonzero
vector v ∈ Cn such that
Av = λv .
The vector v is called an eigenvector of A corresponding to the eigenvalue λ.

Spectral theorem: Let A ∈ Cn×n be Hermitian. Then there exists a unitary matrix U ∈ Cn×n
and real numbers λ1 , . . . , λn such that

A = U diag(λ1 , . . . , λn ) U ∗ .
 
If U = u1 · · · un then Auj = λj uj for j = 1 : n. Hence uj is an eigenvector of A
corresponding to the eigenvalue λj for j = 1 : n.

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Spectral Theorem
Definition. Let A ∈ Cn×n . Then λ ∈ C is said to be an eigenvalue of A if there exists a nonzero
vector v ∈ Cn such that
Av = λv .
The vector v is called an eigenvector of A corresponding to the eigenvalue λ.

Spectral theorem: Let A ∈ Cn×n be Hermitian. Then there exists a unitary matrix U ∈ Cn×n
and real numbers λ1 , . . . , λn such that

A = U diag(λ1 , . . . , λn ) U ∗ .
 
If U = u1 · · · un then Auj = λj uj for j = 1 : n. Hence uj is an eigenvector of A
corresponding to the eigenvalue λj for j = 1 : n.

Example: #∗
  " √1 √1
# " 1 √1
1 2 2 2 −1 0 √2 2
= √−1 √1 −1 √1
.
2 1 2 2
0 3 √ 2 2

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Singular Value Decomposition (SVD)
SVD generalizes spectral theorem for Hermitian matrices to the case of m × n matrices.
Theorem: Let A ∈ Cm×n . Then there exist unitary matrices U ∈ Cm×m and V ∈ Cn×n such
that A = UΣV ∗ ,

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Singular Value Decomposition (SVD)
SVD generalizes spectral theorem for Hermitian matrices to the case of m × n matrices.
Theorem: Let A ∈ Cm×n . Then there exist unitary matrices U ∈ Cm×m and V ∈ Cn×n such
that A = UΣV ∗ , where
 
σ1
 ..  
0  Σr 0

Σ=
 . = ∈ Rm×n ,
 σr  0 0
0 0

σ1 ≥ σ2 ≥ · · · ≥ σr > 0 , Σr = diag(σ1 , . . . , σr ) and r = rank(A).

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Singular Value Decomposition (SVD)
SVD generalizes spectral theorem for Hermitian matrices to the case of m × n matrices.
Theorem: Let A ∈ Cm×n . Then there exist unitary matrices U ∈ Cm×m and V ∈ Cn×n such
that A = UΣV ∗ , where
 
σ1
 ..  
0  Σr 0

Σ=
 . = ∈ Rm×n ,
 σr  0 0
0 0

σ1 ≥ σ2 ≥ · · · ≥ σr > 0 , Σr = diag(σ1 , . . . , σr ) and r = rank(A).

   
Trimmed SVD: Let U and V be given by U = u1 · · · um and V = v1 ··· vn . Then

A = Ur Σr Vr∗ = σ1 u1 v1∗ + · · · + σr ur vr∗ ,

   
where Ur := u1 ··· ur and Vr := v1 · · · vr .

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Example

The Matlab commands [U, S, V] = svd(A) and [U, S, V] = svd(A,0) compute full
and trimmed SVD of an m × n matrix A, respectively.

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Example

The Matlab commands [U, S, V] = svd(A) and [U, S, V] = svd(A,0) compute full
and trimmed SVD of an m × n matrix A, respectively.

The SVD of the 2-by-2 Hermitian matrix is given by

 " √1 #  " √1 #∗
 √1 −1
√
1 2 3 0
= √12 √ −1
2 2 2 ,
2 1 2 2
0 1 √12 √1
2

whereas the spectral decomposition is given by

 " √1 # " 1 #∗
√1 √1

1 2 2 2 −1 0 √2 2
= √ −1 √1 −1 √1
.
2 1 2 2
0 3 √ 2 2

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Properties of SVD

Assume for the moment that the SVD A = UΣV ∗ exists. Then AV = UΣ yields

Avj = σj uj , j = 1 : r =⇒ R(A) = span(u1 , . . . , ur )

Avj = 0, j = r + 1 : n =⇒ N(A) = span(vr +1 , . . . , vn )
and A∗ U = V Σ∗ yields

A∗ uj = σj vj , j = 1 : r , =⇒ R(A∗ ) = span(v1 , . . . , vr )
A∗ uj = 0, j = r + 1 : m =⇒ N(A∗ ) = span(ur +1 , . . . , um )

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Properties of SVD

Assume for the moment that the SVD A = UΣV ∗ exists. Then AV = UΣ yields

Avj = σj uj , j = 1 : r =⇒ R(A) = span(u1 , . . . , ur )

Avj = 0, j = r + 1 : n =⇒ N(A) = span(vr +1 , . . . , vn )
and A∗ U = V Σ∗ yields

A∗ uj = σj vj , j = 1 : r , =⇒ R(A∗ ) = span(v1 , . . . , vr )
A∗ uj = 0, j = r + 1 : m =⇒ N(A∗ ) = span(ur +1 , . . . , um )

• σ1 , . . . , σr are called nonzero singular values of A

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Properties of SVD

Assume for the moment that the SVD A = UΣV ∗ exists. Then AV = UΣ yields

Avj = σj uj , j = 1 : r =⇒ R(A) = span(u1 , . . . , ur )

Avj = 0, j = r + 1 : n =⇒ N(A) = span(vr +1 , . . . , vn )
and A∗ U = V Σ∗ yields

A∗ uj = σj vj , j = 1 : r , =⇒ R(A∗ ) = span(v1 , . . . , vr )
A∗ uj = 0, j = r + 1 : m =⇒ N(A∗ ) = span(ur +1 , . . . , um )

• σ1 , . . . , σr are called nonzero singular values of A

• v1 , . . . , vn are called right singular vectors of A

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Properties of SVD

Assume for the moment that the SVD A = UΣV ∗ exists. Then AV = UΣ yields

Avj = σj uj , j = 1 : r =⇒ R(A) = span(u1 , . . . , ur )

Avj = 0, j = r + 1 : n =⇒ N(A) = span(vr +1 , . . . , vn )
and A∗ U = V Σ∗ yields

A∗ uj = σj vj , j = 1 : r , =⇒ R(A∗ ) = span(v1 , . . . , vr )
A∗ uj = 0, j = r + 1 : m =⇒ N(A∗ ) = span(ur +1 , . . . , um )

• σ1 , . . . , σr are called nonzero singular values of A

• v1 , . . . , vn are called right singular vectors of A
• u1 , . . . , um are called left singular vectors of A

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Properties of SVD
We have
A∗ Avj = σj2 vj , j = 1 : r
 2 
∗ Σr 0
A A=V V∗ =⇒
0 0 A∗ Avj = 0, j = r + 1 : n

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Properties of SVD
We have
A∗ Avj = σj2 vj , j = 1 : r
 2 
∗ Σr 0
A A=V V∗ =⇒
0 0 A∗ Avj = 0, j = r + 1 : n
AA∗ uj = σj2 uj , j = 1 : r
 2 
Σ 0 ∗
AA∗ = U r U =⇒
0 0 AA∗ uj = 0, j = r + 1 : m

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Properties of SVD
We have
A∗ Avj = σj2 vj , j = 1 : r
 2 
∗ Σr 0
A A=V V∗ =⇒
0 0 A∗ Avj = 0, j = r + 1 : n
AA∗ uj = σj2 uj , j = 1 : r
 2 
Σ 0 ∗
AA∗ = U r U =⇒
0 0 AA∗ uj = 0, j = r + 1 : m

This shows that

• the right singular vectors v1 , . . . , vn are orthonormal eigenvectors of the positive
semi-definite matrix A∗ A

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Properties of SVD
We have
A∗ Avj = σj2 vj , j = 1 : r
 2 
∗ Σr 0
A A=V V∗ =⇒
0 0 A∗ Avj = 0, j = r + 1 : n
AA∗ uj = σj2 uj , j = 1 : r
 2 
Σ 0 ∗
AA∗ = U r U =⇒
0 0 AA∗ uj = 0, j = r + 1 : m

This shows that

• the right singular vectors v1 , . . . , vn are orthonormal eigenvectors of the positive
semi-definite matrix A∗ A
• the left singular vectors u1 , . . . , um are orthonormal eigenvectors of the positive
semi-definite matrix AA∗

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Properties of SVD
We have
A∗ Avj = σj2 vj , j = 1 : r
 2 
∗ Σr 0
A A=V V∗ =⇒
0 0 A∗ Avj = 0, j = r + 1 : n
AA∗ uj = σj2 uj , j = 1 : r
 2 
Σ 0 ∗
AA∗ = U r U =⇒
0 0 AA∗ uj = 0, j = r + 1 : m

This shows that

• the right singular vectors v1 , . . . , vn are orthonormal eigenvectors of the positive
semi-definite matrix A∗ A
• the left singular vectors u1 , . . . , um are orthonormal eigenvectors of the positive
semi-definite matrix AA∗
• and the nonzero singular values σ1 , . . . , σr are the square roots of the nonzero eigenvalues
of A∗ A (or equivalently of AA∗ ).

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Existence of SVD
Consider the special case when A ∈ Cn×n is nonsingular. Then A∗ A is positive definite and by
spectral theorem A∗ A has positive eigenvalues:

A∗ A = V diag(λ1 , · · · , λn )V ∗ ,

where λ1 ≥ . . . ≥ λn > 0.

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Existence of SVD
Consider the special case when A ∈ Cn×n is nonsingular. Then A∗ A is positive definite and by
spectral theorem A∗ A has positive eigenvalues:

A∗ A = V diag(λ1 , · · · , λn )V ∗ ,

where λ1 ≥ . . . ≥ λn > 0. Indeed, A∗ Avj = λj vj =⇒ λj = vj∗ A∗ Avj > 0 for j = 1 : n.

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Existence of SVD
Consider the special case when A ∈ Cn×n is nonsingular. Then A∗ A is positive definite and by
spectral theorem A∗ A has positive eigenvalues:

A∗ A = V diag(λ1 , · · · , λn )V ∗ ,

where λ1 ≥ . . . ≥ λn > 0. Indeed, A∗ Avj = λj vj =⇒ λj = vj∗ A∗ Avj > 0 for j = 1 : n.

√ √
Define Σ := diag( λ1 , . . . , λn ) and U := AV Σ−1 . Then A = UΣV ∗ .

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Existence of SVD
Consider the special case when A ∈ Cn×n is nonsingular. Then A∗ A is positive definite and by
spectral theorem A∗ A has positive eigenvalues:

A∗ A = V diag(λ1 , · · · , λn )V ∗ ,

where λ1 ≥ . . . ≥ λn > 0. Indeed, A∗ Avj = λj vj =⇒ λj = vj∗ A∗ Avj > 0 for j = 1 : n.

√ √
Define Σ := diag( λ1 , . . . , λn ) and U := AV Σ−1 . Then A = UΣV ∗ . Now
U ∗ U = Σ−1 V ∗ A∗ AV Σ−1 = I =⇒ U is unitary. Thus A = UΣV ∗ is an SVD.

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Existence of SVD
Consider the special case when A ∈ Cn×n is nonsingular. Then A∗ A is positive definite and by
spectral theorem A∗ A has positive eigenvalues:

A∗ A = V diag(λ1 , · · · , λn )V ∗ ,

where λ1 ≥ . . . ≥ λn > 0. Indeed, A∗ Avj = λj vj =⇒ λj = vj∗ A∗ Avj > 0 for j = 1 : n.

√ √
Define Σ := diag( λ1 , . . . , λn ) and U := AV Σ−1 . Then A = UΣV ∗ . Now
U ∗ U = Σ−1 V ∗ A∗ AV Σ−1 = I =⇒ U is unitary. Thus A = UΣV ∗ is an SVD.

Method 1. SVD of an n × n nonsingular matrix A

1 Compute spectral decomposition A∗ A = V diag(λ1 , · · · , λn )V ∗ .

√ √
2 Define Σ := diag( λ1 , . . . , λn ).
3 Compute U := AV Σ−1 . Then A = UΣV ∗ is an SVD of A.

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Existence of SVD
The SVD of an m-by-n matrix A exists and can be computed from spectral decompositions of
A∗ A and AA∗ in four steps:

Method 2. SVD of an m × n matrix A

1 Compute A∗ A = V diag(σ12 , . . . , σr2 , 0, . . . , 0)V ∗ . Set Σr := diag(σ1 , · · · , σr ) and let Vr

denote the first r columns of V .
2 Compute Ur := AVr Σ−1
r , that is, Ur := [Av1 /σ1 , . . . , Avr /σr ], where vj := Vej , j = 1 : r .

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Existence of SVD
The SVD of an m-by-n matrix A exists and can be computed from spectral decompositions of
A∗ A and AA∗ in four steps:

Method 2. SVD of an m × n matrix A

1 Compute A∗ A = V diag(σ12 , . . . , σr2 , 0, . . . , 0)V ∗ . Set Σr := diag(σ1 , · · · , σr ) and let Vr

denote the first r columns of V .
2 Compute Ur := AVr Σ−1
r , that is, Ur := [Av1 /σ1 , . . . , Avr /σr ], where vj := Vej , j = 1 : r .
Then Ur∗ Ur = Σ−1 ∗ ∗
r Vr A AVr Σr
−1
= Σ−1 2 −1
r Σr Σ r = Ir . Hence Ur is an isometry.

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Existence of SVD
The SVD of an m-by-n matrix A exists and can be computed from spectral decompositions of
A∗ A and AA∗ in four steps:

Method 2. SVD of an m × n matrix A

1 Compute A∗ A = V diag(σ12 , . . . , σr2 , 0, . . . , 0)V ∗ . Set Σr := diag(σ1 , · · · , σr ) and let Vr

denote the first r columns of V .
2 Compute Ur := AVr Σ−1
r , that is, Ur := [Av1 /σ1 , . . . , Avr /σr ], where vj := Vej , j = 1 : r .
Then Ur∗ Ur = Σ−1 ∗ ∗
r Vr A AVr Σr
−1
= Σ−1 2 −1
r Σr Σ r = Ir . Hence Ur is an isometry.
3 Compute AA∗ = Z diag(σ12 , . . . , σr2 , 0, . . . , 0)Z ∗ . Let Um−r denote the last m − r columns
of Z . Then R(Um−r ) = N(A∗ ).

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Existence of SVD
The SVD of an m-by-n matrix A exists and can be computed from spectral decompositions of
A∗ A and AA∗ in four steps:

Method 2. SVD of an m × n matrix A

1 Compute A∗ A = V diag(σ12 , . . . , σr2 , 0, . . . , 0)V ∗ . Set Σr := diag(σ1 , · · · , σr ) and let Vr

denote the first r columns of V .
2 Compute Ur := AVr Σ−1
r , that is, Ur := [Av1 /σ1 , . . . , Avr /σr ], where vj := Vej , j = 1 : r .
Then Ur∗ Ur = Σ−1 ∗ ∗
r Vr A AVr Σr
−1
= Σ−1 2 −1
r Σr Σ r = Ir . Hence Ur is an isometry.
3 Compute AA∗ = Z diag(σ12 , . . . , σr2 , 0, . . . , 0)Z ∗ . Let Um−r denote the last m − r columns
of Z . Then R(Um−r ) = N(A∗ ).
 
Σr 0
. Then U is unitary and A = UΣV ∗ is an SVD
 
4 Set U := Ur Um−r and Σ :=
0 0
of A.

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Example
 
1
Let A = . Then A∗ A = [2] and
1
 " √1 # " 1 #∗
√1 √1

∗1 1 2 0 √2
AA = = √12 −1
2
−1
2 .
1 1 2
√
2
0 0 √12 √
2

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Example
 
1
Let A = . Then A∗ A = [2] and
1
 " √1 # " 1 #∗
√1 √1

1
∗ 1 2 0 √2
AA = = √12 −1
2
−1
2 .
1 1 2
√
2
0 0 √12 √
2

  " √1 −1
√
# √ 
1 2
Thus = √12 √1
2 [1]∗ is an SVD of A.
1 2 2
0

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Example
 
1
Let A = . Then A∗ A = [2] and
1
 " √1 # " 1 #∗
√1 √1

1
∗ 1 2 0 √2
AA = = √12 −1
2
−1
2 .
1 1 2
√
2
0 0 √12 √
2

  " √1 −1
√
# √ 
1 2
Thus = √12 √1
2 [1]∗ is an SVD of A.
1 2 2
0

Similarly, " #∗
√1 −2
√
  √  5 5
B := 1 2 = [1] 5 0 √2 √1
5 5

is an SVD of B.

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

SVD in action

The image of a unit circle in R2 under the action of a 2 × 2 nonsingular matrix A. It follows
that the image of a unit circle is an ellipse with semi-major axis σ1 u1 and semi-minor axis σ2 u2 .

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

SVD in action

Let T denote the unit circle in R2 . Then V ∗ (T) is again a unit circle. Now Σ maps the unit
circle V ∗ (T) to the ellipse E := ΣV ∗ (T). Finally, U maps the ellipse E to the ellipse U(E).

***
R. Alam, IIT Guwahati (January-May 2025) MA571 NLA