MA571 Numerical Linear Algebra

Lecture 1: Introduction

Rafikul Alam
Department of Mathematics
IIT Guwahati

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Course syllabus
Floating point computations: IEEE floating point arithmetic, analysis of roundoff errors;
Sensitivity analysis and condition numbers.

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Course syllabus
Floating point computations: IEEE floating point arithmetic, analysis of roundoff errors;
Sensitivity analysis and condition numbers.
Linear systems: LU decompositions, Gaussian elimination with partial pivoting; Banded
systems, positive definite systems, Cholesky decomposition; stability and sensitivity analysis;
Jacobi, Gauss-Seidel and successive overrelaxation methods.

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Course syllabus
Floating point computations: IEEE floating point arithmetic, analysis of roundoff errors;
Sensitivity analysis and condition numbers.
Linear systems: LU decompositions, Gaussian elimination with partial pivoting; Banded
systems, positive definite systems, Cholesky decomposition; stability and sensitivity analysis;
Jacobi, Gauss-Seidel and successive overrelaxation methods.
Orthogonal matrix factorization: Gram-Schmidt orthonormal process, Householder
transformation, Givens rotations; QR factorization, stability of QR factorization.

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Course syllabus
Floating point computations: IEEE floating point arithmetic, analysis of roundoff errors;
Sensitivity analysis and condition numbers.
Linear systems: LU decompositions, Gaussian elimination with partial pivoting; Banded
systems, positive definite systems, Cholesky decomposition; stability and sensitivity analysis;
Jacobi, Gauss-Seidel and successive overrelaxation methods.
Orthogonal matrix factorization: Gram-Schmidt orthonormal process, Householder
transformation, Givens rotations; QR factorization, stability of QR factorization.
Linear least-squares problems: normal equations, QR method for linear least-squares problems,
singular value decomposition(SVD), polar decomposition, Moore-Penrose inverse; Rank
deficient least-squares problems; Sensitivity analysis of least-squares problems.

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Course syllabus
Floating point computations: IEEE floating point arithmetic, analysis of roundoff errors;
Sensitivity analysis and condition numbers.
Linear systems: LU decompositions, Gaussian elimination with partial pivoting; Banded
systems, positive definite systems, Cholesky decomposition; stability and sensitivity analysis;
Jacobi, Gauss-Seidel and successive overrelaxation methods.
Orthogonal matrix factorization: Gram-Schmidt orthonormal process, Householder
transformation, Givens rotations; QR factorization, stability of QR factorization.
Linear least-squares problems: normal equations, QR method for linear least-squares problems,
singular value decomposition(SVD), polar decomposition, Moore-Penrose inverse; Rank
deficient least-squares problems; Sensitivity analysis of least-squares problems.
Eigenvalue problems: Eigenvalues and eigenvectors, Schur decomposition, Sensitivity of
eigenvalues and eigenvectors; Reduction to Hessenberg and tridiagonal forms; QR algorithm,
implementation of implicit QR algorithm; Power, inverse power and Rayleigh quotient
iterations; Reduction of matrices to bidiagonal forms, QR algorithm for SVD.

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Textbooks

• L. N. Trefethen and David Bau, Numerical Linear Algebra, SIAM, Philadelphia,

1997.
• D. S. Watkins, Fundamentals of Matrix Computations, 2nd Edition, Wiley, 2002.

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Textbooks

• L. N. Trefethen and David Bau, Numerical Linear Algebra, SIAM, Philadelphia,

1997.
• D. S. Watkins, Fundamentals of Matrix Computations, 2nd Edition, Wiley, 2002.

References:
• J. W. Demmel, Applied Numerical Linear Algebra, SIAM, 1997.
• M. L. Overton, Numerical Computing with IEEE Floating Point Arithmetic,
SIAM, 2001.

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Assessment and grading policy

Grading will be based on multiple offline continuous assessments.

• Quiz ( 1 before midsem and 1 after midsem): 20 %
• Mid semester examination: 20 %
• End semester examination: 40 %
• Lab examination and assignments: 20 %

Lab teaching assistants: Dr. Prince Kanhya & Mr. Atul Kumar Rai

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

What is Numerical Linear Algebra?

Numerical Linear Algebra (NLA) is a small but core area in Scientific Computing. Its
main focus is the numerical solution of the following problems:

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

What is Numerical Linear Algebra?

Numerical Linear Algebra (NLA) is a small but core area in Scientific Computing. Its
main focus is the numerical solution of the following problems:

• Systems of linear equations: Solve Ax = b

• Best approximation by least squares: Minimize kAx − bk2 .

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

What is Numerical Linear Algebra?

Numerical Linear Algebra (NLA) is a small but core area in Scientific Computing. Its
main focus is the numerical solution of the following problems:

• Systems of linear equations: Solve Ax = b

• Best approximation by least squares: Minimize kAx − bk2 .
• Solution of eigenvalue and singular value problems:

Eigenvalue problem: Av = λv .
Singular value problem: Av = σu and A∗ u = σv .

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Main issues in NLA
• Development and analysis of numerical algorithms
• matrix factorizations and canonical forms
• stability analysis of algorithms
• effects of finite precision arithmetic upon algorithms
• analyzing the speed of an algorithm

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Main issues in NLA
• Development and analysis of numerical algorithms
• matrix factorizations and canonical forms
• stability analysis of algorithms
• effects of finite precision arithmetic upon algorithms
• analyzing the speed of an algorithm
• Perturbation theory
• condition numbers
• sensitivity of solutions to small perturbations in the input data
• accuracy analysis of computed solutions
• Software

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Main issues in NLA
• Development and analysis of numerical algorithms
• matrix factorizations and canonical forms
• stability analysis of algorithms
• effects of finite precision arithmetic upon algorithms
• analyzing the speed of an algorithm
• Perturbation theory
• condition numbers
• sensitivity of solutions to small perturbations in the input data
• accuracy analysis of computed solutions
• Software

Structured matrices (sparse, banded, Toeplitz, Hankel, etc.)

• require structure preserving algorithms
• have huge impact on computation time and accuracy

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Solving Linear systems

• Direct methods for dense linear systems

• Gaussian elimination: A = LU, PA = LU, PAQ = LU.
• The Cholesky factorization: A = GG >

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Solving Linear systems

• Direct methods for dense linear systems

• Gaussian elimination: A = LU, PA = LU, PAQ = LU.
• The Cholesky factorization: A = GG >
• The QR decomposition: A = QR

• Iterative methods for large-scale (sparse) linear systems

• Classical (Jacobi, Gauss-Seidel, SOR) methods
• Conjugate gradient (CG) method
• Multigrid preconditioning
• GMRES

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Solving Least-Squares Problem

Least-Squares Problem:
Solve min kAx − bk2 .
x

Application: Data science, Machine learning, Signal processing, etc.

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Solving Least-Squares Problem

Least-Squares Problem:
Solve min kAx − bk2 .
x

Application: Data science, Machine learning, Signal processing, etc.

• Choice of Methods (in order of increasing cost and accuracy)

• Normal Equations: A∗ Ax = A∗ b
• The QR factorization: A = QR
• Singular Value Decomposition (SVD): A = UΣV ∗

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Solving Least-Squares Problem

Least-Squares Problem:
Solve min kAx − bk2 .
x

Application: Data science, Machine learning, Signal processing, etc.

• Choice of Methods (in order of increasing cost and accuracy)

• Normal Equations: A∗ Ax = A∗ b
• The QR factorization: A = QR
• Singular Value Decomposition (SVD): A = UΣV ∗
• Popular method: QR factorization
• Householder transformation/ Givens rotation preferred choice
• Modified Gram-Schmidt method is OK but avoid Classical Gram-Schmidt method

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Solving Eigenvalue Problem
Standard eigenproblem (SEP): Av = λv

Compute eigendecomposition A = XDX −1

Generalized eigenproblem (GEP): (A − λB)v = 0

Compute eigendecomposition AX = BXD

• Direct method for dense eigenproblem

• The QR algorithm for SEP
• The QZ algorithm for GEP

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Solving Eigenvalue Problem
Standard eigenproblem (SEP): Av = λv

Compute eigendecomposition A = XDX −1

Generalized eigenproblem (GEP): (A − λB)v = 0

Compute eigendecomposition AX = BXD

• Direct method for dense eigenproblem

• The QR algorithm for SEP
• The QZ algorithm for GEP
• Iterative methods for large-scale (sparse) eigenproblem
• Lanczos method for symmetric eigenproblem,
• Arnoldi method (implicit restart) for nonsymmetric eigenvproblem

Never solve det(A − λI ) = 0 for computing eigenvalue of A!

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA
Applications of Eigenvalue Problems

• Google’s Search Engine

• Structural Dynamics
• Electrical Networks
• Combustion Processes
• Macro-economics
• Quantum Chemistry
• Quantum Physics
• Chemical Reactions
• Markov Chain Techniques
• Magnetohydrodynamics
• Control Theory

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Singular Value Decomposition (SVD)

Given an m-by-n matrix A, there exist unitary matrices U and V such that
 
diag(σ1 , · · · , σr ) 0
A=U V ∗ =: UΣV ∗ ,
0 0

where
• σ1 ≥ σ2 ≥ · · · ≥ σr > 0 are singular values and r = rank(A)
• columns of V are right singular vectors: Avj = σj uj
• columns of U are left singular vectors: A∗ uj = σj vj

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA

Singular Value Decomposition (SVD)

Given an m-by-n matrix A, there exist unitary matrices U and V such that
 
diag(σ1 , · · · , σr ) 0
A=U V ∗ =: UΣV ∗ ,
0 0

where
• σ1 ≥ σ2 ≥ · · · ≥ σr > 0 are singular values and r = rank(A)
• columns of V are right singular vectors: Avj = σj uj
• columns of U are left singular vectors: A∗ uj = σj vj

Popular algorithm:
• Golub-Kahan bidiagonal QR algorithm

R. Alam, IIT Guwahati (January-May 2025) MA571 NLA