Downloaded 07/17/24 to 103.38.69.106 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.

org/terms-privacy

Algebra and
Optimization
Numerical Linear
 Books in the Classics in Applied Mathematics series are monographs and textbooks declared out
of print by their original publishers, though they are of continued importance and interest to the
Downloaded 07/17/24 to 103.38.69.106 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

mathematical community. SIAM publishes this series to ensure that the information presented in
these texts is not lost to today’s students and researchers.

Editor-in-Chief
Susanne C. Brenner, Louisiana State University
Editorial Board
John Boyd, University of Michigan Sven Hammarling, University of Manchester
Bernard Deconinck, University of Washington Nicholas J. Higham, University of Manchester
Irene Fonseca, Carnegie Mellon University Peter Olver, University of Minnesota
Vivette Girault, Laboratoire Jacques-Louis Lions Philip Protter, Columbia University
(LJLL) Frederic Wan, University of California

Classics in Applied Mathematics

C. C. Lin and L. A. Segel, Mathematics Applied to Deterministic Problems in the Natural Sciences
Johan G. F. Belinfante and Bernard Kolman, A Survey of Lie Groups and Lie Algebras with Applications
and Computational Methods
James M. Ortega, Numerical Analysis: A Second Course
Anthony V. Fiacco and Garth P. McCormick, Nonlinear Programming: Sequential Unconstrained
Minimization Techniques
F. H. Clarke, Optimization and Nonsmooth Analysis
George F. Carrier and Carl E. Pearson, Ordinary Differential Equations
Leo Breiman, Probability
R. Bellman and G. M. Wing, An Introduction to Invariant Imbedding
Abraham Berman and Robert J. Plemmons, Nonnegative Matrices in the Mathematical Sciences
Olvi L. Mangasarian, Nonlinear Programming
*Carl Friedrich Gauss, Theory of the Combination of Observations Least Subject to Errors: Part One,
Part Two, Supplement. Translated by G. W. Stewart
U. M. Ascher, R. M. M. Mattheij, and R. D. Russell, Numerical Solution of Boundary Value Problems for
Ordinary Differential Equations
K. E. Brenan, S. L. Campbell, and L. R. Petzold, Numerical Solution of Initial-Value Problems
in Differential-Algebraic Equations
Charles L. Lawson and Richard J. Hanson, Solving Least Squares Problems
J. E. Dennis, Jr. and Robert B. Schnabel, Numerical Methods for Unconstrained Optimization and
Nonlinear Equations
Richard E. Barlow and Frank Proschan, Mathematical Theory of Reliability
Cornelius Lanczos, Linear Differential Operators
Richard Bellman, Introduction to Matrix Analysis, Second Edition
Beresford N. Parlett, The Symmetric Eigenvalue Problem
Richard Haberman, Mathematical Models: Mechanical Vibrations, Population Dynamics, and Traffic Flow
Peter W. M. John, Statistical Design and Analysis of Experiments
Tamer Başar and Geert Jan Olsder, Dynamic Noncooperative Game Theory, Second Edition
Emanuel Parzen, Stochastic Processes
Petar Kokotović, Hassan K. Khalil, and John O’Reilly, Singular Perturbation Methods in Control: Analysis
and Design
Jean Dickinson Gibbons, Ingram Olkin, and Milton Sobel, Selecting and Ordering Populations: A New
Statistical Methodology
James A. Murdock, Perturbations: Theory and Methods

*First time in print.

 Classics in Applied Mathematics (continued)
Downloaded 07/17/24 to 103.38.69.106 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

Ivar Ekeland and Roger Témam, Convex Analysis and Variational Problems
Ivar Stakgold, Boundary Value Problems of Mathematical Physics, Volumes I and II
J. M. Ortega and W. C. Rheinboldt, Iterative Solution of Nonlinear Equations in Several Variables
David Kinderlehrer and Guido Stampacchia, An Introduction to Variational Inequalities and Their
Applications
F. Natterer, The Mathematics of Computerized Tomography
Avinash C. Kak and Malcolm Slaney, Principles of Computerized Tomographic Imaging
R. Wong, Asymptotic Approximations of Integrals
O. Axelsson and V. A. Barker, Finite Element Solution of Boundary Value Problems: Theory and Computation
David R. Brillinger, Time Series: Data Analysis and Theory
Joel N. Franklin, Methods of Mathematical Economics: Linear and Nonlinear Programming, Fixed-Point Theorems
Philip Hartman, Ordinary Differential Equations, Second Edition
Michael D. Intriligator, Mathematical Optimization and Economic Theory
Philippe G. Ciarlet, The Finite Element Method for Elliptic Problems
Jane K. Cullum and Ralph A. Willoughby, Lanczos Algorithms for Large Symmetric Eigenvalue
Computations, Vol. I: Theory
M. Vidyasagar, Nonlinear Systems Analysis, Second Edition
Robert Mattheij and Jaap Molenaar, Ordinary Differential Equations in Theory and Practice
Shanti S. Gupta and S. Panchapakesan, Multiple Decision Procedures: Theory and Methodology
of Selecting and Ranking Populations
Eugene L. Allgower and Kurt Georg, Introduction to Numerical Continuation Methods
Leah Edelstein-Keshet, Mathematical Models in Biology
Heinz-Otto Kreiss and Jens Lorenz, Initial-Boundary Value Problems and the Navier-Stokes Equations
J. L. Hodges, Jr. and E. L. Lehmann, Basic Concepts of Probability and Statistics, Second Edition
George F. Carrier, Max Krook, and Carl E. Pearson, Functions of a Complex Variable: Theory and Technique
Friedrich Pukelsheim, Optimal Design of Experiments
Israel Gohberg, Peter Lancaster, and Leiba Rodman, Invariant Subspaces of Matrices with Applications
Lee A. Segel with G. H. Handelman, Mathematics Applied to Continuum Mechanics
Rajendra Bhatia, Perturbation Bounds for Matrix Eigenvalues
Barry C. Arnold, N. Balakrishnan, and H. N. Nagaraja, A First Course in Order Statistics
Charles A. Desoer and M. Vidyasagar, Feedback Systems: Input-Output Properties
Stephen L. Campbell and Carl D. Meyer, Generalized Inverses of Linear Transformations
Alexander Morgan, Solving Polynomial Systems Using Continuation for Engineering and Scientific Problems
I. Gohberg, P. Lancaster, and L. Rodman, Matrix Polynomials
Galen R. Shorack and Jon A. Wellner, Empirical Processes with Applications to Statistics
Richard W. Cottle, Jong-Shi Pang, and Richard E. Stone, The Linear Complementarity Problem
Rabi N. Bhattacharya and Edward C. Waymire, Stochastic Processes with Applications
Robert J. Adler, The Geometry of Random Fields
Mordecai Avriel, Walter E. Diewert, Siegfried Schaible, and Israel Zang, Generalized Concavity
Rabi N. Bhattacharya and R. Ranga Rao, Normal Approximation and Asymptotic Expansions
Françoise Chatelin, Spectral Approximation of Linear Operators
Yousef Saad, Numerical Methods for Large Eigenvalue Problems, Revised Edition
Achi Brandt and Oren E. Livne, Multigrid Techniques: 1984 Guide with Applications to Fluid Dynamics,
Revised Edition
Bernd Fischer, Polynomial Based Iteration Methods for Symmetric Linear Systems

(continued)
 Classics in Applied Mathematics (continued)
Downloaded 07/17/24 to 103.38.69.106 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

Pierre Grisvard, Elliptic Problems in Nonsmooth Domains

E. J. Hannan and Manfred Deistler, The Statistical Theory of Linear Systems
Françoise Chatelin, Eigenvalues of Matrices, Revised Edition
David Colton and Rainer Kress, Integral Equation Methods in Scattering Theory
Bruce A. Finlayson, The Method of Weighted Residuals and Variational Principles
Roger W. Brockett, Finite Dimensional Linear Systems
P. R. Kumar and Pravin Varaiya, Stochastic Systems: Estimation, Identification, and Adaptive Control
Cornelius Lanczos, Discourse on Fourier Series
S. D. Conte and Carl de Boor, Elementary Numerical Analysis: An Algorithmic Approach
Frederic Y. M. Wan, Mathematical Models and Their Analysis
Michael T. Heath, Scientific Computing: An Introductory Survey, Revised Second Edition
Philip E. Gill, Walter Murray, and Margaret H. Wright, Practical Optimization
N. I. Akhiezer, The Classical Moment Problem and Some Related Questions in Analysis
Philip E. Gill, Walter Murray, and Margaret H. Wright, Numerical Linear Algebra and Optimization
Downloaded 07/17/24 to 103.38.69.106 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

Numerical Linear
Algebra and
Optimization
Philip E. Gill
University of California, San Diego
San Diego, California

Walter Murray
Stanford University
Stanford, California

Margaret H. Wright
New York University
New York City, New York

Society for Industrial and Applied Mathematics

Philadelphia
Downloaded 07/17/24 to 103.38.69.106 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

Copyright © 2021 by the Society for Industrial and Applied Mathematics

This SIAM edition is a republication of the edition published by Addison-Wesley Publishing Company
in 1991.
10 9 8 7 6 5 4 3 2 1
All rights reserved. Printed in the United States of America. No part of this book may be reproduced,
stored, or transmitted in any manner without the written permission of the publisher. For information,
write to the Society for Industrial and Applied Mathematics, 3600 Market Street, 6th Floor,
Philadelphia, PA 19104-2688 USA.
No warranties, express or implied, are made by the publisher, authors, and their employers that the
programs contained in this volume are free of error. They should not be relied on as the sole basis to
solve a problem whose incorrect solution could result in injury to person or property. If the programs
are employed in such a manner, it is at the user’s own risk and the publisher, authors, and their
employers disclaim all liability for such misuse.
Trademarked names may be used in this book without the inclusion of a trademark symbol. These
names are used in an editorial context only; no infringement of trademark is intended.
MATLAB is a registered trademark of The MathWorks, Inc. For MATLAB product information,
please contact The MathWorks, Inc., 3 Apple Hill Drive, Natick, MA 01760-2098 USA,
508-647-7000, Fax: 508-647-7001, [email protected], www.mathworks.com.

Publications Director Kivmars H. Bowling

Executive Editor Elizabeth Greenspan
Acquisitions Editor Paula Callaghan
Developmental Editor Mellisa Pascale
Managing Editor Kelly Thomas
Production Editor Ann Manning Allen
Production Manager Donna Witzleben
Production Coordinator Cally A. Shrader
Graphic Designer Doug Smock

Library of Congress Cataloging-in-Publication Data

Names: Gill, Philip E., author. | Murray, Walter, author. | Wright,
Margaret H., author.
Title: Numerical linear algebra and optimization / Philip E. Gill, Walter
Murray, Margaret H. Wright.
Description: Philadelphia : Society for Industrial and Applied Mathematics,
[2021] | “This SIAM edition is a republication of the edition published
by Addison-Wesley Publishing Company in 1991.” | Includes
bibliographical references and index. | Summary: “This book provides a
unified introduction to the fundamentals of numerical analysis and
scientific computing, techniques for solving linear systems and linear
least-square problems, and numerical optimization methods for both
linear and nonlinear programming”-- Provided by publisher.
Identifiers: LCCN 2021007265 (print) | LCCN 2021007266 (ebook) | ISBN
9781611976564 (paperback) | ISBN 9781611976571 (ebook)
Subjects: LCSH: Algebras, Linear. | Numerical calculations. | Mathematical
optimization.
Classification: LCC QA184 .G53 2021 (print) | LCC QA184 (ebook) | DDC
512/.5--dc23
LC record available at https://lccn.loc.gov/2021007265
LC ebook record available at https://lccn.loc.gov/2021007266

is a registered trademark.
Downloaded 07/17/24 to 103.38.69.106 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

colleague, friend, inspiration

To James Hardy Wilkinson:
Downloaded 07/17/24 to 103.38.69.106 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy
 Downloaded 07/17/24 to 103.38.69.106 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

PREFACE

ix
PREFACE TO THE CLASSICS EDITION
xix
xvii
 Downloaded 07/17/24 to 103.38.69.106 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

x
 Downloaded 07/17/24 to 103.38.69.106 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

xi
 Downloaded 07/17/24 to 103.38.69.106 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

xii
 Downloaded 07/17/24 to 103.38.69.106 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

xiii
 Downloaded 07/17/24 to 103.38.69.106 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

xiv
 Downloaded 07/17/24 to 103.38.69.106 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

xv
Downloaded 07/17/24 to 103.38.69.106 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy
Downloaded 07/17/24 to 103.38.69.106 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

Preface to the Classics Edition

Methods for solving problems in science, engineering, medicine, and business rely
on results and algorithms from linear algebra and optimization—two areas that are
very close scientifically. When we wrote this book in 1990, textbooks that treated
them together were rare. To try to improve that situation, Numerical Linear
Algebra and Optimization was structured to present, from a unified perspective,
the major overlapping content, namely linear systems, linear least-squares, and
linear programming (optimization of a linear objective function subject to linear
constraints). Our motivation throughout was to emphasize the ties between linear
algebra and optimization as much as possible.
The separate fields of optimization and linear algebra have grown enormously
during the intervening decades, but, unfortunately, the lack of pedagogical
connections in textbooks remains striking today, 30 years later. Hence there is a
benefit from republishing, including a further benefit because the book features
material that is difficult to find elsewhere, specifically detailed analyses of LU
updating and of the consequences when using an explicit inverse to solve a linear
system. The latter is especially important today, when many elementary courses
on machine learning are teaching that linear systems should be solved using an
inverse.

xvii
Downloaded 07/17/24 to 103.38.69.106 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy
 Downloaded 07/17/24 to 103.38.69.106 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

xix
 Downloaded 07/17/24 to 103.38.69.106 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

xx
 Downloaded 07/17/24 to 103.38.69.106 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

xxi
 Downloaded 07/17/24 to 103.38.69.106 . Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/terms-privacy

xxii