LECTURE 4

INNER PRODUCT SPACES,

ORTHOGONALITY AND LEAST SQUARES
DUONG MINH BUI, PH.D
LECTURER IN ELECTRICAL POWER ENGINEERING
VIETNAMESE - GERMAN UNIVERSITY, BINH DUONG PROVINCE, VIETNAM
EMAIL: [email protected]
 CONTENTS
• Length and Dot Product in Rn
• Inner Product Spaces
• Angle and Orthogonality in Inner Product Spaces
• Gram–Schmidt Process; QR-Decomposition
• Best Approximation; Least Squares
• Mathematical Modeling Using Least Squares
• Function Approximation; Fourier Series
• Applications to Linear Models

6/1/2023 2
1. LENGTH AND DOT PRODUCT IN RN (1)

6/1/2023 3
1. LENGTH AND DOT PRODUCT IN RN (2)

6/1/2023 4
1. LENGTH AND DOT PRODUCT IN RN (3)

6/1/2023 5
1. LENGTH AND DOT PRODUCT IN RN (4)

6/1/2023 6
 1. LENGTH AND DOT PRODUCT IN RN (5)
1.1 Distance Between Two Vectors in Rn

6/1/2023 7
 1. LENGTH AND DOT PRODUCT IN RN (6)
1.1 Distance Between Two Vectors in Rn (cont.)

6/1/2023 8
 1. LENGTH AND DOT PRODUCT IN RN (7)
1.2 Dot Product and the Angle Between Two Vectors

6/1/2023 9
 1. LENGTH AND DOT PRODUCT IN RN (8)
1.2 Dot Product and the Angle Between Two Vectors (cont.)

6/1/2023 10
 1. LENGTH AND DOT PRODUCT IN RN (9)
1.2 Dot Product and the Angle Between Two Vectors (cont.)
The law of sines:

The law of cosines:

6/1/2023 11
1. LENGTH AND DOT PRODUCT IN RN (10)

6/1/2023 12
1. LENGTH AND DOT PRODUCT IN RN (11)

6/1/2023 13
1. LENGTH AND DOT PRODUCT IN RN (12)

6/1/2023 14
 1. LENGTH AND DOT PRODUCT IN RN (13)
1.2 Dot Product and the Angle Between Two Vectors (cont.)

6/1/2023 15
 1. LENGTH AND DOT PRODUCT IN RN (14)
1.2 Dot Product and the Angle Between Two Vectors (cont.)

Orthogonal

6/1/2023 16
1. LENGTH AND DOT PRODUCT IN RN (15)

6/1/2023 17
1. LENGTH AND DOT PRODUCT IN RN (16)

6/1/2023 18
 1. LENGTH AND DOT PRODUCT IN RN (17)
1.3 The Dot Product and Matrix Multiplication:

6/1/2023 19
EXERCISES (1)

6/1/2023 20
EXERCISES (2)

6/1/2023 21
RELAX!!!

6/1/2023 22
 CONTENTS
• Length and Dot Product in Rn
• Inner Product Spaces
• Angle and Orthogonality in Inner Product Spaces
• Gram–Schmidt Process; QR-Decomposition
• Best Approximation; Least Squares
• Mathematical Modeling Using Least Squares
• Function Approximation; Fourier Series
• Applications to Linear Models

6/1/2023 23
2. INNER PRODUCT SPACES (1)
A function is defined by
different applications

6/1/2023 24
 2. INNER PRODUCT SPACES (2)
• Euclidean dot product:

6/1/2023 25
2. INNER PRODUCT SPACES (3)

6/1/2023 26
2. INNER PRODUCT SPACES (4)

6/1/2023 27
 2. INNER PRODUCT SPACES (5)
• Properties of Inner Products:

• Definitions of Norm, Distance, and Angle:

6/1/2023 28
2. INNER PRODUCT SPACES (6)

6/1/2023 29
 2. INNER PRODUCT SPACES (7)
• The properties of length and distance for general inner product spaces:
For instance, if u and v are vectors in an inner product space, then the following
three properties are true.

6/1/2023 30
2. INNER PRODUCT SPACES (8)

6/1/2023 31
 2. INNER PRODUCT SPACES (9)
• Orthogonal Projections in Inner Product Spaces:
Let u and v be vectors in the plane. If v is nonzero, then u can be orthogonally
projected onto v, as shown in Figure 5.11. This projection is denoted by “projvu”.

If 0, as shown in Figure 5.11(b), then it can be

shown that the orthogonal projection of u onto v is
the same formula.

6/1/2023 32
 2. INNER PRODUCT SPACES (10)
• Orthogonal Projections in Inner Product Spaces (cont.):

6/1/2023 33
 2. INNER PRODUCT SPACES (11)
• Orthogonal Projections in Inner Product Spaces (cont.):

6/1/2023 34
 2. INNER PRODUCT SPACES (12)
• Orthogonal Projections in Inner Product Spaces (cont.):

6/1/2023 35
 2. INNER PRODUCT SPACES (13)
• Orthogonal Projection and Distance:

6/1/2023 36
EXERCISES (3)

6/1/2023 37
EXERCISES (4)

6/1/2023 38
RELAX!!!

6/1/2023 39
 CONTENTS
• Length and Dot Product in Rn
• Inner Product Spaces
• Angle and Orthogonality in Inner Product Spaces
• Gram–Schmidt Process; QR-Decomposition
• Best Approximation; Least Squares
• Mathematical Modeling Using Least Squares
• Function Approximation; Fourier Series
• Applications to Linear Models

6/1/2023 40
 3. ANGLE AND ORTHOGONALITY
IN INNER PRODUCT SPACES (1)
• Cauchy–Schwarz Inequality:

6/1/2023 41
 3. ANGLE AND ORTHOGONALITY
IN INNER PRODUCT SPACES (2)
Cosine
• Angle Between Vectors: function

The “angle” between vectors in a real inner product space.

6/1/2023 42
 3. ANGLE AND ORTHOGONALITY
IN INNER PRODUCT SPACES (3)
• Angle Between Vectors (cont.):

6/1/2023 43
 3. ANGLE AND ORTHOGONALITY
IN INNER PRODUCT SPACES (4)
• Angle Between Vectors (cont.):

6/1/2023 44
 3. ANGLE AND ORTHOGONALITY
IN INNER PRODUCT SPACES (5)
• Orthogonality:

6/1/2023 45
 3. ANGLE AND ORTHOGONALITY
IN INNER PRODUCT SPACES (6)
• Orthogonality (cont.):

6/1/2023 46
3. ANGLE AND ORTHOGONALITY
IN INNER PRODUCT SPACES (7)

6/1/2023 47
EXERCISES (5)

6/1/2023 48
RELAX!!!

6/1/2023 49
 CONTENTS
• Length and Dot Product in Rn
• Inner Product Spaces
• Angle and Orthogonality in Inner Product Spaces
• Gram–Schmidt Process; QR-Decomposition
• Best Approximation; Least Squares
• Mathematical Modeling Using Least Squares
• Function Approximation; Fourier Series
• Applications to Linear Models

6/1/2023 50
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (1)
• Show that a set of vectors is orthogonal and forms an orthonormal basis, and
represent a vector relative to an orthonormal basis;
• Apply the Gram-Schmidt orthonormalization process;
• QR-decomposition;

6/1/2023 51
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (2)
• Orthogonal and Orthonormal Sets:

6/1/2023 52
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (3)
• Orthogonal and Orthonormal Sets (cont.):

6/1/2023 53
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (4)
• Orthogonal and Orthonormal Sets (cont.):

6/1/2023 54
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (5)
• Orthogonal and Orthonormal Sets (cont.):

6/1/2023 55
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (6)
• Orthogonal and Orthonormal Sets (cont.):

Corollary:

6/1/2023 56
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (7)
• Orthogonal and Orthonormal Sets (cont.):

6/1/2023 57
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (8)
• Orthogonal and Orthonormal Sets (cont.):

6/1/2023 58
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (9)
• Orthogonal and Orthonormal Sets (cont.):

6/1/2023 59
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (10)
• Orthogonal and Orthonormal Sets (cont.):

6/1/2023 60
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (11)
• Orthogonal and Orthonormal Sets (cont.):

6/1/2023 61
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (12)
• Coordinates Relative to Orthonormal Bases:

6/1/2023 62
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (13)
• Coordinates Relative to Orthonormal Bases (cont.):

6/1/2023 63
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (14)
• Coordinates Relative to Orthonormal Bases (cont.):

6/1/2023 64
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (15)
• Coordinates Relative to Orthonormal Bases (cont.):

6/1/2023 65
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (16)
• Orthogonal Projections:

6/1/2023 66
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (17)
• Orthogonal Projections (cont.):

6/1/2023 67
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (18)
• Orthogonal Projections (cont.):

6/1/2023 68
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (19)
• Orthogonal Projections (cont.):

6/1/2023 69
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (20)
• Orthogonal Projections (cont.):

subspace

6/1/2023 70
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (21)
• Gram–Schmidt Orthonormalization Process:

6/1/2023 71
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (22)
• Gram–Schmidt Orthonormalization Process (cont.):

6/1/2023 72
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (23)
• Gram–Schmidt Orthonormalization Process (cont.):

6/1/2023 73
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (24)
• Gram–Schmidt Orthonormalization Process (cont.):

6/1/2023 74
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (25)
• Gram–Schmidt Orthonormalization Process (cont.):

6/1/2023 75
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (26)
• Alternative form of the Gram-Schmidt orthonormalization process:

6/1/2023 76
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (27)
• Alternative form of the Gram-Schmidt orthonormalization process (cont.):

6/1/2023 77
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (28)
• Alternative form of the Gram-Schmidt orthonormalization process (cont.):

6/1/2023 78
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (29)
• QR-decomposition:

…
6/1/2023 79
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (30)
• QR-decomposition (cont.):

6/1/2023 80
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (31)
• QR-decomposition (cont.):

6/1/2023 81
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (32)
• QR-decomposition (cont.):

6/1/2023 82
 4. ORTHONORMAL BASES: GRAM–SCHMIDT PROCESS
AND QR-DECOMPOSITION (33)
• QR-decomposition (cont.):

6/1/2023 83
EXERCISES (6)

6/1/2023 84
EXERCISES (7)

6/1/2023 85
EXERCISES (8)

6/1/2023 86
RELAX!!!

6/1/2023 87
 CONTENTS
• Length and Dot Product in Rn
• Inner Product Spaces
• Angle and Orthogonality in Inner Product Spaces
• Gram–Schmidt Process; QR-Decomposition
• Best Approximation; Least Squares
• Mathematical Modeling Using Least Squares
• Function Approximation; Fourier Series
• Applications to Linear Models

6/1/2023 88
 5. BEST APPROXIMATION: LEAST SQUARES (1)
• Define the least squares problem.
• Solve a least squares problem.
• Use least squares for mathematical modeling.

6/1/2023 89
 5. BEST APPROXIMATION: LEAST SQUARES (2)
• Least Squares Regression Line:

6/1/2023 90
 5. BEST APPROXIMATION: LEAST SQUARES (3)
• Least Squares Regression Line (cont.):
We will look for an x that minimizes the norm of the error .

6/1/2023 91
 5. BEST APPROXIMATION: LEAST SQUARES (4)
• Least Squares Regression Line (cont.):

6/1/2023 92
 5. BEST APPROXIMATION: LEAST SQUARES (5)
• Best Approximation:

6/1/2023 93
 5. BEST APPROXIMATION: LEAST SQUARES (6)
• Best Approximation (cont.):

6/1/2023 94
 5. BEST APPROXIMATION: LEAST SQUARES (7)
• Finding Least Squares Solutions:

6/1/2023 95
 5. BEST APPROXIMATION: LEAST SQUARES (8)
• Finding Least Squares Solutions (cont.):

(e1, e2, e3)

6/1/2023 96
 5. BEST APPROXIMATION: LEAST SQUARES (9)
• Finding Least Squares Solutions (cont.):

6/1/2023 97
 5. BEST APPROXIMATION: LEAST SQUARES (10)
• Finding Least Squares Solutions (cont.):

6/1/2023 98
 5. BEST APPROXIMATION: LEAST SQUARES (11)
• Finding Least Squares Solutions (cont.):

6/1/2023 99
 5. BEST APPROXIMATION: LEAST SQUARES (12)
• Conditions for Uniqueness of Least Squares Solutions:

6/1/2023 100
 5. BEST APPROXIMATION: LEAST SQUARES (13)
• Conditions for Uniqueness of Least Squares Solutions (cont.):

17 143
x
95 285
6/1/2023 101
 5. BEST APPROXIMATION: LEAST SQUARES (14)
• More on the Equivalence Theorem:

6/1/2023 102
 5. BEST APPROXIMATION: LEAST SQUARES (15)
• The Role of QR-Decomposition in Least Squares Problems:

6/1/2023 103
 5. BEST APPROXIMATION: LEAST SQUARES (16)
• The Role of QR-Decomposition in Least Squares Problems (cont.):

"

6/1/2023 104
EXERCISES (9)

6/1/2023 105
RELAX!!!

6/1/2023 106
 CONTENTS
• Length and Dot Product in Rn
• Inner Product Spaces
• Angle and Orthogonality in Inner Product Spaces
• Gram–Schmidt Process; QR-Decomposition
• Best Approximation; Least Squares
• Mathematical Modeling Using Least Squares
• Function Approximation; Fourier Series
• Applications to Linear Models

6/1/2023 107
 6. MATHEMATICAL MODELING USING LEAST SQUARES (1)
• Fitting a Curve to Data:

6/1/2023 108
 6. MATHEMATICAL MODELING USING LEAST SQUARES (2)
• Least Squares Fit of a Straight Line:

6/1/2023 109
 6. MATHEMATICAL MODELING USING LEAST SQUARES (3)
• Least Squares Fit of a Straight Line (cont.):
If there are measurement errors in the data, then data points will typically not lie on a line,
and

will be inconsistent.

In this case we look for a least squares approximation to the values of a and b by solving the
normal system

For simplicity, let us assume that the x-coordinates of the data points are not all the same,
so # has linearly independent column vectors and the normal system has the unique solution:

6/1/2023 110
 6. MATHEMATICAL MODELING USING LEAST SQUARES (4)
• Least Squares Fit of a Straight Line (cont.):

6/1/2023 111
 6. MATHEMATICAL MODELING USING LEAST SQUARES (5)
• Least Squares Fit of a Straight Line (cont.):
Since the regression line minimizes the sum of the squares of the data errors, it is
commonly called the least squares line of best fit.

6/1/2023 112
 6. MATHEMATICAL MODELING USING LEAST SQUARES (6)
• Least Squares Fit of a Straight Line (cont.):

6/1/2023 113
 6. MATHEMATICAL MODELING USING LEAST SQUARES (7)
• Least Squares Fit of a Straight Line (cont.):

6/1/2023 114
 6. MATHEMATICAL MODELING USING LEAST SQUARES (8)
• Least Squares Fit of a Polynomial:
Let us attempt to fit a polynomial of fixed degree m:

6/1/2023 115
 6. MATHEMATICAL MODELING USING LEAST SQUARES (9)
• Least Squares Fit of a Polynomial (cont.):

6/1/2023 116
EXERCISES (10)

6/1/2023 117
RELAX!!!

6/1/2023 118
 CONTENTS
• Length and Dot Product in Rn
• Inner Product Spaces
• Angle and Orthogonality in Inner Product Spaces
• Gram–Schmidt Process; QR-Decomposition
• Best Approximation; Least Squares
• Mathematical Modeling Using Least Squares
• Function Approximation; Fourier Series
• Applications to Linear Models

6/1/2023 119
7. FUNCTION APPROXIMATION; FOURIER SERIES (1)
• Least Squares Approximation:

6/1/2023 120
7. FUNCTION APPROXIMATION; FOURIER SERIES (2)
• Least Squares Approximation (cont.):

6/1/2023 121
7. FUNCTION APPROXIMATION; FOURIER SERIES (3)
• Least Squares Approximation (cont.):

6/1/2023 122
7. FUNCTION APPROXIMATION; FOURIER SERIES (4)
• Fourier Series:

6/1/2023 123
7. FUNCTION APPROXIMATION; FOURIER SERIES (5)
• Fourier Series (cont.):

6/1/2023 124
7. FUNCTION APPROXIMATION; FOURIER SERIES (6)
• Fourier Series (cont.):

6/1/2023 125
7. FUNCTION APPROXIMATION; FOURIER SERIES (7)
• Fourier Series (cont.):

6/1/2023 126
7. FUNCTION APPROXIMATION; FOURIER SERIES (8)
• Fourier Series (cont.):

6/1/2023 127
7. FUNCTION APPROXIMATION; FOURIER SERIES (9)
• Fourier Series (cont.):

6/1/2023 128
EXERCISES (11)

6/1/2023 129
RELAX!!!

6/1/2023 130
 CONTENTS
• Length and Dot Product in Rn
• Inner Product Spaces
• Angle and Orthogonality in Inner Product Spaces
• Gram–Schmidt Process; QR-Decomposition
• Best Approximation; Least Squares
• Mathematical Modeling Using Least Squares
• Function Approximation; Fourier Series
• Applications to Linear Models

6/1/2023 131
8. FURTHER EXAMPLES (1)

6/1/2023 132
8. FURTHER EXAMPLES (2)

6/1/2023 133
8. FURTHER EXAMPLES (3)

• Now, considering I to be a function of the variables a0 and

a1, use calculus to determine the values of a0 and a1 that
minimize I. Specifically, by setting the partial derivatives:

6/1/2023 134
8. FURTHER EXAMPLES (4)

6/1/2023 135
8. FURTHER EXAMPLES (5)

6/1/2023 136
8. FURTHER EXAMPLES (6)

6/1/2023 137
8. FURTHER EXAMPLES (7)

6/1/2023 138
8. FURTHER EXAMPLES (8)

(the spanning set)

6/1/2023 139
8. FURTHER EXAMPLES (9)

6/1/2023 140
8. FURTHER EXAMPLES (10)

6/1/2023 141
8. FURTHER EXAMPLES (11)

6/1/2023 142
EXERCISES (12)

77-80

6/1/2023 143
HOMEWORK (1)

6/1/2023 144
HOMEWORK (2)

6/1/2023 145
HOMEWORK (3)

6/1/2023 146
6/1/2023 147
RELAX!!!

6/1/2023 148
VGU pictures (1)

6/1/2023 149
VGU pictures (2)

6/1/2023 150