Scribd
Upload
9 views

Solving Linear Equations(1)

Uploaded by

mrizwan
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
9 views

Solving Linear Equations(1)

Uploaded by

mrizwan
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 14

Fitting lines

Solving Linear Equations Ax = b

Dr Minh Nguyen1

1 Department of Mathematics and Statistics


University of Arkansas at Little Rock
Fitting lines

Outline

1 Fitting lines
Fitting lines

Line fitting problem

A set of data. How to find a straight line that is "best fit" the set?
Fitting lines

Linear equations

Equation of the straight line


Suppose that the line has an equation

y = ax + b.

We will have to determine the best a, b to fit the set of the data.

There are two many (data) equations to determine a and b


 
x1 x2 x3 x4 x5 · · · xN
y1 y2 y3 y4 y5 · · · yN
Fitting lines

Solving equations

Solving equations
By switching a, b to x, y we actually have to solve x, y from the
system

a1 = xb1 + y
a2 = xb2 + y
······
aN = xbN + y .

Usually, when we have more equations than the number of


unknown there will be no solution!
"Optimal solutions"
We will find a solution (x, y ) that minimizes the errors in some
sense.
Fitting lines

Optimal solutions

Some measures of errors


A natural measure is the "total error"

E1 (x, y ) := |a1 −(xb1 +y )|+|a2 −(xb2 +y )|+· · ·+|aN −(xbN +y )|.

Then, we end up with an optimization problem of finding a


vector (x, y )T that minimizes the function E1 (x, y ).

Note that E1 is not differentiable for all x, y .


Least square error
Another measure is the "total error", or least square error

E(x, y ) := |a1 −(xb1 +y )|2 +|a2 −(xb2 +y )|2 +· · ·+|aN −(xbN +y )|2 .

Note that E2 (x, y ) is a quadratic form in x, y , so we can apply


calculus FONC and SONC criteria to find the best x, y .
Fitting lines

Minimizing the quadratic form E(x, y )

N 
X 
E(x, y ) = aJ2 − 2aj (xbj + y ) + (xbj + y )2
j=1
N 
X 
= aJ2 − 2aj bj x − 2aj y + x 2 bj2 + 2bj xy + y 2
j=1
N 
X 
= bj2 x 2 + 2bj xy + y 2 − 2aj bj x − 2aj y + aJ2
j=1
     
N
X XN N
X
= bj2  x 2 + 2  bj  xy + Ny 2 − 2  aj bj  x
j=1 j=1 j=1
   
N
X N
X
− 2 aj  y +  aj2  .
Fitting lines

Minimizing the quadratic form E(x, y )

   
1 x x
E(x, y ) = (x, y )A −B + c,
2 y y

where
 
N N N
" P #
2 N 2 2
PN
b b
j=1 j
X X X
A := Pj=1 j
, B :=  2 a b
j j , 2 a j
 , c = aj2 .
2 N b
j=1 j 2N j=1 j=1 j=1
 
x
Hession(E) = A, ∇E = A − BT .
y
Fitting lines

A General problem

We consider a general problem of solving x from an equation

Ax = b,

where A is a m × n matrix, x is an unknown vector in Rn and b


is a vector in Rm . When the number of equations m is larger
than the number n of unknowns there usually is no solutions at
all. Sufficient conditions from Linear Algebra like b must be in
the range of A must be given to guarantee the existence of an
exact solution.
Therefore, it is reasonable to find a solution x ∗ that minimizes
the "distance" dist(Ax, b). The distance between two vectors in
Rm gives least square criterion to choose x∗ .
Fitting lines

Solutions of linear equations

kAx − bk is minimum if and only if kAx − bk2 is minimum.


kAx − bk2 is a quadratic form of the coordinates of the vector x.
Recall that for x = (x1 , · · · , xn )T ∈ Rn
n
X
kxk2 = xj2 .
j=1

If you use Maple with package Linear Algebra you can compute
x∗ using some command lines. If not, you can use gradient
method with fixed step size small enough.
Fitting lines

A General problem

We consider a general problem of solving x from an equation

Ax = b,

where A is a m × n matrix, x is an unknown vector in Rn and b


is a vector in Rm . When the number of equations m is less than
the number n of unknowns there usually are infinitely solutions.
Finding a solution x with minimum norm is what we want to do.
That means, we want to find x that satisfies Ax = b and that
minimizes kxk.
Fitting lines

Minimum norm solution


Fitting lines

Minimum norm solution


Fitting lines

Maple programs to solve linear equations

In Blackboard two Maple programs are posted to solve linear


equations when the number of equations is larger or less than
the number of unknowns.

You might also like