Copyright Notice

These slides are distributed under the Creative Commons License.

DeepLearning.ai makes these slides available for educational purposes. You may not use or distribute these slides for
commercial purposes. You may make copies of these slides and use or distribute them for educational purposes as long
as you cite DeepLearning.AI as the source of the slides.

For the rest of the details of the license, see https://creativecommons.org/licenses/by-sa/2.0/legalcode

W3 Lesson 1
Vectors and Linear Transformations

Machine Learning motivation

Neural Networks - AI generated images

AI-generated human faces.

● Generative learning: Generating realistic looking images.

Text-to-image and image-to-text generation
Vectors and Linear Transformations

Vectors and their properties

Vectors

4 (4,3)

(4,3,1)
3 3
4
1
How to get from point A to point B?

Taxicab distance =3+4=7

3

4
How to get from point A to point B?

Taxicab distance =3+4=7

3

4
How to get from point A to point B?

2
b
2 +
a b
Helicopter distance = 32 + 42 = 5
3 a
Pythagorean Theorem

4
Norms
(a,b)

L1-norm = ∥(a, b)∥1 = | a | + | b | L2-norm = ∥(a, b)∥2 = a2 + b2

Norm of a vector

4 (4,3)

3
42 + 32 = 25 = 5
Direction of a vector

(4,3)
(2, 1.5) 3
tan(θ) =
3 4
θ θ = arctan(3/4) = 0.64 = 36.87∘
4
Vectors and Linear Transformations

Sum and difference of

vectors
Sum of vectors

(5,4)
(1,3) v
v u+
u (4,1) u + v = (4 + 1,1 + 3) = (5,4)
Difference of vectors

(1,3)

v
u (4,1) u − v = (4 − 1,1 − 3) = (3, − 2)

(3,-2)
General definition: sum and difference
Same number
of components
x = (x1 x2 … xn) y = (y1 y2 … yn)

Sum Difference

x + y = (x1 + y1 x2 + y2 … xn + yn) x − y = (x1 − y1 x2 − y2 … xn − yn)

Sum component by component Subtract component by component

Vectors and Linear Transformations

Distance between vectors

Distances
(1,5)
u−
v= ∥u − v∥1 = | 5 | + | − 3 | = 8
(5, L1-distance
v −3
)
u (6,2)
∥u − v∥2 = 52 + 32 = 5.83
L2-distance
Distances
(1,5)
u−
v= ∥u − v∥1 = | 5 | + | − 3 | = 8
(5, L1-distance
v −3
)
θ u (6,2)
∥u − v∥2 = 52 + 32 = 5.83
L2-distance

cos(θ)
Cosine distance
Vectors and Linear Transformations

Multiplying a vector by a
scalar
Multiplying a vector by a scalar
3u (3,6)

u = (1,2)
λ=3
u (1,2)
λu = (3,6)
If the scalar is negative

u = (1,2)
λ =−2
u (1,2)
λu = (−2, − 4)

−2u
(-2,-4)
General definition

Multiplication by a Scalar

x = (x1 x2 … xn)

λx = (λx1 λx2 … λxn)

Video 6: The dot product
Vectors and Linear Transformations

The dot product

A shortcut for linear operations
Quantities Prices
2 apples apples: $3 Total price
4 bananas bananas: $5
$28
1 cherry cherries: $2

2 x $ 3 = 2×3=6
4 x $ 5 = 4 × 5 = 20
1 x $ 2 = 1×2=2

6 + 20 + 2 = 28
The dot product

2 $ 3
. = $28
4 $ 5
1 $ 2
The dot product

2 $ 3
. = $28
4 $ 5
1 $ 2

(2 × 3) + (4 × 5) + (1 × 2) = 28
The dot product

$ 3
. = $28
28
2 4 1 $ 5
$ 2

(2 × 3) + (4 × 5) + (1 × 2) = 28
The dot product

3
. = 28
2 4 1 5
2

(2 × 3) + (4 × 5) + (1 × 2) = 28
Norm of a vector using dot product

42 + 32 = 25 = 5
4 (4,3)

u 3 4 3
4
= 25
3

L2 − norm = dot0product(u, u)

∥u∥2 = ⟨u, u⟩
Vector Transpose

2 3
4 . 5 = 28
1 2

(2 × 3) + (4 × 5) + (1 × 2) = 28
Vector Transpose
Transpose: convert columns to rows

2 3 3
4 . 5 = 28 2 4 1 . 5 = 28

1 2 2

2 ⋅ 3 + 4 ⋅ 5 + 1 ⋅ 2 = 28 2 ⋅ 3 + 4 ⋅ 5 + 1 ⋅ 2 = 28
Vector Transpose

T
2
4 =
1
Vector Transpose

T
2
4 = 2 4 1
1
Vector Transpose

T
2 T
4 = 2 4 1 2 4 1 =
1
Vector Transpose

T
2 T 2
4 = 2 4 1 2 4 1 = 4
1 1
Matrix Transpose

T
2 5
4 7 = Transpose the columns!
1 3
Matrix Transpose

T
2 5
2 4 1
4 7 =
1 3
Matrix Transpose

T
2 5
2 4 1
4 7 = Columns → Rows
5 7 3
1 3

3×2 3×2
General definition: dot product
Same number
of components

x = (x1 x2 … xn) y = (y1 y2 … yn)

x ⋅ y = (x1 × y1) + (x2 × y2) + … + (xn × yn)

⟨x, y⟩ is another notation for the dot product

y1
y
x ⋅ y T = (x1 x2 … xn) ⋅ 2
⋮
yn
Vectors and Linear Transformations

Geometric dot product

Orthogonal vectors have dot product 0

(-1,3)
(6,2) -1
= 0
u v 6 2
3

⟨u, v⟩ = 0
The dot product

u ⟨u, u⟩ = ∥u∥2

v u ⟨u, v⟩ = 0

v u
⟨u, v⟩ = ?
 The dot product

u ⟨u, u⟩ = ∥u∥2 = ∥u∥ ⋅ ∥u∥

v ⟨u, v⟩ = ∥u∥ ⋅ ∥v∥

u

u v
θ ⟨u, v⟩ = ∥u′∥ ⋅ ∥v∥

u′ = ∥u∥∥v∥ cos(θ)


Geometric dot product

2
(2,4) 6 2 = 20 Positive
4
(-1,3)

(6,2)
-1
(-4,1) 6 2 = 0
+ 3
-

-4
6 2 = -22 Negative
1
 Geometric dot product
Dot product with u > 0

⟨u, v⟩ > 0

u
⟨u, v⟩ = 0

⟨u, v⟩ < 0

Dot product with u < 0

Dot product with u = 0

Vectors and Linear Transformations

Multiplying a matrix by a
vector
Equations as dot product
2a + 4b + c = 28

$ a
. = $
2 4 1 $ b 28
$ c
 Equations as dot product
a + b + c = 10 a + 2b + c = 15 a + b + 2c = 12

$ a $ a $ a
. . .
1 1 1 $ b = $ 10 1 2 1 $ b = $ 15 1 1 2 $ b = $ 12
$ c $ c $ c
Equations as dot product
$ a
.
1 1 1 $ b = $ 10
$ c
a + b + c = 10
$ a
a + 2b + c = 15 .
1 2 1 $ b = $ 15
$ c
a + b + 2c = 12

$ a
.
1 1 2 $ b = $ 12
$ c
Equations as dot product
$ a
.
1 1 1 $ b = $ 10
$ c
a + b + c = 10
$ a
a + 2b + c = 15 .
1 2 1 $ b = $ 15
$ c
a + b + 2c = 12

$ a
.
1 1 2 $ b = $ 12
$ c
Equations as dot product
System of equations Matrix product

a + b + c = 10
1 1 1 $ a 10
a + 2b + c = 15 .
1 2 1 $ b = $ 15
1 1 2 $ c 12
a + b + 2c = 12
Equations as dot product
System of equations Matrix product

a + b + c = 10
1 1 1 a 10
a + 2b + c = 15 1 2 1 = 15
b
1 1 2 c 12
a + b + 2c = 12
Equations as dot product
Matrix product

!
1 1 1 a 10
1 2 1 b = 15
1 1 2 c 12

3×3 Length: 3
# columns = length of vector

Matrix can be rectangular

Equations as dot product
Matrix product

!
1 1 1 a 10
1 2 1 b = 15
1 1 2 c 12
2 1 1 13
# columns = length of vector
4×3 Length: 3 Length: 4

Matrix can be rectangular

W3 Lesson 2
Vectors and Linear Transformations

Matrices as linear
transformations
Matrices as linear transformations
b
3 1 0
1 0
3
1
4
=
1 2 0
1 0
1
2
3

(0,0) (0,0)
a (1,0) (3,1)
(0,1) (1,2)
(1,1) (4,3)
Matrices as linear transformations
b
3 1 -2 -3
=
1 2 3 4

a
Systems of equations as linear transformations
b Second day
3 1 a
1 Day
4 1
=
1 2 b
1 Day
3 2

First day: 3a + b
(4,3)
Second day: a + 2b

(1,1)
a First day
(1,1)
(a,b) (4,3)
Vectors and Linear Transformations

Linear transformations as
matrices
Linear transformations as matrices
b b
? ?
? ?

a a
(0,0) (0,0)
(1,0) (3,-1)
(0,1) (2,3)
(1,1) (5,2)
Linear transformations as matrices
b
? ? 0 2
=
? ? 1 3
2
0 ? ? 1 3 3
=
1 ? ? 0 -1
a
1 (1,0) (3,-1) 3
0 (0,1) (2,3) -1
Vectors and Linear Transformations

Matrix multiplication
Combining linear transformations

3 1 1 3
=
1 2 0 1

3 1 0 1
=
1 2 1 2
Combining linear transformations

3 1 1 3
=
1 2 0 1

3 1 0 1
=
1 2 1 2
Combining linear transformations

2 -1 3 5
=
0 2 1 2

2 -1 1 0
=
0 2 2 4
Combining linear transformations

2 -1 3 5
=
0 2 1 2

2 -1 1 0
=
0 2 2 4
Combining linear transformations
3 1 2 -1
1 2 0 2

? ?
? ?
Combining linear transformations
3 1 2 -1
1 2 0 2

0
4
5
0 2
1
1
0

?
5 ?0
?
2 ?4
Combining linear transformations

Second First

2 -1 3 1 5 0
. =
0 2 1 2 2 4
Dimension of the matrices

2 -1 3 1
2 5
-1
3
1
2 0
-1
1
2
. =
0 2 1 2

2 4
3 1
0 2 0 2
1 2
Dimension of the matrices

2 -1 3 1
2 5
-1
3
1
2 0
-1
1
2
. =
0 2 1 2

2 4
3 1
2×2 2×2 0 2
1
0 2
2

2×2
 Dimension of the matrices

3 0 1 -2
3 1 4
. 1 5 2 0 =
2 -1 2

?
-2 1 4 0
2×3
3×4

2×4
 Dimension of the matrices

3 0 1 -2
2 9 21 -6
3 1 4
. 1 5 2 0 =
2 -1 2

1
? -3 8 -4
-2 1 4 0
2×3
3×4

2×4
• Columns of first matrix must match rows of second (numbers in red match)
• Result takes number of rows from first matrix (numbers in blue match)
• Result takes number of columns from second matrix (numbers in purple match)
Vectors and Linear Transformations

The identity matrix

The identity matrix
a
b
1 0 0 0 0 c = a
d
e

1 0 0 0 0 a a
0 1 0 0 0 b b
0 0 1 0 0 c = c
0 0 0 1 0 d d
0 0 0 0 1 e e
The identity matrix
b
?
1 ?
0 x x
=
?
0 ?
1 y y

(0,0) (0,0)
a (1,0) (1,0)
(0,1) (0,1)
(1,1) (1,1)
Vectors and Linear Transformations

Matrix inverse
Matrix inverses

3 1 ? ?
1 2 ? ?

1 0
0 1
Multiplying matrices

3 1 a b 1 0
. =
1 2 c d 0 1

-1
3 1 2/5 -1/5
=
1 2 -1/5 3/5
How to find an inverse?
3 1 a b 1 0
. =
1 2 c d 0 1

a 2
3 1
c
= 1 3a + 1c = 1 a=
5
b 1
3 1 = 0 3b + 1d = 0 b=−
d 5
a 1
1 2 = 0 1a + 2c = 0 c=−
c 5
b 3
1 2 = 1 1b + 2d = 1 d=
d 5
Quiz
● Find the inverse of the following matrix. If you find that the task is
impossible, feel free to click on “I couldn’t find it”
5 2
1 2
Solution
● By solving the corresponding system of linear equations, we get the
following. 5 2 a b 1 0
. =
1 2 c d 0 1

a
5 2 = 1 • 5a + 2c = 1 • a = 1/4
c
b
5 2 = 0 • 5b + 2d = 0 • b = -1/4
d
a
1 2 = 0 • a + 2c = 0 • c = -1/8
c
b
1 2 = 1 • b + 2d = 1 • d = 5/8
d
Quiz
● Find the inverse of the following matrix. If you find that the task is
impossible, feel free to click on “I’m reaching a dead end”
1 1
2 2
Solutions
● The inverse doesn’t exist!

We need to solve the following system of linear equations:

1 1 a b 1 0
=
2 2 c d 0 1

a+c=1
2b + 2d = 1
2a + 2c = 0
b+d=0
This is clearly a contradiction, since equation 1 says a+c=1, and equation 3 says
2a+2c=0.
Vectors and Linear Transformations

Which matrices have an

inverse?
Which matrices have inverses?

5−1 = 0.2 8−1 = 0.125 0−1 = ???

-1 -1
3 1 0.4 -0.2 5 2 0.25 -0.25 1 1 ? ?
= = =
1 2 -0.2 0.6 1 2 -0.125 0.625 2 2 ? ?

Non-singular matrix Non-singular matrix Singular matrix

Invertible Invertible Non-invertible

Det = 5 Det = 8 Det = 0

Non-zero determinants Zero determinant

Vectors and Linear Transformations

Neural networks and

matrices
Quiz: Natural language processing
Spam Lottery Win Scores: Examples
Yes 1 1 Lottery: ___ points Lottery: 3 point
Win: 2 points
Yes 2 1 “Win, win the lottery!” : 7points
Win: ___ points
No 0 0
Rule:
Yes 0 2
If the number of points of the sentence is bigger than ____,
No 0 1 then the email is spam.
No 1 0
Yes 2 2 Goal: Find the best points and threshold
Yes 2 0 Lottery: ___ point
Win: ___ point
Yes 1 2 Threshold: ___ points
Quiz: Natural language processing
Spam Lottery Win Score > 1.5? Solution:
Lottery: 1 point
Yes 1 1 2 Yes Win: 1 point
Yes 2 1 3 Yes Threshold: 1.5 points

No 0 0 0 No
Yes 0 2 2 Yes
No 0 1 1 No
No 1 0 1 No
Yes 2 2 4 Yes
Yes 2 0 2 Yes
Yes 1 2 3 Yes
Graphical natural language processing
Spam Lottery Win Win
Yes 1 1 Positive zone:
Yes 2 1 Line: 1 ⋅ Win + 1 ⋅ Lottery > 1.5
1 ⋅ Win + 1 ⋅ Lottery = 1.5
No 0 0
Yes 0 2
No 0 1
No 1 0
Yes 2 2
Yes 2 0 Negative zone:
Lottery
Yes 1 2 1 ⋅ Win + 1 ⋅ Lottery < 1.5
Graphical natural language processing
Spam Lottery Win
Yes 1 1
Yes 2 1
Model Check: > 1.5?
No 0 0
Yes 0 2 1
= 3
No 0 1 1
Spam
No 1 0
Yes 2 2
Yes 2 0
Yes 1 2
Dot product between vectors
Spam Lottery Win
Yes 1 1
Yes 2 1
Model Check: > 1.5?
No 0 0
Yes 0 2 1
= 1
No 0 1 1
Not spam
No 1 0
Yes 2 2
Yes 2 0
Yes 1 2
Matrix multiplication
Spam Lottery Win Prod Check
Yes 1 1 2 Yes
Yes 2 1 3 Yes
Check: >1.5?
No 0 0 Model 0 No
Yes 0 2 1 = 2 Yes
No 0 1 1 1 No
No 1 0 1 No
Yes 2 2 4 Yes
Yes 2 0 2 Yes
Yes 1 2 3 Yes
Perceptrons
Spam Word1 Word2 … WordN Prod Check
Yes Yes
Yes Model Yes
Check:
No No
Yes Yes
=
No … No
No No
Yes Yes
Yes Yes
Yes Yes
Threshold and bias
Threshold
Check
Spam Lottery Win Bias 1 ⋅ Win + 1 ⋅ Lottery −1.5 >0
1 ⋅ Win + 1 ⋅ Lottery −1.5 > 0 Bias
Yes 1 1 1
Yes 2 1 1
Model Check: > 0?
1.5?
No 0 0 1
Yes 0 2 1 1

No 0 1 1 1

No 1 0 1 -1.5

Yes 2 2 1
Yes 2 0 1 Bias

Yes 1 2 1
The AND operator

AND x y Dot prod Check

Check: >1.5?
No 0 0 Model 0 No
No 1 0 1 = 1 No
No 0 1 1 1 No
Yes 1 1 2 Yes
The AND operator

AND x y
No 0 0
No 1 0
No 0 1
Yes 1 1
x

1 ⋅ x + 1 ⋅ y = 1.5
The perceptron

x 1
Yes

1 ⋅ x + 1 ⋅ y −1.5
1
No
y

-1.5
Vectors and Linear Transformations

Conclusion