Matrix Norms

Numerical Mathematics
25th March 2020

In these exercises we will consider the matrix norms. First, we recall the definition of a norm
Definition 1. Let V be a linear space over the field R of real numbers. Then, the nonnegative
real-valued function k·k is said to be a norm on the space V if it satisfies the following conditions:
1. kvk = 0 if and only if v = 0 in V,

2. kλvk = |λ|kvk for all λ ∈ R and v ∈ V,

3. ku + vk ≤ kuk + kvk for all u, v ∈ V (the triangle inequality).
Remark 2. The above definition also holds for a linear space V over the field C of complex
numbers, where |λ| signifies the modulus of λ ∈ C. For simplicity, we shall only consider linear
spaces of real numbers.

Vector Norms
We call any norm on a linear space V = Rn , n ≥ 1 a vector norm. The three most common vector
norms are the 1-norm k·k1 , 2-norm (or Euclidean norm) k·k2 , and the ∞-norm k·k∞ .
Definition 3. The 1-norm of the vector v = (v1 , . . . , vn )> ∈ Rn is defined by
n
X
kvk1 = |vi |.
i=1

Definition 4. The 2-norm of the vector v = (v1 , . . . , vn )> ∈ Rn is defined by

n
!1/2
X √ √
kvk2 = |vi |2 = v·v = v > v.
i=1

Definition 5. The ∞-norm of the vector v = (v1 , . . . , vn )> ∈ Rn is defined by

kvk∞ = max |vi |.

1≤i≤n

The 1-norm and 2-norm are special cases of the more general p-norm.
Definition 6. Let p ≥ 1; then, the p-norm of the vector v = (v1 , . . . , vn )> ∈ Rn is defined by

n
!1/p
X
kvkp = |vi |p .
i=1

The following inequality holds on the p-norm.

1
Theorem 7 (Hölder’s inequality). Let p, q ∈ [1, ∞], such that 1/p + 1/q = 1. Then, for any
u, v ∈ Rn , we have that
Xn
ui vi ≤ kukp kvkq .
i=1

Remark 8. If p = ∞ then q = 1 and if p = 1 then q = ∞.

For the case when p = q = 2 we get the Cauchy-Schwarz inequality.
Lemma 9 (Cauchy-Schwarz inequality). For any u, v ∈ Rn , we have that
n
X
ui vi ≤ kuk2 kvk2 .
i=1

Matrix Norms
We call any norm on the linear space Rn×n of n × n matrices a matrix norm. Some matrix norms
satisfy additional properties:
Definition 10. We call a matrix norm k·k a sub-multiplicative matrix norm if
kABk ≤ kAkkBk
n×n
for all A, B ∈ R .
Definition 11. We say that a matrix norm kAk and vector norm kvk are compatible if
kAvk ≤ kAkkvk,
for all A ∈ Rn×n and v ∈ Rn .

Induced Matrix Norm

In this section, we consider the matrix norms induced by a vector norm in the following sense.
Definition 12. Given any vector norm k·k on the space Rn of n-dimensional vectors with real
entries, the induced norm (or subordinate matrix norm) on the space Rn×n of n × n matrices
with real values is defined by
kAvk
kAk = max = max kAvk.
n
v∈R \{0} kvk v∈Rn ,kvk=1

Lemma 13. For any matrix norm k·k induced by the vector norm k·k, the following properties
hold:
1. the matrix and vector norms are compatible, i.e. kAvk ≤ kAkkvk, for all A ∈ Rn×n , v ∈ Rn ,
2. the matrix norm is a sub-multiplicative norm, i.e. kABk ≤ kAkkBk, for all A, B ∈ Rn×n ,
3. kIk = 1, where I ∈ Rn×n is the identity matrix.
While Definition 12 defines an induced norm, it is not helpful for actually computing the norm
of a matrix as it involves maximising a function. Instead, we can show that the following three
theorems hold.
Theorem 14. The matrix norm induced by the vector norm k·k∞ can be expressed, for an n × n
matrix A = (aij )i≤i,j≤n ∈ Rn×n , as
n
X
kAk∞ = max |aij |.
1≤i≤n
j=1

The ∞-norm can be said to be the largest row-sum of the matrix A.

2
Proof. For an arbitrary vector v ∈ Rn \ {0}, write K = kvk∞ , so that |vj | ≤ K for j = 1, . . . , n.
Then,
Xn Xn n
X
|(Av)i | = aij vj ≤ |aij ||vj | ≤ K |aij |.
j=1 j=1 j=1

Define
n
X
C = max |aij |
1≤i≤n
j=1

and note that

Pn
kAvk∞ max1≤i≤n |(Av)i | max1≤i≤n |(Av)i | max1≤i≤n (K j=1 |aij |)
= = ≤ = C, ∀v ∈ Rn \{0}.
v∞ kvk∞ K K

Hence, kAk ≤ C. We now show that kAk ≥ C and, therefore kAk = C. In the definition of C, let
m be the value of i where the maximum is obtained (or one of the maximum values if more than
one exist). We then define a vector v ∈ Rn \ {0}, with such that vj has the same sign as amj and
|vj | = 1, for j = 1, . . . , n (if amj = 0 then vj can be 1 or −1). Due to this definition we have that

n
X n
X
amj vj = |amj ||vj |;
j=1 j=1

therefore,
n
X n
X n
X n
X
kAk∞ kvk∞ ≥ kAvk∞ = max aij vj ≥ amj vj = |amj ||vj | = |amj | = C.
1≤i≤n
j=1 j=1 j=1 j=1

As kvk∞ = 1, then
kAk∞ kvk∞ ≥ kAvk∞ ≥ Ckvk∞ ;
hence, kAk∞ ≥ C.
Theorem 15. The matrix norm induced by the vector norm k·k1 can be expressed, for an n × n
matrix A = (aij )i≤i,j≤n ∈ Rn×n , as
n
X
kAk1 = max |aij |.
1≤j≤n
i=1

The 1-norm can be said to be the largest column-sum of the matrix A.

It can be noted that kAk1 = kA> k∞ . For the definition of the 2-norm we need to recall the
definition of eigenvalues.
Definition 16. Suppose that A ∈ Rn×n . A complex number λ, for which the set of linear equations

Av = λv

has a non-trivial solution v ∈ Cn \ {0}, is called an eigenvalue of A; the associated vector v ∈

Cn \ {0} is called the corresponding eigenvector.
Theorem 17. The matrix norm induced by the vector norm k·k2 can be expressed, for an n × n
matrix A = (aij )i≤i,j≤n ∈ Rn×n , as
q
kAk2 = max λi ,
1≤i≤n

where λi , i = 1, . . . , n, are the eigenvalues of the matrix A> A.

3
Proof.
kAk2 = max kAvk2 , kAvk22 = (Av)> Av = v > A> Av
v∈Rn ,kvk=1

As A> A is a symmetric matrix it has non-negative eigenvalues and there exists an orthogonal
matrix Q (Q> Q = I) and a diagonal matrix D, where Dii = λi , i = 1, . . . , n such that

A> A = Q> DQ
=⇒ kAvk22 = v > Q> DQv = (Qv)> DQv

(Look up eigendecomposition for the above statement). Then, defining y = Qv,

n
X
kAk22 = max
n
kAvk22 = max
n
(Qv)> DQv = max
n
y > Dy = max
n
λi yi2
v∈R ,kvk=1 v∈R ,kvk=1 v∈R ,kvk=1 v∈R ,kvk=1
i=1
 
≤ max λi max kyk22 .
1≤i≤n v∈Rn ,kvk=1

Noting that
kyk22 = (Qv)> Qv = v > Q> Qv = v > Iv = v > v = kvk22 = 1,
and taking the square root of both sides completes the proof.

Frobenius Norm
We finally state one more matrix norm, which is not induced by a vector norm.
Definition 18. The Frobenius norm of the n × n matrix A = (aij )i≤i,j≤n ∈ Rn×n is defined by
 1/2
n X
X n
kAkF =  |aij |2  .
i=1 j=1

Lemma 19. The Frobenius norm k·kF is a sub-multiplicative norm in the sense that

kABkF ≤ kAkF kBkF , for all A, B ∈ Rn×n .

Proof. By Cauchy-Schwarz
 1/2
Xn X
n X
n
kABkF =  aik bkj 
i=1 j=1 k=1

n X
n n
! n !1/2
X X X
≤ aik bkj 
i=1 j=1 k=1 k=1
!1/2  1/2
n X
X n n X
X n
= aik  bkj  = kAkF kBkF
i=1 k=1 k=1 j=1

Properties of Sub-multiplicative Matrix Norms

For sub-multiplicative matrix norms we can state the following properties. For the first property,
we first state for a compatible matrix and vector norm.

4
Lemma 20. Let λ be an eigenvalue of the matrix A ∈ Rn×n . Then, for any matrix norm k·k,
with a compatible vector norm k·k,
|λ| ≤ kAk.
Proof. As λ is an eigenvalue there exists a corresponding eigenvector v such that λx = Av;
therefore,
|λ|kvk = kλvk = kAvk ≤ kAkkvk.

Corollary 21. Let λ be an eigenvalue of the matrix A ∈ Rn×n . Then, for any sub-multiplicative
matrix norm k·k,
|λ| ≤ kAk.
Proof. Define a matrix V ∈ Rn×n such that Vij = vi , for i, j = 1, . . . , n where v is the correspond-
ing eigenvector for the eigenvalue λ. Then,

|λ|kV k = kλV k = kAV k ≤ kAkkV k.

Theorem 22. Let A ∈ Rn×n be a n × n matrix and k·k a sub-multiplicative matrix norm. Then,
if kAk < 1, the matrix I − A is non-singular and
1
k(I − A)−1 k ≤ .
1 − kAk

Proof. For all eigenvalues λi , i = 1, . . . , n

|λi | ≤ kAk < 1.

If v i is an eigenvector for λi then

(I − A)v i = (1 − λi )v i , i = 1, . . . , n,

which implies that 1 − λi 6= 0, i = 1, . . . , n, are the eigenvalues of I − A. Note that for a matrix
B ∈ Rn×n
B non-singular ⇐⇒ all eigenvalues of B are non-zero;
therefore, I − A is non-singular.
Now,

1 = kIk = k(I − A)−1 (I − A)k = k(I − A)−1 − (I − A)−1 Ak

≥ k(I − A)−1 k − k(I − A)−1 Ak
≥ k(I − A)−1 k − k(I − A)−1 kkAk
= k(I − A)−1 k(1 − kAk)

Dividing both sides by (1 − kAk) completes the proof.

Exercises
1. Show that for A ∈ Rn×n any induced matrix norm kAk, defined by Definition 12, satisfies
the conditions of a norm from Definition 1.

2. Prove the three properties from Lemma 13 hold for any induced matrix norm, defined by
Definition 12.

5
3. For a matrix a ∈ Rn×n ,
n
X
|aij | ≤ C, j = 1, . . . , n,
i=1

where
n
X
C = max |aij |.
i≤j≤n
i=1

Show that, for any vector v ∈ Rn ,

n
X
|(Av)i | ≤ Ckvk1 .
i=1

Find a non-zero vector for which equality can be achieved, and deduce that
n
X
kAk1 = max |aij |.
1≤i≤n
j=1

4. Show that the Frobenius norm k·kF is not an induced matrix norm.
Hint. Look at property 3 in Lemma 13.