COMP 414-514: Optimization: Algorithms, Complexity

& Approximations
Homework 1
Kim Tran - kt68

4% of the final grade

Questions
1. [5pts] Given vectors a, b ∈ Rp , write down the triangle and the Cauchy-Schwarz in-
equality. Further, using the Cauchy-Schwarz inequality, show the triangle inequality
(e.g., using the direct proof technique).
Solution
Triangle inequality:
|a + b| ≤ |a| + |b|
Cauchy-Schwarz inequality:
a · b ≤ ∥a∥∥b∥
Triangle inequality proof from Cauchy-Schwarz: From Cauchy-Schwarz, we have:

a · b ≤ ∥a∥∥b∥

Also,

|a + b|2 = ∥a∥2 + 2a · b + ∥b∥2 ≤ ∥a∥2 + 2∥a∥∥b∥ + ∥b∥2 = (∥a∥ + ∥b∥)2

Taking square root of both side we have:

|a + b| ≤ |a| + |b|

, which proved the triangle inequality

2. [5pts] Can p + 1 vectors of p dimensions be linearly independent? Please, elaborate in

your answer.
Solution
No, p + 1 vectors of p dimensions cannot be linearly independent unless the p + 1
vector is a zero vector.

1
 Proof:
A space of p dimensions can be written as a span using p vectors:

y = a1 x1 + a2 x2 + ... + ap xp

Consider the (p + 1)th vector xp+1 . In order for p + 1 vectors to be linearly independent:

a1 x1 + a2 x2 + ... + ap xp + ap+1 xp+1 = 0

needs to have a1 = a2 = ... = ap = ap+1 = 0.

Also, since xp+1 is also in the p dimension space, it can be written as a span using p
vectors:
xp+1 = b1 x1 + b2 x2 + ... + bp xp
Substitute in the equation above, we have:

a1 x1 + a2 x2 + ... + ap xp + ap+1 (b1 x1 + b2 x2 + ... + bp xp ) = 0

⇒ (a1 + ap+1 b1 )x1 + (a2 + ap+1 b2 )x2 + ... + (ap + ap+1 bp )xp = 0
This equation has non-trivial solution and does not satisfy the requirement of a1 =
a2 = ... = ap = ap+1 = 0, which contradict the definition of linearly independent.
Therefore, p + 1 vectors cannot be linearly independent in p dimensions space.
√
3. [5pts] Using direct proof, prove that, for a vector x ∈ Rp , ∥x∥2 ≤ p∥x∥∞ .
Solution
We have:
∥x∥∞ = max |xi | ≥ ∥xi ∥∀i < p
so,
√ √ √
qX qX p
∥x∥2 = x2i ≤ (max |xi |)2 = p(max |xi |)2 = p| max |xi || = p max |xi | = p∥x∥∞
√
Therefore, ∥x∥2 ≤ p∥x∥∞

4. [5pts] Using the definition of the Frobenius norm and showing all steps, what is the
Frobenius norm value of the matrix
 
1 1
A= ?
1 1

Solution
√ √
∥A∥F = 12 + 12 + 12 + 12 = 4=2

5. [5pts] Let  
  −1 −1
−1.5 3 2
A= , B =  0 −2 .
1 −1 0
1 0

2
 Compute the matrix-matrix multiplication A · B.
Solution

 
  −1 −1
−1.5 3 2 
A · B := 0 −2
1 −1 0
1 0
 
(−1.5)(−1) + (3)(0) + (2)(1) (−1.5)(−1) + (3)(−2) + (2)(0)
=
(1)(−1) + (−1)(0) + (0)(1) (1)(−1) + (−1)(−2) + (0)(0)
 
3.5 −4.5
=
−1 1

6. [5pts] Assuming A ∈ Rm×n and B ∈ Rn×p , what is the complexity of A · B in Big-Oh

notation?
Solution
For each entry of the result matrix, we have to perform n times of multiplication since
there are n columns in A and n rows in B. The size of the result matrix is m × p so
there are mp entries. Therefore, the complexity of A · B is O(nmp).
7. [10pts] What is the trace of matrix
 
3 0.3 −1
A = −4 0 1.9?
5 1 −1

Solution

Tr(A) = 3 + 0 + (−1) = 2
   
1 2 1 −7 −7 6
8. [10pts] Argue that A = 4 4 5 has inverse B =  2 1 −1 .
6 7 7 4 5 −4
Solution

  
1 2 1 −7 −7 6
A · B := 4 4 5  2 1 −1
6 7 7 4 5 −4
 
(1)(−7) + (2)(2) + (1)(4) (1)(−7) + (2)(1) + (1)(5) (1)(6) + (2)(−1) + (1)(−4)
= (4)(−7) + (4)(2) + (5)(4) (4)(−7) + (4)(1) + (5)(5) (4)(6) + (4)(−1) + (5)(−4)
(6)(−7) + (7)(2) + (7)(4) (6)(−7) + (7)(1) + (7)(5) (6)(6) + (7)(−1) + (7)(−4)
 
1 0 0
= 0 1 0 = I
0 0 1

Therefore, B is the inverse of A.

3
 9. 
[10pts]
 Find
  the matrix A whose eigenvalues are 1 and 4 and whose eigenvectors are
3 2
and .
1 1
Solution
Using the diagonalization formular we have:

A = XΛX −1

, where X is the eigenvectors matrix and Λ is the eigenvalue matrix

 
3 2
X=
1 1
   
−1 1 1 −2 1 −2
X = =
3 · 1 − 2 · 1 −1 3 −1 3
 
1 0
Λ=
0 4
Then,         
3 2 1 0 1 −2 3 8 1 −2 −5 18
A= = =
1 1 0 4 −1 3 1 4 −1 3 −3 10

10. [10pts] Let u, v ∈ Rp . Prove that

∥u + v∥22 + ∥u − v∥22 = 2∥u∥22 + 2∥v∥22 .

Solution:

r 2 r 2
X X
∥u + v∥22 + ∥u − v∥22 = (u + v)2
+ (u − v)2
X X
=| (u + v)2 | + | (u − v)2 |
X X X X X X
=| u2 + 2 uv + v2 | + | u2 − 2 uv + v2 |
X X
=2 u2 + 2 v2
r r
X 2 X 2
=2 u2 + 2 v2
= 2∥u∥22 + 2∥v∥22

Therefore, ∥u + v∥22 + ∥u − v∥22 = 2∥u∥22 + 2∥v∥22

11. [10pts] Suppose you have a matrix A ∈ Rm×n . If you reshape it into a vector of length
m × n by stacking its columns, show that the Euclidean norm of that vector is exactly
the Frobenius norm ∥A∥F .

4
 Solution: 
Let A = a1 a2 · · · an , where ai is each column of matrix A of dimension m × 1
The Frobenius norm of A:
∥A∥F = m
P Pn
i j aij
The Euclidean norm of the reshaped vector v of length m × n:
pPmn qP P
2 m n
∥v∥2 = i ai = i j aij
Therefore, the Euclidean norm of the reshaped vector is exactly the Frobenius norm
of matrix A

12. [10pts] Let D ∈ Rp×p be a diagonal matrix with diagonal entries d1 , . . . , dp . Show that
D2 is also diagonal with diagonal entries (d21 , . . . , d2p ). Then, if Q = U DU −1 (where U
is invertible), prove that Q2 = U D2 U −1 .
Solution: 
d1 0 · · · 0
 0 d2 · · · 0 
Let D =  .. .. . . .. . We have:
 
. . . .
0 0 · · · dp
    
d1 0 · · · 0 d1 0 · · · 0 d21 0 · · · 0
 0 d2 · · · 0   0 d2 · · · 0   0 d2 · · · 0 
2 2
D =  .. .. . . ..  =  .. .. . .
    
..   .. .. . . .. 
. . . .  . . . . . . . .
0 0 · · · dp 0 0 · · · dp 0 0 · · · d2p

Thus, D2 is also diagonal with diagonal (d21 , d22 , · · · , d2p )

Q2 = (UDU−1 )(UDU−1 )
= UD(U−1 U)DU−1
= UDIDU−1
= U(DD)U−1
= UD2 U−1

Therefore, Q2 = UD2 U−1

13. [10pts] Show that for any x, y ∈ Rp , |x⊤ y| ≤ ∥x∥1 ∥y∥∞ .

Solution:
We have: p
X
⊺
|x y| = xi y i
i=1

For every element in the sum:

xi yi ≤ xi max |yi |
i

5
Thus,
p p
X X
xi yi ≤ max |yi | xi = ∥y∞ ∥∥x1 ∥
i
i=1 i=1
⊺
Therefore, |x y| = ∥x1 ∥∥y∞ ∥