Vector and Matrix Calculus
Herman Kamper
[email protected]
30 January 2013
1 Introduction
As explained in detail in [1], there unfortunately exists multiple competing notations concerning
the layout of matrix derivatives. This can cause a lot of difficulty when consulting several
sources, since different sources might use different conventions. Some sources, for example [2]
(from which I use a lot of identities), even use a mixed layout (according to [1, Notes]). Identities
for both the numerator layout (sometimes called the Jacobian formulation) and the denominator
layout (sometimes called the Hessian formulation) is given in [1], so this makes it easy to check
what layout a particular source uses. I will aim to stick to the denominator layout, which seems
to be the most widely used in the field of statistics and pattern recognition (e.g. [3] and [4,
pp. 327–332]). Other useful references concerning matrix calculus include [5] and [6]. In this
document column vectors are assumed in all cases expect where specifically stated otherwise.
Table 1: Derivatives of scalars, vector functions and matrices [1, 6].
column vector
scalar y matrix Y ∈ Rm×n
y ∈ Rm
∂y row vector matrix ∂Y
∂x (only
scalar x scalar ∂y m
∂x ∈ R
∂x numerator layout)
column vector matrix
column vector x ∈ Rn ∂y n ∂y
∂x ∈ R ∂x ∈ Rn×m
∂y
matrix X ∈ Rp×q matrix ∂X ∈ Rp×q
2 Definitions
Table 1 indicates the six possible kinds of derivatives when using the denominator layout. Using
this layout notation consistently, we have the following definitions.
The derivative of a scalar function f : Rn → R with respect to vector x ∈ Rn is
∂f (x)
∂x1
∂f (x)
∂f (x) def ∂x2
= ..
(1)
∂x .
∂f (x)
∂xn
This is the transpose of the gradient (some authors simply call this the gradient, irrespective of
whether numerator or denominator layout is used).
1
T
The derivative of a vector function f : Rn → Rm , where f (x) = f1 (x) f2 (x) . . . fm (x)
and x ∈ Rn , with respect to scalar xi is
∂f (x) def h ∂f1 (x) ∂f2 (x) ∂fm (x)
i
= ∂xi ∂xi ... ∂xi
(2)
∂xi
T
The derivative of a vector function f : Rn → Rm , where f (x) = f1 (x) f2 (x) . . . fm (x) ,
with respect to vector x ∈ Rn is
∂f (x) ∂f1 (x) ∂f2 (x) ∂fm (x)
. . .
∂x1 ∂x1 ∂x1 ∂x1
∂f (x) ∂f1 (x) ∂f2 (x) ∂fm (x)
∂f (x) def ∂x2 ...
= ∂x2 ∂x2 ∂x2
= .. .. .. .. ..
(3)
∂x . . . . .
∂f (x) ∂f1 (x) ∂f2 (x) ∂fm (x)
∂xn ∂xn ∂xn ... ∂xn
This is just the transpose of the Jacobian matrix.
The derivative of a scalar function f : Rm×n → R with respect to matrix X ∈ Rm×n is
∂f (X) ∂f (X) ∂f (X)
· · ·
∂X11 ∂X12 ∂X1n
∂f (X) ∂f (X) ∂f (X)
∂f (X) def ∂X21 ∂X22 · · · ∂X2n
= .. .. .. ..
(4)
∂X . . . .
∂f (X) ∂f (X) ∂f (X)
∂Xm1 ∂Xm2 ··· ∂Xmn
Observe that the (1) is just a special case of (4) for column vectors. Often (as in [3]) the gradient
notation is used as an alternative to the notation used above, for example:
∂f (x)
∇x f (x) = (5)
∂x
∂f (X)
∇X f (X) = (6)
∂X
3 Identities
3.1 Scalar-by-vector product rule
If a ∈ Rm , b ∈ Rn and C ∈ Rm×n then
m
X m
X n
X m X
X n
aT Cb = ai (Cb)i = ai Cij bj = Cij ai bj (7)
i=1 i=1 j=1 i=1 j=1
Now assume we have vector functions u : Rm → Rm , v = Rn → Rn and A ∈ Rm×n . The vector
functions u and v are functions of x ∈ Rq , but A is not. We want to find an identity for
∂uT Av
(8)
∂x
2
From (7), we have:
T m n
∂uT Av
∂u Av ∂ XX
= = Aij ui vj
∂x l ∂xl ∂xl
i=1 j=1
m X
n
X ∂
= Aijui vj
∂xl
i=1 j=1
m X n
X ∂ui ∂vj
= Aij vj + ui
∂xl ∂xl
i=1 j=1
m X
n m n
X ∂ui X X ∂vj
= Aij vj + Aij ui (9)
∂xl ∂xl
i=1 j=1 i=1 j=1
Now we can show (by writing out the elements [Notebook, 2012-05-22]) that:
m Xn m n
∂u ∂v T X ∂ui X X T ∂vj
Av + A u = Aij vj + (A )ji ui
∂x ∂x l ∂xl ∂xl
i=1 j=1 i=1 j=1
m X n m X n
X ∂ui X ∂vj
= Aij vj + Aij ui (10)
∂xl ∂xl
i=1 j=1 i=1 j=1
A comparison of (9) and (10) completes the proof that
∂uT Av ∂u ∂v T
= Av + A u (11)
∂x ∂x ∂x
3.2 Useful identities from scalar-by-vector product rule
From (11) it follows, with vectors and matrices b ∈ Rm , d ∈ Rq , x ∈ Rn , B ∈ Rm×n , C ∈ Rm×q ,
D ∈ Rq×n , that
∂(Bx + b)T C(Dx + d) ∂(Ax + b) ∂(Dx + d)T T
= C(Dx + d) + C (Bx + b) (12)
∂x ∂x ∂x
resulting in the identity:
∂(Bx + b)T C(Dx + d)
= BT C(Dx + d) + DT CT (Bx + b) (13)
∂x
by using the easily verifiable identities:
∂(u(x) + v(x)) ∂u(x) ∂v(x)
= + (14)
∂x ∂x ∂x
∂Ax
= AT (15)
∂x
∂a
=0 (16)
∂x
Some other useful special cases of (11):
∂xT Ab
= Ab (17)
∂x
3
∂xT Ax
= (A + AT )x (18)
∂x
∂xT Ax
= 2Ax if A is symmetric (19)
∂x
3.3 Derivatives of determinant
See [7, p. 374] for definition of cofactors. Also see [Notebook, 2012-05-22].
We can write the determinant of matrix X ∈ Rn×n as
n
X
|X| = Xi1 Ci1 + Xi2 Ci2 + . . . + Xin Cin = Xij Cij+ (20)
j=1
Thus the derivative will be
∂|X| ∂
= {Xi1 Ci1 + Xi2 Ci2 + . . . + Xin Cin }
∂X kl ∂Xkl
∂
= {Xk1 Ck1 + Xk2 Ck2 + . . . + Xkn Ckn }
∂Xkl
(can choose i any number, so choose i = k)
= Ckl (21)
Thus (see [7, p. 386])
∂|X|
= cofactor X = (adj X)T (22)
∂X
But we know that the inverse of X is given by [7, p. 387]
1
X−1 = adj X (23)
|X|
thus
adj X = |X|X−1 (24)
which, when substituted into (22), results in the identity
∂|X|
= |X|(X−1 )T (25)
∂X
From (25) we can also write
∂ ln |X| ∂ ln |X| 1 ∂|X| 1
= = = |X|(X−1 )T (26)
∂X kl ∂Xkl |X| ∂X |X|
giving the identity
∂ ln |X|
= (X−1 )T (27)
∂X
4
References
[1] Matrix calculus. [Online]. Available: http://en.wikipedia.org/wiki/Matrix calculus
[2] K. B. Petersen and M. S. Pedersen, “The matrix cookbook,” 2008.
[3] A. Ng, Machine Learning. Class notes for CS229, Stanford Engineering Everywhere,
Stanford University, 2008. [Online]. Available: http://see.stanford.edu
[4] S. R. Searle, Matrix Algebra Useful for Statistics. New York, NY: John Wiley & Sons, 1982.
[5] J. R. Schott, Matrix Analysis for Statistics. New York, NY: John Wiley & Sons, 1996.
[6] T. P. Minka, “Old and new matrix algebra useful for statistics,” 2000. [Online]. Available:
http://research.microsoft.com/en-us/um/people/minka/papers/matrix
[7] D. G. Zill and M. R. Cullen, Advanced Engineering Mathematics, 3rd ed. Jones and Bartlett,
2006.
