Foundations

of
Game Engine Development
 Foundations
of
Game Engine Development

VOLUME 1
MATHEMATICS
by Eric Lengyel

Terathon Software LLC

Lincoln, California
Foundations of Game Engine Development
Volume 1: Mathematics

ISBN-13: 978-0-9858117-4-7

Copyright © 2016, by Eric Lengyel

All rights reserved. No part of this publication may be reproduced, stored in an

information retrieval system, transmitted, or utilized in any form, electronic or
mechanical, including photocopying, scanning, digitizing, or recording, without
the prior permission of the copyright owner.

Fifth printing

Published by Terathon Software LLC

www.terathon.com

Series website: foundationsofgameenginedev.com

About the cover: The cover image is a scene from a game entitled The 31st,
developed with the Tombstone Engine. Artwork by Javier Moya Pérez.
3.2 Normal Vectors
A normal vector, or just normal for short, is a vector that is perpendicular to a
surface, and the direction in which it points is said to be normal to the surface. A
flat plane has only one normal direction, but most surfaces aren’t so simple and
thus have normal vectors that vary from point to point. Normal vectors are used
for a wide variety of reasons in game engine development that include surface
shading, collision detection, and physical interaction.

3.2.1 Calculating Normal Vectors

There are a few ways in which normal vectors can be calculated, and the best
method in any particular case really depends on how a surface is described from a
mathematical standpoint. In the case that a surface is defined implicitly by a scalar
function f (  ), the normal vector at  = ( x, y, z ) is given by the gradient ∇f (  )
because it is perpendicular to every direction tangent to the level surface of f at .
For example, suppose we have the ellipsoid shown in Figure 3.2, defined by the
equation

y2
f (  )= x 2 + + z 2 − 1= 0 . (3.2)
4
The point  = ( 46 ,1, 46 ) lies on the surface of this ellipsoid, and the normal vector
n at that point is given by

 ∂f ∂f ∂f 
n = ∇f (  )  ,= ,  = y   6 , 1 , 6 . (3.3)
=  2 x, , 2 z   
 ∂x ∂y ∂z    2   2 2 2 

Calculating normal vectors with the gradient is something that’s usually done
only in the process of constructing a triangle mesh to approximate an ideal surface
described by some mathematical formula. The normal vectors are typically scaled
to unit length and stored with the vertex coordinates that they’re associated with.
Most of the time, a game engine is working with a triangle mesh having an arbi-
trary shape that was created in a modeling program, and the only information avail-
able is the set of vertex coordinates and the list of indices that tell how vertices are
grouped into triangles.
As illustrated in Figure 3.3, the normal vector n for a single triangular face can
be calculated by taking the cross product between vectors aligned with two of the

1
2 Chapter 3 Geometry

triangle’s edges. Let 0, 1, and 2 be the vertices of a triangle wound in the coun-
terclockwise direction. An outward-facing normal vector is then given by

n = ( 1 − 0 ) × ( 2 − 0 ). (3.4)

Any permutation of the subscripts that keeps them in the same cyclic order pro-
duces the same normal vector. It doesn’t matter which vertex is chosen to be sub-
tracted from the other two as long as the first factor in the cross product involves
the next vertex in the counterclockwise order. If the order is reversed, then the
calculated normal vector still lies along the same line, but it points in the opposite
direction.
To calculate per-vertex normal vectors for a triangle mesh, it is typical for a
game engine’s model processing pipeline to calculate all of the per-face normal
vectors and then take an average at each vertex over all of the faces that use that
vertex. The average may be weighted based on triangle area or other factors to
create a smooth field of normal vectors over a curved surface. In cases in which a
hard edge is desired, such as for a cube or the pyramid in Figure 3.3, vertex posi-
tions are typically duplicated, and different normal vectors corresponding to dif-
ferent faces are associated with the various copies of the vertex coordinates.

n = f ( )

y
x

Figure 3.2. The normal vector n at a particular point  on a surface implicitly defined by
the equation f (  ) = 0 is given by the gradient ∇f (  ).
3.2 Normal Vectors 3

2

n = ( 1 − 0 ) × ( 2 − 0 )

1

0

Figure 3.3. The normal vector n for a triangular face having vertices 0, 1, and 2 is given
by the cross product between vectors corresponding to two edges of the triangle.

3.2.2 Transforming Normal Vectors

When a model is transformed by a matrix M in order to alter its geometry, every
point  belonging to the original model becomes a point M in the transformed
model. Since a tangent vector t can be approximated by the difference of points 
and  on a surface, or is often exactly equal to such a difference, it is transformed
in the same way as a point to become Mt because the difference between the new
points M and M is tangent to the new surface. Problems arise, however, if we
try to apply the same transformation to normal vectors.
Consider the shape shown in Figure 3.4 that has a normal vector n on its
slanted side. Let M be a transformation matrix that scales by a factor of two in the
horizontal direction but does not scale in the vertical direction. If the matrix M is
multiplied by n, then the resulting vector Mn is stretched horizontally and, as
clearly visible in the figure, is no longer perpendicular to the surface. This indicates
that something is inherently different about normal vectors, and if we want to pre-
serve perpendicularity, then we must find another way to transform them that pro-
duces the correct results. Taking a closer look at how a matrix transforms a vector
provides some insight. We restrict our discussion to 3 × 3 matrices here since nor-
mal vectors are not affected by translation, but the same conclusions will apply to
4 × 4 matrices in the discussion of planes later in this chapter.
4 Chapter 3 Geometry

n nM −1
Mn
M
b b

a 2a

Figure 3.4. A shape is transformed by a matrix M that scales by a factor of two only in the
horizontal direction. The normal vector n is perpendicular to the original surface, but if it
is treated as a column vector and transformed by the matrix M, then it is not perpendicular
to the transformed surface. The normal vector is correctly transformed by treating it as a
row vector and multiplying by M −1. (The original normal vector is shown in light gray on
the transformed surface.)

Let v A be a vector whose coordinates are expressed in coordinate system A as

indicated by its superscript, and consider the fact that the components of v A meas-
ure distances along the coordinate axes. A distance Δx A along the x direction in
coordinate system A is equivalent to the sum of distances Δx B, Δy B, and Δz B along
the axis-aligned directions in another coordinate system B. The x component of v A
can therefore be expressed as the vector

 Δx B A Δy B A Δz B A 
 Δx A v x , Δx A v x , Δx A v x  (3.5)
 
in coordinate system B. Similar expressions with Δy A and Δz A in the denominators
can be used to express the y and z components of v in coordinate system B. Adding
them up gives us a transformed vector v B in coordinate system B that corresponds
to the original vector v A in its entirety, and this can be written as the matrix trans-
formation
 ΔxB ΔxB ΔxB 
 Δx A Δy A Δz A 
 
 Δy B Δy B Δy B  A
vB =  A v . (3.6)
 Δx Δy A Δz A 
 Δz B Δz B Δz B 
 
 Δ x A Δy A Δ z A 
3.2 Normal Vectors 5

Each entry of this matrix multiplies a component of v A by a ratio of axis-aligned

distances, and the axis appearing in the denominator of each ratio corresponds to
the component of v A by which the ratio is multiplied. This has the effect of cancel-
ling the distances in coordinate system A and replacing them with distances in co-
ordinate system B.
Now let us consider a normal vector that was calculated as a gradient. The key
to understanding how such normal vectors transform is realizing that the compo-
nents of a gradient do not measure distances along the coordinate axes, but instead
measure reciprocal distances. In the partial derivatives that compose a vector
( ∂f ∂x , ∂f ∂y , ∂f ∂z ), distances along the x, y, and z axes appear in the denomi-
nators. This is fundamentally different from the measurements made by the com-
ponents of an ordinary vector, and it’s the source of the problem exemplified by
the nonuniform scale shown in Figure 3.4. Whereas an ordinary vector v is treated
as a column matrix with components ( v x , v y , v z ), we write a normal vector n as the
row matrix

1 1 1 
n= . (3.7)
 nx ny n z 

It then becomes apparent that multiplying this vector on the right by the matrix in
Equation (3.6) has the effect of cancelling reciprocal distances in coordinate sys-
tem B and replacing them with reciprocal distances in coordinate system A. Calling
the matrix M, we can state that it is simultaneously the transform that takes ordi-
nary vectors from A to B through the product v B = Mv A and the transform that
takes normal vectors, in the opposite sense, from B to A through the product
n A = n B M . Inverting the matrix reverses both of these transformations, so we con-
clude that the correct transformation from A to B for a normal vector, at least one
calculated with a gradient, is given by

n B = n A M −1 . (3.8)

The correctness of Equation (3.8) can be verified by demonstrating that a trans-

formed normal vector remains perpendicular to any transformed tangent vector.
Suppose that n A and t A are normal and tangent to a surface at some point in coor-
dinate system A. By definition, they are perpendicular, and we must have
nA ⋅t A = 0. (Since n A is a row vector and t A is a column vector, the matrix product
n A t A is actually what we’re calculating here, but the dot is still included by con-
vention, even though the notation is not technically correct, to make it clear that
we are producing a scalar quantity.) Let M be a matrix that transforms ordinary
6 Chapter 3 Geometry

vectors from coordinate system A to coordinate system B. Then the transformed

normal n B is given by n A M −1, and the transformed tangent t B is given by Mt A.
Their product is
n B ⋅ t B = n A M −1Mt A = n A ⋅ t A = 0, (3.9)
and this establishes the fact that they are still perpendicular in coordinate system B
after the transformation by M.
Getting back to the example in Figure 3.4, the transformation matrix M is

2 0 0
M = 0 1 0 (3.10)
 
 0 0 1 
when we align the x axis with the horizontal direction and the y axis with the ver-
tical direction. We can take the normal vector before the transformation to be
n = [ b a 0 ]. Transforming this with Equation (3.8) gives us a new normal vector
equal to [ b 2 a 0 ], which is perpendicular to the transformed surface. The im-
portant observation to make is that the matrix M scales x values by a factor of two,
but because normal vectors use reciprocal coordinates as shown in Equation (3.7),
multiplying n x by two is equivalent to multiplying the x component of n by a factor
of one half, which is exactly what M −1 does.
In the case that a normal vector n A is calculated as the cross product s × t be-
tween two tangent vectors s and t, the transformed normal vector n B should be
equal to the cross product between the transformed tangent vectors. Again, let M
be a matrix that transforms ordinary vectors from coordinate system A to coordi-
nate system B. Then n= B
Ms × Mt, but we need to be able to calculate n B without
any knowledge of the vectors s and t. Expanding the matrix-vector products by
columns with Equation (1.28), we can write

nB = ( s x M [ 0 ] + s y M [1] + s z M [ 2 ] ) × ( t x M [ 0 ] + t y M [1] + t z M [ 2 ] ), (3.11)

where we are using the notation M [ j ] to mean column j of the matrix M (matching
the meaning of the [] operator in our matrix data structures). After distributing the
cross product to all of these terms and simplifying, we arrive at

( s y t z − sz t y ) ( M [1] × M [ 2 ] )
nB =
+ ( sz t x − sx t z ) ( M [ 2 ] × M [ 0 ] )
+ ( sx t y − s y t x ) ( M [ 0 ] × M [1] ). (3.12)
3.2 Normal Vectors 7

The cross product n A = s × t is clearly visible here, but it may be a little less obvious
that the cross products of the matrix columns form the rows of det ( M ) M −1, which
follows from Equation (1.95). We conclude that a normal vector calculated with a
cross product is correctly transformed according to

n B = n A det ( M ) M −1 . (3.13)

Using the adjugate of M, defined in Section 1.7.5, we can also write this as

n B = n A adj ( M ) . (3.14)

This is not only how normal vectors transform, but it’s how any vector resulting
from a cross product between ordinary vectors transforms.
Equation (3.13) differs from Equation (3.8) only by the additional factor of
det ( M ), showing that the two types of normal vectors are closely related. Since
normal vectors are almost always rescaled to unit length after they’re calculated,
in practice, the size of det ( M ) is inconsequential and often ignored, making the
two normal vector transformation equations identical. However, there is one situ-
ation in which det ( M ) may have an impact, and that is the case when the transform
performed by M contains a reflection. When the vertices of a triangle are reflected
in a mirror, their winding orientation is reversed, and this causes a normal vector
calculated with the cross product of the triangle’s edges to reverse direction as well.
This is exactly the effect that a negative determinant of M would have on a normal
vector that is transformed by Equation (3.13).
The code in Listing 3.1 multiplies a normal vector, stored in a Vector3D data
structure and treated as a row matrix, by a Transform4D data structure on the right,
but it ignores the fourth column of the transformation matrix. When positions and
normals are being transformed by a 4 × 4 matrix H, a point  is transformed as H,
but a normal n has to be transformed as nM −1, where M is the upper-left 3 × 3 por-
tion of H. Being able to multiply by a Transform4D data structure is convenient
when both H and H −1 are already available so that the matrix M −1 doesn’t need to
be extracted.
In the general case, both H and M −1 are needed to transform both positions and
normals. If M happens to be orthogonal, which is often the case, then its inverse is
simply equal to its transpose, so the transformed normal is just nM T , but this is
equivalent to Mn if we treat n as a column matrix. Thus, it is common to see game
engines treat ordinary vectors and normal vectors as the same kind of mathematical
element and use multiplication by the same matrix H on the left to transform both
kinds among different coordinate systems.
8 Chapter 3 Geometry

Listing 3.1. This multiplication operator multiplies a Vector3D data structure representing a normal
vector as a row matrix on the right by a Transform4D data structure to transform a normal vector
from coordinate system B to coordinate system A. The transformation matrix is treated as a 3 × 3
matrix, ignoring the fourth column. Note that this transforms a normal vector in the opposite sense
in relation to how the same matrix would transform an ordinary vector from coordinate system A to
coordinate system B.
Vector3D operator *(const Vector3D& n, const Transform4D& H)
{
return (Vector3D(n.x * H(0,0) + n.y * H(1,0) + n.z * H(2,0),
n.x * H(0,1) + n.y * H(1,1) + n.z * H(2,1),
n.x * H(0,2) + n.y * H(1,2) + n.z * H(2,2)));
}