See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/228832149

Theano: A CPU and GPU math compiler in Python

Article · January 2010

CITATIONS READS
440 5,278

9 authors, including:

James Bergstra Frederic Bastien

Harvard University Université de Montréal
27 PUBLICATIONS 21,470 CITATIONS 13 PUBLICATIONS 4,243 CITATIONS

SEE PROFILE SEE PROFILE

Pascal Lamblin Razvan Pascanu

Université de Montréal Université de Montréal
16 PUBLICATIONS 13,155 CITATIONS 117 PUBLICATIONS 23,562 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Razvan Pascanu on 03 March 2014.

The user has requested enhancement of the downloaded file.

PROC. OF THE 9th PYTHON IN SCIENCE CONF. (SCIPY 2010) 1

Theano: A CPU and GPU Math Compiler in Python

James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins,
Joseph Turian, David Warde-Farley, Yoshua Bengio

Abstract— Theano is a compiler for mathematical expressions in program optimization can be time-consuming and error-prone,
Python that combines the convenience of NumPy’s syntax with the making an automated approach to performance optimization
speed of optimized native machine language. The user composes highly desirable.
mathematical expressions in a high-level description that mimics
NumPy’s syntax and semantics, while being statically typed and Theano, on the other hand, works on a symbolic represen-
functional (as opposed to imperative). These expressions allow tation of mathematical expressions, provided by the user in
Theano to provide symbolic differentiation. Before performing a NumPy-like syntax. Access to the full computational graph
computation, Theano optimizes the choice of expressions, translates of an expression opens the door to advanced features such
them into C++ (or CUDA for GPU), compiles them into dynamically as symbolic differentiation of complex expressions, but more
loaded Python modules, all automatically. Common machine learn-
ing algorithms implemented with Theano are from 1.6× to 7.5× importantly allows Theano to perform local graph transforma-
faster than competitive alternatives (including those implemented tions that can correct many unnecessary, slow or numerically
with C/C++, NumPy/SciPy and MATLAB) when compiled for the unstable expression patterns. Once optimized, the same graph
CPU and between 6.5× and 44× faster when compiled for the GPU. can be used to generate CPU as well as GPU implementations
This paper illustrates how to use Theano, outlines the scope of the (the latter using CUDA) without requiring changes to user
compiler, provides benchmarks on both CPU and GPU processors,
and explains its overall design. code.
Theano is similar to [SymPy], in that both libraries ma-
nipulate symbolic mathematical graphs, but the two projects
Introduction have a distinctly different focus. While SymPy implements a
richer set of mathematical operations of the kind expected in
Python is a powerful and flexible language for describing
a modern computer algebra system, Theano focuses on fast,
large-scale mathematical calculations, but the Python inter-
efficient evaluation of primarily array-valued expressions.
preter is in many cases a poor engine for executing them.
Theano is free open source software, licensed under the
One reason is that Python uses full-fledged Python objects
New (3-clause) BSD license. It depends upon NumPy, and
on the heap to represent simple numeric scalars. To reduce
can optionally use SciPy. Theano includes many custom C
the overhead in numeric calculations, it is important to use
and CUDA code generators which are able to specialize
array types such as NumPy’s ndarray so that single Python
for particular types, sizes, and shapes of inputs; leveraging
objects on the heap can stand for multidimensional arrays of
these code generators requires gcc (CPU) and nvcc (GPU)
numeric scalars, each stored efficiently in the host processor’s
compilers, respectively. Theano can be extended with custom
native format.
graph expressions, which can leverage scipy.weave, Py-
[NumPy] provides an N-dimensional array data type, and
CUDA, Cython, and other numerical libraries and compilation
many functions for indexing, reshaping, and performing ele-
technologies at the user’s discretion. Theano has been actively
mentary computations (exp, log, sin, etc.) on entire arrays
and continuously developed and used since January 2008.
at once. These functions are implemented in C for use within
It has been used in the preparation of numerous scientific
Python programs. However, the composition of many such
papers and as a teaching platform for machine learning in
NumPy functions can be unnecessarily slow when each call
graduate courses at l’Université de Montréal. Documentation
is dominated by the cost of transferring memory rather than
and installation instructions can be found on Theano’s website
the cost of performing calculations [Alted]. [numexpr] goes
[theano]. All Theano users should subscribe to the announce1
one step further by providing a loop fusion optimization that
mailing list (low traffic). There are medium traffic mailing lists
can glue several element-wise computations together. Unfor-
for developer discussion2 and user support3 .
tunately, numexpr requires an unusual syntax (the expression
must be encoded as a string within the code), and at the time This paper is divided as follows: Case Study: Logistic
of this writing, numexpr is limited to optimizing element-wise Regression shows how Theano can be used to solve a sim-
computations. [Cython] and [scipy.weave] address Python’s ple problem in statistical prediction. Benchmarking Results
performance issue by offering a simple way to hand-write presents some results of performance benchmarking on prob-
crucial segments of code in C (or a dialect of Python which lems related to machine learning and expression evaluation.
can be easily compiled to C, in Cython’s case). While this What’s in Theano gives an overview of the design of Theano.
approach can yield significant speed gains, it is labor-intensive: Limitations and Future Work outlines current limitations of our
if the bottleneck of a program is a large mathematical expres- implementation and currently planned additions to Theano.
sion comprising hundreds of elementary operations, manual 1 http://groups.google.com/group/theano-announce
2 http://groups.google.com/group/theano-dev
The corresponding author is with Université de Montréal, e-mail:
[email protected]. 3 http://groups.google.com/group/theano-users
2 PROC. OF THE 9th PYTHON IN SCIENCE CONF. (SCIPY 2010)

Case Study: Logistic Regression calling said functions to perform numerical computations.
To get a sense of how Theano feels from a user’s perspec- The code listings in Figures 1 - 4 illustrate these steps with
tive, we will look at how to solve a binary logistic regression a working program that fits a logistic regression model to
problem. Binary logistic regression is a classification model random data.
parameterized by a weight matrix W and bias vector b. The 1: import numpy
model estimates the probability P(Y = 1|x) (which we will 2: import theano.tensor as T
3: from theano import shared, function
denote with shorthand p) that the input x belongs to class 4:
y = 1 as: 5: x = T.matrix()
(i)
6: y = T.lvector()
eW x +b 7: w = shared(numpy.random.randn(100))
P(Y = 1|x(i) ) = p(i) = (i)
(1) 8: b = shared(numpy.zeros(()))
1 + eW x +b 9: print "Initial model:"
10: print w.get_value(), b.get_value()
The goal is to optimize the log probability of N training
examples, D = {(x(i) , y(i) ), 0 < i ≤ N}), with respect to W and Figure 1: Logistic regression, part 1: declaring variables.
b. To maximize the log likelihood we will instead minimize
the (average) negative log likelihood4 :
The code in Figure 1 declares four symbolic variables x, y
1 w, and b to represent the data and parameters of the model.
`(W, b) = − ∑ y(i) log p(i) + (1 − y(i) ) log(1 − p(i) ) (2)
N i Each tensor variable is strictly typed to include its data type,
its number of dimensions, and the dimensions along which it
To make it a bit more interesting, we can also include an may broadcast (like NumPy’s broadcasting) in element-wise
`2 penalty on W , giving a cost function E(W, b) defined as: expressions. The variable x is a matrix of the default data type
(float64), and y is a vector of type long (or int64). Each
E(W, b) = `(W, b) + 0.01 ∑ ∑ w2i j (3)
i j
row of x will store an example x(i) , and each element of y will
store the corresponding label y(i) . The number of examples to
In this example, tuning parameters W and b will be use at once represents a tradeoff between computational and
done through stochastic gradient descent (SGD) on E(W, b). statistical efficiency.
Stochastic gradient descent is a method for minimizing a The shared() function creates shared variables for W
differentiable loss function which is the expectation of some and b and assigns them initial values. Shared variables behave
per-example loss over a set of training examples. SGD esti- much like other Theano variables, with the exception that
mates this expectation with an average over one or several they also have a persistent value. A shared variable’s value is
examples and performs a step in the approximate direction maintained throughout the execution of the program and can
of steepest descent. Though more sophisticated algorithms for be accessed with .get_value() and .set_value(), as
numerical optimization exist, in particular for smooth convex shown in line 10.
functions such as E(W, b), stochastic gradient descent remains
11: p_1 = 1 / (1 + T.exp(-T.dot(x, w)-b))
the method of choice when the number of training examples 12: xent = -y*T.log(p_1) - (1-y)*T.log(1-p_1)
is too large to fit in memory, or in the setting where training 13: cost = xent.mean() + 0.01*(w**2).sum()
examples arrive in a continuous stream. Even with relatively 14: gw,gb = T.grad(cost, [w,b])
15: prediction = p_1 > 0.5
manageable dataset sizes, SGD can be particularly advanta-
geous for non-convex loss functions (such as those explored Figure 2: Logistic regression, part 2: the computation graph.
in Benchmarking Results), where the stochasticity can allow
the optimizer to escape shallow local minima [Bottou]. The code in Figure 2 specifies the computational graph
According to the SGD algorithm, the update on W is required to perform stochastic gradient descent on the pa-
1 ∂ E(W, b, x, y) rameters of our cost function. Since Theano’s interface shares
W ←W −µ 0 ∑ , (4)
N i ∂W x=x(i) ,y=y(i)
much in common with that of NumPy, lines 11-15 should
be self-explanatory for anyone familiar with that module. On
where µ = 0.1 is the step size and N is the number of line 11, we start by defining P(Y = 1|x(i) ) = 1 as the sym-
examples with which we will approximate the gradient (i.e. bolic variable p_1. Notice that the matrix multiplication and
the number of rows of x). The update on b is likewise element-wise exponential functions are simply called via the
1 ∂ E(W, b, x, y) T.dot and T.exp functions, analogous to numpy.dot and
b ← b−µ ∑ . (5)
0
N i ∂b x=x(i) ,y=y(i)
numpy.exp. xent defines the cross-entropy loss function,
which is then combined with the `2 penalty on line 13, to form
Implementing this minimization procedure in Theano in- the cost function of Eq (3) and denoted by cost.
volves the following four conceptual steps: (1) declaring sym- Line 14 is crucial to our implementation of SGD, as it
bolic variables, (2) using these variables to build a symbolic performs symbolic differentiation of the scalar-valued cost
expression graph, (3) compiling Theano functions, and (4) variable with respect to variables w and b. T.grad operates
4 Taking the mean in this fashion decouples the choice of the regularization
by iterating backwards over the expression graph, applying the
coefficient and the stochastic gradient step size from the number of training chain rule of differentiation and building symbolic expressions
examples. for the gradients on w and b. As such, gw and gb are also
THEANO: A CPU AND GPU MATH COMPILER IN PYTHON 3

symbolic Theano variables, representing ∂ E/∂W and ∂ E/∂ b

respectively. Finally, line 15 defines the actual prediction
(prediction) of the logistic regression by thresholding
P(Y = 1|x(i) ).

16: predict = function(inputs=[x],

17: outputs=prediction)
18: train = function(
19: inputs=[x,y],
20: outputs=[prediction, xent],
21: updates={w:w-0.1*gw, b:b-0.1*gb})

Figure 3: Logistic regression, part 3: compilation.

The code of Figure 3 creates the two functions required to

train and test our logistic regression model. Theano functions
are callable objects that compute zero or more outputs from
values given for one or more symbolic inputs. For example,
the predict function computes and returns the value of Figure 5: Fitting a multi-layer perceptron to simulated data with
prediction for a given value of x. Parameters w and b various implementations of stochastic gradient descent. These models
are passed implicitly - all shared variables are available as have 784 inputs, 500 hidden units, a 10-way classification, and are
trained 60 examples at a time.
inputs to all functions as a convenience to the user.
Line 18 (Figure 3) which creates the train function
highlights two other important features of Theano functions:
Benchmarking Results
the potential for multiple outputs and updates. In our example,
train computes both the prediction (prediction) of the Theano was developed to simplify the implementation of
classifier as well as the cross-entropy error function (xent). complex high-performance machine learning algorithms. This
Computing both outputs together is computationally efficient section presents performance in two processor-intensive tasks
since it allows for the reuse of intermediate computations, such from that domain: training a multi-layer perceptron (MLP) and
as dot(x,w). The optional updates parameter enables training a convolutional network. We chose these architectures
functions to have side-effects on shared variables. The updates because of their popularity in the machine learning community
argument is a dictionary which specifies how shared variables and their different computational demands. Large matrix-
should be updated after all other computation for the function matrix multiplications dominate in the MLP example and two-
takes place, just before the function returns. In our example, dimensional image convolutions with small kernels are the
calling the train function will update the parameters w and major bottleneck in a convolutional network. More information
b with new values as per the SGD algorithm. about these models and their associated learning algorithms
is available from the Deep Learning Tutorials [DLT]. The
22: N = 4 implementations used in these benchmarks are available online
23: feats = 100
24: D = (numpy.random.randn(N, feats), [dlb].
25: numpy.random.randint(size=N,low=0, high=2)) CPU timing was carried out on an an Intel(R) Core(TM)2
26: training_steps = 10 Duo CPU E8500 @ 3.16GHz with 2 GB of RAM. All
27: for i in range(training_steps):
28: pred, err = train(D[0], D[1]) implementations were linked against the BLAS implemented
29: print "Final model:", in the Intel Math Kernel Library, version 10.2.4.032 and
30: print w.get_value(), b.get_value() allowed to use only one thread. GPU timing was done on a
31: print "target values for D", D[1]
32: print "prediction on D", predict(D[0]) GeForce GTX 285. CPU computations were done at double-
precision, whereas GPU computations were done at single-
Figure 4: Logistic regression, part 4: computation. precision.
Our first benchmark involves training a single layer MLP by
Our example concludes (Figure 4) by using the functions stochastic gradient descent. Each implementation repeatedly
train and predict to fit the logistic regression model. Our carried out the following steps: (1) multiply 60 784-element
data D in this example is just four random vectors and labels. input vectors by a 784 × 500 weight matrix, (2) apply an
Repeatedly calling the train function (lines 27-28) fits our element-wise hyperbolic tangent operator (tanh) to the result,
parameters to the data. Note that calling a Theano function (3) multiply the result of the tanh operation by a 500 × 10
is no different than calling a standard Python function: the matrix, (4) classify the result using a multi-class generalization
graph transformations, optimizations, compilation and calling of logistic regression, (5) compute the gradient by performing
of efficient C-functions (whether targeted for the CPU or GPU) similar calculations but in reverse, and finally (6) add the
have all been done under the hood. The arguments and return gradients to the parameters. This program stresses element-
values of these functions are NumPy ndarray objects that wise computations and the use of BLAS routines.
interoperate normally with other scientific Python libraries and Figure 5 compares the number of examples processed
tools. per second across different implementations. We compared
4 PROC. OF THE 9th PYTHON IN SCIENCE CONF. (SCIPY 2010)

7 a**2 + b**2 + 2*a*b 4.0 2*a + 3*b

Speed up vs NumPy
6 3.5
5 3.0
4 2.5
2.0
3 1.5
2 1.0
1 0.5
01e3 1e5 1e7 0.01e3 1e5 1e7
1.0 a+1 40 2*a + b**10

Speed up vs NumPy
0.9 35
0.8 30
0.7 25
0.6 20
0.5
0.4 15
0.3 10
0.2 5
0.11e3 1e5 0
1e7 1e3 1e5 1e7
Dimension of vectors a and b Dimension of vectors a and b

Figure 6: Fitting a convolutional network using different software. Figure 7: Speed comparison between NumPy, numexpr, and Theano
The benchmark stresses convolutions of medium-sized (256 by 256) for different sizes of input on four element-wise formulae. In each
images with small (7 by 7) filters. subplot, the solid blue line represents Theano, the dashed red line
represent numexpr, and performance is plotted with respect to NumPy.

Theano (revision #ec057beb6c) against NumPy 1.4.1, MAT-

LAB 7.9.0.529, and Torch 5 (a machine learning library For small amounts of data, the extra function-call overhead of
written in C/C++) [torch5] on the CPU and GPUMat 0.25 numexpr and Theano makes them slower. For larger amounts
for MATLAB ([gpumat]) on the GPU. of data, and for more complicated expressions, Theano is
When running on the CPU, Theano is 1.8x faster than fastest because it uses an implementation specialized for each
NumPy, 1.6x faster than MATLAB, and 7.5x faster than Torch expression.
5.5 Theano’s speed increases 5.8x on the GPU from the CPU,
a total increase of 11x over NumPy (CPU) and 44x over Torch What kinds of work does Theano support?
5 (CPU). GPUmat brings about a speed increase of only 1.4x
when switching to the GPU for the MATLAB implementation, Theano’s expression types cover much of the same func-
far less than the 5.8x increase Theano achieves through CUDA tionality as NumPy, and include some of what can be found
specializations. in SciPy. Table 1 lists some of the most-used expressions in
Because of the difficulty in implementing efficient convolu- Theano. More extensive reference documentation is available
tional networks, we only benchmark against known libraries online [theano].
that offer a pre-existing implementation. We compare against Theano’s strong suit is its support for strided N-dimensional
EBLearn [EBL] and Torch, two libraries written in C++. arrays of integers and floating point values. Signed and
EBLearn was implemented by Yann LeCun’s lab at NYU, who unsigned integers of all native bit widths are supported, as
have done extensive research in convolutional networks. To put are both single-precision and double-precision floats. Single-
these results into perspective, we implemented approximately precision and double-precision complex numbers are also
half (no gradient calculation) of the algorithm using SciPy’s supported, but less so - for example, gradients through several
signal.convolve2d function. This benchmark uses con- mathematical functions are not implemented. Roughly 90%
volutions of medium sized images (256 × 256) with small of expressions for single-precision N-dimensional arrays have
filters (7 × 7). Figure 6 compares the performance of Theano GPU implementations. Our goal is to provide GPU implemen-
(both CPU and GPU) with that of competing implementations. tations for all expressions supported by Theano.
On the CPU, Theano is 2.2x faster than EBLearn, its best
competitor. This advantage is owed to the fact that Theano
compiles more specialized convolution routines. Theano’s Random numbers are provided in two ways: via NumPy’s
speed increases 4.9x on the GPU from the CPU, a total of random module, and via an internal generator from the
10.7x over EBLearn (CPU). On the CPU, Theano is 5.8x MRG family [Ecu]. Theano’s RandomStreams replicates
faster than SciPy even though SciPy is doing only half the the numpy.random.RandomState interface, and acts as
computations. This is because SciPy’s convolution routine has a proxy to NumPy’s random number generator and the various
not been optimized for this application. random distributions that use it. The MRG_RandomStreams
We also compared Theano with numexpr and NumPy for class implements a different random number generation algo-
evaluating element-wise expressions on the CPU (Figure 7). rithm (called MRG31k3p) that maps naturally to GPU archi-
tectures. It is implemented for both the CPU and GPU so that
5 Torch was designed and implemented with flexibility in mind, not speed programs can produce the same results on either architecture
(Ronan Collobert, p.c.). without sacrificing speed. The MRG_RandomStreams class
THEANO: A CPU AND GPU MATH COMPILER IN PYTHON 5

Operators +, -, /, *, **, //, eq, neq, <, <=, >, >=,

&, |, ^ Canonicalization
Allocation alloc, eye, [ones,zeros]_like,
identity{_like}
Stabilization
Indexing* basic slicing (see set_subtensor and
inc_subtensor for slicing lvalues); lim-
ited support for advanced indexing Specialization
Mathematical exp, log, tan[h], cos[h], sin[h],
Functions real, imag, sqrt, floor, ceil,
round, abs GPU Transfer
Tensor all, any, mean, sum, min, max, var,
Operations prod, argmin, argmax, reshape,
flatten, dimshuffle Code Generation
Conditional cond, switch
Figure 8: The compilation pipeline for functions compiled for GPU.
Looping Scan Functions compiled for the CPU omit the GPU transfer step. v2

Linear dot, outer, tensordot, diag,

Algebra cholesky, inv, solve research, and that has motivated the inclusion of more special-
Calculus* grad ized expression types such as the logistic sigmoid, the softmax
function, and multi-class hinge loss.
Signal conv2d, FFT, max_pool_2d
Processing
Compilation by theano.function
Random RandomStreams, What happens under the hood when creating a function?
MRG_RandomStreams
This section outlines, in broad strokes, the stages of the com-
Printing Print pilation pipeline. Prior to these stages, the expression graph
is copied so that the compilation process does not change
Sparse compressed row/col storage, limited opera- anything in the graph built by the user. As illustrated in Figure
tor support, dot, transpose, conversion 8, the expression graph is subjected to several transformations:
to/from dense
(1) canonicalization, (2) stabilization, (3) specialization, (4)
Machine sigmoid, softmax, multi-class hinge loss optional GPU transfer, (5) code generation. There is some
Learning overlap between these transformations, but at a high level they
TABLE I.: Overview of Theano’s core functionality. This list is have different objectives. (The interested reader should note
not exhaustive, and is superseded by the online documentation. that these transformations correspond roughly, but not exactly
More details are given in text for items marked with an asterisk. to the optimization objects that are implemented in the project
dimshuffle is like numpy.swapaxes. source code.)
Canonicalization: The canonicalization transformation puts
the user’s expression graph into a standard form. For example,
offers a more limited selection of random number distributions duplicate expressions are merged into a single expression. Two
than NumPy though: uniform, normal, and multinomial. expressions are considered duplicates if they carry out the
Sparse vectors and matrices are supported via SciPy’s same operation and have the same inputs. Since Theano ex-
sparse module. Only compressed-row and compressed- pressions are purely functional (i.e., cannot have side effects),
column formats are supported by most expressions. There these expressions must return the same value and thus it is
are expressions for packing and unpacking these sparse types, safe to perform the operation once and reuse the result. The
some operator support (e.g. scaling, negation), matrix transpo- symbolic gradient mechanism often introduces redundancy,
sition, and matrix multiplication with both sparse and dense so this step is quite important. For another example, sub-
matrices. Sparse expressions currently have no GPU equiva- expressions involving only multiplication and division are
lents. put into a standard fraction form (e.g. a / (((a * b)
There is also support in Theano for arbitrary Python objects. / c) / d) -> (a * c * d) / (a * b) -> (c *
However, there are very few expressions that make use of that d) / (b)). Some useless calculations are eliminated in this
support because the compilation pipeline works on the basis of phase, for instance cancelling out uses of the a term in the
inferring properties of intermediate results. If an intermediate previous example, but also reducing exp(log(x)) to x, and
result can be an arbitrary Python object, very little can be computing outright the values of any expression whose inputs
inferred. Still, it is occasionally useful to have such objects in are fully known at compile time. Canonicalization simplifies
Theano graphs. and optimizes the graph to some extent, but its primary
Theano has been developed to support machine learning function is to collapse many different expressions into a single
6 PROC. OF THE 9th PYTHON IN SCIENCE CONF. (SCIPY 2010)

normal form so that it is easier to recognize expression patterns default GPU optimizations.
in subsequent compilation stages. Tensors stored on the GPU use a special internal data type
Stabilization: The stabilization transformation improves the with an interface similar to the ndarray. This datatype fully
numerical stability of the computations implied by the ex- supports strided tensors, and arbitrary numbers of dimensions.
pression graph. For instance, consider the function log(1 + The support for strides means that several operations such as
exp(x)), which tends toward zero as limx→−∞ , and x as the transpose and simple slice indexing can be performed in
limx→−∞ . Due to limitations in the representation of double constant time.
precision numbers, the computation as written yields infinity Code Generation: The code generation phase of the com-
for x > 709. The stabilization phase replaces patterns like pilation process produces and loads dynamically-compiled
one with an expression that simply returns x when x is Python modules with specialized implementations for the
sufficiently large (using doubles, this is accurate beyond the expressions in the computation graph. Not all expressions
least significant digit). It should be noted that this phase have C (technically C++) implementations, but many (roughly
cannot guarantee the stability of computations. It helps in some 80%) of Theano’s expressions generate and compile C or
cases, but the user is still advised to be wary of numerically CUDA code during theano.function. The majority of
problematic computations. expressions that generate C code specialize the code based on
Specialization: The specialization transformation replaces the dtype, broadcasting pattern, and number of dimensions of
expressions with faster ones. Expressions like pow(x,2) be- their arguments. A few expressions, such as the small-filter
come sqr(x). Theano also performs more elaborate special- convolution (conv2d), further specialize code based on the
izations: for example, expressions involving scalar-multiplied size the arguments will have.
matrix additions and multiplications may become BLAS Why is it so important to specialize C code in this way?
General matrix multiply (GEMM) nodes and reshape, Modern x86 architectures are relatively forgiving of code that
transpose, and subtensor expressions (which create does not make good use techniques such as loop unrolling
copies by default) are replaced by constant-time versions and prefetching contiguous blocks of memory, and only the
that work by aliasing memory. Expressions subgraphs in- conv2d expression goes to any great length to generate many
volving element-wise operations are fused together (as in special case implementations for the CPU. By comparison,
numexpr) in order to avoid the creation and use of unnec- GPU architectures are much less forgiving of code that is
essary temporary variables. For instance, if we denote the not carefully specialized for the size and physical layout of
a + b operation on tensors as map(+, a, b), then an function arguments. Consequently, the code generators for
expression such as map(+, map(*, a, b), c) would GPU expressions like GpuSum, GpuElementwise, and
become map(lambda ai,bi,ci: ai*bi+ci, a, b, GpuConv2d generate a wider variety of implementations than
c). If the user desires to use the GPU, expressions with corre- their respective host expressions. With the current generation
sponding GPU implementations are substituted in, and transfer of graphics cards, the difference in speed between a naïve
expressions are introduced where needed. Specialization also implementation and an optimal implementation of an expres-
introduces expressions that treat inputs as workspace buffers. sion as simple as matrix row summation can be an order of
Such expressions use less memory and make better use of magnitude or more. The fact that Theano’s GPU ndarray-
hierarchical memory, but they must be used with care because like type supports strided tensors makes it even more important
they effectively destroy intermediate results. Many expressions for the GPU code generators to support a variety of memory
(e.g. GEMM and all element-wise ones) have such equivalents. layouts. These compile-time specialized CUDA kernels are
Reusing memory this way allows more computation to take integral to Theano’s GPU performance.
place on GPUs, where memory is at a premium.
Moving Computation to the GPU: Each expression in Limitations and Future Work
Theano is associated with an implementation that runs on While most of the development effort has been directed
either the host (a host expression) or a GPU device (a at making Theano produce fast code, not as much attention
GPU expression). The GPU-transfer transformation replaces has been paid to the optimization of the compilation process
host expressions with GPU expressions. The majority of host itself. At present, the compilation time tends to grow super-
expression types have GPU equivalents and the proportion is linearly with the size of the expression graph. Theano can deal
always growing. with graphs up to a few thousand nodes, with compilation
The heuristic that guides GPU allocation is simple: if any times typically on the order of seconds. Beyond that, it can
input or output of an expression resides on the GPU and the be impractically slow, unless some of the more expensive
expression has a GPU equivalent, then the GPU equivalent optimizations are disabled, or pieces of the graph are com-
is substituted in. Shared variables storing float32 tensors piledseparately.
default to GPU storage, and the expressions derived from A Theano function call also requires more overhead (on the
them consequently default to using GPU implementations. It is order of microseconds) than a native Python function call. For
possible to explicitly force any float32 variable to reside on this reason, Theano is suited to applications where functions
the GPU, so one can start the chain reaction of optimizations correspond to expressions that are not too small (see Figure
and use the GPU even in graphs with no shared variables. It is 5).
possible (though awkward, and discouraged) to specify exactly The set of types and operations that Theano provides
which computations to perform on the GPU by disabling the continues to grow, but it does not cover all the function-
THEANO: A CPU AND GPU MATH COMPILER IN PYTHON 7

ality of NumPy and covers only a few features of SciPy. Acknowledgements

Wrapping functions from these and other libraries is often Theano has benefited from the contributions of many
straightforward, but implementing their gradients or related members of Yoshua Bengio’s machine learning group in
graph transformations can be more difficult. Theano does not the computer science department (Départment d’Informatique
yet have expressions for sparse or dense matrix inversion, et de Recherche Operationelle) at l’Université de Montréal,
nor linear algebra decompositions, although work on these is especially Arnaud Bergeron, Thierry Bertin-Mahieux, Olivier
underway outside of the Theano trunk. Support for complex Delalleau, Douglas Eck, Dumitru Erhan, Philippe Hamel, Si-
numbers is also not as widely implemented or as well-tested mon Lemieux, Pierre-Antoine Manzagol, and François Savard.
as for integers and floating point numbers. NumPy arrays with The authors acknowledge the support of the following agencies
non-numeric dtypes (strings, Unicode, Python objects) are not for research funding and computing support: NSERC, RQCHP,
supported at present. CIFAR, SHARCNET and CLUMEQ.
We expect to improve support for advanced indexing and
linear algebra in the coming months. Documentation online R EFERENCES
describes how to add new operations and new graph trans-
formations. There is currently an experimental GPU version [theano] Theano. http://www.deeplearning.net/
software/theano
of the scan operation, used for looping, and an experimental
[NumPy] T. E. Oliphant. “Python for Scientific Computing”.
lazy-evaluation scheme for branching conditionals. Computing in Science & Engineering 9, 10 (2007).
The library has been tuned towards expressions related [Bottou] L. Bottou. “Online Algorithms and Stochastic Ap-
to machine learning with neural networks, and it is not as proximations”. In D. Saad, ed. Online Learning and
well tested outside of this domain. Theano is not a powerful Neural Networks (1998). Cambridge University Press,
Cambridge, UK. Online: http://leon.bottou.
computer algebra system, and it is an important area of future
org/papers/bottou-98x
work to improve its ability to recognize numerical instability [numexpr] D. Cooke et al. “numexpr”. http://code.
in complicated element-wise expression graphs. google.com/p/numexpr/
Debugging Theano functions can require non-standard tech- [Cython] S. Behnel, R. Bradshaw, and D. S. Seljebotn. “Cython:
niques and Theano specific tools. The reason is two-fold: C-Extensions for Python”. http://www.cython.
1) definition of Theano expressions is separate from their org/
[scipy.weave] SciPy Weave module. http://docs.scipy.
execution, and 2) optimizations can introduce many changes org/doc/scipy/reference/tutorial/
to the computation graph. Theano thus provides separate weave.html
execution modes for Theano functions, which allows for auto- [Alted] F. Alted. “Why Modern CPUs Are Starving And What
mated debugging and profiling. Debugging entails automated Can Be Done About It”. Computing in Science and
sanity checks, which ensure that all optimizations and graph Engineering 12(2):68-71, 2010.
[SymPy] SymPy Development Team. “SymPy: Python Library
transformations are safe (Theano compares the results before for Symbolic Mathematics”. http://www.sympy.
and after their application), as well as comparing the outputs org/
of both C and Python implementations. [BLAS] J. J. Dongarra, J. Du Croz, I. S. Duff, and
We plan to extend GPU support to the full range of C data S. Hammarling. “Algorithm 679: A set of
types, but only float32 tensors are supported as of this writing. Level 3 Basic Linear Algebra Subprograms”.
ACM Trans. Math. Soft., 16:18-28, 1990.
There is also no support for sparse vectors or matrices on http://www.netlib.org/blas
the GPU, although algorithms from the CUSPARSE package [LAPACK] E. Anderson et al. “LAPACK Users’ Guide, Third
should make it easy to add at least basic support for sparse Edition”. http://www.netlib.org/lapack/
GPU objects. lug/index.html
[DLT] Deep Learning Tutorials. http://
Conclusion deeplearning.net/tutorial/
[dlb] Benchmarking code: http://github.com/
Theano is a mathematical expression compiler for Python pascanur/DeepLearningBenchmarks
that translates high level NumPy-like code into machine [torch5] R. Collobert. “Torch 5”. http://torch5.
language for efficient CPU and GPU computation. Theano sourceforge.net
[EBL] “EBLearn: Energy Based Learning, a C++
achieves good performance by minimizing the use of tem- Machine Learning Library”. http://eblearn.
porary variables, minimizing pressure on fast memory caches, sourceforge.net/
making full use of gemm and gemv BLAS subroutines, and [gpumat] “GPUmat: GPU toolbox for MATLAB”. http://
generating fast C code that is specialized to sizes and constants gp-you.org
in the expression graph. Theano implementations of machine [Ecu] P. L’Ecuyer, F. Blouin, and R. Couture. “A Search for
Good Multiple Recursive Generators”. ACM Transac-
learning algorithms related to neural networks on one core of tions on Modeling and Computer Simulation, 3:87-98,
an E8500 CPU are up to 1.8 times faster than implementations 1993.
in NumPy, 1.6 times faster than MATLAB, and 7.6 times faster
than a related C++ library. Using a Nvidia GeForce GTX285
GPU, Theano is an additional 5.8 times faster. One of Theano’s
greatest strengths is its ability to generate custom-made CUDA
kernels, which can not only significantly outperform CPU
implementations but alternative GPU implementations as well.

View publication stats