CENG 545

Lecture 2:
Introduction to CUDA

1
 Credits
• The material used in this presentation is based on code
available in:
– the Tutorial on CUDA in Dr. Dobbs Journal
– Supercomputing 2008 Education Program
– David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 ECE
498AL Spring 2010, University of Illinois, Urbana-Champaign
– Andrew Bellenir’s code for matrix multiplication
– Igor Majdandzic’s code for Voronoi diagrams
– NVIDIA’s CUDA programming guide
 Software Requirements/Tools

• CUDA device driver

• CUDA Software Development Kit
– Emulator
• CUDA Toolkit

• Occupancy calculator
• Visual profiler
 What is Cuda?
• CUDA is a scalable parallel programming model
and a software environment for parallel computing
– Minimal extensions to familiar C/C++ environment
– Heterogeneous serial-parallel programming model
• NVIDIA’s TESLA architecture accelerates CUDA
– Expose the computational horsepower of NVIDIA GPUs
– Enable GPU computing
• CUDA also maps well to multicore CPUs

4
• CUDA Architecture
– Expose general-purpose GPU computing as first-class
capability
– Retain traditional DirectX/OpenGL graphics performance
• CUDA C
– Based on industry-standard C
– A handful of language extensions to allow heterogeneous
programs
– Straightforward APIs to manage devices, memory, etc.
5
 CUDA C Prerequisites

• You (probably) need experience with C or C++

• You do not need any GPU experience
• You do not need any graphics experience
• You do not need any parallel programming experience

6
 The Basics

• Host –The CPU and its memory (host memory)

• Device –The GPU and its memory (device memory)

• Kernel = function that runs on the device

7
 A GPU is a specialized computer
• We need to allocate space in the graphic card’s
memory for the variables.
• The graphic card does not have I/O devices, hence we
need to copy the input data from the memory in the
host computer into the memory in the graphic card,
using the variable allocated in the previous step.
• We need to specify code to execute.
• Copy the results back to the memory in the host
computer.

Supercomputing 2008
Education Program
 CUDA – C with no shader limitations!
• Integrated host+device app C program
– Serial or modestly parallel parts in host C code
– Highly parallel parts in device SPMD kernel C code

Serial Code (host)

Parallel Kernel (device)

KernelA<<< nBlk, nTid >>>(args); ...

Serial Code (host)

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 9

ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Initially:

array
Host’s Memory GPU Card’s Memory

Supercomputing10
2008
Education Program
 Allocate Memory in the GPU card

array array_d
Host’s Memory GPU Card’s Memory

Supercomputing11
2008
Education Program
Copy content from the host’s memory to the
GPU card memory

array array_d
Host’s Memory GPU Card’s Memory

Supercomputing12
2008
Education Program
 Execute code on the GPU

GPU MPs

array array_d
Host’s Memory GPU Card’s Memory

Supercomputing13
2008
Education Program
Copy results back to the host memory

array array_d
Host’s Memory GPU Card’s Memory

Supercomputing14
2008
Education Program
 CUDA Devices and Threads
• A compute device
– Is a coprocessor to the CPU or host
– Has its own DRAM (device memory)
– Runs many threads in parallel
– Is typically a GPU but can also be another type of parallel processing
device
• Data-parallel portions of an application are expressed as device
kernels which run on many threads
• Differences between GPU and CPU threads
– GPU threads are extremely lightweight
• Very little creation overhead
– GPU needs 1000s of threads for full efficiency
• Multi-core CPU needs only a few
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 15
ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Extended C
• Type Qualifiers
__device__ float filter[N];
– global, device, shared,
local, constant __global__ void convolve (float *image) {

__shared__ float region[M];

• Keywords ...

– threadIdx, blockIdx region[threadIdx] = image[i];

• Intrinsics __syncthreads()
– __syncthreads ...

image[j] = result;
• Runtime API }

– Memory, symbol, // Allocate GPU memory

execution management void *myimage = cudaMalloc(bytes)

• // 100 blocks, 10 threads per block

Function launch convolve<<<100, 10>>> (myimage);

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 16

ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Extended C
Integrated source
(foo.cu)

cudacc
EDG C/C++ frontend
Open64 Global Optimizer

GPU Assembly CPU Host Code

foo.s foo.cpp

OCG gcc / cl

G80 SASS Mark Murphy, “

foo.sass NVIDIA’s Experience with Open64,”
www.capsl.udel.edu/conferences/
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 open64/2008/Papers/101.doc 17
ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Arrays of Parallel Threads
• A CUDA kernel is executed by an array of
threads
– All threads run the same code (SPMD)
– Each thread has an ID that it uses to compute
memory addresses and make control decisions
threadID 0 1 2 3 4 5 6 7

…
float x = input[threadID];
float y = func(x);
output[threadID] = y;
…

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 18

ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Thread Blocks: Scalable Cooperation
• Divide monolithic thread array into multiple blocks
– Threads within a block cooperate via shared memory,
atomic operations and barrier synchronization
– Threads in different blocks cannot cooperate

Thread Block 0 Thread Block 1 Thread Block N - 1

0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7
threadID

… … …
float x = float x = float x =
input[threadID];
float y = func(x);
output[threadID] = y;
input[threadID];
float y = func(x);
output[threadID] = y;
… input[threadID];
float y = func(x);
output[threadID] = y;
… … …

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 19

ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Block Size
• Recall that the “stream processors” of the GPU are
organized as MPs (multi-processors) and every MP
has its own set of resources:
– Registers
– Local memory
• The block size needs to be chosen such that there are
enough resources in an MP to execute a block at a
time.

Supercomputing 2008
Education Program
 Grid Size and Block Size
• Programmers need to specify:
– The grid size: The size and shape of the data that the
program will be working on
– The block size: The block size indicates the sub-area of the
original grid that will be assigned to an MP (a set of stream
processors that share local memory)

Supercomputing 2008
Education Program
 In the GPU:

Processing Elements

Array Elements
Block 0 Block 1
Supercomputing22
2008
Education Program
 Block IDs and Thread IDs
Host Device
• Each thread uses IDs to decide Grid 1
what data to work on Kernel Block Block
– Block ID: 1D or 2D 1 (0, 0) (1, 0)

– Thread ID: 1D, 2D, or 3D Block Block

(0, 1) (1, 1)

• Simplifies memory Grid 2

addressing when processing Kernel

2
multidimensional data Block (1, 1)
– Image processing (0,0,1) (1,0,1) (2,0,1) (3,0,1)

– Solving PDEs on volumes Thread Thread Thread Thread

– … (0,0,0) (1,0,0) (2,0,0) (3,0,0)

Thread Thread Thread Thread

(0,1,0) (1,1,0) (2,1,0) (3,1,0)

Courtesy: NDVIA
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 23
ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
Figure 3.2. An Example of CUDA Thread Org
 CUDA Memory Model Overview
• Global memory
– Main means of
communicating R/W Grid

Data between host and

Block (0, 0) Block (1, 0)
device
– Contents visible to all Shared Memory Shared Memory

threads Registers Registers Registers Registers

– Long latency access

Thread (0, 0) Thread (1, 0) Thread (0, 0) Thread (1, 0)

• We will focus on global

memory for now Host Global Memory

– Constant and texture

memory will come later
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 24
ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 CUDA API Highlights:
Easy and Lightweight
• The API is an extension to the ANSI C programming
language
Low learning curve

• The hardware is designed to enable lightweight

runtime and driver
High performance

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 25

ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 CUDA Device Memory Allocation
• cudaMalloc()
– Allocates object in the Grid

device Global Memory Block (0, 0) Block (1, 0)

– Requires two parameters Shared Memory Shared Memory

• Address of a pointer to the Registers Registers Registers Registers

allocated object
Thread (0, 0) Thread (1, 0) Thread (0, 0) Thread (1, 0)
• Size of of allocated object
• cudaFree() Host Global
Memory

– Frees object from device

Global Memory
• Pointer to freed object
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 26
ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 CUDA Device Memory Allocation (cont.)
• Code example:
– Allocate a 64 * 64 single precision float array
– Attach the allocated storage to Md
– “d” is often used to indicate a device data structure

TILE_WIDTH = 64;
Float* Md
int size = TILE_WIDTH * TILE_WIDTH * sizeof(float);

cudaMalloc((void**)&Md, size);
cudaFree(Md);
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 27
ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 CUDA Host-Device Data Transfer
• cudaMemcpy()
– memory data transfer Grid

– Requires four parameters Block (0, 0) Block (1, 0)

• Pointer to destination Shared Memory Shared Memory

• Pointer to source Registers Registers Registers Registers

• Number of bytes copied
• Type of transfer Thread (0, 0) Thread (1, 0) Thread (0, 0) Thread (1, 0)

– Host to Host
– Host to Device Host Global
Memory
– Device to Host
– Device to Device
• Asynchronous transfer
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 28
ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 CUDA Host-Device Data Transfer
(cont.)
• Code example:
– Transfer a 64 * 64 single precision float array
– M is in host memory and Md is in device memory
– cudaMemcpyHostToDevice and
cudaMemcpyDeviceToHost are symbolic constants

cudaMemcpy(Md, M, size, cudaMemcpyHostToDevice);

cudaMemcpy(M, Md, size, cudaMemcpyDeviceToHost);

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 29

ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 CUDA Keywords

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 30

ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Function Qualifiers
• Kernels designated by function qualifier:
– __global__
• Function called from host and executed on device
• Must return void
• Other CUDA function qualifiers
– __device__
• Function called from device and run on device
• Cannot be called from host code
– __host__
• Function called from host and executed on host (default)
• __host__ and __device__ qualifiers can be combined to generate
both CPU and GPU code
31
 CUDA Function Declarations
Executed Only callable
on the: from the:
__device__ float DeviceFunc() device device
__global__ void KernelFunc() device host
__host__ float HostFunc() host host

• __global__ defines a kernel function

– Must return void
• __device__ and __host__ can be used
together
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 32
ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 CUDA Function Declarations (cont.)

• __device__ functions cannot have their

address taken
• For functions executed on the device:
– No recursion
– No static variable declarations inside the function
– No variable number of arguments

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 33

ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Launching Kernels
• Modified C function call syntax:
kernel<<<dim3 dG, dim3 dB>>>(…)
• Execution Configuration (“<<< >>>”)
– dG - dimension and size of grid in blocks
• Two-dimensional: x and y
• Blocks launched in the grid: dG.x * dG.y
– dB - dimension and size of blocks in threads:
• Three-dimensional: x, y, and z
• Threads per block: dB.x * dB.y * dB.z
– Unspecified dim3 fields initialize to 1

34
 CUDA Built-in Device Variables
• All __global__ and __device__ functions have
access to these automatically defined variables
– dim3 gridDim;
• Dimensions of the grid in blocks (at most 2D)
– dim3 blockDim;
• Dimensions of the block in threads
– dim3 blockIdx;
• Block index within the grid
– dim3 threadIdx;
• Thread index within the block

35
 A More Complex Example: add()

• Using our add()kernel:

__global__ void add( int *a, int *b, int *c ) {
*c = *a + *b;
}
• Let’s take a look at main()…

36
 A More Complex Example: main()
İnt main( void ) {
int a, b, c; // host copies of a, b, c
int *dev_a, *dev_b, *dev_c; // device copies of a, b, c
int size = sizeof( int); // we need space for an integer

// allocate device copies of a, b, c

cudaMalloc( (void**)&dev_a, size );
cudaMalloc( (void**)&dev_b, size );
cudaMalloc( (void**)&dev_c, size );

a = 2;
b = 7;

37
A More Complex Example: main()(cont)
// copy inputs to device
cudaMemcpy( dev_a, &a, size, cudaMemcpyHostToDevice);
cudaMemcpy( dev_b, &b, size, cudaMemcpyHostToDevice);

// launch add() kernel on GPU, passing parameters

add<<< 1, 1 >>>( dev_a, dev_b, dev_c);

// copy device result back to host copy of c

cudaMemcpy( &c, dev_c, size, cudaMemcpyDeviceToHost);

cudaFree( dev_a);
cudaFree( dev_b);
cudaFree( dev_c);
return0;
}
38
 Parallel Programming in CUDA C
• But wait…GPU computing is about massive
parallelism
• So how do we run code in parallel on the device?
• Solution lies in the parameters between the triple
angle brackets:
add<<< 1, 1 >>>( dev_a, dev_b, dev_c);
add<<< N, 1 >>>( dev_a, dev_b, dev_c);
• Instead of executing add()once, add()executed Ntimes
in parallel

39
 Parallel Programming in CUDA C

• With add() running in parallel…let’s do vector addition

• Terminology: Each parallel invocation of add()referred to as a block
• Kernel can refer to its block’s index with the variable blockIdx.x

• Each block adds a value from a[]and b[], storing the result in c[]:
__global__ void add( int*a, int*b, int*c ) {
c[blockIdx.x] = a[blockIdx.x] + b[blockIdx.x];
}
• By using blockIdx.xto index arrays, each block handles different indices

40
 Parallel Programming in CUDA C
• We write this code:
__global__ void add( int*a, int*b, int*c ) {
c[blockIdx.x] = a[blockIdx.x] + b[blockIdx.x];
}
• This is what runs in parallel on the device:
Block 0
– c[0] = a[0] + b[0];
Block 1
– c[1] = a[1] + b[1];
Block 2
– c[2] = a[2] + b[2];
Block 3
– c[3] = a[3] + b[3];

41
#define N 512
int main( void ) {
int *a, *b, *c; // host copies of a, b, c
int *dev_a, *dev_b, *dev_c; // device copies of a, b, c
int size = N *sizeof( int); // we need space for 512 integers
// allocate device copies of a, b, c
cudaMalloc( (void**)&dev_a, size );
cudaMalloc( (void**)&dev_b, size );
cudaMalloc( (void**)&dev_c, size );
a = (int*)malloc( size );
b = (int*)malloc( size );
c = (int*)malloc( size );
random_ints( a, N );
random_ints( b, N );

42
// copy inputs to device
cudaMemcpy( dev_a, a, size, cudaMemcpyHostToDevice);
cudaMemcpy( dev_b, b, size, cudaMemcpyHostToDevice);
// launch add() kernel with N parallel blocks
add<<< N, 1 >>>( dev_a, dev_b, dev_c);
// copy device result back to host copy of c
cudaMemcpy( c, dev_c, size, cudaMemcpyDeviceToHost);
free( a ); free( b ); free( c );
cudaFree( dev_a);
cudaFree( dev_b);
cudaFree( dev_c);
return0;
}

43
 Threads

• Terminology: A block can be split into parallel

threads
• Let’s change vector addition to use parallel threads
instead of parallel blocks:
__global__ voidadd( int*a, int*b, int*c ) {
c[ threadIdx.x] = a[threadIdx.x ] + b[threadIdx.x ];
}

• We use threadIdx.xinstead of blockIdx.x in add()

• main()will require one change as well…

44
 intmain( void ) {
int*a, *b, *c; //host copies of a, b, c
int*dev_a, *dev_b, *dev_c; //device copies of a, b, c
intsize = N * sizeof( int); //we need space for 512 integers
// allocate device copies of a, b, c
cudaMalloc( (void**)&dev_a, size );
cudaMalloc( (void**)&dev_b, size );
cudaMalloc( (void**)&dev_c, size );
a = (int*)malloc( size );
b = (int*)malloc( size );
c = (int*)malloc( size );
random_ints( a, N );
random_ints( b, N );

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007- 45

2009
ECE 498AL, University of Illinois, Urbana-Champaign
// copy inputs to device
cudaMemcpy( dev_a, a, size, cudaMemcpyHostToDevice);
cudaMemcpy( dev_b, b, size, cudaMemcpyHostToDevice);
// launch add() kernel with N
add<<< >>>( dev_a, dev_b, dev_c);
// copy device result back to host copy of c
cudaMemcpy( c, dev_c, size, cudaMemcpyDeviceToHost);
free( a ); free( b ); free( c );
cudaFree( dev_a);
cudaFree( dev_b);
cudaFree( dev_c);
return0;
}

46
 Using Threads AndBlocks

• We’ve seen parallel vector addition using

– Many blocks with 1 thread apiece
– 1 block with many threads
• Let’s adapt vector addition to use lots of bothblocks
and threads
• After using threads and blocks together, we’ll talk
about whythreads
• First let’s discuss data indexing…

47
 Indexing Arrays With Threads And
Blocks
• No longer as simple as just using threadIdx.xor blockIdx.xas
indices
• To index array with 1 thread per entry (using 8 threads/block)

If we have M threads/block, a unique array index for each entry

given by
int index = threadIdx.x + blockIdx.x * M;
int index = x + y * width;
48
 Indexing Arrays: Example

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007- 49

2009
ECE 498AL, University of Illinois, Urbana-Champaign
 Addition with Threads and Blocks
• The blockDim.x is a built-in variable for threads per
block:
intindex= threadIdx.x + blockIdx.x * blockDim.x;
• A combined version of our vector addition kernel to
use blocks and threads:
__global__ voidadd( int*a, int*b, int*c ) {
intindex = threadIdx.x + blockIdx.x * blockDim.x;
c[index] = a[index] + b[index];
}

• So what changes in main() when we use both blocks

and threads?

50
 Parallel Addition (Blocks/Threads):
main()
#define N (2048*2048)
#define THREADS_PER_BLOCK 512
intmain( void ) {
int*a, *b, *c; // host copies of a, b, c
int*dev_a, *dev_b, *dev_c; // device copies of a, b, c
intsize = N * sizeof( int); // we need space for N integers
// allocate device copies of a, b, c
cudaMalloc( (void**)&dev_a, size );
cudaMalloc( (void**)&dev_b, size );
cudaMalloc( (void**)&dev_c, size );
a = (int*)malloc( size );
b = (int*)malloc( size );
c = (int*)malloc( size );
random_ints( a, N );
random_ints( b, N );

51
 // copy inputs to device
cudaMemcpy( dev_a, a, size, cudaMemcpyHostToDevice);
cudaMemcpy( dev_b, b, size, cudaMemcpyHostToDevice);
// launch add() kernel with blocks and threads
add<<< N/THREADS_PER_BLOCK, THREADS_PER_BLOCK >>>( dev_a,
dev_b, dev_c);
// copy device result back to host copy of c
cudaMemcpy( c, dev_c, size, cudaMemcpyDeviceToHost);
free( a ); free( b ); free( c );
cudaFree( dev_a);
cudaFree( dev_b);
cudaFree( dev_c);

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007- 52

2009
ECE 498AL, University of Illinois, Urbana-Champaign
 Let’s look at a very simple example
• The code has been divided into two files:
– simple.c
– simple.cu
• simple.c is ordinary code in C
• It allocates an array of integers, initializes it to values
corresponding to the indices in the array and prints the
array.
• It calls a function that modifies the array
• The array is printed again.

Supercomputing 2008
Education Program
 simple.c
•
#include <stdio.h>
#define SIZEOFARRAY 64
extern void fillArray(int *a,int size);
/* The main program */
int main(int argc,char *argv[])
{
/* Declare the array that will be modified by the GPU */
int a[SIZEOFARRAY];
int i;
/* Initialize the array to 0s */
for(i=0;i < SIZEOFARRAY;i++) {
a[i]=i;
}
/* Print the initial array */
printf("Initial state of the array:\n");
for(i = 0;i < SIZEOFARRAY;i++) {
printf("%d ",a[i]);
}
printf("\n");
/* Call the function that will in turn call the function in the GPU that will fill
the array */
fillArray(a,SIZEOFARRAY);
/* Now print the array after calling fillArray */
printf("Final state of the array:\n");
for(i = 0;i < SIZEOFARRAY;i++) {
printf("%d ",a[i]);
}
printf("\n");
return 0;
}

Supercomputing 2008
Education Program
 simple.cu
• simple.cu contains two functions
– fillArray(): A function that will be executed on the host and
which takes care of:
• Allocating variables in the global GPU memory
• Copying the array from the host to the GPU memory
• Setting the grid and block sizes
• Invoking the kernel that is executed on the GPU
• Copying the values back to the host memory
• Freeing the GPU memory

Supercomputing 2008
Education Program
 fillArray (part 1)
#define BLOCK_SIZE 32
extern "C" void fillArray(int *array,int arraySize){
/* a_d is the GPU counterpart of the array that exists on the host memory */
int *array_d;

/* allocate memory on device */

/* cudaMalloc allocates space in the memory of the GPU card */
result = cudaMalloc((void**)&array_d,sizeof(int)*arraySize);
/* copy the array into the variable array_d in the device */
/* The memory from the host is being copied to the corresponding variable in the GPU global memory */
result = cudaMemcpy(array_d,array,sizeof(int)*arraySize,
cudaMemcpyHostToDevice);

Supercomputing 2008
Education Program
 fillArray (part 2)
/* execution configuration... */
/* Indicate the dimension of the block */
dim3 dimblock(BLOCK_SIZE);
/* Indicate the dimension of the grid in blocks */
dim3 dimgrid(arraySize/BLOCK_SIZE);
/* actual computation: Call the kernel, the function that is */
/* executed by each and every processing element on the GPU
card */
cu_fillArray<<<dimgrid,dimblock>>>(array_d);
/* read results back: */
/* Copy the results from the GPU back to the memory on the host
*/
result =
cudaMemcpy(array,array_d,sizeof(int)*arraySize,cudaMemcpyDevice
ToHost);
/* Release the memory on the GPU card */
cudaFree(array_d);
}

Supercomputing 2008
Education Program
 simple.cu (cont.)
• The other function in simple.cu is
– cu_fillArray()
• This is the kernel that will be executed in every stream processor in
the GPU
• It is identified as a kernel by the use of the keyword: __global__
• This function uses the built-in variables
– blockIdx.x and
– threadIdx.x
to identify a particular position in the array

Supercomputing 2008
Education Program
 cu_fillArray
__global__ void cu_fillArray(int *array_d){
int x;
/* blockIdx.x is a built-in variable in CUDA
that returns the blockId in the x axis
of the block that is executing this block of code
threadIdx.x is another built-in variable in CUDA
that returns the threadId in the x axis
of the thread that is being executed by this
stream processor in this particular block
*/
x=blockIdx.x*BLOCK_SIZE+threadIdx.x;
array_d[x]+=array_d[x];
}

Supercomputing 2008
Education Program
 A Simple Running Example
Matrix Multiplication
• A simple matrix multiplication example that illustrates
the basic features of memory and thread management
in CUDA programs
– Leave shared memory usage until later
– Local, register usage
– Thread ID usage
– Memory data transfer API between host and device
– Assume square matrix for simplicity

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 60

ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Programming Model:
Square Matrix Multiplication Example
• P = M * N of size WIDTH x WIDTH N

• Without tiling:

WIDTH
– One thread calculates one element of P
– M and N are loaded WIDTH times from
global memory
M P

WIDTH
WIDTH WIDTH
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 61
ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Memory Layout of a Matrix in C
M0,0 M0,1 M0,2 M0,3

M1,0 M1,1 M1,2 M1,3

M2,0 M2,1 M2,2 M2,3

M3,0 M3,1 M3,2 M3,3

M0,0 M0,1 M0,2 M0,3 M1,0 M1,1 M1,2 M1,3 M2,0 M2,1 M2,2 M2,3 M3,0 M3,1 M3,2 M3,3

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 62

ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Step 1: Matrix Multiplication
A Simple Host Version in C
// Matrix multiplication on the (CPU) host in double
precision
N

void MatrixMulOnHost(float* M, float* N, float* P, int Width) k

{
j

WIDTH
for (int i = 0; i < Width; ++i)
for (int j = 0; j < Width; ++j) {
double sum = 0;
for (int k = 0; k < Width; ++k) {
double a = M[i * width + k];
double b = N[k * widthM+ j]; P

sum += a * b; i
}
P[i * Width + j] = sum;

WIDTH
}
} k
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 WIDTH WIDTH
63
ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Step 2: Input Matrix Data Transfer
(Host-side Code)
void MatrixMulOnDevice(float* M, float* N, float* P, int Width)
{
int size = Width * Width * sizeof(float);
float* Md, Nd, Pd;
…
1. // Allocate and Load M, N to device memory
cudaMalloc(&Md, size);
cudaMemcpy(Md, M, size, cudaMemcpyHostToDevice);

cudaMalloc(&Nd, size);
cudaMemcpy(Nd, N, size, cudaMemcpyHostToDevice);

// Allocate P on the device

cudaMalloc(&Pd, size);
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 64
ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Step 3: Output Matrix Data Transfer
(Host-side Code)

2. // Kernel invocation code – to be shown later

…

3. // Read P from the device

cudaMemcpy(P, Pd, size, cudaMemcpyDeviceToHost);

// Free device matrices

cudaFree(Md); cudaFree(Nd); cudaFree (Pd);
}

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 65

ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Step 4: Kernel Function

// Matrix multiplication kernel – per thread code

__global__ void MatrixMulKernel(float* Md, float* Nd, float* Pd, int Width)
{

// Pvalue is used to store the element of the matrix

// that is computed by the thread
float Pvalue = 0;

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 66

ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Step 4: Kernel Function (cont.)
for (int k = 0; k < Width; ++k) { Nd

float Melement = Md[threadIdx.y*Width+k];

float Nelement = Nd[k*Width+threadIdx.x]; k
Pvalue += Melement * Nelement;

WIDTH
} tx

Pd[threadIdx.y*Width+threadIdx.x] = Pvalue;
} Md Pd

ty ty

WIDTH
k tx
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 WIDTH WIDTH 67
ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Step 5: Kernel Invocation
(Host-side Code)

// Setup the execution configuration

dim3 dimGrid(1, 1);
dim3 dimBlock(Width, Width);

// Launch the device computation threads!

MatrixMulKernel<<<dimGrid, dimBlock>>>(Md, Nd, Pd, Width);

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 68

ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Only One Thread Block Used
Grid 1 Nd
• One Block of threads compute Block 1
2
matrix Pd
4
– Each thread computes one
element of Pd Thread
(2, 2)
2
• Each thread 6
– Loads a row of matrix Md
– Loads a column of matrix Nd
– Perform one multiply and
addition for each pair of Md and
Nd elements
– Compute to off-chip memory 3 2 5 4 48
access ratio close to 1:1 (not very
high)
• Size of matrix limited by the
WIDTH
number of threads allowed in a
thread block
Md Pd
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 69
ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Step 7: Handling Arbitrary Sized Square
Matrices
• Have each 2D thread block to compute Nd

a (TILE_WIDTH)2 sub-matrix (tile) of

the result matrix

WIDTH
– Each has (TILE_WIDTH)2 threads
• Generate a 2D Grid of
(WIDTH/TILE_WIDTH)2 blocks Pd
Md

You still need to put a loop by

around the kernel call for TILE_WIDTH
cases where ty

WIDTH
WIDTH/TILE_WIDTH is
greater than max grid size bx tx
(64K)!
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 WIDTH WIDTH 70
ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Some Useful Information on
Tools

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 71

ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Compiling a CUDA Program
C/C++ CUDA float4 me = gx[gtid];
Application me.x += me.y * me.z;
• Parallel Thread
eXecution (PTX)
– Virtual Machine
NVCC CPU Code and ISA
– Programming
model
PTX Code – Execution
Virtual resources and
state
PhysicalPTX to Target ld.global.v4.f32 {$f1,$f3,$f5,$f7}, [$r9+0];
mad.f32 $f1, $f5, $f3, $f1;
Compiler

G80 … GPU

Target code
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 72
ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Compilation
• Any source file containing CUDA language
extensions must be compiled with NVCC
• NVCC is a compiler driver
– Works by invoking all the necessary tools and
compilers like cudacc, g++, cl, ...
• NVCC outputs:
– C code (host CPU Code)
• Must then be compiled with the rest of the application using another tool
– PTX
• Object code directly
• Or, PTX source, interpreted at runtime

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 73

ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Linking
• Any executable with CUDA code requires two
dynamic libraries:
– The CUDA runtime library (cudart)
– The CUDA core library (cuda)

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 74

ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Debugging Using the
Device Emulation Mode
• An executable compiled in device emulation mode
(nvcc -deviceemu) runs completely on the host
using the CUDA runtime
– No need of any device and CUDA driver
– Each device thread is emulated with a host thread

• Running in device emulation mode, one can:

– Use host native debug support (breakpoints, inspection, etc.)
– Access any device-specific data from host code and vice-versa
– Call any host function from device code (e.g. printf) and vice-
versa
– Detect deadlock situations caused by improper usage of
__syncthreads
© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 75
ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Device Emulation Mode Pitfalls
• Emulated device threads execute sequentially, so
simultaneous accesses of the same memory location
by multiple threads could produce different results.
• Dereferencing device pointers on the host or host
pointers on the device can produce correct results in
device emulation mode, but will generate an error in
device execution mode

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 76

ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign
 Floating Point
• Results of floating-point computations will slightly
differ because of:
– Different compiler outputs, instruction sets
– Use of extended precision for intermediate results
• There are various options to force strict single precision on the host

© David Kirk/NVIDIA and Wen-mei W. Hwu, 2007-2009 77

ECE 498AL Spring 2010, University of Illinois, Urbana-Champaign