CSE 599 I
Accelerated Computing -
Programming GPUS
CUDA Dynamic Parallelism
Objective
● Introduce dynamic parallelism, a relatively recent CUDA technique in which
kernels launch kernels
● Learn about various rules and restrictions that apply to dynamic parallelism
● Study some prototypical applications of dynamic parallelism
What is Dynamic Parallelism
An extension to the CUDA programming model which allows a thread to launch
another grid of threads executing another kernel
First introduced with the Kepler architecture (2012)
Uses for Dynamic Parallelism
● Recursive algorithms
● Processing at different levels of detail for different parts of the input (i.e.
irregular grid structure)
● Algorithms in which new work is “uncovered” along the way
Work Discovery Without Dynamic Parallelism
__global__ void workDiscoveryKernel(const int * starts, const int * ends, float * data) {
int i = threadIdx.x + blockDim.x * blockIdx.x;
for (int j = starts[i]; j < ends[i]; ++j) {
process(data[j]);
}
}
Work Discovery
Thread
0
Thread
1
CPU
Thread
2
Thread
3
Without dynamic parallelism
Work Discovery With Dynamic Parallelism
__global__ void workDiscoveryKernel(const int * starts, const int * ends, float * data) {
int i = threadIdx.x + blockDim.x * blockIdx.x;
const int N = ends[i] - starts[i];
workDiscoveryChildKernel<<<(N-1)/128+1,128>>>(data + starts[i], N);
__global__ void workDiscoveryChildKernel(float * data, const int N) {
int j = threadIdx.x + blockDim.x * blockIdx.x;
if (j < N) {
process(data[j]);
}
}
Work Discovery
Thread Thread Thread Thread
0 1
0 0
Thread Thread Thread Thread Thread Thread
0 1 2 3
1 1
CPU CPU
Thread Thread Thread
0
2 2
Thread Thread Thread Thread Thread
0 1 3
3 3
Without dynamic parallelism With dynamic parallelism
Work Discovery
Thread Thread Thread Thread
0 1
0 0
Thread Thread Thread Thread Thread Thread
0 1 0 1
1 1
CPU CPU
Thread Thread Thread
2 0 These can be done
2
in parallel!
Thread Thread Thread Thread Thread
0 1 0
3 3
Without dynamic parallelism With dynamic parallelism
Global Memory and Dynamic Parallelism
Parent and child grids have two points of guaranteed global memory
consistency:
1. When the child grid is launched by the parent; all memory operations
performed by the parent thread before launching the child are visible to the
child grid when it starts
2. When the child grid finishes; all memory operations by any thread in the
child grid are visible to the parent thread once the parent thread has
synchronized with the completed child grid
Constant Memory and Dynamic Parallelism
Constant memory also cannot be changed from within a child grid or before
launching a child grid
Thus, all constant memory must be set on the host before launching the parent
kernel and remain constant for the duration of the entire kernel tree
Local Memory and Dynamic Parallelism
Local memory is private to a thread, and dynamic parallelism is not exception
Child grids have no privileged access to the parent thread’s local data
Not OK OK
__global__ void badParentKernel() { __global__ void goodParentKernel(float * data)
{
float data[10];
childKernel<<<...>>>(data); childKernel<<<...>>>(data);
} }
__global__ void badParentKernel() { __device__ float value;
__global__ void goodParentKernel(float * data)
float value; {
childKernel<<<...>>>(&value);
childKernel<<<...>>>(&value);
}
}
Shared Memory and Dynamic Parallelism
Shared memory is private to a block of threads, and dynamic parallelism is no
exception
Parent threads have no privileged access to a child block’s shared memory
Memory Allocation from within a Kernel
In addition to kernel launches, dynamic parallelism allows memory allocation
from within a kernel via cudaMalloc() and cudaFree()
A few differences about allocating memory from within a kernel:
● Cannot allocate zero-copy memory
● The allocation limit is the device malloc heap size, which may be smaller
than the total device memory size
○ You can get or set this limit using cudaDevice[Get/Set]Limit() with the parameter
cudaLimitMallocHeapSize
● Memory allocated with cudaMalloc() inside a kernel must be freed with
cudaFree() from inside a kernel, and a kernel cannot call cudaFree() with a
pointer that was allocated on the host
Kernels All the Way Down
A kernel launched from within a kernel can launch a kernel, which can also
launch a kernel, etc.
The total “nesting depth” allowed with dynamic parallelism is limited to 24
There are other limits that tend to come up before the maximum nesting depth
Dynamic Parallelism with Multiple GPUs
Kernels launched from within a kernel cannot be executed on another GPU
Pending Launch Pool
The pending launch pool is a buffer that keeps track of kernels that are currently
being executed or waiting to be executed
By default, the pending launch pool has room for 2048 kernels before spilling
into a virtualized pool, which is very slow
Like the device malloc heap size, this limit can be queried or set using
cudaDevice[Get/Set]Limit(), this time with parameter
cudaLimitDevRuntimePendingLaunchCount
Implicit Synchronization
A parent thread is implicitly synchronized with its children before terminating
CPU
Parent grid completes
Parent grid Implicit sync
Child grid completes
Child grid
Explicit Synchronization
A parent thread can also explicitly synchronize with child grids using
cudaDeviceSynchronize()
This blocks the calling thread on all child grids created by all threads in the block
Blocking all threads can be done by calling cudaDeviceSynchronize() from all
threads or following a call by one thread with __syncthreads()
Synchronization Depth
A parent kernel that performs explicit synchronization on a child grid may be
swapped out while waiting for the child grid to finish
This requires storing the entire state of the kernel, i.e. registers, shared memory,
program counters, etc.
The deepest nesting level at which synchronization is performed is referred to
as the synchronization depth
Synchronization depth is limited by the size of the backing store, which can be
checked or set using cudaDevice[Get/Set]Limit() and the parameter
cudaLimitDevRuntimeSyncDepth
Streams and Dynamic Parallelism
● Kernels can launch new kernels in both the default and non-default streams
to be executed concurrently
● Child kernels launched in explicit streams must use streams that were
allocated from within the kernel that launched them
● The scope of a stream is a block; there can be no sharing of streams
between host and device, between blocks, or between parent and child
Streams and Dynamic Parallelism
● If no stream is specified, the default stream is used, serializing all kernels
launched in the same block (even by different threads)
● cudaStreamSynchronize() cannot be called by device code;
cudaDeviceSynchronize() must be used to wait for all child grids luanched
by the block
● All device streams must be non-blocking. To force awareness of this on the
programmer, streams created by the device must use
cudaStreamCreateWithFlags(&stream, cudaStreamNonBlocking)
Events and Dynamic Parallelism
Events also have some support in device code, but not the full functionality
Currently, only cudaStreamWaitEvent() is allowed to be called from a kernel (no
timing or event synchronization)
Events are scoped to the block (like streams)
Events consume device memory, so there is no limit, but too many events risks
reduced concurrency
Example: Drawing Bezier Curves
A Bezier curve is a smooth curved defined by a set of n control points, where n
determines the degree of the curve
For n = 3, the curve is a quadratic Bezier curve defined by control points P0, P1,
and P2, and the following equation:
B(t) = (1-t)2 P0 + 2(1-t)t P1 + t2 P2
Example: Drawing Bezier Curves
A Bezier curve is defined over a continuous domain
We’ll be looking at a kernel to compute a set of discrete points along a
user-defined Bezier curve
To make the curve look smooth, we’ll want to compute more points in
high-curvature regions
Low curvature: High curvature:
Example: Drawing Bezier Curves
#define MAX_NUM_POINTS 128
struct BezierCurve {
float2 controlPoints[3];
float2 vertices[MAX_NUM_POINTS];
int numVertices;
};
__device__ float computeCurvature(const BezierCurve * curve) {
return length(curve->controlPoints[1] - 0.5*(curve->controlPoints[0] +
curve->controlPoints[2])) / length(curve->controlPoints[2] -
curve->controlPoints[0]);
}
We’ll be given a curves with controlPoints set, and we want to compute
vertices
Example: Drawing Bezier Curves
__global__ void computeBezierCurvesKernel(BezierCurve * curves, const int N) {
if (blockIdx.x < N) {
const float curvature = computeCurvature(&curves[blockIdx.x]);
// compute number of points based on curvature, between 4 and MAX_NUM_POINTS
const int nVertices = min(max((int)(curvature * 64.f),4,MAX_NUM_POINTS);
curves[blockIdx.x].numVertices = nVertices;
for (int p = threadIdx.x; p < nVertices; p += blockDim.x) {
const float t = p / (float)(nVertices - 1);
const float oneMinusT = 1.f - t;
float2 position = oneMinusT * oneMinusT * curves[blockIdx.x].controlPoints[0] +
2.f * t * oneMinusT * curves[blockIdx.x].controlPoints[1] +
t * t * curves[blockIdx.x].controlPoints[2];
curves[blockIdx.x].vertices[p] = position;
}
Example: Drawing Bezier Curves
#define MAX_NUM_POINTS 128
struct BezierCurve {
float2 controlPoints[3];
float2 * vertices;
int numVertices;
};
With dynamic parallelism, we won’t need to statically declare the size of the
vertices buffer
Example: Drawing Bezier Curves
__global__ void computeBezierCurvesParentKernel(BezierCurve * curves, const int N) {
const int i = threadIdx.x + blockDim.x * blockIdx.x;
if (i < N) {
const float curvature = computeCurvature(&curves[i]);
// compute number of points based on curvature, between 4 and MAX_NUM_POINTS
curves[i].numVertices = min(max((int)(curvature * 64.f),4,MAX_NUM_POINTS);
cudaMalloc(&curves[i].vertices, curves[i].numVertices * sizeof(float2));
cudaStream_t stream;
cudaStreamCreateWithFlags(&stream, cudaStreamNonBlocking);
computeBezierCurvesChildKernel<<<(curves[i].numVertices-1)/32+1,32,0,stream>>>(&curves[i]);
cudaStreamDestroy(stream);
}
Example: Drawing Bezier Curves
__global__ void computeBezierCurveChildKernel(BezierCurve * curve) {
const int p = threadIdx.x + blockDim.x * blockIdx.x;
if (p < curve->numVertices) {
const float t = p / (float)(nVertices - 1);
const float oneMinusT = 1.f - t;
float2 position = oneMinusT * oneMinusT * curves[blockIdx.x].controlPoints[0] +
2.f * t * oneMinusT * curves[blockIdx.x].controlPoints[1] +
t * t * curves[blockIdx.x].controlPoints[2];
curve->vertices[p] = position;
}
Example: Drawing Bezier Curves
__global__ void cleanupKernel(BezierCurve * curves, const int N) {
const int i = threadIdx.x + blockDim.x * blockIdx.x;
if (i < N) {
cudaFree(curves[i]->vertices);
}
Recursive Example: Quadtrees
A quadtree is a tree specially designed for storing 2D points
Each node represents a square in the plane, and has exactly 4 children, each
representing a quadrant of the square
D
A
A B
C
G
F
E J
G H I
D K
I
H
B
F
C
J
K
Recursive Example: Quadtrees
A quadtree is a tree specially designed for storing 2D points
Each node represents a square in the plane, and has exactly 4 children, each
representing a quadrant of the square
D
A
A B
C
G
F
E J
G H I
D K
?
I
H
B
F
C
J
K
Recursive Example: Quadtrees
A quadtree is a tree specially designed for storing 2D points
Each node represents a square in the plane, and has exactly 4 children, each
representing a quadrant of the square
D
A
A B
C
G
F
E J
G H I
D K
?
I
H
B
F
C
J
K
Recursive Example: Quadtrees
A quadtree is a tree specially designed for storing 2D points
Each node represents a square in the plane, and has exactly 4 children, each
representing a quadrant of the square
D
A
A B
C
G
F
E J
G H I
D K
?
I
H
B
F
C
J
K
Recursive Example: Quadtrees
points: A B C D E F G H I J K
D
A
A B
C
G
F
I
H
K
Recursive Example: Quadtrees
reorder
points: B C D E F G D E G B C F
D
A
A B
C
G
F
E
G
D
I
H
K
Recursive Example: Quadtrees
A D E G B C F H I J K
D
A
0 B
C
G
F
1 9
3 7 8 10
2
I
H
4
6
5
J
K
Recursive Example: Quadtrees
__global__ void buildQuadtreeKernel(QuadtreeNode * nodes, float2 * pointsA, float2 * pointsB, Parameters params) {
__shared__ int smem[8];
QuadtreeNode & node = nodes[blockIdx.x];
const int numPoints = node.numPoints;
// recursive base case
if (numPoints < params.pointThreshold || node.depth() > params.maxDepth) return;
const BoundingBox & bbox = node.boundingBox;
const float2 center = bbox.center();
const int pointsStart = node.pointsStart;
const int pointsEnd = node.pointsEnd;
// compute number of points for each child and store result in shared memory
countPointsInChildNodes(pointsA + pointsStart, pointsEnd - pointsStart, center, smem);
// do a scan on the number of points for each child to compute offsets
scanForOffsets(smem);
// move the points
reorderPoints(pointsA + pointsStart, pointsB + pointsStart, pointsEnd - pointsStart, center, smem);
if (threadIdx.x == blockDim.x - 1) {
cudaMalloc(&node.children, 4 * sizeof(QuadtreeNode)); // allocate memory for the four children
prepareChildren(node, smem); // set bounding boxes, etc. for the children
buildQuadtreeKernel<<<4, blockDim.x>>>(node.children, pointsB, pointsA, params);
}
}
Recursive Algorithms in CUDA Before 2012
Technically, recursion has always been possible
However, it required awkward loop unrolling
Essentially, one had to implement a call stack within the kernel
Conclusion / Takeaways
● Dynamic parallelism is a powerful new tool allowing kernels to perform
recursive functions and dynamically redistribute work for better load
balancing
Sources
https://www.wikipedia.org/
Kirk, David B., and W. Hwu Wen-Mei. Programming massively parallel processors:
a hands-on approach. Morgan Kaufmann, 2016.
