Mastering DirectX 11

with Unity
Renaldas Zioma
Simon Green

Thursday, March 15, 2012

 Overview

Unity DirectX 11 Renderer

Renaldas Zioma, Unity Technologies
Twitter: @__ReJ__
New features and DirectX 11 Effects
Simon Green, NVIDIA

Thursday, March 15, 2012

 New Unity Features

Unity DirectX 11 Renderer

Unity Physically Based shaders
Modified lighting pipeline
Catmull-Clark subdivision
Blend shapes, PointCache (PC2) support
Reflection (cubemaps, quads) probes
Post processing
Incremental lightmap baking

Thursday, March 15, 2012

Hello, this is a short list of main features that we have developed recently. Ill mostly talk about Physically Based shaders and Catmull-Clark subdivision.
 Unity DirectX 11

Compute shader
Tessellation
3D textures
other technologies from 2002 that we added: single channel float
textures!

Thursday, March 15, 2012

We have implemented DirectX11 renderer in Unity. Main highlights are compute shaders and tesselation. We finally bothered to implement 3D texture support
too.
 Compute example - 1M Particles in Unity

particleCS.SetBuffer(0, "posBuffer", posBuffer);

particleCS.SetBuffer(0, "velBuffer", velBuffer);
particleCS.SetFloat("dt", Time.deltaTime);
...
particleCS.Dispatch(0, size, 1, 1);
Thursday, March 15, 2012
Compute shader API is really simple and can be accessed directly from C#/Javascript.
 Unity Physically Based shader

Thursday, March 15, 2012

 Unity Physically Based shader

Inspired by Mental Ray Architectural (MIA) shaders

good enough for high quality offline rendering
most hard-surface materials (metal, wood, glass, clay)
Familiar to CG / non-game artists
Physically based
reduces number of parameters
Energy conserving
(kinda) impossible to setup material breaking laws of physics
Predictable results
avoid hacks
intuitive
Just 1 shader for almost everything
Thursday, March 15, 2012
It is important to get shading and lighting foundation right. We cant really fix broken image with shiny posts.
Modern hardware provides abundant power (esp ALU, texfetch) which can be used for physically based shading. I believe
that by avoiding ad-hoc solutions in shading we can achieve more intuitive and predictable workflow for artists.

As an inspiration we have picked MR Architectural shader library - mainly because of familiarity to CG artists, wide range of
supported materials while keeping code kernel small and simple and energy conserving.

Our shaders primarily attempts to be physically accurate by keeping the track of the light distribution - light is either
absorbed, reflected or transmitted further, hence no light magically (dis)appears.
 Unity Physically Based shader

Diffuse: Oren-Nayar
1 parameter: roughness
Lambert (roughness == 0)
different approximations

Thursday, March 15, 2012

Oren-Nayar allows rough and powdery surfaces - brick, concrete, clay. Moon surface is another good example of non-
Lambertian scattering. OrenNayar can also be used for clothing materials such as velvet. And when roughness is 0, Oren-
Nayar degenerates into Lambert.

There are several Oren-Nayar approximations varying in cost and quality.

 Unity Physically Based shader

Specular: Cook-Torrance
2 parameters: roughness (glosiness), reflectivity
energy conserving
physically based
micro-facet theory

Thursday, March 15, 2012

Cook-Torrance lighting is targeted at metals and plastics and allows to represent specular reflectivity for rough surfaces.
Unlike Phong, Cook-Torrance proves accurate way of computing the color of the specular term an approach based more
on physics than pure mathematics.
 Unity Physically Based shaders

Fresnel curve - how reflectivity depends on view angle

2 parameters: facing camera (at 0 degrees), at 90 degrees to camera
interpolated using Schlick approximation: lerp (Reflat0, Reflat90, (N.V)5)
called BRDF in MIA

Thursday, March 15, 2012

View dependent reflectivity - Fresnel curve is a simple way to simulate full Bi-directional Reflectance Distribution
Function.
 Energy conservation

Diffuse + Reflection (+ Refraction) <= 1

1st law of thermodynamics
Reflectivity takes energy from both Diffuse and Transparency
adding reflectivity will reduce diffuse energy
100% reflective will never be diffuse or transparent!
Transparency takes energy from Diffuse
standard Alpha blending
Both Cook-Torrance and Oren-Nayar are energy conserving
unlike original Blinn-Phong
intense highlights are narrow, wide highlights are less intense

Thursday, March 15, 2012

Energy conservation - one of the most important features of the material.
No additional energy is created and the incoming light energy is properly distributed to the diffuse, reflection and refraction
components in a way that maintains the 1st law of thermodynamics.

Adding more reflectivity (as energy must be taken from somewhere) the diffuse level and the transparency will be reduced
automatically. Adding transparency will reduce diffuse level.

Energy conservation applies to specular highlight of CookTorrance making intense highlights narrow and wide highlights less
intense. Analogous behavior is achieved with modified energy conserving BlinnPhong.
 Reflectivity takes energy from Diffuse and
Transparency
Adding reflectivity will reduce diffuse energy

Thursday, March 15, 2012

Apple Terminator

Reflectivity varies from 0 on the left to 1 on the right

 Reflectivity takes energy from Diffuse and
Transparency
100% reflective will never be diffuse or transparent!

Transparency
Reflectivity

Thursday, March 15, 2012

Reflectivity varies from 0.1 on the top to 0.7 on the bottom
Transparency varies from 0.1 on the left to 0.8 on the right
 Blurry reflections

Thursday, March 15, 2012

 Blurry reflections

Low quality blur by sampling different miplevels (LODbias)

DX9/GL requires fixup across cubemap edges
does not work for planar reflections
only Box filtering
Increase blur quality by sampling multiple times (1..8)
Bend surface normal
emulates micro-facets of the rough surface
Reflect view direction against bent normal
Can simulate various normal distributions N N

Thursday, March 15, 2012

DirectX11 saves headache of fixing up pixels when sampling on the edge of the cubemap.
Using lower miplevels doesnt work well for planar reflections as they are usually rendered in screen space. Blurring should
offset rays along the reflection vector, but not in screen space.

Multiple jittered samples could be used to improve quality of the blurry reflections beyond box filtering.
 Combining 2 normal maps

Artists wanted detail normal maps

Blending 2 normal maps just flattens both
Want to get results as if blending 2 heightmaps
Warp 2nd normal map
using normals from the 1st normal map:

float3x3 nBasis = float3x3(

float3 (n1.z, n1.x,-n1.y),
float3 (n1.x, n1.z,-n1.y),
float3 (n1.x, n1.y, n1.z ));
n = normalize (n2.x*nBasis[0] + n2.y*nBasis[1] + n2.z*nBasis[2]);

Thursday, March 15, 2012

 Without normals

Thursday, March 15, 2012

 Detail normals

Thursday, March 15, 2012

 Combined normals

Thursday, March 15, 2012

 Optimizing Physically Based shaders

Small code kernel - easy to optimize

same code for Skin and Car shading too
Costs
90 ALU instructions on average
270 ALU instructions at max
up to 24 texture fetches
Permutations - pick depending on parameter values
if diffuse roughness = 0, use Lambert instead of OrenNayar
different OrenNayar approximations
glossiness value affects number of reflection texture samples
etc

Thursday, March 15, 2012

 Catmull-Clark Subdivision

Thursday, March 15, 2012

 Catmull-Clark Subdivision

Standard in modeling and offline rendering

Does NOT map directly to DirectX11 tessellation pipeline
PN-Triangles or Phong Tesselation does not exactly match
results from modeling tools
Ongoing research on Catmull-Clark approximation on GPU
requires complex mesh preprocessing
or too expensive for required quality

Thursday, March 15, 2012

 Catmull-Clark Subdivision

Instead we perform 1st subdivision level on CPU

most visually important subd level
7.4M verts/sec
per single core on Core i7 2600K
plain C implementation
not fast, but works!
Higher subd levels on GPU using Phong Tessellation
Future plan: move Catmull-Clark code to COMPUTE shader
otherwise bandwidth bound due to CPU -> GPU uploads

Thursday, March 15, 2012

We went for a pragmatic approach: all characters would be modelled with subdivision surfaces like usual in offline CG. Then
at runtime, we perform one level of Catmull-Clark subdivision - this is enough to get the surface be approximately the
right shape. Subdivision is performed after animation, mesh skinning and blend shapes (morph targets) are calculated, and
it computes new vertex positions, normals and tangent vectors.

In our case the initial mesh is already quite dense to support animation, and after one level of Catmull-Clark subdivision
the shape is close to the limit surface.
 Global Illumination

Unity supports lightmap baking using Beast

Only indirect lighting is stored in lightmaps
Light probes
lighting for dynamic objects
Reflection probes
cubemaps (infinite distance reflections)
quads (local reflections)

Thursday, March 15, 2012

 Light probes - tetrahedras

Thursday, March 15, 2012

 Ambient occlusion

Horizon Based Ambient Occlusion (HBAO)

developed by NVIDIA
Ambient Occlusion should NOT affect direct illumination
by definition!
otherwise gets dirty look
Calculate SSAO early, feed results into pixel shaders
min (bakedAO, objectAO, SSAO)
apply only to indirect lightmap, light probes and reflections

Thursday, March 15, 2012

Ambient occlusion should affect only indirect (ambient) illumination. Direct illumination should be affected only by direct
occlusion (shadowmaps).
 Old Unity SSAO

Thursday, March 15, 2012

Comparison between old Unity SSAO and new HBAO algorithms
 HBAO

Thursday, March 15, 2012

Comparison between old Unity SSAO and new HBAO algorithms
 HBAO

Thursday, March 15, 2012

 Skin rendering

Based on the same code as hard-surface shader

But in multiple layers
Double specular
Texture space diffusion

Thursday, March 15, 2012

Mulitple layers with varying blur amounts
One specular layer is used to mask second specular layer - artistic decision
Thursday, March 15, 2012
 Skin rendering

Based on the same code as hard-surface shader

But in multiple layers
Double specular
Texture space diffusion

Thursday, March 15, 2012

eyes could look better ;(
 DirectX 11 Effects

APEX Destruction
Hair pipeline
Volumetric explosions
Motion blur

Thursday, March 15, 2012

 APEX Destruction

Unity already uses PhysX

Rigid body simulation

APEX Destruction enables real-time destruction effects

Fracturing objects
Breaking walls, smashing windows etc.

NVIDIA integrated APEX runtime into Unity

Thursday, March 15, 2012

 NVIDIA APEX Scalable Dynamics Framework

Turbulence

Vegetation

Clothing

Destruction

Thursday, March 15, 2012

APEX is a scalable Dynamics Framework (aka Adaptive Physics EXtensions)

Extendible framework: the building blocks are modules that solve certain problems. Their runtime complexity varies, but a common to all is attention to
ease of authoring. Apex is artist-focused.

Many modules are currently available, or will be soon. Destruction and Clothing are ready now, in our 0.9 release, as well as particles and forcefields.
Vegetation (which integrates with SpeedTree) and Turbulence will be available soon.
 APEX Destruction
Mesh data & Fracture Map

OBJ FBX BMP USER

PhysXLab Core

Destruction
APEX Core
Renderer PhysX SDK

APX

APEX asset file

Thursday, March 15, 2012
Now well see how to author destructibles.

Destruction is authored in PhysXLab. PhysXLab is a standalone tool that allows artists to fracture arbitrary objects and immediately try them
out within the tool.

You can import for example meshes from Alias OBJ or Autodesk FBX format files, then fracture it using a variety of methods. The result is
stored in an Destructible Asset file (pda). This includes the fractured graphics mesh as well as references to resources such as materials and
particle systems.

The tool allows you to select the internal material, used for the newly-generated internal faces.

What you see is what you get the APEX file output is used for simulation preview within the tool, as well as the runtime loadable asset.
Theres no difference.
 APEX Destruction Demo

Thursday, March 15, 2012

 Hair Pipeline

Hair and fur is the next big challenge for game characters
as animation and skin shading gets better, low quality hair more obvious
Unity hair system was designed to be similar as possible to off-
line systems
Hair styles (groom) can be modeled in Softimage XSI / Maya / MAX
Exported to Unity using PC2 point cache format
Rendered using DirectX 11 tessellation
Generation of render hair geometry done entirely in hardware
Geometry amplification
GeForce GTX 580 tessellation hardware is very fast

Thursday, March 15, 2012

 Early DirectX 11 Hair Test

Thursday, March 15, 2012

 Hair Styling

Hair defined by two items:

Emitter mesh
Guide hairs
defined at vertices of each triangle in emitter mesh
typically 15 control vertices per guide hair
stored in DirectX 11 structured buffer

Thursday, March 15, 2012

 Guide Hair Curves and Emitter Mesh

Thursday, March 15, 2012

 Generated Render Hairs

Thursday, March 15, 2012

 Hair using DirectX 11 Tessellation

Uses isoline domain

Hardware can generate up to 64 isolines / patch, with up to 64 verts / line
Domain shader
Generates render hairs over surface of each triangle
chooses a random position on triangle (using integer hash function)
render hairs interpolated between guide hairs using barycentric coords
Evaluates position on each curve using B-Spline
Number of curves / tri and vertices/curve can be varied
supports LOD and back-face culling
Use 3D noise based on position to add detail, wind effects
Geometry shader
Expands line strips to camera-facing triangle strips

Thursday, March 15, 2012

Using splines allows actual number of vertices generated along curve to be varied
 Hair

Thursday, March 15, 2012

after NVIDIA party
 Add Noise

Thursday, March 15, 2012

 Clumping

Thursday, March 15, 2012

 Hair Shading

System supports thin geometric hair

or wide textured strips with images of multiple hairs
8x multisample anti-aliasing works surprisingly well
use Alpha-To-Coverage with added random dithering
gives order independent transparency without blending
transparency super-sampling using shader coverage output would also
possible, but expensive
shader includes fading at root and tip
Use Kajiya-Kay anisotropic shading model
supports self-shadowing
fake rim-lighting effect

Thursday, March 15, 2012

 Hair UI

Thursday, March 15, 2012

 Not just for hair!

Thursday, March 15, 2012

Supports fur mode, doesnt use guide hairs, fur generated based on mesh tangent and normal.
 Explosions

Explosions are a common element in modern video games!

Thursday, March 15, 2012

 Inspiration FumeFX

Thursday, March 15, 2012

 Explosion Approaches

Particle system
Can look good, especially with pre-rendered sprites
But hard to get fireball look
Sprites can look flat, especially with a moving camera
Fluid simulation
Possible in DirectX 11, but expensive
Hard for artists to control
Procedural techniques
Practical on todays PC hardware

Thursday, March 15, 2012

 Great Presentation from GDC 2005

Thursday, March 15, 2012

 !!

Thursday, March 15, 2012

 Distance Fields

Aka Level Sets

Popular in demo scene
Store distance to nearest surface

Advantages
Can render very quickly using sphere tracing
ray marching using distance information to take larger steps
Can calculate normal as gradient of distance field
Nice ambient occlusion approximations

Thursday, March 15, 2012

 Volumetric Explosions

Basic shape defined using spheres

Spheres can be animated in Unity
Detail added using pyroclastic noise
Simply offset distance from surface using Perlin noise at position
Noise stored in small 3D texture (64^3). Use 4-6 octaves
Shade based on displacement amount
Rendered using hybrid distance field / volume renderer
Sphere trace to surface
32 steps often enough
Then ray-march through volume
~16 steps

Thursday, March 15, 2012

 Start with some spheres

Thursday, March 15, 2012

 Add Displacement (Pyroclastic Noise)

Thursday, March 15, 2012

 2 Octaves

Thursday, March 15, 2012

 4 Octaves

Thursday, March 15, 2012

 Shade Based On Displacement Distance

Thursday, March 15, 2012

 Add Color Gradient

Thursday, March 15, 2012

 Add some lighting

Thursday, March 15, 2012

 Add Volume Rendering

Thursday, March 15, 2012

 Smoke Stack

Thursday, March 15, 2012

 Smoke Stack

Thursday, March 15, 2012

 Smoke Stack

Thursday, March 15, 2012

 Smoke Stack

Thursday, March 15, 2012

 Another Example

Thursday, March 15, 2012

 Motion Blur

Thursday, March 15, 2012

 Motion Blur

Improves readability in fast moving scenes

Gives cinematic look
maybe this will change with movies moving to 48 fps and higher?

Unity has an existing motion blur image effect

but not really motion blur
just leaves trails behind objects by blending new frames on top of old

We wanted to implement a modern velocity-based motion blur in

Unity

Thursday, March 15, 2012

 Velocity Buffer Motion Blur

Generate velocity buffer

Shader calculates difference between previous and current positions in
screen-space
Unity now provides
previous model matrix (UNITY_MATRIX_PREV_M)
Previous world space position for skinned objects (POSITION1)
Camera motion blur calculated from depth buffer and previous model
view projection matrix
Have to be careful about motion blur on first frame of camera cuts!

Use these velocities to blur an image of the current scene in

direction of motion

Thursday, March 15, 2012

 Reconstruction Filter Motion Blur

A Reconstruction Filter for Plausible Motion Blur

McGuire et al I3D 2012
Uses current frame image and depth
Plus screen-space velocity buffer
Simulates scatter as gather
accounts for occlusion using depth
blurs correctly outside object silhouettes
Still has some artifacts
Especially at tile boundaries

Thursday, March 15, 2012

 Velocity Buffer Visualization

Thursday, March 15, 2012

 Motion Blur - Off

Thursday, March 15, 2012

 Simple Motion Blur

Thursday, March 15, 2012

 Reconstruction Filter Motion Blur

Thursday, March 15, 2012

 Things I Like About Unity

Nice editor
Very customizable
Very fast to prototype new effects
Automatically generates sliders for shader parameters
Shader parameters can be animated over time using curves
C# is fine once you get used to it!
like Java without the stupid bits
Helpful developers and community

Thursday, March 15, 2012

 Example: Gradient Editor script

Thursday, March 15, 2012

 Conclusion

DirectX 11 brings state-of-the-art graphics to Unity

Techniques from the off-line CG world are becoming possible in
real-time
Aim to make Unity as artist-friendly as possible

Thursday, March 15, 2012

 Thanks

Everyone at Passion Pictures

Quentin Vien
Jamie Franks
Julian Hodgson
Erland Krner
Aras Pranckeviius
Ole Ciliox
Robert and Kuba Cupisz

Thursday, March 15, 2012

 Questions?

Thursday, March 15, 2012

 References

Sphere Tracing: A Geometric Method for the Antialiased Ray

Tracing of Implicit Surfaces
John C. Hart, 1994
Volume Rendering for Games
Simon Green GDC 2005
Rendering Worlds with Two Triangles
Iigo Quilez, NVSCENE 08
Production Volume Rendering
Magnus Wrenninge et al, Siggraph 2011

Thursday, March 15, 2012

 References (cont.)

Stupid OpenGL Shader Tricks

Simon Green, GDC 2003
A Reconstruction Filter for Plausible Motion Blur
Morgan McGuire, Padraic Hennessy, Michael Bukowski, Brian Osman, I3D
2012

Thursday, March 15, 2012