Stereo Vision

- Applications
Lu Yang
Email: [email protected]
School of Automation Engineering, UESTC
Stereo Matching
Given two or more images of the same scene or
object, compute a representation of its shape
?

What are some possible applications?

2
Face modeling
From one stereo pair to a 3D head model

[Frederic Deverney, INRIA]

3
Z-keying: mix live and synthetic
Takeo Kanade, CMU (Stereo Machine)

4
View Interpolation
Given two images with correspondences,
morph (warp and cross-dissolve) between
them [Chen & Williams, SIGGRAPH’93]

input depth image novel view

[Matthies,Szeliski,Kanade’88]

5
More view interpolation
Spline-based depth map

input depth image novel view

[Szeliski & Kang ‘95]

6
View Morphing
Morph between pair of images using epipolar
geometry [Seitz & Dyer, SIGGRAPH’96]

7
Video view interpolation

8
Virtualized RealityTM
[Takeo Kanade et al., CMU]
• collect video from 50+ stream
• reconstruct 3D model sequences

• steerable version used for

SuperBowl XXV “eye vision”
http://www.cs.cmu.edu/afs/cs/project/VirtualizedR/www/VirtualizedR.html

9
Real-time stereo

Nomad robot searches for meteorites in Antartica

http://www.frc.ri.cmu.edu/projects/meteorobot/index.html

Used for robot navigation (and other tasks)

• Software-based real-time stereo techniques
10
Additional applications
• Real-time people tracking (systems from Pt.
Gray Research and SRI)
• “Gaze” correction for video conferencing
[Ott,Lewis,Cox InterChi’93]
• Other ideas?

11
Stereo Matching
Given two or more images of the same scene or
object, compute a representation of its shape

What are some possible representations?

• depth maps
• volumetric models
• 3D surface models
• planar (or offset) layers

12
Stereo Matching
What are some possible algorithms?
• match “features” and interpolate
• match edges and interpolate
• match all pixels with windows (coarse-fine)
• use optimization:
– iterative updating
– dynamic programming
– energy minimization (regularization, stochastic)
– graph algorithms

13
Outline (remainder of lecture)
Image rectification
Matching criteria
Local algorithms (aggregation)
• iterative updating
Optimization algorithms:
• energy (cost) formulation & Markov Random Fields
• mean-field, stochastic, and graph algorithms
Multi-View stereo & occlusions

14
Stereo: epipolar geometry
Match features along epipolar lines

epipolar line

epipolar plane viewing ray

15
Stereo image pair

16
Anaglyphs

http://www.rainbowsymph
ony.com/freestuff.html

(Wikipedia for images)

Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923

17
Stereo: epipolar geometry
for two images (or images with collinear camera
centers), can find epipolar lines
epipolar lines are the projection of the pencil of
planes passing through the centers

Rectification: warping the input images

(perspective transformation) so that epipolar
lines are horizontal

18
Rectification
Project each image onto same plane, which is parallel
to the epipole
Resample lines (and shear/stretch) to place lines in
correspondence, and minimize distortion

[Loop and Zhang, CVPR’99]

19
Rectification

BAD!

20
Rectification

GOOD!

21
Finding correspondences
apply feature matching criterion (e.g.,
correlation or Lucas-Kanade) at all pixels
simultaneously
search only over epipolar lines (many fewer
candidate positions)

22
Your basic stereo algorithm

For each epipolar line

For each pixel in the left image
• compare with every pixel on same epipolar line in right image
• pick pixel with minimum match cost
Improvement: match windows
• This should look familar...
23
Image registration (revisited)
How do we determine correspondences?
• block matching or SSD (sum squared differences)

d is the disparity (horizontal motion)

How big should the neighborhood be?

24
Neighborhood size
Smaller neighborhood: more details
Larger neighborhood: fewer isolated mistakes

w=3 w = 20

25
Matching criteria
Raw pixel values (correlation)
Band-pass filtered images [Jones & Malik 92]
“Corner” like features [Zhang, …]
Edges [many people…]
Gradients [Seitz 89; Scharstein 94]
Rank statistics [Zabih & Woodfill 94]

26
Stereo: certainty modeling
Compute certainty map from correlations

input depth map certainty map

27
Plane Sweep Stereo
Sweep family of planes through volume
 projective re-sampling of (X,Y,Z)

input image
composite

virtual camera

• each plane defines an image  composite homography

28
Plane Sweep Stereo
For each depth plane
• compute composite (mosaic) image — mean

• compute error image — variance

• convert to confidence and aggregate spatially
Select winning depth at each pixel

29
Plane sweep stereo
Re-order (pixel / disparity) evaluation loops

for every pixel, for every disparity

for every disparity for every pixel
compute cost compute cost
30
Stereo matching framework
1. For every disparity, compute raw matching
costs

Why use a robust function?

• occlusions, other outliers

Can also use alternative match criteria

31
Stereo matching framework
2. Aggregate costs spatially

• Here, we are using a box filter

(efficient moving average
implementation)
• Can also use weighted average,
[non-linear] diffusion…

32
Stereo matching framework
3. Choose winning disparity at each pixel

4. Interpolate to sub-pixel accuracy

E(d)

d* d

33
Traditional Stereo Matching
Advantages:
• gives detailed surface estimates
• fast algorithms based on moving averages
• sub-pixel disparity estimates and confidence
Limitations:
• narrow baseline  noisy estimates
• fails in textureless areas
• gets confused near occlusion boundaries

34
Stereo with Non-Linear Diffusion
Problem with traditional approach:
• gets confused near discontinuities
New approach:
• use iterative (non-linear) aggregation to obtain
better estimate
• provably equivalent to mean-field estimate of
Markov Random Field

35
Linear diffusion
Average energy with neighbors + starting value

window diffusion

36
Feature-based stereo
Match “corner” (interest) points

Interpolate complete solution

37
Data interpolation
Given a sparse set of 3D points, how do we
interpolate to a full 3D surface?
Scattered data interpolation [Nielson93]
• triangulate
• put onto a grid and fill (use pyramid?)
• place a kernel function over each data point
• minimize an energy function

38
Energy minimization
1-D example: approximating splines

dx,y

zx,y

39
Relaxation
How can we get the best solution?
Differentiate energy function, set to 0

40
Relaxation
Iteratively improve a solution by locally
minimizing the energy: relax to solution

zx,y

dx+1,y dx,y dx+1,y

Earliest application: WWII numerical simulations

41
Dynamic programming
Evaluate best cumulative cost at each pixel

0 0 0 0 0

1 1 1 1 1

42
Dynamic programming
1-D cost function

Stero matching 43
Dynamic programming
Disparity space image and min. cost path

44
Dynamic programming
Sample result
(note horizontal
streaks)

[Intille & Bobick]

45
Dynamic programming
Can we apply this trick in 2D as well?

dx-1,y dx,y

dx-1,y-1 dx,y-1

No: dx,y-1 and dx-1,y may depend on different values of dx-1,y-1

46
Graph cuts
Solution technique for general 2D problem

47
Graph cuts
a-b swap
a expansion
modify smoothness penalty based on edges
compute best possible match within integer
disparity

48
Graph cuts
Two different kinds of moves:

49
Bayesian inference
Formulate as statistical inference problem
Prior model pP(d)
Measurement model pM(IL, IR| d)
Posterior model
pM(d | IL, IR)  pP(d) pM(IL, IR| d)
Maximum a Posteriori (MAP estimate):
maximize pM(d | IL, IR)

50
Markov Random Field
Probability distribution on disparity field d(x,y)

Enforces smoothness or coherence on field

51
Measurement model
Likelihood of intensity correspondence

Corresponds to Gaussian noise for quadratic r

52
MAP estimate
Maximize posterior likelihood

Equivalent to regularization (energy

minimization with smoothness constraints)

53
Why Bayesian estimation?
Principled way of determining cost function
Explicit model of noise and prior knowledge
Admits a wider variety of optimization
algorithms:
• gradient descent (local minimization)
• stochastic optimization (Gibbs Sampler)
• mean-field optimization
• graph theoretic (actually deterministic) [Zabih]
• [loopy] belief propagation
• large stochastic flips [Swendsen-Wang]
54
Depth Map Results

Input image Sum Abs Diff

Mean field Graph cuts

55
Stereo evaluation

56
Stereo—best algorithms

CSE 576, Spring 2008 Stereo matching 57

Traditional stereo
Advantages:
• works very well in non-occluded regions
Disadvantages:
• restricted to two images (not)
• gets confused in occluded regions
• can’t handle mixed pixels

58
Stereo Reconstruction
X
Steps
• Calibrate cameras
• Rectify images
z
• Compute disparity u u’
• Estimate depth
f f
C baseline C’

59
 Choosing the Baseline

Large Baseline Small Baseline

What’s the optimal baseline?
• Too small: large depth error
• Too large: difficult search problem
60
Effect of Baseline on Estimation

61
62
Multibaseline Stereo
Basic Approach
• Choose a reference view
• Use your favorite stereo algorithm BUT
– replace two-view SSD with SSD over all baselines

Limitations
• Must choose a reference view
• Visibility: select which frames to match
[Kang, Szeliski, Chai, CVPR’01]

63
 Epipolar-Plane Images [Bolles 87]
http://www.graphics.lcs.mit.edu/~aisaksen/projects/drlf/epi/

Lesson: Beware of occlusions

64
 Active stereo with structured light

Li Zhang’s one-shot stereo

camera 1 camera 1

projector projector

camera 2
Project “structured” light patterns onto the object
• simplifies the correspondence problem
65
 Li Zhang, Noah Snavely,
Brian Curless, Steve Seitz
Spacetime Stereo

http://grail.cs.washington.edu/projects/stfaces/
66
Summary
Applications
Image rectification
Matching criteria
Local algorithms (aggregation & diffusion)
Optimization algorithms
• energy (cost) formulation & Markov Random Fields
• mean-field; dynamic programming; stochastic;
graph algorithms
Multi-View stereo
• visibility, occlusion-ordered sweeps
67
Bibliography
D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-
frame stereo correspondence algorithms. International Journal of
Computer Vision, 47(1):7-42, May 2002.
R. Szeliski. Stereo algorithms and representations for image-based
rendering. In British Machine Vision Conference (BMVC'99), volume 2,
pages 314-328, Nottingham, England, September 1999.
G. M. Nielson, Scattered Data Modeling, IEEE Computer Graphics and
Applications, 13(1), January 1993, pp. 60-70.
S. B. Kang, R. Szeliski, and J. Chai. Handling occlusions in dense multi-
view stereo. In CVPR'2001, vol. I, pages 103-110, December 2001.
Y. Boykov, O. Veksler, and Ramin Zabih, Fast Approximate Energy
Minimization via Graph Cuts, Unpublished manuscript, 2000.
A.F. Bobick and S.S. Intille. Large occlusion stereo. International Journal of
Computer Vision, 33(3), September 1999. pp. 181-200
D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion.
International Journal of Computer Vision, 28(2):155-174, July 1998
68
 Bibliography
Volume Intersection
• Martin & Aggarwal, “Volumetric description of objects from multiple views”, Trans.
Pattern Analysis and Machine Intelligence, 5(2), 1991, pp. 150-158.
• Szeliski, “Rapid Octree Construction from Image Sequences”, Computer Vision,
Graphics, and Image Processing: Image Understanding, 58(1), 1993, pp. 23-32.

Voxel Coloring and Space Carving

• Seitz & Dyer, “Photorealistic Scene Reconstruction by Voxel Coloring”, Proc.
Computer Vision and Pattern Recognition (CVPR), 1997, pp. 1067-1073.
• Seitz & Kutulakos, “Plenoptic Image Editing”, Proc. Int. Conf. on Computer Vision
(ICCV), 1998, pp. 17-24.
• Kutulakos & Seitz, “A Theory of Shape by Space Carving”, Proc. ICCV, 1998, pp.
307-314.

69
 Bibliography
Related References
• Bolles, Baker, and Marimont, “Epipolar-Plane Image Analysis: An Approach to
Determining Structure from Motion”, International Journal of Computer Vision, vol 1,
no 1, 1987, pp. 7-55.
• Faugeras & Keriven, “Variational principles, surface evolution, PDE's, level set
methods and the stereo problem", IEEE Trans. on Image Processing, 7(3), 1998, pp.
336-344.
• Szeliski & Golland, “Stereo Matching with Transparency and Matting”, Proc. Int. Conf.
on Computer Vision (ICCV), 1998, 517-524.
• Roy & Cox, “A Maximum-Flow Formulation of the N-camera Stereo Correspondence
Problem”, Proc. ICCV, 1998, pp. 492-499.
• Fua & Leclerc, “Object-centered surface reconstruction: Combining multi-image
stereo and shading", International Journal of Computer Vision, 16, 1995, pp. 35-56.
• Narayanan, Rander, & Kanade, “Constructing Virtual Worlds Using Dense Stereo”,
Proc. ICCV, 1998, pp. 3-10.

70