Vision Review: Image Processing

Course web page: www.cis.udel.edu/~cer/arv

September 17, 2002

Announcements
Homework and paper presentation guidelines are up on web page Readings for next Tuesday: Chapters 6, 11.1, and 18 For next Thursday: Stochastic Road Shape Estimation

Computer Vision Review Outline

Image formation Image processing Motion & Estimation Classification

Outline
Images Binary operators Filtering
Smoothing Edge, corner detection

Modeling, matching Scale space

Images
An image is a matrix of pixels Note: Matlab uses Resolution
Digital cameras: 1600 X 1200 at a minimum Video cameras: ~640 X 480

Grayscale: generally 8 bits per pixel Intensities in range [0255] RGB color: 3 8-bit color planes

Image Conversion
RGB Grayscale: Mean color value, or weight by perceptual importance (Matlab: rgb2gray)

Grayscale Binary: Choose threshold based on histogram of image intensities (Matlab: imhist)

Color Representation
RGB, HSV (hue, saturation, value), YUV, etc. Luminance: Perceived intensity Chrominance: Perceived color
HS(V), (Y)UV, etc. Normalized RGB removes some illumination dependence:

Binary Operations
Dilation, erosion (Matlab: imdilate, imerode)
Dilation: All 0s next to a 1 1 (Enlarge foreground) Erosion: All 1s next to a 0 0 (Enlarge background)

Connected components
Uniquely label each n-connected region in binary image 4- and 8-connectedness Matlab: bwfill, bwselect

Moments: Region statistics

Zeroth-order: Size First-order: Position (centroid) Second-order: Orientation

Image Transformations
Geometric: Compute new pixel locations
Rotate Scale Undistort (e.g., radial distortion from lens)

Photometric: How to compute new pixel values when non-integral

Nearest neighbor: Value of closest pixel Bilinear interpolation (2 x 2 neighborhood) Bicubic interpolation (4 x 4)

Bilinear Interpolation
Idea: Blend four pixel values surrounding source, weighted by nearness

Vertical blend

Horizontal blend

Image Comparison: SSD

Given a template image and an image , how to quantify the similarity between them for a given alignment? Sum of squared differences (SSD)

Cross-Correlation for Template Matching

Note that SSD formula can be written:

First two terms fixed last term measures mismatchthe cross-correlation:

In practice, normalize by image magnitude when shifting template to search for matches

Filtering
Idea: Analyze neighborhood around some point in image with filter function ; put result in new image at corresponding location System properties
Shift invariance: Same inputs give same outputs, regardless of location Superposition: Output on sum of images = Sum of outputs on separate images Scaling: Output on scaled image = Scaled output on image

Linear shift invariance Convolution

Convolution
Definition:

Shorter notation: Properties

Commutative Associative

Fourier theorem: Convolution in spatial domain = Multiplication in frequency domain

More on Fourier transforms on Thursday

Discrete Filtering
1 -1 -1 Linear filter: Weighted sum of pixels over rectangular neighborhoodkernel defines 1 2 -1 1 1 1 weights
Think of kernel as template being matched by correlation (Matlab: imfilter, filter2) Convolution: Correlation with kernel rotated 180
Matlab: conv2

Dealing with image edges

Zero-padding Border replication

Filtering Example 1:
1 -1 -1
1 1
Rotate

2 -1 1 1

2 2 2 1

2 1 2 3

2 3 1 2

3 3 2 2

-1 2

-1 -1 1

1 1

2 1 2 3

2 3 1 2

3 3 2 2

-1 2

-1 -1 1 1 1 1 2 2 3 2 3 1 2 3 3 2 2

Step 1

2 2 1

2 -1 4 2 1

2 -1 -2 1

1 1

2 1 2 3

2 3 1 2

3 3 2 2

-1 2

-1 -1 1 1 1 1 2 1 2 3 3 2 2

Step 2

2 2 1

2 2 -2 4 2 1 2 3

2 1 -2 -1 3

1 1

2 1 2 3

2 3 1 2

3 3 2 2

-1 2

-1 -1 1 1 1 1 3 2 2

Step 3

2 2 1

2 -2 4 2 2 2 1 2 3 1 2

2 -1 -3 3 1 3

1 1

2 1 2 3

2 3 1 2

3 3 2 2

-1 2

-1 -1 1 1 2 2 2 1 2 1 2 3 1

Step 4
1 1 5 4

2 2 1

2 6 -2 3 1 2 2 2

4 -2

3 3 -3 -3 1

1 1

2 1 2 3

2 3 1 2

3 3 2 2

-1 2

-1 -1 1

Step 5

2 2 1

2 1 3

2 3 1 2

3 3 2 2

5 9

4 -2

-1 4 2 1

-1 -2 2 2

1 1

2 1 2 3

2 3 1 2

3 3 2 2

-1 2

-1 -1 1

Step 6

2 2 1

2 3 2

3 3 2 2

5 9

4 6

4 -2

-2 2 2 1 1 3

2 2 -2 -2 1

Final Result
5 4 4 -2 6 5 5

6 14 5

11 7

9 12 8

Separability
Definition: 2-D kernel can be written as convolution of two 1-D kernels Advantage: EfficiencyConvolving image with kernel requires multiplies vs. for nonseparable kernel

Smoothing (Low-Pass) Filters

Replace each pixel with average of neighbors Benefits: Suppress noise, aliasing Disadvantage: Sharp features blurred Types 1 1 1
Mean filter (box) Median (nonlinear) Gaussian 1 1 1

3 x 3 box filter

Box Filter: Smoothing

Original image

7 x 7 kernel

Gaussian Kernel
Idea: Weight contributions of neighboring pixels by nearness

Matlab: fspecial(gaussian,)

Gaussian: Benefits
Rotational symmetry treats features of all orientations equally (isotropy) Smooth roll-off reduces ringing Efficient: Rule of thumb is kernel width 5
Separable Cascadable: Approach to large comes from identity

Gaussian: Smoothing

Original image

7x7 kernel =1 =3

Gradient
Think of image intensities as a function . Gradient of image is a vector field as for a normal 2-D height function:

Edge: Place where gradient magnitude is high; orthogonal to gradient direction

Edge Causes
Depth discontinuity Surface orientation discontinuity Reflectance discontinuity (i.e., change in surface material properties) Illumination discontinuity (e.g., shadow)

Edge Detection
Edge Types
Step edge (ramp) Line edge (roof)

Searching for Edges:

Filter: Smooth image Enhance: Apply numerical derivative approximation Detect: Threshold to find strong edges Localize/analyze: Reject spurious edges, include weak but justified edges

Step edge detection

First derivative edge detectors: Look for extrema Sobel operator 1 0 -1 1 2 1 (Matlab: edge(I, sobel)) 2 0 -2 0 0 0 Prewitt, Roberts cross 1 0 -1 -1 -2 -1
Derivative of Gaussian
Sobel x Sobel y

Second derivative: Look for zero-crossings

Laplacian : Isotropic Second directional derivative Laplacian of Gaussian/Difference of Gaussians

Derivative of Gaussian

Laplacian of Gaussian

Matlab: fspecial(log,)

Edge Filtering Example

1
2 1
Rotate

0 -1
0 -2 0 -1 0 0 0 1 0 0 0 0 0 2 2 2 2 2 2 2 2

-1 0

-2 0
-1 0

2
1

-1 0 -2 0 -1 0

1 2 1 1 2 0 1 0 0 0 2 2 1 2 2 2 2 2 0

0 0 0 0

2 2 2 2

2 2 2 2

Step 1

0 0 0

-1 0 2 -2 0 2 -1 0 0 0

-1 0 -2 0 -1 0

1 2 1 -1 0 2 0 2 0 0 0 2 0 1 0 0 0 1 2 4 3 2 2 2 2 2 2 2 0 6

0 0 0 0

2 2 2 2

2 2 2 2

Step 2

0 0 0

-1 0 -2 0 -1 0

1 2 1 -1 0 0 0 0 0 2 0 1 0 0 0 2 0 3 0 2 2 1 3 4 3 2 2 2 0 6

0 0 0 0

2 2 2 2

2 2 2 2

Step 3

0 0 0 6

-1 0 -2 0 -1 0

1 2 1 -1 0 0 0 0 0 0 0 0 0 2 0 -4 3 3 0 -2 3 2 2 2 2

0 0 0 0

2 2 2 2

2 2 2 2

Step 4
1 2 1 0 6

0 0 0

6 -6
edge effect

Sobel Edge Detection: Gradient Approximation

Horizontal

Vertical

Sobel vs. LoG Edge Detection: Matlab Automatic Thresholds

Sobel

LoG

Canny Edge Detection

Derivative of Gaussian Non-maximum suppression
Thin multi-pixel wide ridges down to single pixel

Thresholding
Low, high edge-strength thresholds Accept all edges over low threshold that are connected to edge over high threshold

Matlab: edge(I, canny)

Canny Edge Detection: Example

(Matlab automatically set thresholds)

Corner Detection
Basic idea: Find points where two edges meeti.e., high gradient in orthogonal directions Examine gradient over window (Shi & Tomasi, 1994)

Edge strength encoded by eigenvalues ; corner is where over threshold Harris corners (Harris & Stephens, 1988), Susan corners (Smith & Brady, 1997)

Example: Corner Detection

courtesy of S. Smith

SUSAN corners

Edge-Based Image Comparison

Chamfer, Hausdorff distance, etc.
Transform edge map based on distance to nearest edge before correlating as usual

courtesy of D. Gavrila

Scale Space
How thick an edge? How big a dot? Must consider what scale we are interested in when designing filters Efficiency a major consideration: Finegrained template matching is expensive over a full image

Image Pyramids
Idea: Represent image at different scales, allowing efficient coarse-to-fine search Downsampling: Simplest scale change: Decimationjust downsample

from Forsyth & Ponce

Gaussian, Laplacian Pyramids

Gaussian pyramid of image: and

Laplacian pyramid
Difference of image and Gaussian at each level of

courtesy of Wolfram

Gaussian pyramid

Laplacian pyramid

Color-based Image Comparison

Color histograms (Swain & Ballard, 1991)
Steps
Histogram RGB/HSV triplets over two images to be compared Normalize each histogram by respective total number of pixels to get frequencies Similarity is Euclidean distance between color frequency vectors

Sensitive to lighting changes Works for different-sized images Matlab: imhist, hist