What is image filtering?

Modify the pixels in an image based on

some function of a local neighborhood of
Image Filtering the pixels.
&
Edge Detection Some function

Reading:
Chapter 7 and 8, F&P

Linear Functions
Simplest: linear filtering.
Replace each pixel by a linear combination of its
neighbors.
The prescription for the linear combination is
called the convolution kernel.

Convolution Key properties

Let I be the image and g be the kernel. The Linearity:
output of convolving I with g is denoted I g filter(I1 + I2 ) = filter(I1) + filter(I2)
Shift invariance:
f [ m, n] = I g = I[ m k , n l ] g[ k , l ] same behavior regardless of pixel location
k ,l filter(shift(I)) = shift(filter(I))
Theoretical result:
Any linear shift-invariant operator can be
represented as a convolution
I

1
Properties in more detail
Commutative: a * b = b * a
Conceptually no difference between filter and signal
Associative: a * (b * c) = (a * b) * c
Often apply several filters one after another: (((a * b1) * b2) * b3)
This is equivalent to applying one filter: a * (b1 * b2 * b3)
Distributes over addition: a * (b + c) = (a * b) + (a * c)
Scalars factor out: ka * b = a * kb = k (a * b)
Identity: unit impulse e = [, 0, 0, 1, 0, 0, ],
a*e=a

2
 Yucky details
What is the size of the output?
MATLAB: filter2(g, I, shape)
shape = full: output size is sum of sizes of I and g
shape = same: output size is same as I
shape = valid:output size is of difference sizes for I & g
full same valid
g g g g
g g

I I I

g g g
g
g g

Implementation details Implementation details

What about near the edge? What about near the edge?
the filter window falls off the edge of the image the filter window falls off the edge of the image
need to extrapolate need to extrapolate
methods: methods (MATLAB):
clip filter (black) clip filter (black): imfilter(f, g, 0)
wrap around wrap around: imfilter(f, g, circular)
copy edge copy edge: imfilter(f, g, replicate)
reflect across edge reflect across edge: imfilter(f, g, symmetric)

Source: S. Marschner Source: S. Marschner

3
Linear filtering (warm-up slide) Linear filtering (warm-up slide)

coefficient

coefficient
1.0
? 1.0

0 0
Pixel offset Pixel offset
original original Filtered
0 0 0 (no change)
0 1 0
0 0 0

Linear filtering Shift

coefficient

coefficient
1.0 1.0

0
? 0
Pixel offset Pixel offset
original original shifted
0 0 0
0 0 1
0 0 0

Linear filtering Blurring

coefficient

coefficient

0.3

0
? 0.3

0
Pixel offset Pixel offset
original original Blurred (filter
1 1 1 applied in both
Box filter: 1 1 1 dimensions).
1 1 1

4
 Blur examples Blur examples

coefficient

coefficient
8 2.4 8 2.4
impulse impulse
0.3 0.3

0 0
original Pixel offset filtered original Pixel offset filtered

coefficient
8 8
edge 4 4
0.3

0
original Pixel offset filtered

Smoothing with box filter revisited Smoothing with box filter revisited
Smoothing with an average actually doesnt Smoothing with an average actually doesnt compare at
compare at all well with a defocused lens all well with a defocused lens
Most obvious difference is that a single point of Most obvious difference is that a single point of light
light viewed in a defocused lens looks like a fuzzy viewed in a defocused lens looks like a fuzzy blob; but
blob; but the averaging process would give a little
square the averaging process would give a little square
Better idea: to eliminate edge effects, weight
contribution of neighborhood pixels according to their
closeness to the center, like so:

Source: D. Forsyth fuzzy blob

Gaussian Kernel Choosing kernel width

Gaussian filters have infinite support, but
discrete filters use finite kernels

0.003 0.013 0.022 0.013 0.003

0.013 0.059 0.097 0.059 0.013
0.022 0.097 0.159 0.097 0.022
0.013 0.059 0.097 0.059 0.013
0.003 0.013 0.022 0.013 0.003

5 x 5, = 1

Constant factor at front makes volume sum to 1 (can be

ignored, as we should re-normalize weights to sum to 1
in any case)
Source: C. Rasmussen Source: K. Grauman

5
Gaussian filtering Example: Smoothing with a
A Gaussian kernel gives less weight to pixels further Gaussian
from the center of the window
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0 1 2 1
0 0 0 90 90 90 90 90 0 0 2 4 2
0 0 0 90 0 90 90 90 0 0 1 2 1
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0

This kernel is an approximation of a Gaussian function:

Mean vs. Gaussian filtering Separability of the Gaussian filter

Source: D. Lowe

Separability example Gaussian filters

Remove high-frequency components from the image
2D convolution (low-pass filter)
(center location only)
Convolution with self is another Gaussian
So can smooth with small-width kernel, repeat, and get same
The filter factors result as larger-width kernel would have
into a product of 1D
Convolving two times with Gaussian kernel of width is same
filters:
as convolving once with kernel of width sqrt(2)
Separable kernel
Perform convolution Factors into product of two 1D Gaussians
along rows: * =

Useful: can convolve all rows, then all columns

How does this change the computational complexity?
Followed by convolution Linear vs. quadratic in mask size
* =
along the remaining column:

Source: K. Grauman Source: K. Grauman

6
Review: Linear filtering Noise
What are the defining mathematical properties Salt and pepper
of a convolution? noise: contains
random occurrences
What is the difference between blurring with a of black and white
box filter and blurring with a Gaussian? pixels
Impulse noise:
What happens when we convolve a Gaussian Original Salt and pepper noise contains random
with another Gaussian? occurrences of white
What is separability? pixels
Gaussian noise:
How does separability affect computational variations in intensity
complexity? drawn from a
Gaussian normal
distribution
Impulse noise Gaussian noise
Source: S. Seitz

Gaussian noise Reducing Gaussian noise

Mathematical model: sum of many independent
factors
Good for small standard deviations
Assumption: independent, zero-mean noise

Smoothing with larger standard deviations suppresses noise,

but also blurs the image
Source: K. Grauman

Reducing salt-and-pepper noise Alternative idea: Median filtering

3x3 5x5 7x7 A median filter operates over a window by
selecting the median intensity in the window

Whats wrong with the results?

Is median filtering linear?

No. Not a convolution Source: K. Grauman

7
Median filter Median filter
Salt-and-pepper noise Median filtered
What advantage does median filtering have
over Gaussian filtering?
Robustness to outliers

MATLAB: medfilt2(image, [h w])

Source: K. Grauman Source: K. Grauman

Median vs. Gaussian filtering Linear filtering (warm-up slide)

3x3 5x5 7x7

2.0
Gaussian
1.0

0 0
?
original
Median

Linear filtering (no change) Linear filtering

2.0 2.0

1.0

0 0 0
0.33

0
?
original Filtered original
(no change)

8
(Remember blurring) Sharpening

2.0
coefficient

0.3 0.33

0 0
0
Pixel offset
original Sharpened
original Blurred (filter
original
applied in both
dimensions).

Sharpening Sharpening example

1.7
11.2

coefficient
8 8

-0.25
-0.3
original Sharpened
(differences are
accentuated; constant
areas are left untouched).

Sharpening Sharpening

0 0 0 1 1 1 0 0 0 1 1 1
0 2 0
0 0 0
- 1 1 1
1 1 1
? 0 2 0
0 0 0
- 1 1 1
1 1 1

(Note that filter sums to 1)

Original Original Sharpening filter
- Accentuates differences
with local average

Source: D. Lowe Source: D. Lowe

9
Sharpening Unsharp mask filter
I + ( I I g ) = ( 1 + ) I I g = I (( 1 + ) e g )

image blurred unit impulse

image (identity)

before after
unit impulse
Gaussian Laplacian of Gaussian

Sharpening Revisited Edge detection

What does blurring take away? Goal: Identify sudden
changes (discontinuities)
in an image
=
Intuitively, most semantic
and shape information from
original smoothed (5x5) detail
the image can be encoded in
Lets add it back: the edges
More compact than pixels

+ = Ideal: artists line drawing

(but artist is also using
original detail sharpened object-level knowledge)
Source: D. Lowe

Origin of Edges Characterizing edges

An edge is a place of rapid change in the image
surface normal discontinuity intensity function
depth discontinuity intensity function
image (along horizontal scanline) first derivative
surface color discontinuity

illumination discontinuity

Edges are caused by a variety of factors

edges correspond to
extrema of derivative
Source: Steve Seitz

10
Image gradient Differentiation and convolution
The gradient of an image: Recall, for 2D We could
function, f(x,y): approximate this as
The gradient points in the direction of most rapid change in intensity f f (x + , y) f (x, y ) f f (xn+1 , y ) f (xn , y)
= lim
x 0 x x

This is linear and shift (which is obviously a

The gradient direction is given by: invariant, so must be convolution)
the result of a
-1 1
how does this relate to the direction of the edge? perpendicular
convolution.
The edge strength is given by the gradient magnitude

Source: D. Forsyth, D. Lowe

Finite differences: example Finite difference filters

Other approximations of derivative filters exist:

Which one is the gradient in the x-direction

(resp. y-direction)? Source: K. Grauman

Effects of noise Effects of noise

Consider a single row or column of the image Finite difference filters respond strongly to noise
Plotting intensity as a function of position gives a Image noise results in pixels that look very different
signal from their neighbors
Generally, the larger the noise the stronger the
response
What is to be done?
How to compute a derivative?

Where is the edge? Source: D. Forsyth

11
Effects of noise Solution: smooth first
Finite difference filters respond strongly to noise
Image noise results in pixels that look very different
from their neighbors
Generally, the larger the noise the stronger the
response
What is to be done?
Smoothing the image should help, by forcing pixels
different to their neighbors (=noise pixels?) to look
more like neighbors

Source: D. Forsyth Where is the edge? Look for peaks in

Derivative theorem of convolution Derivative of Gaussian filter

Differentiation is convolution, and convolution is
associative: d d
( f g) = f g
dx dx

* [1 -1] =
This saves us one operation:

d
g
dx

d
f g
dx
Source: S. Seitz

Derivative of Gaussian filter Summary: Filter mask properties

Filters act as templates
Highest response for regions that look the most like
the filter
Dot product as correlation
Smoothing masks
Values positive
x-direction y-direction Sum to 1 constant regions are unchanged
Amount of smoothing proportional to mask size
Derivative masks
Opposite signs used to get high response in regions of
high contrast
Sum to 0 no response in constant regions
Which one finds horizontal/vertical edges? High absolute value at points of high contrast
Source: K. Grauman

12
 Tradeoff between smoothing and Implementation issues
localization

The gradient magnitude is large along a thick

1 pixel 3 pixels 7 pixels trail or ridge, so how do we identify the
Smoothed derivative removes noise, but blurs actual edge points?
edge. Also finds edges at different scales. How do we link the edge points to form curves?
Source: D. Forsyth Source: D. Forsyth

Laplacian of Gaussian 2D edge detection filters

Consider
Laplacian of Gaussian

Gaussian derivative of Gaussian

Laplacian of Gaussian
operator

is the Laplacian operator:

Where is the edge? Zero-crossings of bottom graph

MATLAB demo Edge finding

g = fspecial('gaussian',15,2);
imagesc(g)
surfl(g)
gclown = conv2(clown,g,'same');
imagesc(conv2(clown,[-1 1],'same'));
imagesc(conv2(gclown,[-1 1],'same'));
dx = conv2(g,[-1 1],'same');
imagesc(conv2(clown,dx,'same'));
lg = fspecial('log',15,2);
lclown = conv2(clown,lg,'same');
imagesc(lclown) We wish to mark points along the curve where the magnitude is biggest.
We can do this by looking for a maximum along a slice normal to the
imagesc(clown + .2*lclown) curve (non-maximum suppression). These points should form a curve.
There are then two algorithmic issues: at which point is the maximum,
and where is the next one?
Source: D. Forsyth

13
 Non-maximum suppression Predicting the next edge point
At q, we have a Assume the marked
maximum if the point is an edge point.
value is larger Then we construct the
than those at tangent to the edge
both p and at r. curve (which is normal
Interpolate to to the gradient at that
get these point) and use this to
values. predict the next points
(here either r or s).

Source: D. Forsyth Source: D. Forsyth

Designing an edge detector Canny edge detector

Criteria for an optimal edge detector:
This is probably the most widely used edge
Good detection: the optimal detector must
detector in computer vision
minimize the probability of false positives (detecting
spurious edges caused by noise), as well as that of Theoretical model: step-edges corrupted by
false negatives (missing real edges) additive Gaussian noise
Good localization: the edges detected must be as Canny has shown that the first derivative of the
close as possible to the true edges Gaussian closely approximates the operator
Single response: the detector must return one that optimizes the product of signal-to-noise
point only for each true edge point; that is, minimize
ratio and localization
the number of local maxima around the true edge

J. Canny, A Computational Approach To Edge Detection, IEEE

Trans. Pattern Analysis and Machine Intelligence, 8:679-714, 1986.

Source: L. Fei-Fei Source: L. Fei-Fei

Canny edge detector The Canny edge detector

1. Filter image with derivative of Gaussian
2. Find magnitude and orientation of gradient
3. Non-maximum suppression:
Thin multi-pixel wide ridges down to single pixel
width
4. Linking and thresholding (hysteresis):
Define two thresholds: low and high
Use the high threshold to start edge curves and the
low threshold to continue them

MATLAB: edge(image, canny) original image (Lena)

Source: D. Lowe, L. Fei-Fei

14
The Canny edge detector The Canny edge detector

norm of the gradient thresholding

The Canny edge detector Hysteresis thresholding

original image

thinning high threshold low threshold hysteresis threshold

(strong edges) (weak edges)
(non-maximum suppression)
Source: L. Fei-Fei

Effect of (Gaussian kernel spread/size) Edge detection is just the beginning

image human segmentation gradient magnitude

original Canny with Canny with

The choice of depends on desired behavior

large detects large scale edges Berkeley segmentation database:
http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbe
small detects fine features Source: S. Seitz
nch/

15