Outline

• Types of transformations and invariance

– Scale invariance
Lecture 13: Local invariant features • Local features: detectors and descriptors
– SIFT
Tuesday, Oct 30
Prof. Kristen Grauman

Geometric transformations
• What would we like our image descriptions
to be invariant to?

Figure from T. Tuytelaars ECCV 2006 tutorial

Photometric transformations And other nuisances…

• Noise
• Blur
• Compression artifacts
• Appearance variation for a category

Figure from T. Tuytelaars ECCV 2006 tutorial

 Classes of transformations Exhaustive search
• Euclidean/rigid:
A multi-scale approach
Translation + rotation
• Similarity: Translation +
rotation + uniform scale Similarity
Translation transformation
AffineTranslation
Projectivetransformation
and Scaling
transformation
• Affine: Similarity + shear
– Valid for orthographic
camera, locally planar
object
• (Projective: Affine +
projective warps)
• Photometric: affine
intensity change
– I -> aI + b
Slide from T. Tuytelaars ECCV 2006 tutorial

Exhaustive search Exhaustive search

A multi-scale approach A multi-scale approach

Slide from T. Tuytelaars ECCV 2006 tutorial Slide from T. Tuytelaars ECCV 2006 tutorial

Exhaustive search Key idea of invariance

A multi-scale approach We want to extract the patches from
each image independently.

Slide from T. Tuytelaars ECCV 2006 tutorial Slide adapted from T. Tuytelaars ECCV 2006 tutorial
 Invariant local features (Good) invariant local features
Subset of local feature types • Reliably detected
designed to be invariant to y1
– Scale y2 • Distinctive
…
– Translation
yd • Robust to noise, blur, etc.
– Rotation
– Affine transformations • Description normalized properly
– Illumination x1
x2
1) Detect distinctive interest points …
2) Extract invariant descriptors xd

[Mikolajczyk & Schmid, Matas et al., Tuytelaars & Van Gool, Lowe, Kadir et al.,… ]

Interest points: From stereo to recognition Review: corner detection as an interest

operator
• Feature detectors previously used for
stereo, motion tracking
• Now also for recognition
– Schmid & Mohr 1997
• Harris corners to select interest points
• Rotationally invariant descriptor of local image
“flat” region: “edge”: “corner”:
regions
no change in no change along significant change
• Identify consistent clusters of matched features all directions the edge direction in all directions
to do recognition

C.Harris, M.Stephens. “A Combined Corner and Edge Detector”. 1988

[Slide credit: Darya Frolova and Denis Simakov]

Review: Harris Detector Workflow Review: Harris Detector Workflow

Compute corner response R
 Review: Harris Detector Workflow Review: Harris Detector Workflow
Find points with large corner response: R>threshold Take only the points of local maxima of R

Review: Harris Detector Workflow

Harris Detector
• Rotation invariance

Ellipse rotates but its shape (i.e.

eigenvalues) remains the same
Corner response R is invariant to image rotation

But, for corner detection we must search windows at

a pre-determined scale.

Scale space (Witkin 83) Scale space

first derivative peaks contours of f’’ = 0 in scale-space
Scale space insights:
• edge position may shift with increasing scale (σ)
• two edges may merge with increasing scale
(edges can disappear)
larger • an edge may not split into two with increasing
scale (new edges do not appear)

Gaussian filtered 1d signal x

Adapted from Steve Seitz, UW

 Scale Invariant Detection Scale Invariant Detection
• Consider regions of different sizes around a • The problem: how do we choose
point corresponding circles independently in each
• At the right scale, regions of corresponding image?
content will look the same in both images

[Slides by Darya Frolova and Denis Simakov]

Scale Invariant Detection Scale Invariant Detection

• Solution: • Common approach:
– Design a function on the region (circle), which is
Take a local maximum of this function
“scale invariant” (the same for corresponding
regions, even if they are at different scales) Observation: region size, for which the maximum is achieved,
Example: average intensity. For corresponding should be invariant to image scale.
regions (even of different sizes) it will be the same.
– For a point in one image, we can consider it as Important: this scale invariant region size
a function of region size (circle radius) is found in each image independently!

f Image 1 f Image 2 f Image 1 f Image 2

scale = 1/2 scale = 1/2

region size region size s1 region size s2 region size

Scale Invariant Detection

Following example was created by T.

[Images from T. Tuytelaars] Tuytelaars, ECCV 2006 tutorial
 Scale Invariant Detection Scale space
• A “good” function for scale detection:
Scale space insights:
has one stable sharp peak
• edge position may shift with increasing scale (σ)
f f f Good ! • two edges may merge with increasing scale
bad bad (edges can disappear)
region size region size
region size
• an edge may not split into two with increasing
scale (new edges do not appear)
• For usual images: a good function would be a
one which responds to contrast (sharp local What could be an approximation of
intensity change) an image’s scale space?

Scale invariant detection Scale selection principle

Requires a method to repeatably select points in • Intrinsic scale is the scale at which
location and scale: normalized derivative assumes a
– Only reasonable scale-space kernel is a Gaussian maximum -- marks a feature containing
(Koenderink, 1984; Lindeberg, 1994)
– An efficient choice is to detect peaks in the difference
r
lu
B

trac
b
u
S

interesting structure. (T. Lindeberg ’94)

of Gaussian pyramid (Burt & Adelson, 1983; Crowley
& Parker, 1984)
Î Maxima/minima of Laplacian
– Difference-of-Gaussian is a close approximation to
Laplacian

Slide adapted from David Lowe, UBC

 Scale Invariant Detection

Kernels:
f = Kernel ∗ Image
L = σ 2 ( Gxx ( x, y, σ ) + Gyy ( x, y, σ ) )
(Laplacian)

DoG = G ( x, y, kσ ) − G ( x, y , σ )
(Difference of Gaussians)

where Gaussian

x2 + y2
−
G ( x, y , σ ) = 1
2πσ
e 2σ 2
Scale space images: repeatedly Adjacent Gaussian images
convolve with Gaussian subtracted
[Slide by Darya Frolova and Denis Simakov]

SIFT: Key point localization SIFT: Example of keypoint detection

Threshold on value at DOG peak and on ratio of principle
n Detect maxima and minima curvatures (Harris approach)
of difference-of-Gaussian in
scale space (a) 233x189 image
(b) 832 DOG extrema
n Then reject points with low (c) 729 left after peak
contrast (threshold) value threshold
(d) 536 left after testing
n Eliminate edge responses ratio of principle
r
lu
B

trac
b
u
S

curvatures
(use ratio of principal
curvatures)

Candidate keypoints:
list of (x,y,σ)
Adapted from David Lowe, UBC Slide from David Lowe, UBC

Scale Invariant Detectors Scale Invariant Detection:

• Experimental evaluation of detectors Summary
w.r.t. scale change
• Given: two images of the same scene with a
large scale difference between them
Repeatability rate: • Goal: find the same interest points
# correspondences independently in each image
# possible correspondences
• Solution: search for maxima of suitable
functions in scale and in space (over the
image)

K.Mikolajczyk, C.Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV 2001
 Affine Invariant Detection Affine Invariant Detection
• Above we considered: • Intensity-based regions (IBR):
Similarity transform (rotation + uniform scale) – Start from a local intensity extrema
– Consider intensity profile along rays
– Select maximum of invariant function f(t) along each ray
– Connect local maxima
• Now we go on to: – Fit an ellipse
Affine transform (rotation + non-uniform scale)

T.Tuytelaars, L.V.Gool. “Wide Baseline Stereo Matching Based on Local, Affinely

Invariant Regions”. BMVC 2000.

Affine Invariant Detection Point Descriptors

• We know how to detect points
• Next question:
How to describe them for matching?
• Maximally Stable Extremal Regions
(MSER)
– Threshold image intensities:
I > I0
– Extract connected components
(“Extremal Regions”)
?
– Seek extremal regions that
remain “Maximally Stable” under
range of thresholds

Point descriptor should be:

1. Invariant
2. Distinctive
Matas et al. Robust Wide Baseline Stereo from Maximally Stable Extremal Regions. BMVC 2002.

Rotation Invariant Descriptors Rotation Invariant Descriptors

• Find local orientation
• Harris corner response measure:
Dominant direction of gradient
depends only on the eigenvalues of the
matrix M

• Rotate description relative to dominant

orientation

1 K.Mikolajczyk, C.Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV 2001
C.Harris, M.Stephens. “A Combined Corner and Edge Detector”. 1988 2 D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. Accepted to IJCV 2004
 Scale Invariant Descriptors SIFT descriptors: Select canonical orientation

n Create histogram of local

• Use the scale determined by detector to gradient directions computed
compute descriptor in a normalized frame at selected scale
n Assign canonical orientation
at peak of smoothed
histogram
n Each key specifies stable 2D
coordinates (x, y, scale,
orientation)

0 2π

[Images from T. Tuytelaars] Slide by David Lowe, UBC

SIFT descriptors: vector formation

SIFT properties
n Thresholded image gradients are sampled over 16x16
array of locations in scale space
• Invariant to
n Create array of orientation histograms
n 8 orientations x 4x4 histogram array = 128 dimensions
– Scale
– Rotation
• Partially invariant to
– Illumination changes
– Camera viewpoint
– Occlusion, clutter

Slide by David Lowe, UBC

SIFT matching and recognition

Value of local (invariant) features
n Index descriptors
n Generalized Hough transform: vote for object poses
• Complexity reduction via selection of
n Refine with geometric verification: affine fit, check for
agreement between image features and model distinctive points
• Describe images, objects, parts without
requiring segmentation
– Local character means robustness to clutter,
occlusion
• Robustness: similar descriptors in spite of
noise, blur, etc.

SIFT Features
Adapted from David Lowe, UBC
 Coming up
• Problem set 3 due 11/13
– Stereo matching
– Local invariant feature indexing

• Thursday: image indexing with bags of words

– Read Video Google paper