Keypoint detection

 Many applications benefit from features localized in (x,y)

 Edges well localized only in one direction  detect corners?

 Desirable properties of keypoint detector

 Accurate localization
 Invariance against shift, rotation, scale, brightness change
 Robustness against noise, high repeatability

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection 1
 Keypoint detection
 Laplacian detector
 Determinant of Hessian detector
 Harris detector
 FAST detector

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection 2
 applications
Local Feature Detection and Extraction
 Laplacian keypoint detector
LoG convolution Thresholding

keypoints
Input Detect local
f [x,y]
min/max

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection 4
 Input images

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection 5
 LoG response

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection 6
 Thresholded LoG response

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection 7
 Local extrema of thresholded LoG response

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection 8
 Superimposed LoG keypoints

500 strongest
keypoints

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection 9
 Determinant of Hessian keypoint detector

Dxx Dyy Dxy

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection
 Input images

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection
 DoH response

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection
 Thresholded DoH response

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection
 Local maxima of DoH response

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection
 Superimposed DoH keypoints

500 strongest
keypoints

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection
 ???
Which of the following images is the Laplacian of Gaussian and which is
the Determinant of Hessian of the above image?

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection
 What patterns can be localized most
accurately?
 Local displacement sensitivity (assuming continuous f(x,y))


Linear approximation for small x, y fx(x,y) – horizontal image gradient
fy(x,y) – vertical image gradient

2
  x  
S x,
 
 fy x, 
x

f y x,
y
 
 y 
x , y window
 

y 
  
 

2

f x x,
y f x x, 2y  f y x,
   x 
  
  x y  
 
fx 
 
f y x,   y 
x , y window
x, y f  x, y 
y 
y

  x  
 M
 
 y

 y
  x y
 Iso-sensitivity curves are ellipses
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection
 Harris detector
λ2
Based on eigenvalues λ1, λ2 of “structure
“Edge” matrix” (aka ”normal matrix” aka “second-moment
λ2 >> λ1 matrix”)
“Corner”
λ1 and λ2 are large

fx[x,y] – horizontal image gradient

fy[x,y] – vertical image gradient

“Flat” “Edge”
region λ1 >> λ2 λ1

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection
 Harris cornerness
2 2
C  det(M     1 2  k  1  2 
) k  trace 

5
M 5

14
18

2
1

4
-4

8
16

12
4

0
4 4

-1
-3 10

6
14

0
3
-2

2
λ2 λ2 12

k = 0.2 k = 0.05
2
3 2 3 8

1
10

4
6
-1

1
8

0
2 1 2
0
0

2 6
4
-1
0 4
1 -2 1 2
2
0 -1 -3
-4 0 0
0 -2

0
0
0 00
0 1 2 3 4 5 0 1 2 3 4 5
λ1 λ1

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection
 Input images

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection
 Harris cornerness

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection
 Thresholded cornerness

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection
 Local maxima of cornerness

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection
 Superimposed Harris keypoints

500 strongest
keypoints

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection
 Robustness of Harris detector
 Invariant to brightness offset: f [x,y]  f [x,y] + c

 Invariant to shift and rotation 

 Not invariant to scaling 1

0.8

Repeatability
0.6

 0.4

0.2

edge corner 0
1 2 3 4 5 6
Scale factor

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection
 The following structure matrices are examples of the three different cases

???
“Flat”, “Edge” and “Corner” of the Harris detector. Which is which?

0.01
M 5 1
 M  12 0.1 
 M  
.05
0.1  0.01
 1 4   0.02 
.003

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection
 Features from Accelerated Segment Test (FAST)

1 2
16

15 3

14 4
13 P 5
12 6

11 7
10 9 8

 Compare “nucleus” p to circle of sixteen pixels

 Nucleus is feature point, iff at least n=9 contiguous circle
pixels are either all brighter, or all darker, by θ
 Optimize pixel comparisons to reject non-corners early
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection
 Input images

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection
 FAST corners superimposed

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection
 FAST corner detection on smartphone

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection
 FAST keypoint tracking on smartphone

Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Keypoint Detection