Computer Vision

Feature Extraction
(Corner and interest point detection, Local
variant feature detectors and descriptors)
Contents

• Corner and interest point detection

• Local variant feature detectors and descriptors
 Why Feature Descriptor?

• Easy to classify cat and dog

• Shape and edges are known
• shape of the object, color, edges, • Can not be used to match two objects
background, etc. are known • Feature descriptor is a simplified
representation of the image/ patches
 Challenges of object Detection

• Apply thresholding to convert it to binary

• More complex
• Compare geometric properties to
recognize the objects • Can recognize letter and identify license
plates
Challenges of Object Detection
• Find template in image
• Objects in image are
• Rotated
• scaled
• occluded
• Create several templates with different
Image
rotations and scales
• Instead determine feature descriptors
Local and Global Descriptors
Local and Global Descriptors
• Global Descriptors
• Describe the image as a whole
• Contour representations, Shape descriptors, Texture features
• DoG, HOG, histograms of optical flow (HOF) etc are few examples
• Limitations: difficult to detect local objects and objects with occlusions
• Local Descriptors
• Describes a patch within an image
• More robust against occlusions
• SIFT, SURF, LBP, BRISK, MSER, and FREAK are examples
Overview of Local Features Matching
Overview of Local Features Matching
Overview of Local Features Matching

• Match the local descriptors between

images
Overview of Local Features Matching

• Match the local descriptors between

images
Overview of Local Features Matching

• Match the local descriptors between

images
• If matched, determine correspondence
between two patches
• Apply same correspondence to other
interest points
Feature matching process
• Identify interest points
• Choose patches surrounding key points

Patches around interest points

 Feature matching process

• Match interest points

• Remove outliers
Applications of interest points
• Stereo image matching
• Choose corner interest points on both images
• Apply image matching between left and right image for stereo
correspondence
 Interest point Detection and Applications

• Join two images to get a bigger image

• Choose interest point (corner point)
• Use correspondence between interest
points for image alignment

Panorama Stitching
Characteristics of good features of interest points
• Repeatability
• Find same feature in multiple images irrespective of geometric and photometric
transformations
• Saliency
• Each feature is unique
• Compactness
• fewer features than number of image pixels which represent entire image
Characteristics of good features
• Efficiency
• Computationally faster for real time applications
• Locality
• Occupy small area of image
• Robust to clutter and occlusion
• Covariant
• Should be detected in corresponding locations despite geometric and photometric
variations
Patches of images for matching
• Select a patch (square) in one image
• Match it with a patch in the other image

• Patch should have unique shape in the image

Patch Matching

• Corners are unique points in images

Corner Point

• Corner points are special case of interest points

• Corner –
• is intersection of two or more edge segments
• point at which the direction of the object’s border rapidly changes
• have a well-defined location in the image space
• maintain their stability when scale and rotation
• can compute the interest points accurately
How to determine corner Point?

Flat: No change in pixel intensities in all directions

Edge: No change in pixel intensities along edge direction
Corner: Significant change in pixel intensities in all directions
• Difference in pixel intensities in window can be used to identify corners
Corner detector
• Change in intensity is tested for the shift (u,v) of a window
𝐸 𝑢, 𝑣 = ෍ 𝑤(𝑥, 𝑦) 𝐼 𝑥 + 𝑢, 𝑦 + 𝑣 − 𝐼(𝑥, 𝑦) 2

𝑥,𝑦
• For nearly constant patches, change of intensity, E(u,v) is almost 0
• For different patches, E(u,v) is large
• Then it is corner point
 Corner detector
𝐸 𝑢, 𝑣 = ෍ 𝑤(𝑥, 𝑦) 𝐼 𝑥 + 𝑢, 𝑦 + 𝑣 − 𝐼(𝑥, 𝑦) 2

𝑥,𝑦
Maximum value: black, Minimum value: white

• Entire process of calculation is computationally expensive, Requires several computations

especially for large images
Examples of Derivatives to determine change in intensity

image

X derivative

Y derivative
Examples of Derivatives to determine change in intensity

image

X derivative

Y derivative
Examples of Derivatives to determine change in intensity

image

X derivative

Y derivative
Examples of Derivatives to determine change in intensity

image

X derivative

Y derivative
Examples of Derivatives to determine change in intensity

image

X derivative

Y derivative
Examples of Derivatives to determine change in intensity

image

X derivative

Y derivative
Examples of Derivatives to determine change in intensity

image

X derivative

Y derivative
Examples of Derivatives to determine change in intensity

image

X derivative

Y derivative
Examples of Derivatives to determine change in intensity

image

X derivative

Y derivative
Harris Corner Detector
• For small shifts, (u,v) of window, E(u,v) can be approximated as
𝑢
𝐸 𝑢, 𝑣 = 𝑢, 𝑣 𝑀
𝑣
𝐼𝑥2 𝐼𝑥 𝐼𝑦
𝑀 = ෍ 𝑤(𝑥, 𝑦)
𝐼𝑥 𝐼𝑦 𝐼𝑦2
𝑥,𝑦

• Ix and Iy are image derivatives in x and y directions in a window/ patch

Harris Corner Detector

Prewitt operator

1 0 -1
1 0 -1
1 0 -1
Mask for Ix

1 1 1
0 0 0
Ix Iy -1 -1 -1
Image
Mask for Iy
 Harris Corner Detector

Window, w(x,y)

Image
 Harris Corner Detector

Window, w(x,y)

Image Ix
 Harris Corner Detector

Window, w(x,y)

Image Ix Iy
 Harris Corner Detector

Window, w(x,y)

Image Ix Iy Gradients of points in window

𝐼𝑥2 𝐼𝑥 𝐼𝑦
𝑀 = ෍ 𝑤(𝑥, 𝑦)
𝐼𝑥 𝐼𝑦 𝐼𝑦2
𝑥,𝑦

• λ1 and λ2 are Eigen values of M

λ1 and λ2 are large

Point is at the corner
Harris Corner Detector
𝐼𝑥2 𝐼𝑥 𝐼𝑦
𝑀 = ෍ 𝑤(𝑥, 𝑦)
𝐼𝑥 𝐼𝑦 𝐼𝑦2
𝑥,𝑦

Image Ix Iy
Point is on the edge
Harris Corner Detector
𝐼𝑥2 𝐼𝑥 𝐼𝑦
𝑀 = ෍ 𝑤(𝑥, 𝑦)
𝐼𝑥 𝐼𝑦 𝐼𝑦2
𝑥,𝑦

Image Ix Iy
Point is in flat portion of image
Eigen Values for Corner Point Detection
Harris Corner Detector
𝐼𝑥2 𝐼𝑥 𝐼𝑦
𝑀 = ෍ 𝑤(𝑥, 𝑦)
𝐼𝑥 𝐼𝑦 𝐼𝑦2
𝑥,𝑦

• Eigen value computation is costly

• R = det{M} – k (trace {M})2
Where Det{M} = λ1x λ2 and Trace{M} = λ1 + λ2
• Or,
Det{M} = Ix2 Iy2 - Ixy2
Trace{M}=Ix2 +Iy2
• Empirically determined value of k is 0.04 – 0.06
• R is large for corner points
• R is –ve and large for edge
• |R| is small for flat region
 Ex: Harris Corner Detector (window 3x3)
0 0 0 0 0 1 0 -1 1 2 1
0 10 10 10 10 2 0 -2 0 0 0
0 10 0 0 0 1 0 -1 -1 -2 -1
0 10 0 0 0 Sobel mask for Ix Sobel mask for Iy
0 10 0 0 0

window Image

𝐼𝑥2 𝐼𝑥 𝐼𝑦
𝑀 = ෍ 𝑤(𝑥, 𝑦)
𝐼𝑥 𝐼𝑦 𝐼𝑦2
𝑥,𝑦

For Corner, threshold

for R > 3000000
 Ex: Harris Corner Detector (window 3x3)
0 0 0 0 0 1 0 -1 1 2 1
0 10 10 10 10 2 0 -2 0 0 0
0 10 0 0 0 1 0 -1 -1 -2 -1
0 10 0 0 0 Sobel mask for Ix Sobel mask for Iy
0 10 0 0 0

window Image 0 0 0 0 0 0 0 0 0 0
0 10 0 10
0 0
𝐼𝑥2 𝐼𝑥 𝐼𝑦 0 0
𝑀 = ෍ 𝑤(𝑥, 𝑦) 0 0 0 0
𝐼𝑥 𝐼𝑦 𝐼𝑦2
𝑥,𝑦 0 10 0 0 0 0 10 0 0 0
Ix Iy
For Corner, threshold
for R > 3000000
 Ex: Harris Corner Detector
0 0 0 0 0 0 0 0 0 0 𝐼𝑥2 𝐼𝑥 𝐼𝑦
0 -20 10 0 10 0 -20 -10 0 10 𝑀 = ෍ 𝑤(𝑥, 𝑦)
𝐼𝑥 𝐼𝑦 𝐼𝑦2
0 -10 30 0 0 0 10 30 40 0 𝑥,𝑦
0 0 40 0 0 0 0 0 0 0
0 10 0 0 0 0 10 0 0 0 𝑀=
Ix Iy

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 10 0 10 0 10
0 0 0 0 0 0
0 0 0 0 0 0
0 10 0 0 0 0 10 0 0 0 0 10 0 0 0
Ix2 Iy2 IxIy
 Ex: Harris Corner Detector (window 3x3)
0 0 0 0 0
0 10 10 10 10 3100 1100 Det{M} = Ix2 Iy2 - Ixy2
𝑀=
0 10 0 0 0 1100 3100 Trace{M}=Ix2 +Iy2
0 10 0 0 0
0 10 0 0 0
R = det{M} – k (trace {M})2
Image
0 0 0 0 0
For Corner, threshold
0 10
for R > 3000000
0 6862400 0
It is a corner point
0 0
0 10 0 0 0
Corner Response, R
Ex: Harris Corner Detector (window 3x3)
0 0 0 0 0 1 0 -1 1 2 1
0 0 0 0 0 2 0 -2 0 0 0
10 10 10 10 10 1 0 -1 -1 -2 -1
0 0 0 0 0 Sobel mask for Ix Sobel mask for Iy
0 0 0 0 0
Image
Ex: Harris Corner Detector
0 0 0 0 0 1 0 -1 1 2 1
0 0 0 0 0 2 0 -2 0 0 0
10 10 10 10 10 1 0 -1 -1 -2 -1
0 0 0 0 0 Sobel mask for Ix Sobel mask for Iy
0 0 0 0 0
Image

0 0 0 0 0 0 0 0 0 0
0 0 0 0
10 10 10 10
0 0 0 0
0 0 0 0 0 0 0 0 0 0
Ix Iy
 Ex: Harris Corner Detector
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
10 10 10 10 10 10 10 0 10 0 0 10
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Image Ix2 Iy2 IxIy

𝐼𝑥2 𝐼𝑥 𝐼𝑦
𝑀 = ෍ 𝑤(𝑥, 𝑦) 𝑀=
𝐼𝑥 𝐼𝑦 𝐼𝑦2
𝑥,𝑦

Determinant of M Trace of M
෍ 𝐼𝑥2 = ෍ 𝐼𝑦2 = = =

σ 𝐼𝑥 𝐼𝑦 =0 R = det{M} – k (trace {M})2

 Ex: Harris Corner Detector (window 3x3)
0 0 0 0 0
0 0 0 0 0
0 0 0 0 0
0 0
10 10 10 10 10 10 -3686400 10
0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Image Corner Response, R

It is an edge point

For Edge, threshold for R < - 3000000

Harris Corner Detector

Image Threshold on R<-10000 Threshold on R > 10000

edges corners

Threshold for edge, -10,000 < R

For corner, R > 10000
Choice of Window Function
Choice of Window Function
• If uniform (box) window is chosen as the window function
• Detector is not rotation invariant
Choice of Window Function
• If uniform (box) window is chosen as the window function
• Detector is not rotation invariant
Harris Corner Detector

Yellow is
maximum
magnitude
and blue is
minimum
Harris Corner Detector

Yellow is
maximum
magnitude
and blue is
minimum
Harris Corner Detector

Yellow is
maximum
magnitude
and blue is
minimum

Yellow is
maximum
magnitude
and blue is
minimum
Harris Corner Detector

Yellow is
maximum
magnitude
and blue is
minimum

Yellow is
maximum
magnitude
and blue is
minimum
Harris Corner Detector with Gaussian Window

Image Image after rotation

and variation in
illumination
Harris Corner Detector with Gaussian Window

Image Image after rotation

and variation in
illumination
Harris Corner Detector with Gaussian Window

Image after rotation Corner response, R

Image
and variation in
illumination Blue for negative R
Red for positive R
Harris Corner Detector with Gaussian Window

Corner response, R Points with large value of R (R> threshold)

Hessian Corner Detector
• Harris and Hessian detectors are similar and both are rotation invariant
• Herris detector uses first moment
• Hessian detector uses second moment
• Hessian detector provides good performance in terms of computation time and accuracy
• Given a pixel, the Hessian matrix is

𝐼𝑥𝑥 𝐼𝑥𝑦 Ixx and Iyy are second moment in x and y direction respectively
𝐻=
𝐼𝑥𝑦 𝐼𝑦𝑦 Ixy is first moment in x direction and then in y direction

Det(H) = IxxIyy – (Ixy)2

If det(H) is high then it is a corner point
Ex: Hessian Corner Detector
0 0 0 0 0 0 0 0 0 1 0 1 0 -1 1 2 1
0 10 10 10 10 1 -2 1 0 -2 0 2 0 -2 0 0 0
0 10 0 0 0 0 0 0 0 1 0 1 0 -1 -1 -2 -1
0 10 0 0 0 Mask for Second Mask for Second Mask for first Mask for first
moment, Ixx in x moment, Iyy in y moment, Ix in x moment of , Ix
0 10 0 0 0
direction direction direction that is , Ixy
Image in y direction
 Ex: Hessian Corner Detector
0 0 0 0 0 0 0 0 0 1 0 1 0 -1 1 2 1
0 10 10 10 10 1 -2 1 0 -2 0 2 0 -2 0 0 0
0 10 0 0 0 0 0 0 0 1 0 1 0 -1 -1 -2 -1
0 10 0 0 0 Mask for Second Mask for Second Mask for first Mask for first
moment, Ixx moment, Iyy moment, Ix moment of , Ix
0 10 0 0 0
that is , Ixy
Image in y direction

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 10 0 10 0 10 0 10
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 10 0 0 0 0 10 0 0 0 0 10 0 0 0 0 10 0 0 0
Ixx Iyy Ix Ixy
 Ex: Hessian Corner Detector
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 10 10 10 10 0 -10 0 0 10 0 -10 -20 -20 10 0 -20 10 0 10
0 10 0 0 0 0 -20 10 0 0 0 0 10 10 0 0 -10 30 0 0
0 10 0 0 0 0 -20 10 0 0 0 0 0 0 0 0 0 40 0 0
0 10 0 0 0 0 10 0 0 0 0 10 0 0 0 0 10 0 0 0
Image Ixx Iyy Ix

0 0 0 0 0 𝐼𝑥𝑥 𝐼𝑥𝑦
𝐻𝑒𝑠𝑠𝑖𝑎𝑛, 𝐻 = ෍ 𝑤𝑖𝑛𝑑𝑜𝑤
0 -10 -50 -30 10 𝐼𝑥𝑦 𝐼𝑦𝑦
0 -70 -80 -20 0
−30 −200
0 -10 40 30 0 𝐻𝑒𝑠𝑠𝑖𝑎𝑛, 𝐻 =
−200 −30
0 10 0 0 0

Ixy |Det(H)| = 39100

 Harris and Hessian Corner Detectors
• Photometric/ intensity transformation

• Intensity shift (I I + b) does not change derivative

• Therefore R is invariant to shift in intensity
• If I is scaled (I  a*I)
• Then false point appears after scaling
Applications of Interest points
• Image alignment
• 3D reconstruction
• Motion tracking
• Object recognition
• Indexing and database retrieval
 References
• https://nptel.ac.in/courses/108103174
• https://sbme-tutorials.github.io/2018/cv/notes/6_week6.html
• https://medium.com/data-breach/introduction-to-harris-corner-detector-
32a88850b3f6
• https://www.baeldung.com/cs/harris-corner-detection
• https://www.codingninjas.com/codestudio/library/harris-corner-detection
• https://www.google.com/url?sa=t&source=web&rct=j&url=https://www.cs.umd.ed
u/class/fall2019/cmsc426-
0201/files/12_HarrisCornerDetection.pdf&ved=2ahUKEwj_q4fY5br-AhWu-
jgGHeTBCJAQFnoECD8QAQ&usg=AOvVaw0WjY5eRFeu-vCUFu-g6o90
• https://fiveko.com/feature-points-using-harris-corner-detector/