Object Recognition

Prepared By: Eman Abed AlWahhab

 Outline
• Introduction (Computer Vision)
• History
• Human Vision Vs. Computer Vision
• Main Goal of Computer Vision
• Significance of Computer Vision
• Connections to other Disciplines
• Key Stages in Digital Image Processing

• Object Recognition
• What is Object Recognition?
• What is Pattern Recognition?
• Approaches
• Applications
• Main Components
• Gender Example

2
 Outline
• Fingerprint Recognition
• Definition
• Fingerprint Matching Using Ridge-End and Burification
• Fingerprint Image
• Binarization
• Thinning
• Minutiae Extraction

• Car Number Plate Recognition

• What is an ALPR System?
• ALPR Procedure
• Characters Recognition
• Characters Segmentation
• Normalization of Characters

• New Innovations in Object Recognition

• References 3
Brief History of Computer Vision
• 1966: Minsky assigns computer vision
as an undergrad summer project
• 1960’s: interpretation of synthetic
worlds
Guzman ‘68
• 1970’s: some progress on interpreting
selected images
• 1980’s: ANNs come and go; shift
toward geometry and increased
mathematical rigor
• 1990’s: face recognition; statistical Ohta Kanade ‘78
analysis in vogue
• 2000’s: broader recognition; large
annotated datasets available; video
processing starts
• 2030’s: robot uprising? 4
Turk and Pentland ‘91
 Human Vision
 Vision is the process of discovering what is present
in the world and where it is by looking

 Computational algo implemented in this

massive network of neurons; they obtain their
inputs from retina, & produce as output an
“understanding” of the scene in view
 But what does it mean to “understand”
the scene? What algos & data
representation are used by brain?

5
 Computer Vision
 Computer Vision is the study of analysis of pictures and
videos in order to achieve results similar to those as by
humans

 Analogously, given a set of TV camera

 What computer architectures, data
structures & algorithms should use
to create a machine that can “see”
as we do?

6
Human Vision VS Computer Vision

What we see What a computer sees

7
Main Goal of Computer Vision

Every picture tells a story!!

8
* write computer programs that can interpret images *
Significance of Computer Vision

Safety Health Security

Comfort Fun Access 9

Connections to other Disciplines

10
What is Digital Image Processing?

•The continuum from image processing to

computer vision can be broken up into low-,
mid- and high-level processes
Low Level Process Mid Level Process High Level Process
Input: Image Input: Image Input: Attributes
Output: Image Output: Attributes Output: Understanding
Examples: Noise Examples: Object Examples: Scene
removal, image recognition, understanding,
sharpening segmentation autonomous navigation

11
Key Stages in Digital Image Processing

Image Morphological
Restoration Processing

Image
Segmentation
Enhancement

Image Object
Acquisition Recognition

Representation
Problem Domain
& Description
Colour Image Image
Processing Compression 12
 Key Stages in Digital Image Processing:
Image Acquisition
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

Image Morphological
Restoration Processing

Image
Segmentation
Enhancement

Image Object
Acquisition Recognition

Representation
Problem Domain
& Description
Colour Image Image
Processing Compression 13
 Key Stages in Digital Image Processing:
Image Enhancement
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

Image Morphological
Restoration Processing

Image
Segmentation
Enhancement

Image Object
Acquisition Recognition

Representation
Problem Domain
& Description
Colour Image Image
Processing Compression 14
 Key Stages in Digital Image Processing:
Image Restoration
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

Image Morphological
Restoration Processing

Image
Segmentation
Enhancement

Image Object
Acquisition Recognition

Representation
Problem Domain
& Description
Colour Image Image
Processing Compression 15
 Key Stages in Digital Image Processing:
Morphological Processing
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

Image Morphological
Restoration Processing

Image
Segmentation
Enhancement

Image Object
Acquisition Recognition

Representation
Problem Domain
& Description
Colour Image Image
Processing Compression 16
 Key Stages in Digital Image Processing:
Segmentation
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

Image Morphological
Restoration Processing

Image
Segmentation
Enhancement

Image Object
Acquisition Recognition

Representation
Problem Domain
& Description
Colour Image Image
Processing Compression 17
 Key Stages in Digital Image Processing:
Object Recognition
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

Image Morphological
Restoration Processing

Image
Segmentation
Enhancement

Image Object
Acquisition Recognition

Representation
Problem Domain
& Description
Colour Image Image
Processing Compression 18
 Key Stages in Digital Image Processing:
Representation & Description
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

Image Morphological
Restoration Processing

Image
Segmentation
Enhancement

Image Object
Acquisition Recognition

Representation
Problem Domain
& Description
Colour Image Image
Processing Compression 19
Key Stages in Digital Image Processing:
Image Compression

Image Morphological
Restoration Processing

Image
Segmentation
Enhancement

Image Object
Acquisition Recognition

Representation
Problem Domain
& Description
Colour Image Image
Processing Compression 20
 Key Stages in Digital Image
Processing:
Colour Image Processing
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

Image Morphological
Restoration Processing

Image
Segmentation
Enhancement

Image Object
Acquisition Recognition

Representation
Problem Domain
& Description
Colour Image Image
Processing Compression 21
 What is Object Recognition?
• Last step in image processing

• It is the task of finding and identifying objects in an image

or video sequence

Like human understanding, it includes :

• Detection – of separate objects

• Description – of their geometry and positions in 3D
• Classification – as being one of a known class
• Identification – of the particular instance
• Understanding – of spatial relationships between objects
22
So what does object recognition involve?
Detection: are there people?
Verification: is that a lamp?
Identification: is that Potala Palace?
Object Categorization
mountain

tree
building
banner

street lamp

vendor
people
Scene and context categorization/Understanding

• outdoor
• city
•…

Slide credit Fei-Fei, Fergus, Torralba CVPR07 Short Course

Learning and Adaptation
• Supervised learning
– A teacher provides a category label or cost for
each pattern in the training set

• Unsupervised learning
– The system forms clusters or “natural groupings”
of the input patterns

29
The Design Cycle
What is Pattern Recognition?

“The assignment of a physical object or event to one of several

prespecified categeries” -- Duda & Hart

• A pattern is an object, process or event that can be given a

name.
• A pattern class (or category) is a set of patterns sharing
common attributes and usually originating from the same
source.
• During recognition (or classification) given objects are
assigned to prescribed classes.
• A classifier is a machine which performs classification.
Basic Components of a Pattern Recognition System

32
Components of Pattern Recognition
(Cont’d)
• Data acquisition and sensing
• Pre-processing
 Removal of noise in data.
 Isolation of patterns of interest from the background.
• Feature extraction
 Finding a new representation in terms of features.
(Better for further processing)

33
Components of Pattern Recognition
(Cont’d)
• Model learning and estimation
 Learning a mapping between features and pattern groups.
• Classification
 Using learned models to assign a pattern to a predefined
category
• Post-processing
 Evaluation of confidence in decisions.
 Exploitation of context to improve performances.

34
Examples of Pattern Recognition Applications

35
Pattern Representation
• A pattern is represented by a set of d features,
or attributes, viewed as a d-dimensional
feature vector.
T
x ( x1 , x 2 ,  , x d )
Basic concepts
Pattern
x1 Feature vector x X
- A vector of observations
x2 (measurements).
x
y  - x is a point in feature space X.

xn

Hidden state y Y
- Cannot be directly measured.
- Patterns with equal hidden state belong to the same class.

Task
- To design a classifer (decision rule) q : X Y
which decides about a hidden state based on an onbservation.
 Feature Extraction

Task: to extract features which are good for classification.

Good features: • Objects from the same class have similar feature values.
• Objects from different classes have different values.

“Good” features “Bad” features

 Feature Extraction Methods
Feature extraction Feature selection
m1
m1 φ1 x1 x1
m2
m2 φ2 x2 x2
m3
  

mk φn xn xn
mk

Problem can be expressed as optimization of parameters of featrure extractor φ(θ)

Supervised methods: objective function is a criterion of separability (discriminability)
of labeled examples, e.g., linear discriminat analysis (LDA).
Unsupervised methods: lower dimesional representation which preserves important
characteristics of input data is sought for, e.g., principal component analysis (PCA).
 Classifier
A classifier partitions feature space X into class-labeled regions such that
X X1 X2  X |Y | and X 1 X2  X |Y | { 0}

X1
X1 X3 X1

X2
X2 X3

The classification consists of determining to which region a feature vector x belongs to.
Borders between decision boundaries are called decision regions.
 Representation of classifier

A classifier is typically represented as a set of discriminant functions

fi (x) : X ,i 1,  , | Y |
The classifier assigns a feature vector x to the i-the class if f i ( x ) f j (x) j i

f1 ( x )

f 2 (x)
x max y
Feature vector  Class identifier

f |Y | ( x )

Discriminant function
Block diagram
Both definitions may be depicted by the following
block diagram.
The process consists of two major operations:
Classification
Class /
Category
Feature extraction

Class /
Object Pattern Category

Class /
Category
of 15
Example : Gender
Assume an algorithm to recognize the gender of a
student in a university, where the available input is
several features of the students (of course, the gender
cannot be one of the features).

The student to be classified is

the Object, The gender (Male
or Female) are the Classes,
and the input which is referred
to the student is the Pattern.

of 15
What is a Feature?
Feature is a scalar x which is quantitatively
describes a property of the Object.

Example: Possible features of a student:

• Number of eyes x {0, 1, 2}
• Hair color x {0 =Black, 1 =Blond, 2 =Red ,…}
• Wear glasses or not x {0, 1}
• Hair length [cm] x [0..100]
• Shoe size [U.S] x [3.5, 4, 4.5, .. ,14]
• Height [cm] x [40..240]
• Weight [kg] x [30..600]

of 15
What is Feature Extraction?
“When we have two or more classes, feature extraction
consist of choosing those features which are most
effective for preserving class separability”(Fukunaga p. 441)

Assume we choose the shoe size of the

student as a feature. The selection is
heuristically and seems reasonable.

of 15
What is a Pattern?
Pattern is a n-tuple X (vector) of N scalars xi
i [1,N], which are called the Features.
Conventional form of a pattern is:
T
X x1 x2  xN , xi , X V

Where V is known as the Feature Space, and

N is the dimension of V.

Alon Slapak of 15
Possible patterns for the gender problem:
We can use the shoe size alone to have:
X Shoe size
We can combine the height and the weight to have:
T
X height , w eight
We can even combine the height, weight and the shoe size to be on the safe side:

T
X height , w eight , shoe size
Or, we can use them all:
T
X # of eyes , hair color , glasses , hair length , height , w eight , shoe size

of 15
 Example Since the male students and the
Assume we are using the
height and the weight of female students differ from each
other in height and weight, we are
each of the students in 120
expected to have two separated
the university as a 110
clusters.
pattern. 100
Males
The height and the weight 90
are both features, which
weight [kg]
80
span a feature space V of
dimension 2. 70

60
Each of the students is
represented as a point in 50
Female
the feature space. Patterns 40 s
of male students are 30
depicted in blue, and those 120 130 140 150 160 170
height [cm]
180 190 200 210 220

of female students – in red.

Each student is characterized by a
vector of two feature: (height,
weight).
of 15
What is a Class?
“Class is a set of patterns that share some
common properties” (Wang p.10)

In our example, the Male students and the Female

students are two classes of objects that share a
common gender.

of 15
What is Classification?
Classification is a mathematical function or
algorithm which assigns a feature to one
of the classes. 120

110

Example: 100
Males
We can draw a line
90
between the two
weight [kg]

clusters in the gender 80

example, and every 70

student will be classified 60

as a female or male
according to this line. 50
Females
40

30
120 130 140 150 160 170 180 190 200 210 220
height [cm]
of 15
 Clusters Separation
Misclassifications are a consequence of the
separation of the clusters. The separation
of clusters is quantified using two major
methods:
1. Mathematically: there are several separation criteria’s.
2. “Intuitively”: overlapping of the clusters.
100 100 100

90 90 90

80 80 80

70 70 70

60 60 60

50 50 50

40 40 40

30 30 30

20 20 20

10 10 10

0 0 0
0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100

Separable clusters Almost separable clusters Non-separable clusters

of 15
Classification Quality
WARNING!!! Although the idea is well illustrated, it is a bad habit to
judge a classification quality according to the visual representation
of clusters.

The classification quality The clusters separation

is strongly depends on strongly depends on the
the clusters separation features selection

Feature selection is of
paramount importance in
classification quality

of 15
53
 Fingerprint
The popular Biometric used to authenticate person is Fingerprint which
is unique and permanent throughout a person’s life

Fingerprint is the pattern of ridges and valleys

The ridges have characteristics, called minutiae, are the ridge ending
and the ridge bifurcation

Ridge ending is defined as the point where ridge ends abruptly

Ridge bifurcation is defined as the point where a ridge forks into branch
ridges

54
Valleys and Ridges

55
Ridge Ending and Bifurcation

56
 Fingerprint Recognition
Fingerprint recognition or fingerprint authentication refers to the
method of verifying a match between two human fingerprint

Fingerprint recognition techniques have the advantage to use low-

cost standard capturing device

However , recognition of the fingerprint becomes a complex

computer vision problem , especially when dealing with noisy and
low quality images

A minutia matching is widely used for fingerprint recognition and can be

classified as ridge ending and ridge bifurcation
57
Fingerprint Matching using Ridge-End and
Bifurcation Points

58
Fingerprint Image
• The input fingerprint image is the gray scale
image of a person, which has intensity values
ranging from 0 to 255
• A number of methods are used to acquire
fingerprints
• The inked impression method remains the
most popular one
• Inkless fingerprint scanners are also present
59
Inked method Inkless method

60
Binarization

Binarization is used to convert gray scale image into

binary image by fixing the threshold value

The pixel values above the threshold are set to ‘1’ and the
pixel values below the threshold are set to ‘0’ respectively

61
Original Fingerprint Binarized Fingerprint

62
 Thinning

The binarized image is thinned

using Block Filter

To reduce the thickness of all ridge

lines to a single pixel width to
extract minutiae points effectively

Thinning does not change the

location of minutiae points
compared to original fingerprint
63
Binarized Fingerprint Image after Thining

64
 Minutiae
Extraction

Classification of ridge-end and ridge

bifurcation points is done by creating matrix

Crossing Number is used to locate the

minutiae points in fingerprint image

Crossing Number is defined as half of the

sum of differences between intensity values
of two adjacent pixels

65
• If crossing Number is 1 minutiae points are
classified as Termination

• If crossing Number is 2 minutiae points are

classified as Normal ridge

• If crossing Number 3 or greater than 3

minutiae points are classified as Bifurcation

66
Crossing Number and Type of Minutiae

67
Gray-scale Fingerprint Minutiae points

68
Minutiae Matching
Image Acquisition

Computation of Points

Location Detection of Points

Amount and Location Matching

69
Image Acquisition

Iimage.jpg = Input Image

acquisition from reader.

Timage.jpg = Template
Image retrieve from
database.

70
 Computation of
Points
After the detection of minutiae points,
matching algorithm require to calculate
total number of available points in the
fingerprint image separately

To perform this computation two

counter variables are used to count
both ridge-end and bifurcation points

71
Minutiae Point Calculation

72
 Location Detection
of Points
Each minutiae point in the fingerprint image
has a specific location.

This location information of particular point is

significant to store for further matching of
fingerprints.

The location of every point in the digital image

is given by pixel position, so that it can be
taken and stored separately for both ridge-end
and bifurcation points.

73
Minutiae Point Extracted in Input Image

74
Amount and Location
Matching
In the previous steps, all the required
information about points is computed and
stored

Now, this is the matching step, here the

algorithm compares the computed values
with the stored values

This algorithm first, compares the

combination of both amounts of ridge-end
and bifurcation points with stored data

If the match occurs, the algorithm then

compares the location of ridge points with
stored location data

75
76
77
78
 What is an ALPR System?

• Some toll road

requirements we encounter:
– 99.9% image capture
– 99% overall plate read
accuracy
– 99% OCR accuracy on 90%
of capture plate
images – number and state

79
 Recognized character
The procedure is based on extraction of plate
region, segmentation of plate characters and
recognition of characters 80
81
Characters Segmentation
• In the segmentation of plate characters, the car
number plate is segmented into its constituent
parts to obtain its characters individually . Image
filtering from unwanted spots and noise.

• Dilation of image to separate characters from each

others.

Segmented plate number

82
Separating the plate characters

It is done by finding starting and end points of

characters in horizontal direction.

Characters separated individually

83
 Normalization of characters
Normalization is to refine the characters into a
block containing no extra white spaces (pixels) in all
the four sides of the characters.

Sometimes called contrast stretching.

Then each character should be equal in size.

84
Below an example of normalized character where the
character fill all the 4 sides

Normalized character

85
86
Supermarket scanner recognizes objects without
barcodes

87
88
Supermarket scanner recognizes objects without barcodes

Uses object recognition to identify foods at the supermarket checkout line.

The technology uses a camera that compares the food that is being scanned to a
large, expandable database of products.

That camera filters out background "noise" in its picture, so that it only sees
objects held close to its lens against a neutral black background.

The technology recognizes supermarket items at check out without requiring a

bar code...making bar codes obsolete for check out purposes.

It uses proprietary pattern recognition technology and claims it can operate at

high speeds.

This object recognition system requires a database that contains the information
about the items in the supermarket.

This system claims to be able to make very precise identification of produce.

89
Google patents new object recognition technology, likely has
plans to use with YouTube

90
 Google patents new object recognition technology,
likely has plans to use with YouTube

It’s known as “automatic large scale video object recognition.”

It can actually recognize the difference between a variety of

objects, not just human faces.

After recognizing an object it then labels it with certain tags. If you

are wondering how it does this, there is a special object name
repository involved.

This database would hold at least 50,000 object names,

information and shapes that would allow for easy identification.
91
92
 Android Eye
Android Eye is an advanced Object Recognition app. Take a picture of any object, and
Android Eye will tell you what it is.

Take a picture of a car... Android Eye will tell you the make and model of the car.

Take a picture of a foreign t-shirt label... Android Eye will tell you the brand, and where
the shirt is from. Take a picture of a tree... a ball... a person... the results are endless.

This is the new version; it works very well, particularly with vehicles, products, brands,
and well-known "things".

It also guesses celebrity names.

Software that does this is usually only available to government agencies and research
93
facilities.
A Google Glass App

94
 A Google Glass App Knows What You're Looking At

An app for Google’s wearable computer Glass can

recognize objects in front of a person wearing the device.

Google has shown that the camera integrated into Google

Glass, the company’s head-worn computer, can capture
some striking video.

They built an app that uses that camera to recognize what

a person is looking at.

The app was built at an employee hack session held by the

company this month to experiment with ways to
demonstrate their new image recognition service.

The app can either work on photos taken by a person

wearing Glass, or constantly grab images from the device’s
camera.

Those are sent to the cloud or a nearby computer for

processing by AlchemyAPI’s image recognition software.
The software sends back its best guess at what it sees and 95
then Glass will display, or speak, the verdict.
SELF-PARKING CAR

96
97
References
• http://arxiv.org/ftp/arxiv/papers/1001/1001.4186.pdf
• http://www4.comp.polyu.edu.hk/~csajaykr/myhome/teaching/biometrics/final_re
port.pdf
• http://www.google.com.lb/ Fingerprint Analysis and Representation.ppt
• http://www.academia.edu/2537762/Fingerprint_Matching_using_Ridge-
End_and_Bifurcation_Points
• http://ai.pku.edu.cn/aiwebsite/research.files/collected%20papers%20-
%20fingerprint/Minutiae%20detection%20algorithm%20for%20fingerprint%20reco
gnition.pdf
• http://en.wikipedia.org/wiki/Computer_vision

98
99