Lecture 10 (4.14.

07)

Image Representation and

Description

Shahram Ebadollahi

4/15/2008 DIP ELEN E4830 1

 Image

Description

Recognition

High-level Image
Representation

Image
4/15/2008 Understanding 2
Lecture Outline

1 Image Description
1 Shape Descriptors
1 Texture & Texture Descriptors
1 SIFT
1 Motion Descriptors
1 Color Descriptors

4/15/2008 3
Shape Description
1 Shape Represented by its Boundary
1 Shape Numbers,
1 Fourier Descriptors,
1 Statistical Moments

1 Shape Represented by its Interior

1 Topological Descriptors
1 Moment Invariants

4/15/2008 4
Boundary Representation:
(Freeman) Chain Code
Boundary representation = 0766666453321212

Original boundary Sub-sampled boundary Chain code of boundary

Chain code for Chain code for

4-neighborhood 8-neighborhood

4/15/2008 5
 Chain Code: example

8-directional chain code 2 00006066666666444444242222202202

Starting point normalized chain code 2 00006066666666444444242222202202

Rotation normalized chain code 2 0006200000006000006260000620626

4/15/2008 First difference of chain code 6
Shape Number –
A boundary descriptor

4/15/2008 7
 Boundary descriptor – Fourier
s (k ) = x(k ) + jy (k ) k = 0,1,2, 1 , K − 1
K −1
a (u ) = ∑ s (k )e − j 2πuk / K u = 0,1,2, 1 , K − 1 Fourier Descriptor
k =0

K −1
1
s (k ) =
K
∑ a
u =0
(u ) e j 2πuk / K
k = 0,1,2, 1 , K − 1

4/15/2008 8
 Boundary Reconstruction using
Fourier Descriptors
100% 50% 10% 5%

2868
descriptors

Only 8
descriptors

2.5% 1.25% 0.63% 0.28%

4/15/2008 9
 Boundary Representation: Signatures
• Represent 2-D boundary shape using 1-D signature signal

4/15/2008 10
Boundary Representation: Signatures

4/15/2008 11
Boundary Description using
Statistical Moments
A−1
µ n (v) = ∑ (vi − m) n p(vi ) n-th moment of v
i =0

A−1
m = ∑ vi p (vi )
i =1

4/15/2008 12
Region Descriptors - Simple

1 Area
1 Perimeter
1 Compactness (perimeter)2/Area
1
r
Circularity Ratio 2
1 Mean/Median intensity r b
πr
1 Max/Min intensity a=2b
1 Normalized area
C : 4π 5π 16
A
Rc = 2 4 ≈ 0.8 π ≈ 0.78
P / 4π Rc : 1 5 4
Area of circle with same
perimeter as the shape

4/15/2008 13
4/15/2008 14
 Topological Region Descriptors
• Topological properties: Properties of image preserved under
rubber-sheet distortions

H: # holes in the image

H=2, C=1, E=-1
C: # connected components

E = C-H: Euler Number

H=0, C=3, E=3

V −Q + F = C − H = E
H=1, C=1, E=0 H=2, C=1, E=-1

4/15/2008 15
Geometric Moment Invariants

m pq = ∫ ∫ x p y q f ( x, y )dxdy (p+q)-th 2D geometric moment

M −1 N −1 Projection of f(x,y) onto monomial x p yq
m pq = ∑∑ x p y q f ( x, y )
x =0 y =0

• Why use moments?

• Geometric moments of different orders represent spatial
characteristics of the image intensity distribution

m00 Total intensity of image. For binary image 2 area

x0 = m10 / m00 Intensity centroid

y0 = m01 / m 00 binary image 2 geometrical center

4/15/2008 16
 Central Moments

M −1 N −1
µ pq = ∑∑ ( x − x0 ) p ( y − y0 ) q f ( x, y ) µ 00 = m00
x =0 y =0
µ10 = µ 01 = 0
[Translation invariance]
µ 02 , µ 20 Variance about the centroid

µ11 covariance

Scaled Central Moment

µ pq
λ pq = µ ' pq /( µ '00 ) ( p + q + 2) / 2
Scale and translation invariant µ ' pq =
α p+q+2

Normalized Un-Scaled Central Moment

4/15/2008
η pq = µ pq /( µ00 ) ( p + q + 2 ) / 2 17
 Moment Invariants
(translation, scale, mirroring, rotation)

φ1 = η 20 + η 02
φ2 = (η 20 − η 02 ) 2 + 4η112
φ3 = (η30 − η12 ) 2 + (η 21 − η 03 ) 2
φ4 = (η30 + η12 ) 2 + (η 21 + η 03 ) 2
ϕ1 = η 20 + η
φ5 = 1
φ6 = 1
φ7 = 1

4/15/2008 18
Affine Transform & Affine Moment Invariants
m m−r
x' = ∑∑ ark x r y k
x' = Tx ( x, y ) r =0 k =0
m m−r
y ' = Ty ( x, y ) y ' = ∑∑ brk x r y k
r =0 k =0

In practice: bilinear transform

x' = a0 + a1 x + a2 y + a3 xy
y ' = b0 + b1 x + b2 y + b3 xy
4 pairs of corresponding points
needed to find coefficients

In practice: affine transform

x' = a0 + a1 x + a2 y
3 pairs of corresponding points
needed to find coefficients
y ' = b0 + b1 x + b2 y

Rotation: Scale: Skew:

x' = x cos φ + y sin φ x' = ax x' = x + y tan φ

4/15/2008
y ' = − x sin φ + y cos φ y ' = by y' = y 19
 Elliptical Shape Descriptors

Principal moment of inertia:

Image ellipse characterizes
( µ 20 + µ 02 ) + [( µ 20 − µ 02 ) + 4µ ]
2 2 1/ 2
fundamental shape features and
I1 = 11
2 also 2D position and orientation
( µ 20 + µ 02 ) − [( µ 20 − µ 02 ) 2 + 4µ112 ]1/ 2
I2 =
2

θ
( I1 + I 2 ) / m00
2
spreadness 2µ11
θ = 0.5 tan −1 ( )
µ 20 − µ 02
( I 2 − I1 ) /( I1 + I 2 ) elongation
a = 2( I1 / µ 00 )1/ 2 b = 2( I 2 / µ 00 )1/ 2

4/15/2008 20
Texture - Definition

4/15/2008 21
 Texture – Quantification Methods

1 Statistical: compute local features at

each point in image and derive a set
of statistics from the distribution of
local features
1 1st, 2nd, and higher-order statistics
based on how many points are
used to define local features

1 Structural: texture is considered to

be composed of “texture elements”.
Properties of the “texture elements”
and their spatial placement rules
characterizes the texture
1 Original texture can be
reconstructed from its structural
description

4/15/2008 22
Statistical Texture Analysis
1st order statistics

image f → hf histogram

• Obtain statistics of the histogram:

L −1
Mean: ∑ ih(i)
i =0
: average intensity

L −1
Variance:
∑ (i − µ )
i =0
2
h(i ) : measure of intensity contrast

L −1
Skewness: ∑ (i
i =0
− µ ) 3
h(i )
L −1
Entropy: − ∑ h(i ) log h(i ) Measure of variability of intensity
4/15/2008 i =0 23
4/15/2008 24
 Statistical Texture Analysis
1st order statistics
image
histogram
statistics

Skewness = 2.08 Skewness = 2.44 Skewness = -0.092

Entropy = 0.88 Entropy = 0.77 Entropy = 0.97

4/15/2008 25
 Statistical Texture Analysis
2nd order statistics: Co-occurrence
f (m2 , n2 ) = j
d
θ P( d ,θ ) (i, j ) ≈ Pr[ f (m1 , n1 ) = i, f (m2 , n2 ) = j ]
f (m1 , n1 ) = i
1 2 j L

1
1
Joint gray-level histogram of pairs of
2
pixels P( d ,θ ) ( i , j )
1 2D histogram
i

4/15/2008 26
 Statistical Texture Analysis
2nd order statistics: Co-occurrence
P( d ,θ =0° ) (i, j ) =| {((k , l ), (m, n)) ∈ ( M × N ) × ( M × N ) :
k − m = 0, | l − n | = d , f (k , l ) = i, f (m, n) = j} |
P( d ,θ = 45° ) (i, j ) =| {((k , l ), (m, n)) ∈ ( M × N ) × ( M × N ) :
(k − m = d , l − n = −d ) ∨ (k = m = −d , l − n = d ), f (k , l ) = i, f (m, n) = j} |

P( d ,θ =90° ) (i, j )
90°
P( d ,θ =135° ) (i, j )
135° 45°
k

{1} is set cardinality

0°
j
4/15/2008 27
Statistical Texture Analysis
2nd order statistics: Co-occurrence (example)
image Co-occurrence matrix

4/15/2008 28
 Statistical Texture Analysis
2nd order statistics: Co-occurrence (statistics)
L L

Angular 2nd moment (energy): ∑∑ ( d ,θ ) (i, j )

P 2

i =1 j =1
(measure of image homogeneity)

Maximum Probability: max P( d ,θ ) (i, j )

i, j

L L
− ∑∑ P( d ,θ ) (i, j ) logP( d ,θ ) (i, j )
Entropy: i =1 j =1

L L

∑∑ λ κ
i − j P( d ,θ ) (i , j )
Contrast: i =1 j =1
(measure of local variations) L L

∑∑ [ijP
i =1 j =1
( d ,θ ) (i, j )] − µ x µ y
Correlation:
(measure of image linearity) σ xσ y
L L L L
4/15/2008 µ x = ∑ i ∑ P( d ,θ ) (i, j ) σ x = ∑ (i − µ x ) 2
∑P ( d ,θ ) (i, j ) 29
i =1 j =1 i =1 j =1
4/15/2008 30
 Statistical Texture Analysis
2nd order statistics: Difference Statistics
P( d ,θ ) (k ) = ∑P ( d ,θ )
i , j∈{1,1, L}
(i, j ) is a subset of co-occurrence matrix

|i − j | = k

L −1

Angular 2nd moment (energy): ∑ (d ,θ ) (k )

P 2

k =0

L −1
Mean: ∑ kP
k =0
( d ,θ ) (k )

L −1

Entropy: − ∑ P( d ,θ ) (k ) log P( d ,θ ) (k )
k =0

L −1
Contrast: ∑ P(d ,θ ) (k )
k 2

k =0

4/15/2008 31
 Statistical Texture Analysis
2nd order statistics: Autocorrelation

M − pN −q

MN ∑ ∑ f ( k , l ) f ( k + p, l + q )
C ff ( p, q ) = k =1 l =1
( M − p )( N − q ) M N

∑∑
k =1 l =1
f 2 (k , l )

Large texture elements 3 autoccorrelation decreases slowly with increasing distance

Small texture elements 3 autoccorrelation decreases rapidly with increasing distance

Periodic texture elements 3 periodic increase & decrease in autocorrelation value

4/15/2008 32
Statistical Texture Analysis
2nd order statistics: Fourier Power Spectrum
f ( x, y ) ↔ F (u , v)
Note:
Power Spectrum P(u, v) =| F (u , v) |2
C ff = F −1{| F (u, v) |2 }
v
π Indicator for size of
P ( r ) = 2∑ P ( r , θ ) dominant texture element
u or texture coarseness
θ =0

L/2
v
P(θ ) = ∑ P(r , θ )
Indicator for the
r =0
directionality of the texture
u
4/15/2008 33
Law’s Texture Energy Measures
L3 = [1,2,1] E3 = [−1,0,1] S3 = [−1,2,−1]

L5 = [1,4,6,4,1]
E5 = [−1,−2,0,2,1]
S5 = [−1,0,2,0,−1]
R5 = [1,−4,6,−4,1]
W5 = [−1,2,0,−2,−1]

 −1 0 0 − 1
2
− 4 0 0 − 4
8
 •Convolute different Law’s
L5 × S5 = − 6
T
0 12 0 − 6 masks with image
 
− 4 0 8 0 − 4 • Compute energy statistics

4/15/2008
 − 1 0 2 0 − 1  34
4/15/2008 35
4/15/2008 36
4/15/2008 37
4/15/2008 38
4/15/2008 39
 Motion – object

Difference image:
d (i, j ) = 0 if | f1 (i, j ) − f 2 (i, j ) | ≤ ε object
motion
1 otherwise

No motion direction information !

f1 f2

n
d cum (i, j ) = ∑ ak | f1 (i, j ) − f k (i, j ) |
k =1 Cumulative difference image

Tells us how often the image gray

level was different from gray-level
of reference image

absolute positive negative

4/15/2008 40
Motion Field
• A velocity vector is assigned to each pixel in the image
• Velocities due to relative motion between camera and the 3D
scene
• Image change due to motion during a time interval dt
• Velocity field that represents 3-dimensional motion of object
points across 2-dimensional image

Motion field
4/15/2008 41
 Optical flow
• Motion of brightness patterns in image sequence

• Assumptions for computing optical flow:

• Observed brightness of any object point is
constant over time
• Nearby points in the image plane move in a
similar manner

∂f ∂f ∂f
f ( x + dx, y + dy, t + dt ) = f ( x, y, t ) + dx + dy + dt + O(∂ 2 )
∂x ∂y ∂t
Gray-level difference at same
= f ( x, y, t ) + f x dx + f y dy + f t dt + O(∂ )
2 location over time is
equivalent to product of spatial
gray-level difference and
dx dy velocity
f ( x + dx, y + dy, t + dt ) = f ( x, y, t ) ⇒ − f t = f x + fy
dt dt
− f t = f x u + f y v = ∇f .c
dx dy
c=( , ) = (u, v)
dt dt
4/15/2008 known unknown 42
Optical Flow Constraints

− f t = f xu + f y v

• no spatial change in brightness, induce no temporal

change in brightness 2 no discernible motion

• motion perpendicular to local gradient induce no temporal

change in brightness 2 no discernible motion

• motion in direction of local gradient, induce temporal

change in brightness 2 discernible motion

• only motion in direction of local gradient induces temporal

change in brightness and discernible motion

4/15/2008 43
Optical flow != Motion Field

MF ≠ 0 MF = 0
OF = 0 OF ≠ 0

4/15/2008 44
Which descriptors?
Image Feature Evaluation

1. Prototype Performance
1 Classify (Segment) image using
different features
1 Evaluate which feature is optimal
(minimum classification error)

2. Figure of Merit
1 Establish functional distance
measurements between set of
image features (large distance 2
low classification error)
1 Bhattacharyya distance

4/15/2008 45