Lecture

4
Single View Metrology

Professor Silvio Savarese

Computa(onal Vision and Geometry Lab

Silvio Savarese Lecture 4 - 12-Apr-17

Lecture 4
Single View Metrology

Announcements:
- Can you hear my voice well enough?
- Please ll in a short survey on how the class is doing so far
- Remember to review the basics of linear algebra e.g.
SVD, etc!!

Silvio Savarese Lecture 4 - 12-Apr-17

Lecture 4
Single View Metrology

Review calibraHon and 2D transformaHons
Vanishing points and lines
EsHmaHng geometry from a single image
Extensions
Reading:
[HZ] Chapter 2 ProjecHve Geometry and TransformaHon in 2D
[HZ] Chapter 3 ProjecHve Geometry and TransformaHon in 3D
[HZ] Chapter 8 More Single View Geometry
[Hoeim & Savarese] Chapter 2

Silvio Savarese Lecture 4 - 12-Apr-17

 Calibration Problem

jC

! u $
pi = # i
&=M Pi M = K[R T]
#" vi &%
11 unknowns
In pixels World ref. system
Need at least 6 correspondences
 Once the camera is calibrated...
Pinhole perspective projection

Line of
sight
P
p
Ow
C

M = K[R T]
-Internal parameters K are known
-R, T are known but these can only relate C to the calibration rig
Can I estimate P from the measurement p from a single image?

No - in general L (P can be anywhere along the line defined by C and p)

 Recovering structure from a single view

http://www.robots.ox.ac.uk/~vgg/projects/SingleView/models/hut/hutme.wrl
 Transformation in 2D

-Isometries

-Similarities

-Affinity

-Projective
 TransformaHon in 2D

x ' x x
y' = R t y
Isometries: 0
1 y = H e
[Eq. 4]
[Euclideans] 1 1 1

- Preserve distance (areas)

- 3 DOF
- Regulate moHon
of rigid object
 TransformaHon in 2D

! x' $ ! x $ ! x $
# & ! SR t $# & # &
SimilariHes: # y' & = # &# y & = H s # y &
# 1 & #" 0 1 &%# & # 1 &
" % " 1 % " %
! s 0 $ [Eq. 5]
S =# &
" 0 s %

- Preserve
- raHo of lengths
- angles
-4 DOF
 TransformaHon in 2D

x ' x x
y' = A t y
AniHes: 0
1 y = H a
1 1 1 [Eq. 6]

a 11 a12 s x 0
A = = R ( ) R ( ) D R ( ) D =
a 21 a 22 s y
0
[Eq. 7]
 TransformaHon in 2D

x ' x x
y' = A t y
AniHes: 0
1 y = H a
1 1 1 [Eq. 6]

a 11 a12 s x 0
A = = R ( ) R ( ) D R ( ) D =
a 21 a 22 s y
0
[Eq. 7]
-Preserve:
- Parallel lines
- RaHo of areas
- RaHo of lengths on
collinear lines
- others
- 6 DOF

 TransformaHon in 2D

x ' x x
y' = A t y
ProjecHve: v
b y = H p
[Eq. 8]
1 1 1

- 8 DOF
- Preserve:
- collinearity
- cross raHo of 4 collinear points
- and a few others
 The cross raHo
The cross-raHo of 4 collinear points is dened as

[Eq. 9]
P4 X i
P3 P3 P1 P4 P2 Y
Pi = i
Z i
P2 P3 P2 P4 P1
P1 1
Lecture 4
Single View Metrology

Review calibraHon and 2D transformaHons
Vanishing points and lines
EsHmaHng geometry from a single image
Extensions
Reading:
[HZ] Chapter 2 ProjecHve Geometry and TransformaHon in 2D
[HZ] Chapter 3 ProjecHve Geometry and TransformaHon in 3D
[HZ] Chapter 8 More Single View Geometry
[Hoeim & Savarese] Chapter 2

Silvio Savarese Lecture 4 - 12-Apr-17

 Lines in a 2D plane
y
ax + by + c = 0 -a/b l
a -c/b

l = b x

c T
x1 a
If x = [ x1, x2]T l x b = 0
2
1 c
[Eq. 10]
 Lines in a 2D plane
y
Intersecting lines l
x
x = l l [Eq. 11] l
x
Proof

l l l (l l) l = 0 x l [Eq. 12]
l l l (l l) l = 0 x l [Eq. 13]
x

x is the intersecting point

 2D Points at infinity (ideal points)
x1 a
a / b = a ' / b'
x = x 2 , x 3 0 l = b
x 3 c
l a '
x'1
x = x'2 l l = b'
0 c

b
Lets intersect two parallel lines:
l l a = x [Eq.13]

0 = ideal point!
In Euclidian coordinates this point is at infinity
Agree with the general idea of two lines intersecting at infinity
 2D Points at infinity (ideal points)
x1 a
a / b = a ' / b'
x = x 2 , x 3 0 l = b
x 3 c
l a '
l l = b'
Note: the line l = [a b c]T pass trough the ideal point x c

b
l T x = [a b c ] a = 0 [Eq. 15]
0

So does the line l since a b = a b

 Lines infinity l
Set of ideal points lies on a line called the line at infinity.
How does it look like?

0 b'

l = 0 x' = a'
0
1
b' '
T ''
x1 0 x = a ' '
0 = 0 0
Indeed: x 2
0 1 l
A line at infinity can be thought of the set of directions of lines in the plane
Projective transformation of a point at infinity

A t
H =
v b

p' = H p is it a point at infinity?

1 p ' x
A t 1 = p '
H p = ? =
b y no!
v 0 p ' z
[Eq. 17]
An affine
1 p ' x transformation
A t 1 = p ' of a point
H A p = ? =
b y at infinity is
0 0 0
still a point at
[Eq. 18] infinity
Projective transformation of a line (in 2D)

A t
H =
v b

T
l = H l is it a line at infinity?

[Eq. 19]
T 0 t x
T A t 0 = t
H l = ? = v b y no!
1 b
[Eq. 20]

T 0 T
0 0
T A t 0 = A 0
H A l = ? = t T AT 0 = 0
0 1 1
[Eq. 21] 1 1 1
 Points and planes in 3D

x 1 a
y

x b
2
x= =
x
x 3 c z

1 d

x xT = 0 ax + by + cz + d = 0
[Eq. 22] [Eq. 23]
 Lines in 3D

Lines have 4 degrees of freedom - hard to

represent in 3D-space

Can be defined as intersection of 2 planes

d = direction of the line

= [a, b, c]T
 Points at infinity in 3D
Points where parallel lines intersect in 3D

point at infinity x1
world x
x = 2
x3

0
Parallel lines
 Vanishing points
The projective projection of a point at infinity into the image
plane defines a vanishing point.

point at infinity x1
p
x M
world
x = 2
x3

0 p1
Parallel lines = direcHon of
corresponding p = p2
parallel lines in 3D
p3
 Vanishing points and directions
d

v
d = direction of the line
= [a, b, c]T
C
1
K v
v=Kd d = 1
K v
[Eq. 24] [Eq. 25]

Proof: " a %
$ ' " a %
" a %
b M $ '
X = $ '
v = M X = K [ I 0] $
b ' = K$ b '
$ c ' $ c ' $ '
$ ' $ ' $ c '
0 0 # &
# & # &
Vanishing (horizon) line
l

Projective transformation M

horizon

T
lhor = H P l
[Eq. 26]
Image
Example of horizon line

The orange line is the horizon!

Are these two lines parallel or not?

Recognition helps reconstruction!

Humans have learnt this

- Recognize the horizon line

- Measure if the 2 lines meet at the horizon
- if yes, these 2 lines are // in 3D
 Vanishing points and planes

n
l
l horiz
C

T
n = K l horiz
[Eq. 27]

See sec. 8.6.2 [HZ] for details

 Planes at infinity
0
y
0
=
0
x
1
z plane at infinity

Parallel planes intersect at infinity in a common line the

line at infinity

A set of 2 or more lines at infinity defines the plane at

infinity
 Angle between 2 vanishing points
x2

x1 v2
d1

v1
d2 C

T
v v2
1
cos = = (K K ) T 1
T T
[Eq. 28] v v1 v v 2
1 2 [Eq. 30]

T
If = 90 v v2 = 0
1 [Eq. 29]
Scalar equation
Projective transformation of a conic

p

C

p = M T M 1
HZ page 73, eq. 3.16
Projective transformation of Absolute conic

T
= M M 1 T 1
= (K K )
HZ page 73
 Properties of

= (K K ) T 1 M = K !" R T #$
[Eq. 30]

1 2 4
1. = 2 3 5 symmetric and known up scale

4 5 6
2 = 0
2. 2 = 0 zero-skew 3. 1 = 3 square pixel
 Summary
T
v=Kd n = K l horiz
[Eq. 24]
[Eq. 27]

v1T v 2 = 90
cos = T
v v2 = 0
T T 1
v v1 v v 2
1 2
[Eq. 29]
[Eq. 28]

T 1
Useful to: = (K K )
[Eq. 30]
To calibrate the camera
To estimate the geometry of the 3D world
Lecture 4
Single View Metrology

Review calibraHon
Vanishing points and line
EsHmaHng geometry from a single image
Extensions
Reading:
[HZ] Chapter 2 ProjecHve Geometry and TransformaHon in 3D
[HZ] Chapter 3 ProjecHve Geometry and TransformaHon in 3D
[HZ] Chapter 8 More Single View Geometry
[Hoeim & Savarese] Chapter 2

Silvio Savarese Lecture 4 - 12-Apr-17

 Single view calibration - example
v1T v 2
[Eq. 28]

cos =
v1T v1 v T2 v 2

v2
v1
o
= 90
[Eq. 29]
T
v v2 = 0
1 Do we have enough constraints to
T 1 estimate K?
= (K K ) K has 5 degrees of freedom and Eq.29
is a scalar equation L
 Single view calibration - example
v1T v 2
[Eq. 28]

cos =
v1T v1 v T2 v 2

v2
v1 v3

T
v v2 = 0
1
[Eqs. 31]

T
v v3 = 0
1
T
v v3 = 0
2
 Single view calibration - example
1 2 4

= 2 3 5
4 5 6

v2
v1
2 = 0 v3
Square pixels

No skew 1 = 3

T
v v2 = 0
1
[Eqs. 31]

T
v v3 = 0
1
T
v v3 = 0
2
 Single view calibration - example
! 0 $

known up to scale
# 1 4
&
= # 0 1 5 &
# &
#" 4 5 6 &%
v2
v1
2 = 0 v3
Square pixels

No skew 1 = 3

T
v v2 = 0
1
[Eqs. 31]

T
v v3 = 0
1
Compute !
T
v v3 = 0
2
 Single view calibration - example
! 0 $
# 1 4
&
= # 0 1 5 &
# &
#" 4 5 6 &%
v2
v1
2 = 0 v3
Square pixels

No skew 1 = 3

T
v v2 = 0
1
[Eqs. 31]

T
v v3 = 0
1
Once is calculated, we get K:
T 1
T
= (K K ) K
v v3 = 0
2 (Cholesky factorization; HZ pag 582)
 Single view reconstruction - example

lh

[Eq. 27]
T
K known n = K l horiz = Scene plane orientation in
the camera reference system

Select orientation discontinuities

 Single view reconstruction - example

Recover the structure within the camera reference system

Notice: the actual scale of the scene is NOT recovered

Recognition helps reconstruction!

Humans have learnt this
Lecture 4
Single View Metrology

Review calibraHon
Vanishing points and lines
EsHmaHng geometry from a single image
Extensions
Reading:
[HZ] Chapter 2 ProjecHve Geometry and TransformaHon in 3D
[HZ] Chapter 3 ProjecHve Geometry and TransformaHon in 3D
[HZ] Chapter 8 More Single View Geometry
[Hoeim & Savarese] Chapter 2

Silvio Savarese Lecture 4 - 12-Apr-17

 Criminisi & Zisserman, 99

http://www.robots.ox.ac.uk/~vgg/projects/SingleView/models/merton/merton.wrl
 Criminisi & Zisserman, 99

http://www.robots.ox.ac.uk/~vgg/projects/SingleView/models/merton/merton.wrl
La Trinita' (1426)
Firenze, Santa Maria
Novella; by Masaccio
(1401-1428)
La Trinita' (1426)
Firenze, Santa Maria
Novella; by Masaccio
(1401-1428)
http://www.robots.ox.ac.uk/~vgg/projects/SingleView/models/hut/hutme.wrl
 Single view reconstruction - drawbacks

Manually select:
Vanishing points and lines;
Planar surfaces;
Occluding boundaries;
Etc..
Automatic Photo Pop-up
Hoiem et al, 05
Automatic Photo Pop-up
Hoiem et al, 05
 Automatic Photo Pop-up
Hoiem et al, 05

Software:
http://www.cs.uiuc.edu/homes/dhoiem/projects/software.html
 Make3D
Saxena, Sun, Ng, 05

Training Prediction

Image Depth

Plane Parameter MRF

Planar Surface
Segmentation

youtube Connectivity Co-Planarity

 Make3D
Saxena, Sun, Ng, 05

A software: Make3D
Convert your image into 3d model

http://make3d.stanford.edu/
http://make3d.cs.cornell.edu/
Depth map reconstruc5on using deep learning
Eigen et al., 2014

Depth Map Prediction from a Single Image using a Multi-Scale Deep Network,
Eigen, D., Puhrsch, C. and Fergus, R. Proc. Neural Information Processing Systems 2014,
3D Layout estimation
Dasgupta, et al. CVPR 2016

58
3D Layout estimation
 Coherent object detection and scene
layout estimation from a single image
Y. Bao, M. Sun, S. Savarese, CVPR 2010,
BMVC 2010

M. Sun Y. Bao
Next lecture:
Multi-view geometry (epipolar geometry)
Appendix
 Vanishing points - example
d
v1, v2: measurements star
K = known and constant v2

Can I compute R?
No rotation around z
C2
1
v1
K v1
d1 = 1
K v1 R,T
C1
K 1 v 2
d 2 = 1
K v2 In 2D
d1
R
d2

R d1 = d 2 R
R =
 K 1 v1
d1 = 1
K v1
v1
K 1 v 2
d 2 = 1
K v2

v2