Lecture 02
Image Formation
Davide Scaramuzza
Outline of this lecture
• Image Formation
• Other camera parameters
• Digital camera
• Perspective camera model
• Lens distortion
• Camera calibration
– DLT algorithm
Historical context
• Pinhole model: Mozi (470-390 BCE),
Aristotle (384-322 BCE)
• Principles of optics (including lenses):
Alhacen (965-1039 CE) Alhacen’s notes
• Camera obscura: Leonardo da Vinci
(1452-1519), Johann Zahn (1631-1707)
• First photo: Joseph Nicephore Niepce (1822)
• Daguerréotypes (1839)
• Photographic film (Eastman, 1889)
• Cinema (Lumière Brothers, 1895) Niepce, “La Table Servie,” 1822
• Color Photography (Lumière Brothers, 1908)
• Television (Baird, Farnsworth, Zworykin, 1920s)
• First consumer camera with CCD:
Sony Mavica (1981)
• First fully digital camera: Kodak DCS100 (1990)
CCD chip
Image formation
• How are objects in the world captured in an
image?
How to form an image
object film
• Place a piece of film in front of an object
Do we get a reasonable image?
Pinhole camera
object barrier film
• Add a barrier to block off most of the rays
– This reduces blurring
– The opening is known as the aperture
Camera obscura
In Latin, means ‘dark room’
• Basic principle known to Mozi (470-390
BC), Aristotle (384-322 BC)
• Drawing aid for artists: described by
Leonardo da Vinci (1452-1519)
• Image is inverted
• Depth of the room (box) is the effective
focal length
"Reinerus Gemma-Frisius, observed an eclipse of the sun at Louvain on January 24, 1544,
and later he used this illustration of the event in his book De Radio Astronomica et
Geometrica, 1545. It is thought to be the first published illustration of a camera obscura..."
Hammond, John H., The Camera Obscura, A Chronicle
Camera obscura at home
http://www.youtube.com/watch?v=B2aOs8RWntg
Sketch from http://www.funsci.com/fun3_en/sky/sky.htm
Home-made pinhole camera
What can we do
to reduce the blur?
Effects of the Aperture Size
• In an ideal pinhole, only one ray of light reaches each point on
the film the image can be very dim
• Making aperture bigger makes the image blurry
Shrinking the aperture
Why not make the aperture
as small as possible?
Shrinking the aperture
Why not make the aperture
as small as possible?
• Less light gets through
(must increase the
exposure)
• Diffraction effects…
Image formation using a converging lens
object Lens film
• A lens focuses light onto the film
• Rays passing through the Optical Center are not deviated
Image formation using a converging lens
Lens
object
Focal Point
Optical Axis
Focal Length: f
• All rays parallel to the Optical Axis converge at the Focal
Point
Thin lens equation
Object Lens
A Focal Point
B
Image
f
z e
• Similar Triangles: B e
A z
Find a relationship between f, z, and e
Thin lens equation
Object Lens
A A Focal Point
B
Image
f
z e
B e “Thin lens equation”
• Similar Triangles:
A z e e 1 1 1
1
B e f e f z f z e
1
A f f Any object point satisfying
this equation is in focus
“In focus”
Lens film
object
Optical Axis Focal Point
“Circle of Confusion”
f or
“Blur Circle”
• There is a specific distance from the lens, at which world points are “in focus” in the
image
• Other points project to a “blur circle” in the image
Blur Circle
Lens
Focal Plane
Object Image Plane
f
L
Blur Circle of radius R
z e
L
• Object is out of focus Blur Circle has radius: R
2e
– A minimal L (pinhole) gives minimal R
– To capture a ‘good’ image:adjust camera settings, such that R remains smaller
than the image resolution
The Pin-hole approximation
• What happens if 𝑧 ≫ 𝑓 ?
Object Lens
A Focal Point
A
B
Image
f
z e
• We need to adjust the image plane such that objects at infinity are in
focus
1 1 1 1 1
f e
f z e f e
0
The Pin-hole approximation
• What happens if 𝑧 ≫ 𝑓 ?
Focal Plane
Object Lens
Focal Point
h
Optical Center
C
h'
or
Center of Projection f
z
• We need to adjust the image plane such that objects at infinity are in
focus
1 1 1 1 1
f e
f z e f e
0
The Pin-hole approximation
• What happens if 𝑧 ≫ 𝑓 ?
Object Lens
Focal Point
h
Optical Center
C
h'
or
Center or Projection f
z
• We need to adjust the image plane such that objects at infinity are in focus
h' f f
h' h
h z z
• The dependence of the apparent size of an object on its depth (i.e. distance from
the camera) is known as perspective
Perspective effects
• Far away objects appear smaller
Perspective effects
Perspective and art
• Use of correct perspective projection indicated in 1st century BCE
frescoes
• During Renaissance time, artists developped systematic methods to
determine perspective projection (around 1480-1515)
Raphael Durer
Playing with Perspective
• Perspective gives us very strong depth cues
hence we can perceive a 3D scene by viewing its 2D representation (i.e. image)
• An example where perception of 3D scenes is misleading is the Ames room
“Ames room”
A clip from "The computer that
ate Hollywood" documentary.
Dr. Vilayanur S. Ramachandran.
Projective Geometry
What is preserved?
• Straight lines are still straight
Projective Geometry
What is lost?
• Length
• Angles
Parallel?
Perpendicular?
Vanishing points and lines
Parallel lines in the world intersect in the image at a “vanishing point”
Vanishing points and lines
Parallel lines in the world intersect in the image at a “vanishing point”
Vertical vanishing
point
(at infinity)
Vanishing
line
Vanishing
Vanishing
point
point
Vanishing points and lines
Parallel planes in the world intersect in the image at a
“vanishing line”
Vanishing Point Vanishing Point
o o
Vanishing Line
Outline of this lecture
• Image Formation
• Other camera parameters
• Digital camera
• Perspective camera model
• Lens distortion
• Camera calibration
– DLT algorithm
Focus and depth of field
• Depth of field (DOF) is the distance between the nearest and farthest objects in a scene that
appear acceptably sharp in an image.
• Although a lens can precisely focus at only one distance at a time, the decrease in sharpness
is gradual on each side of the focused distance, so that within the DOF, the unsharpness is
imperceptible under normal viewing conditions
Depth of field
Focus and depth of field
• How does the aperture affect the depth of field?
• A smaller aperture increases the range in which the object appears
approximately in focus but reduces the amount of light into the
camera
Field of view depends on focal length
• As f gets smaller, image
becomes more wide angle
– more world points project
onto the finite image plane
• As f gets larger, image
becomes more narrow
angle
– smaller part of the world
projects onto the finite
image plane
Field of view
Smaller FOV = larger Focal Length
Field of view
Angular measure of portion of 3d space seen by the camera
Outline of this lecture
• Image Formation
• Other camera parameters
• Digital camera
• Perspective camera model
• Lens distortion
• Camera calibration
– DLT algorithm
Digital cameras
• Film sensor array
• Often an array of charge coupled devices
• Each CCD/CMOS is light sensitive diode that converts photons (light energy)
to electrons
Digital images width
j=1 500
Pixel Intensity with 8 bits
i=1
ranges between [0,255]
height
300
im[176][201] has value 164 im[194][203] has value 37
NB. Matlab coordinates: [rows, cols]; C/C++ [cols, rows]
Color sensing in digital cameras
Bayer grid • The Bayer pattern (Bayer 1976)
places green filters over half of the
sensors (in a checkerboard
pattern), and red and blue filters
over the remaining ones.
• This is because the luminance signal
is mostly determined by green
values and the human visual system
is much more sensitive to high
frequency detail in luminance than
in chrominance.
Color sensing in digital cameras
Bayer grid
Estimate missing
components from
neighboring values
(demosaicing)
Foveon chip design
(http://www.foveon.com) stacks
the red, green, and blue sensors
beneath each other but has not
gained widespread adoption.
Color images:
RGB color space
… but there are
also many other
color spaces… (e.g.,
YUV)
R G B
An example camera datasheet
Outline of this lecture
• Image Formation
• Other camera parameters
• Digital camera
• Perspective camera model
• Lens distortion
• Camera calibration
– DLT algorithm
Perspective Camera
Zc = optical axis
Zc
Pc
u
v
O
O = principal point
p
f Image plane
C Xc
C = optical center = center of the lens
Yc
For convenience, the image plane is usually represented in front of C such that the
image preserves the same orientation (i.e. not flipped)
Note: a camera does not measure distances but angles!
a camera is a “bearing sensor”
From World to Pixel coordinates
Find pixel coordinates (u,v) of point Pw
in the world frame:
Pc P
0. Convert world point Pw to u w
camera point Pc
v
O
Find pixel coordinates (u,v) of point Pc
in the camera frame: p x
y
Zc Zw Xw
1. Convert Pc to image-plane C Xc
coordinates (x,y)
W
Yc Yw
2. Convert Pc to (discretised) pixel
[R|T]
coordinates (u,v)
Perspective Projection (1)
From the Camera frame to the image plane
Pc=( Xc , 0 , Zc )T
Xc
p
Zc Xc
C x
O
f Image Plane
The Camera point Pc=( Xc , 0 , Zc )T projects to p=(x, y) onto the image plane
From similar triangles: x Xc fX c
x
f Zc Zc
1. Convert Pc to image-plane
Similarly, in the general case: coordinates (x,y)
y Yc fY
y c
f Zc Zc
2. Convert Pc to (discretised) pixel
coordinates (u,v)
Perspective Projection (2)
From the Camera frame to pixel coordinates
To convert p from the local image plane coords (x,y) to the pixel coords (u,v), we need to
account for:
the pixel coords of the camera optical center O (u0 , v0 ) (0,0) u Image plane
Scale factors ku , kv for the pixel-size in both dimensions
v
O (u0,v0) x
So: k fX
u u0 ku x u u0 u c
Zc
y p
k fY
v v0 kv y v v0 v c
Zc
Use Homogeneous Coordinates for linear mapping from 3D to 2D, by introducing an extra
element (scale):
u~ u
u
p p v~ v
~
v ~
w 1
Perspective Projection (3)
So: ku fX c
u u0
Zc
u
kv fYc Zc
v v0
Zc Pc
Expressed in matrix form and homogenerous v
O
coordinates:
p
u k u f 0 u0 X c f
v 0 k f v Y
v 0 c
0 0 1 Z c C Xc Image plane (CCD)
Or alternatively Yc
u u 0 u0 X c Xc
v 0 v v0 Yc K Yc Focal length in pixels
0 0 1 Z c Z c
K is called “Calibration matrix” or “Matrix of Intrinsic Parameters”
Sometimes, it is common to assume a skew factor (𝐾12 ≠ 0) to account for possible misalignments between CCD and lens.
However, the camera manufacturing process today is so good that we can safely assume 𝐾12 = 0 and 𝛼𝑢 = 𝛼𝑣 .
Exercise 1
• Determine the Intrinsic Parameter Matrix (K) for a digital camera with image size
640 × 480 pixels and horizontal field of view equal to 90°
• Assume the principal point in the center of the image and squared pixels
• What is the vertical field of view?
Exercise 1
• Determine the Intrinsic Parameter Matrix (K) for a digital camera with image size
640 × 480 pixels and horizontal field of view equal to 90°
• Assume the principal point in the center of the image and squared pixels
640
𝑓= = 320 𝒑𝒊𝒙𝒆𝒍𝒔
𝜃
2 tan 2
320 0 320
K 0 320 240
0 0 1
• What is the vertical field of view?
𝐻 480
𝜃𝑉 = 2 tan−1 = 2 tan −1 = 73.74°
2𝑓 2 ∙ 320
Exercise 2
• Prove that world’s parallel lines intersect at a vanishing point in the camera image
Vertical vanishing
point
(at infinity)
Vanishing
line
Vanishing
Vanishing
point
point
Slide from Efros, Photo from Criminisi
Exercise 2
• Prove that world’s parallel lines intersect at a vanishing point in the camera image
• Let’s consider the perspective projection equation in standard coordinates:
X
u u0
Z
Y
v v0
Z
• Let’s parameterize a 3D line with:
X X0 l
Y Y s m
0
Z Z 0 n
• Now substitute this into the camera perspective projection equation and compute
the limit for 𝑠 → ∞
• What is the intuitive interpretation of this?
Perspective Projection (4)
From the Camera frame to the World frame
X c r11 r12 r13 X w t1
Y r r23 Yw t 2
c 21 r 22
Z c r31 r32 r33 Z w t3 P c= P w
u
v O
Xw Xw Zw Xw
X c r11 r12 r13 t1 p
Y r
t2 Yw
T Yw
W
c 21 r22 r23
Zw
R
Z w Zc Yw
Z c r31 r32 r33 t3 C Xc
1 1
Yc
[R|T]
Projection Matrix (M)
X w Extrinsic Parameters
u Xc
u Y
v K Yc v K R T w
Zw
1 Z c 1
1
Perspective Projection Equation
Outline of this lecture
• Perspective camera model
• Lens distortion
• Camera calibration
– DLT algorithm
Radial Distortion
No distortion Barrel distortion Pincushion
Radial Distortion
• The standard model of radial distortion is a transformation from the ideal coordinates
(𝑢, 𝑣) (i.e., undistorted) to the real observable coordinates (distorted) (𝑢𝑑 , 𝑣𝑑 )
• The amount of distortion of the coordinates of the observed image is a nonlinear function
of their radial distance . For most lenses, a simple quadratic model of distortion produces
good results
𝑢𝑑 2
𝑢 − 𝑢0 𝑢0
𝑣𝑑 = 1 + 𝑘1 𝑟 𝑣 − 𝑣0 + 𝑣0
where 𝑟 2 = 𝑢 − 𝑢0 2 + 𝑣 − 𝑣0 2
• Depending on the amount of distortion (an thus on the camera field of fiew), higher order
terms can be introduced:
𝑢𝑑 2 + 𝑘 𝑟4 + 𝑘 𝑟6
𝑢 − 𝑢0 𝑢0
𝑣𝑑 = 1 + 𝑘1 𝑟 2 3 𝑣 − 𝑣0 + 𝑣0
𝑟 2 = 𝑢 − 𝑢0 2 + 𝑣 − 𝑣0 2
Summary: Perspective projection equations
• To recap, a 3D world point 𝑃 = 𝑋𝑤 , 𝑌𝑤 , 𝑍𝑤 projects into the image point
𝑝 = 𝑢, 𝑣
~ X w 0 u0
u u Y 0 v
p v v K R T
~ ~ w where K 0
~ Zw 0 0 1
w
1
1
and λ is the depth (λ = 𝑍𝐶 ) of the scene point
• If we want to take into account the radial distortion, then the distorted
coordinates 𝑢𝑑 , 𝑣𝑑 (in pixels) can be obtained as
𝑢𝑑 2
𝑢 − 𝑢0 𝑢0
𝑣𝑑 = 1 + 𝑘1 𝑟 𝑣 − 𝑣0 + 𝑣0
where 𝑟 2 = 𝑢 − 𝑢0 2 + 𝑣 − 𝑣0 2
Summary (things to remember)
• Perspective Projection Equation
• Intrinsic and extrinsic parameters (K, R, t)
• Homogeneous coordinates
• Normalized image coordinates
• Image formation equations (including radial distortion)
