Lane Detection and Tracking by Video Sensors

Jens Goldbeck Bernd Huertgen

Robert Bosch GmbH, FV/SLH Robert Bosch GmbH, FV/SLH
P.O. Box 77 77 77, P.O. Box 77 77 77,
3 1132 Hildesheim, Germany, 3 1132 Hildesheim, Germany,
jemgoldbeck @ fr.bosch.de bernd.huertgen @ fr.bosch.de

Abstract system design have to be pursued. Only by considering the

whole signal flow from optics through sensor, sensor
The paper describes a video sensor system f o r vehicle electronics, digital signal processing electronics, algorithms
environment sensing. The system comprises a high dynamic up to the final actuator and/or driver information system a
range CMOS camera, dedicated image evaluation circuitry powerful video sensor system can be realized.
and image processing software f o r capturing vehicle This paper presents an appropriate system design mainly
environment data. The main part of the paper is concerned consisting of a high luminance dynamic CMOS video
with the description of a real-time lane detection and camera, a video signal processing and evaluation system
tracking algorithm which is able to determine the course of together with interface circuitry connecting the sensor to the
the road ahead of the vehicle together with the position of vehicle’s CAN infrastructure. The system is used for the
the vehicle relative to the road with high accuracy. Keji development of new comfort, assistance, and control
feature of the presented system is its ability to adapt to functions. The contribution focuses on the description of a
nearly all weather conditions and road types without any lane detection and tracking algorithm. Nevertheless the
external adjustments together with its selfassessment overall system design together with other possible
capabilities, both of which are crucial f o r automotive applications is scetched at a glance [7].
applications. The remainder of the paper is organized as follows: The
overall system design along with camera technology and
image processing hardware is described in the next chapter.
1 Introduction Special emphasis is put on the detection and tracking of lane
boundaries which is examined in detail in the subsequent
chapters. The paper concludes with some results and a brief
With the rapid development of hard- and software
summary.
components image processing systems gain more and more
attraction as sensor and measurement devices. A rather new
field of image processing systems, partly due to cost and 2 Overall system design
safety constraints, are automotive applications for which
increasing interest and effort can be observed within the last The entire system comprises an image acquisition unit, an
decade. Up to now numerous contributions to this field have image processing unit, a data interface, and a data
been made, among of them such pioneering ones like [2] visualization unit.
[3][5][8] and others without claiming completeness. The
aims of all researches are somewhat different ranging from
simple warning devices and more complex driver support 2.1 Image acquisition
systems to highly sophisticated automated highway systems
or even automated driving in public traffic. However, those The image acquisition system is mounted below the
different applications necessitate very similar basic image interior rear mirror and observes the area in front of the car.
processing tasks like determination of lane boundaries,
In automotive applications the high luminance dynamic of
detection of obstacles and interpretation of road signs.
the natural outdoor scenes poses a problem for the
The paper is concerned with the description of a video
commonly used CCD cameras. This is the reason why we
sensor system suitable for analysing the environment of a
feel that only a high dynamic range camera is able to cope
vehicle. Current CCD technology and video components with high luminance dynamic which frequently occurs in
from the consumer market do not fulfill automotive normal traffic. Typical situations include rising or setting
requirements, i.e. thermal stability, shock resistance, sun, headlights or reflections thereof on wet road surfaces
environmental illumination range, reliability, functional
during night which overtax standard CCD cameras resulting
robustness and more. To meet these challenges new paths in

-74-
0 1999 IEEE
0-7803-4975-X/98/$10.00
in extremely degraded image quality or even complete pointing to front (along the tangent to the middle of the
blindness of the camera for a couple of frames. It is obvious lane), left, and upwards direction, respectively (figure 1).
that in these cases image processing software and therewith
the entire system fails. Hence it is essential to enhance the
camera’s brightness dynamic capabilities.
A possible solution to this problem is to realize a
monotonic non-linear mapping function whose sensitivity
decreases with increasing light intensity. A natural choice
are modern CMOS imagers which exploit the logarithmic
behavior of MOS transistors operating in the subthreshold
region. Sensors of this type are able to cope with most of the
above described critical situations and therefore are first
choice for the vision system described in this paper. e

2.2 CMOS camera Figure 1. Used axis systems.

Due to the application of non-standard CMOS imagers an The video sensor models horizontal curvature of the
entirely new camera has been designed. Its highlights are parallel road boundaries by a third order polynomial as a
high brightness dynamic range of 1: 100.000, high resolution close approximation of the clothoid:
of 512x256 pixels, 2:l aspect ratio, progressive scan, partial
image read out capability, 10 bit resolution, logarithmic
compression, digital image data output, and on board CAN
interface for camera control. Due to the partial image read
out capability the frame rate is not fixed to 25 or 30 frames
per second but may vary in a wide range depending on the
with k= {-+ 1 left boundary
1 right boundary
desired application and size of the image area to be read out B : widthof lane
in one frame. With the current design 20 Mega pixel per c h,O : horizontal curvature of lane
second can be read out leading to a maximum full frame rate
of 160 frames per second. c h , l: changeof horizontal curvatureof lane

Likewise, vertical bending of the lane is modeled by a third

2.3 Image evaluation hardware order polynomial:

The main task of the image evaluation hardware is to

provide a physical basis for the application software which
evaluates the digitized video sequence provided by the
camera. In the case described here this means determination with c ”,, : vertical curvature of lane
of lane course and vehicle’s position regarding the lane. c : change of vertical curvature of lane
Image processing is carried out by a standard Pentium PC
based system. A 200 MHz MMX type is sufficient to If the vehicle is first translated and then rotated with respect
evaluate 50 full frames per second. In order to reduce the to the earth fixed axis system, a point f i E has the following
computational burden of the main processor for simple tasks representation in the vehicle’s axis system (figure 1):
like prefiltering or subsampling, a freely configurable FPGA -
coprocessor is employed. fi, =REV (2, -t,v) (3)
-..
with REV =Rx,R U E &, 7 tE, = (0, 70) .
3 Roadmodel
If the camera is first translated and then rotated about the
A fundamental component of video-based lane detection Z , -axis by the pan angle y V c , the Y, -axis by the tilt
and tracking is road-modeling enabling model based image angle avC,and the X , -axis by the roll angle 1 v c ,a point
analysis of road scenes. Related to the well known X, can be represented in the camera axis system as follows
approaches of other researchers [2][3][5] our video system
(figure 1):
models the course of the lane by an approximation of a
+
clothoid.
2, =Eve (2, -tvc) (4)
Let X E =(X,,YE,Z,)T denote a 3d-point in an earth
fixed axis system with its origin represented by the where elements of i,, = ( X v c , Yvc,Zvc) T and
projection of the centre of the vehicle’s rear axle onto the T
0°C = (Ux, WY” 3 > wzv 1 = (Yvc “vc, wvc ) T
3 are the so-
ground moved to the middle of the lane and X,, Y,,Z,

-75-
called external camera parameters which are determined by Inserting (l), (2), (9), (IO), and (1 1) into (12) we get
an extrinsic camera calibration [4]. Inserting (3) into (4) we
get the representation of a point 2 , in the camera axis x = g " ( X E , P ) and (1 3 )

--
system: Y=K(xE,P). (14)

fie = ~ v c ~ 2E, v- ~ v c ~ iEE v -~vcivc (5) 6 = ( B , ch,O > C h . 1 , c v , O , '",I, 'EV 9 QEV 3 YEV WEV )T

REC !i
cc denotes the vector of model parameters to .be estimated.
Resolving (14) leads to
For small angles the linearized rotation matrices are given
by xE =h(y,P) (1 5)
1 wvc -avc and putting (15) into (1 3) we get

x = g(y,I3) . (1 6)

1'
V/EV -aEV
Vehicle model
REV

and
[
= -WEV
aEV

EEC =RVCREV
-;EV

.
"
(6)
Since the system uses merely data provided by the wheel
spin sensors to determine the curvature co of the vehicle's
trajectory, only a simple dynamic system is applied here:

Under the assumption that the product of two rotation angles

is much smaller than one the rotation matrix EECsimplifies
1 0 0 0 0 0 0 0 0
to:
0 1 vkAtk.k+l 0 0 0 0 0 0
0 0 1 0 0 0 0 0 0
1 w V C +'l/EV -(ffVC +aEV)
0 0 0 1 vkAtk,k+, 0 0 0 0
1 YVC YEV 0 0 0 I 0 0 0 0
!Pk =
aVC + f f E V -(YVC + YEV ) 1 (vkAtk,k+l)2 (vkAtk,k+\)3
'kAtk.k+l
6
0 0 0 0 1 0 0
0 0 0 0 0 0 1 0
Inserting (6), (7) and -'kAtk,k+l -___
(vkAtk,k+l)' I
+ - 2
R v c tvc = m vc

into ( 5 ) leads to

g i k=

where v denotes the vehicle's velocity in X E -direction

and At the time difference between two successive frames.
central projection yields the image coordinates U is called the input vector and the transition matrix 9
ii,, = ( ~ , y ofapoint
) ~ G~ : describes the dynamical behaviour of the system. The index
k is related to the instant t k at which the corresponding
(12 ) frame is taken.

where F, , F,. denote the focal lengths in pixel coordinates 5 Parameter estimation
and c, , c ,. are the coordinates of the principal point in pel.
In this case the fitting of the 2d-measurements to the
These interior camera parameters are determined by an above derived 3d-model is a non-linear task which can be
intrinsic camera calibration [4]. solved by use of an extended Kalman filter (EKF) [ 1][5][6].

-16-
On the basis of least squares estimation the EKF computes To minimize the influence of outliers, variances of
an optimal estimation of the inaccessible parameter vector measurements are modeled by use of a non-quadratic
p during a prediction (time update) and an innovation weighting function considering the normalized quality of the
(measurement update) step. measured points.
Taking into account that the above discrete and time After completion of parameter estimation a measure of
invariant system is subject to random noise, it can be confidence and reliability is computed to quantify the
described by its linear system function current performance of the sensor based on the covariance
- matrix and the model fit.
P k + l = fk ( I 3 k 2 G k 1+ q k (1 8) Finally the estimated model parameters and the self-
with system noise q and by its non-linear measuring assessment information are provided on the CAN bus for
further processing.
function

X k = g k (Pk , Y k ) + f k (19) 6 Detection of border points

with measurement noise f . The so-called Jacobi matrix can The task is to detect such image points which lie on the
be determined by linearizing (19): inside of visual lane boundaries. These lane border points
are characterized by grayvalue based gradients formed by a
difference in grayvalue between the surface of the road and
lane boundaries, e.g. lane markings, soft shoulders or
gutters. To be independent of absolute grayvalues, the
At the beginning of the estimation process the parameter
following gradient based method similar to [9] is used.
vector pk,o at tk,O with its covariance matrix Po have to Search lines are placed in the image at a constant distance
be initialized. Then the system state at instant tk+l can be in the earth fixed axis system. The position of the centre of
predicted using (17) and each search line is defined by the projection of the predicted
road model onto the image plane at the corresponding
distance. The width of search lines and their maximal
distance are adapted according to the variances of the
Q of the system noise. The
with the covariance matrix - predicted model parameters. The smaller the predicted
predicted system state can be improved within the variances are, the more one can trust into the estimated
innovation step using measurements (X, Y ) k + , : model parameters leading to narrowing the search lines.
To detect the position of possible lane border points, two
non-parallel directional derivations of the current image
6 k + l = f j k + l + K k + l (%k+l -gk+l ( p k + l 3 f k + l ) ) (22) signal are computed using isotropic filter kernels. The
where the Kalman gain can be calculated as follows: advantage of isotropic over anisotropic filters is the simple
implementation of the former in hardware.
Kk+l = %+I c5+1G+,C+I 2k+l +%+I )-I (23)
Along each search line the derivational signals are
evaluated in the direction perpendicular to the predicted lane
with the covariance matrix 11 of the measurement noise. course at the centre of the search line. At first the resulting
The covariance matrix of the estimation error can be gradient signal is searched for dark-light-dark (DLD)
grayvalue transitions. DLD-transitions describe white or
improved to:
yellow lane markers on a gray surface and they consist of
!k+I =fk+l -Kk+l C k + l 2 k + l ' (24) two (single) gradients which are similar by magnitude but
antiparallel in orientation, the so-called double gradients.
Parameter estimation in this way leads to the following From amplitude, position of the (single) gradients forming a
problems. The first and the second one which are due to double gradient and their orientation a fitting value q is
digital rounding errors are already solved in [ 11: calculated that quantifies the quality of each lane border
To avoid covariance matrix becoming point. This value of quality is used in the parameter
unsymmetrical, it can be splitted into a diagonal matrix estimation process to fix the weight of the corresponding
-
D and an upper triangular matrix : = mT measurement.
If no double gradient can be found on a search line, in a
To avoid covariance matrix becoming indefinite, a second step the gradient signal is searched for single
sequential innovation step of the EKF can be maxima which are caused by a simple light-dark (LD)
implemented assuming measurements are statistical grayvalue transition describing a single gradient. These
independent. transitions occur if lane markers are missing or not
Because of its least squares minimization the EKF is completely covered by a search line. Likewise, single
susceptible to outliers and therefore not robust. gradients are assessed by a value of quality.

-11-
 Important issues in lane tracking are the reliability of the
estimated lane and the robustness of the estimation process.
Reliability and robustness can be improved if measurements
feeding the EKF contain as little outliers as possible. For
that reason a measurement point on the corresponding lane
border is rejected and therefore not passed to the estimation
process if its set of features gained during the course of
measurement detection does not meet the criteria demanded
for inliers.
Frame 3

7 System test

System tests have been performed online in the test

vehicle as well as with image sequences taken in a variety of
scenes with different environmental conditions.
Our video sensor is able to detect and track lane
boundaries not only during diffuse daylight on dry asphalt
but also during rain or back-lighting situations on wet
pavement. Furthermore lane tracking is possible, both on
Frame 4
highways and winding country roads even if one lane
boundary leaves the detection area of the camera.
Results are occasionally presented by a challenging scene
which shows an access road to a highway at bright daylight
(figure 2). The scene describes a small curve to the right at
an ascending road whose lane width is decreasing. Although
several line structures are in close vicinity of the lane
markers and the right lane border gets lost later on, lane
recognition works reliably. It must be said that if one lane
boundary cannot be detected any more, lane width and tilt
angle cannot be updated. Frame 5
Time difference of successive frames belonging to this
Figure 2. Access road to highway.
scene is 40 ms and temporal difference of the five presented
images is about one second.
8 Summary and conclusion
A system has been presented which enables lane sensing
for different applications, e.g. warning devices, driver
assistance systems or vehicle guidance, using a high
dynamic CMOS video camera to compute the ego-state of
the vehicle as well as lane geometry in real time.
The lane to be described is either bounded by light-
coloured lane markers or by transitions from pavement to
road borders, e.g. soft shoulders or gutters.
Both lane curvature and vertical bending are modeled by
Frame 1 a third order polynomial as a close approximation to the
clothoid. Identification of lane model is performed by a
combination of lane boundary and robust estimation.
The lane boundary detection is based on an edge oriented
method and is able to assess quality of the determined
measurements and of the corresponding lane boundary.
For parameter estimation a general extended Kalman
filter is used. The influence of outliers on the Kalman filter
is reduced by the use of a non-quadratic error function to
model variances of measurements.
Frame 2 The video sensor quantifies its current performance and
provides this self-assessment information, namely

-18-
confidence and reliability, together with model parameters
to the vehicle’s CAN bus for further processing.
Results have been presented, demonstrating that a precise
and reliable lane description is possible even in difficult
situations.

References

[I] G.J. Bierman. Factorization Methods for Discrete Sequential

Esfirnation. Academic Press, New York, 1977
[2] E.D. Dickmanns, A. Zapp. A curvature-based scheme for
improving vehicle guidance by computer vision.
Proc. SPIE Conjerence on Mobile Robots, vol. 727, 1986
[3] W. Enkelmann et al. ROMA - A system for model-based
analysis of road markings. Proc. of the 1995 IEEE In?. Veh.
Sym., Sep. 25-26, 1995, DetroiUMi, USA
[4] S. Ernst et al. Camera Calibration for Lane and Obstacle
Detection. Proceedings ITSC, Oct. 5-8, 1999, Tokyo, Japan,
to be printed
[5] U. Franke. Real time 3D-road modeling for autonomous
vehicle guidance. 7th Scandinavian Conference on Image
Analysis, August 13-18, 1991, Aalborg, Denmark
[6] M.S. Grewal, A.D. Andrews. Kalman Filtering - Theory and
Practice. Prentice Hall, 1993
[7] B. Huertgen et al. Vehicle Environment Sensing by Video
Sensors. 1999 SAE hit. Congress and Exposition, march 1-4,
1999, Detroit/Mi, USA, to be published
[8] D.A. Pomerleau. RALPH: Rapidly Adaptive Lateral Position
Handler, Proc. of the 1995 IEEE Int. Veh. Sym., Sep. 25-26,
1995, Detroit/Mi, USA
[9] R. Risack et al. Robust lane recognition embedded in a real-
time driver assistance system,
Proc. of the 1998 IEEE Cor$ 011 Int. Veh., Oct. 28-30,
Stuttgart, Germany, vol. 1 , pp. 35-40

-79-