This article has been accepted for inclusion in a future issue of this journal.

Content is final as presented, with the exception of pagination.

IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS 1

Automatic Road Crack Detection Using Random

Structured Forests
Yong Shi, Limeng Cui, Zhiquan Qi, Fan Meng, and Zhensong Chen

Abstract—Cracks are a growing threat to road conditions and attentions in recent years. As it is known, traditional man-
have drawn much attention to the construction of intelligent ual road crack detection approaches are very time-consuming,
transportation systems. However, as the key part of an intelli- dangerous, labor-intensive and subjective [2]–[5]. Thus, the
gent transportation system, automatic road crack detection has
been challenged because of the intense inhomogeneity along the slow and subjective traditional procedures have been replaced
cracks, the topology complexity of cracks, the inference of noises gradually by automatic crack detection, which is developed for
with similar texture to the cracks, and so on. In this paper, we fast and reliable crack analysis in intelligent transportation sys-
propose CrackForest, a novel road crack detection framework tems [6]. Automated crack detection systems can quantify the
based on random structured forests, to address these issues. Our quality of road surfaces and assist in prioritizing and planning
contributions are shown as follows: 1) apply the integral channel
features to redefine the tokens that constitute a crack and get the maintenance of the road network and thereby accomplish
better representation of the cracks with intensity inhomogeneity; the objective of preserving the roads in good condition and
2) introduce random structured forests to generate a high- extending the service life.
performance crack detector, which can identify arbitrarily com- With the development of image processing techniques, road
plex cracks; and 3) propose a new crack descriptor to characterize crack detection and recognition have been widely discussed
cracks and discern them from noises effectively. In addition, our
method is faster and easier to parallel. Experimental results prove in the past few decades [7]–[11]. In early methods [12], [13],
the state-of-the-art detection precision of CrackForest compared researchers usually use threshold-based approaches to find
with competing methods. crack regions based on the assumption that real crack pixel is
Index Terms—Road crack detection, structured learning, consistently darker than its surroundings. These methods are
machine learning, random structured forests, crack descriptor, very sensitive to noises, since only brightness feature is taken
crack characterization. into consideration. Moreover, these approaches are performed
on individual pixels. Lack of global view also makes these
I. I NTRODUCTION methods unsatisfying.
In terms of the current methods [5], [8], [9], [11], [14], [15],

C RACK is a form of road distresses that may potentially

reduce the road performance and threaten the road safety
[1]. Governments have made a great effort to achieve the goal
most researchers try to suppress the inference of noises by
incorporating features such as gray-level value [11], the mean
and the standard deviation value [5], [6], [16]. In addition, to
of constructing a high quality road network. They are now, improve the continuity of the existing methods, researchers
more than ever, fully aware of the need for adequate road attempt to conduct crack detection from a global view by
inspection and maintenance. Crack detection is an essential introducing methods such as Minimal Path Selection (MPS)
part of road maintenance systems and has attracted growing [17]–[19], Minimum Spanning Tree (MST) [20], [21], Crack
Fundamental Element (CFE) [22], [23] and so on. These meth-
ods can partly eliminate noises and enhance the continuity of
Manuscript received October 17, 2015; revised February 10, 2016; accepted detected cracks.
April 4, 2016. This work was supported by the National Natural Sci- However, these methods do not perform well while dealing
ence Foundation of China under Grants 91546201, 71331005, 71110107026, with cracks with intensity inhomogeneity or complex topology.
and 61402429. The Associate Editor for this paper was A. Eskandarian.
(Corresponding author: Zhiquan Qi.) A possible explanation is that the used features only roughly
Y. Shi is with the School of Economics and Management, University of capture the gray-level information but some unique charac-
Chinese Academy of Sciences, Beijing 100049, China; with Key Laboratory of teristics of crack may not be presented and utilized properly.
Big Data Mining and Knowledge Management, Chinese Academy of Sciences,
Beijing 100190, China; and also with University of Nebraska Omaha, Omaha, Besides, local structured information is ignored by existing
NE 68106 USA (e-mail: yshi@[Link]). methods. In fact, cracks in a local image patch are highly
L. Cui is with the School of Computer and Control Engineering, University interdependent, which often contain well-known patterns, such
of Chinese Academy of Sciences, Beijing 101408, China (e-mail: lmcui932@
[Link]). as longitudinal, transverse, diagonal and so on. Therefore,
Z. Qi is with the Research Center on Fictitious Economy and Data Science, structured learning is proposed to solve similar problems in
Chinese Academy of Sciences, Beijing 100190, China (e-mail: qizhiquan@ recent years. For example, in [24], researchers apply structured
[Link]).
F. Meng and Z. Chen is with the School of Economics and Management, learning to semantic image labeling where image labels are also
University of Chinese Academy of Sciences, Beijing 100049, China (e-mail: interdependent.
im_feixie@[Link]; wxzmczs@[Link]). In order to overcome these two shortages mentioned above,
Color versions of one or more of the figures in this paper are available online
at [Link] we propose a novel road crack detection method (called
Digital Object Identifier 10.1109/TITS.2016.2552248 CrackForest) based on random structured forests, which is
1524-9050 © 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See [Link] for more information.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

2 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

superior to other state-of-the-art detecting techniques like maps are built by a 2D continuous wavelet transform, wavelet
CrackTree [20], CrackIT [6], FFA [25] and MPS [17], [19]. coefficients maximal values are obtained for crack detection.
CrackForest incorporates complementary features from multi- As a result, differences between crack regions and crack free
ple levels to characterize cracks and to take advantage of the regions could be raised up. However, due to the anisotropic
structured information in crack patches. In specific, we first ex- characteristic of wavelets, these approaches may not handle the
tend the traditional road crack detection feature set by introduc- cracks with low continuity or high curvature properly.
ing the integral channel features [26] to re-define crack tokens Minimal Path Selection: Give both endpoints of the curve
with structured information. After that, we apply random struc- as user’s input, minimal path based method can extract simple
tured forests [27] to exploit such structured information. Ran- open curves in images, that is first proposed by Kass et al. [39].
dom structured forests predict a patch crack of structured tokens In [40], Kaul et al. propose a method that is able to detect
that are aggregated across the image to compute our prelimi- the same types of contour-like image structures with less prior
nary crack detection result. In this step, the structured tokens knowledge about both the topology and the endpoints of the
assigned to each image patch can be obtained simultaneously. desired curves. To avoid false detections that are assimilating
Then, the structured tokens are used to construct the crack de- loops, Amhaz et al. [17], [19] propose an improved algorithm
scriptor which consists of two statistical histograms to charac- to select endpoints at the local scale and then to select minimal
terize cracks with arbitrary topology. With the crack descriptor, paths at the global scale. It can also detect the width of the
a classification method is applied to discriminate the cracks crack. In [25], Nguyen et al. propose a method which takes into
from noises. In addition, we also propose a quantitative eval- account intensity and crack form features for crack detection
uation method for road crack detection task. Extensive exper- simultaneously by introducing Free-Form Anisotropy.
iments demonstrate the efficiency of CrackForest on real road Machine Learning: With the increasing size of image data,
crack dataset and our method shows state-of-the-art precision. machine learning based methods [3], [5], [15], [41]–[43] have
become an important branch in detecting road cracks. In [3],
artificial neural network models are used to separate crack
II. R ELATED W ORK pixels from the background by selecting proper thresholds.
In this section, we first give a brief review of crack detection, [41] deals with the detection of poorly contrasted cracks in
after that, the related crack characterization methods are dis- textured areas using a Markov random field model. In [43],
cussed. Crack characterization exploits the spatial distribution Cord et al. use AdaBoost to distinguish images of road surfaces
of image tokens composing the detected cracks and thereby with defects from road surfaces based on textual information
transforms the structured tokens into discrete labels. with patterns. For all these methods, the training and classifi-
cation are conducted on each sub-image and as local method,
they have drawbacks in finding complete crack curves over the
A. Crack Detection
whole image.
Numerous papers have been written on road crack detection
over the past 30 years. Early works [1], [28]–[30] are mainly
B. Crack Characterization
based on intensity-thresholding for its simplicity and efficiency.
Most recent work explores crack detection under more chal- Existing methods on crack characterization are mainly based
lenging conditions and can be divided into five branches: on shape descriptor, crack seeds and assigning crack type on
methods based on saliency detection, textured-analysis, wavelet each image block.
transform, minimal path and machine learning. An assessment Reference [14] gives the definition of cracks based on math-
of various pavement distress detection methods can be found in ematical morphology and proposes that a crack is thought to
[31] and [32]. be a succession of saddle points with linear features. But this
Salient Detection: Salient regions are visually more con- definition is pretty vague. Reference [2], [44] use the direc-
spicuous due to their contrast with the surroundings. Although tion indices of each pixels and extensible directions for each
existing methods [33], [34] demonstrate their effectiveness in direction to characterize cracks. A chromosome representation
detecting salient regions in the Berkeley database [35], they is applied to encode the different ensemble of directions and its
perform poor on the completeness and continuity of detected extensible directions. Therefore, a crack can be represented as
crack. a long sequence of 0 and 1.
Textured-Analysis: Since road surface images are often Reference [31], [42] categorize the cracks into five types:
highly textured, textured-analysis methods [8], [36], [37] are longitudinal, transverse, diagonal, block, and alligator. Refer-
introduced in road crack detection. In order to distinguish the ence [42] uses a neural network based method to search patterns
cracks and the backgrounds, [8], [36] use the Wigner model, of various crack types horizontally and vertically. Reference
and [37] uses classification method. These methods use a [31] uses curves and buffers to describe certain regions of a
local binary pattern operator to determine whether each pixel crack. Reference [9] uses longitudinal, transverse, or diagonal
belongs to a crack and the local neighbor information is not crack seeds to identify longitudinal and transverse cracks. Ori-
taken into consideration. Therefore, the cracks with intensity entation and strength information are taken into consideration
inhomogeneity can not be detected precisely. by [20], which largely improves the diversity of crack seeds.
Wavelet Transform: Wavelet transform is applied to sepa- In [6], cracks are classified into three types as defined by
rate distresses from noises [38]. In [4], complex coefficient the Portuguese Distress Catalog. They use two block feature
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

SHI et al.: AUTOMATIC ROAD CRACK DETECTION USING RANDOM STRUCTURED FORESTS 3

Fig. 3. Procedure of the proposed automatic road crack detection method.

detection method by introducing the integral channel features.

These features extracted from multiple levels and orientations
Fig. 1. Consider the pavement surface shown in (a). (b) Preliminary detection
results after applying random structured forests. Darker color indicates that the allow us to re-define representative crack tokens with richer
pixel is more likely to contain a crack. After eroding and dilating, the result is structured information. In the second part, random structured
shown in (c). (d) Final result after the classification stage. forests are introduced to exploit such structured information,
and thereby a preliminary result of crack detection can be
obtained. In the third part, we propose a new crack descriptor
by using the statistical character of tokens. This descriptor
can characterize the cracks with arbitrary topology. And a
classification algorithm (KNN, SVM or One-Class SVM) is
applied to discriminate cracks from noises effectively. Please
find the results in every part of our method in Fig. 1.

A. Structured Tokens
Token (segmentation mask) indicates the crack regions of
an image patch. Current block-based methods [6], [38] are
Fig. 2. Examples of tokens learned from a manually labeled image database.
usually used to extract small patches and calculate mean
(a) Most representative token for each token set. (b) Mean contour structure for and standard deviation value on these patches to represent
each token set. an image token. These traditional features are computed on
gray level images and applied to describe the brightness and
including the mean and the standard deviation values of pixel- gradient information. However, local structured information is
normalized intensities to categorize an image block as longi- not taken into consideration. So in the first step, we re-define
tudinal, transversal or miscellaneous. Reference [5] computes the tokens by introducing the integral channel features which
CTA (Conditional Texture Anisotropy) values over the distrib- incorporate the color, gradient information from multiple levels
ution of the mean and the standard deviation values calculated and facets.
on pixels to distinguish crack pixels from defect free pixels. 1) Learning the Tokens: Assume that we have a set of im-
However, there are two main drawbacks in these methods. ages I with a corresponding set of binary images G representing
On the one hand, new types of crack cannot be generated. By the manually labeled crack edge from the sketches. We use a
applying the structured tokens, we extend the crack types into 16 × 16 sliding window to extract image patches x ∈ X from
thousands of dimensions. On the other hand, these methods the original image. Image patch x which contains a labeled
perform poor on the cracks with complex topology. To address crack edge at its center pixel, will be regarded as positive
this issue, we propose a novel crack descriptor to describe the instance and vice versa. y ∈ Y encodes the corresponding local
cracks with arbitrary complex topology. image annotation (crack region or crack free region), which also
indicates the local structured information of the original image.
These tokens cover the diversity of various cracks, which are
III. AUTOMATIC ROAD C RACK D ETECTION
not limited to straight lines, corners, curves, etc.
In this section, we will introduce our novel crack detection From Fig. 4, we can see the extracted image patches and their
method which can take advantage of the structured information hand drawn contour tokens. These image patches and tokens
of cracks. Fig. 3 shows the overall procedure of our proposed will be used to train CrackForest later.
method. This framework can be divided into three parts: In 2) Feature Extraction: To describe the above tokens, fea-
the first part, we extend the feature set of traditional crack tures are computed on the image patches x extracted from the
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

4 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

Fig. 5. Routing path of an image patch.

token in each leaf to represent the token class. The class number
Fig. 4. (Top) Example of original image and its ground truth. (Bottom)
Example of extracted image patches and their hand-drawn contours. Notice the
of tokens equals to the number of leaves.
variety of sketches. A forest T can be seen as an ensemble of decision trees ft .
Each tree ft (x) gives a prediction of a sample x ∈ X . The final
training images I, and considered to be weak classifiers in the class prediction of multiple trees is integrated by a majority
next step. voting algorithm. A leaf L(π) ∈ ft can assign a class prediction
We use mean and standard deviation value as features. Two for samples it is reached by, where π stands for the most
matrices are computed for each original image: the mean ma- represented token in the leaf. Each node N (h, ftL , ftR ) ∈ ft is
trix Mm with each block’s average intensity and the standard associated with a binary split function
deviation matrix STDm with corresponding standard devia-
tion value std. Each image patch yields a mean value and a h(x, θj ) ∈ {0, 1} (1)
16 × 16 standard deviation matrix. with feature θj for each node j. If h(x, θj ) = 0, sample x
To characterize the cracks more comprehensively, we also should be branched to the left sub-tree ftL , otherwise the right
apply a set of channel features composed with color, gradient sub-tree ftR .
and oriented gradient information. Integral channel features not 1) Class Prediction: Given a tree ft ∈ T , the class pre-
only perform better than other features including histogram of diction of an image patch x ∈ X can be obtained by recur-
oriented gradient (HOG), but also achieve fast detecting results sively branching it forward until a leaf is reached. An intuitive
and integrate heterogeneous sources of information [26]. example has shown in Fig. 5. The prediction function ψ(x|ft ) :
3 color, 2 magnitude and 8 orientation channels, for a total X → Y for node j is
of 13 channels yield 3328 candidate features. Each of the chan-   
nel captures a different aspect of information. Self-similarity    ψ x|ftL , for h(x, θj ) = 0
L
ψ x|N h, ft , ft R
=  
features are compute for each channel. These features capture ψ x|ftR , for h(x, θj ) = 1
the portion that an image patch contains similar textures based
on color or gradient information [45]. Texture information is ψ (x|L(π)) = π. (2)
  on a m × m grid over the patch. These differences
computed The final class prediction of x is obtained from the prediction
yield 5·5
2 = 300 more features per channel. of each tree as the one receiving the majority voting.
2) Randomized Training: Each tree is trained individually.
B. Structured Learning For a given node Nj and training set Sj ⊂ X × Y, the goal is
In previous step, a set of tokens y which indicate the struc- to find the optimal feature θj that results in a good split of the
tured information of local patches, and features which describe data. In other words, the discrepancy of tokens in the same leaf
such tokens, are acquired. In this step, we cluster these tokens should be as small as possible. We apply information gain to
by using a state-of-the-art structured learning framework, ran- measure this discrepancy and maximize the information gain to
dom structured forests, to generate an effective crack detector. choose θj . The form of information gain for node j is defined
Random structured forests can exploit the structured informa- as follow:
tion and predict the segmentation mask (token) of a given image  
Ij = I Sj , SjL , SjR (3)
patch. Thereby we can obtain the preliminary result of crack
detection. where Sj = SjL ∪ SjR , SjL = {(x, y) ∈ Sj |h(x, θj ) = 0}
In random structured forests, each decision tree ft (x) clas- stands for a set of samples that reaches the left sub-tree of the
sifies an image patch x ∈ X by recursively branching left or current node and SjR = {(x, y) ∈ Sj |h(x, θj ) = 1} refers to
right down to the tree until a leaf is reached. And the class of the other set of samples that reaches the right sub-tree.
the node is assigned to patch x. The leaf stores the prediction Whether a terminal node should be further split depends on
of the input x, which is a target label y ∈ Y or a distribution the maximum depth, the minimum size of node or the entropy
over Y. By training such a tree, tokens with the same structure of the class distribution. If the node is no longer splitting, a
will be gathered at one leaf. We use the most representative leaf is grown where the class prediction π is set to the most
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

SHI et al.: AUTOMATIC ROAD CRACK DETECTION USING RANDOM STRUCTURED FORESTS 5

representative token in the training data. Otherwise a node

N (h, ftL , ftR ) is grown where h is a split function regulated
by parameter θj , maximizing the information gain about the
label distribution due to the split {SjL , SjR } of the training
data S.
For multi-class classification (Y ⊂ Z), the definition of in-
formation gain is
 
 Sjk   
Ij = H(Sj ) − H Sjk (4)
|Sj |
k∈{L,R}

where H(Sj ) = − y py log(py ) denotes the Shannon entropy
and py stands for the proportion of elements
 in S with label y.
Alternatively, Gini impurity H(Sj ) = y py (1 − py ) can also
be applied in Equation (4).
Individual decision tree tends to overfit, which may nega- Fig. 6. Assigning y to each image patch. The image patches have been
tively affected accuracy. To overcome this drawback, random assigned to the tokens below (both from left to right).
structured forests combine multiple decision trees together to
assign the final label. Random structured forests have shown
promising flexibility and generalization ability, and most im-
portantly, this method is easy to parallel and extremely fast.
The randomness is embodied by randomly subsampling the
data used to train each tree and each node, and randomly
subsampling the features used to split each node. In order to
maintain the diversity of trees, only a small pool of features is
used to select the optimal θj when choosing the split function.
3) Structured Mapping: Random structured forests change
the discrete outer space of the traditional decision forests into a Fig. 7. (a) Binarization result based on threshold when α = 0.1 (remov-
structured space Y. While dealing with structured label y ∈ Y ing pixels of low probability according to the given probability map).
directly may cause significant computing expense, the struc- (b) Result after erosion and dilation with a 4 × 4 rectangular structuring
element.
tured labels y ∈ Y at a leaf is mapped into a set of discrete
labels c ∈ C, where C = {1, . . . , k}. Given the discrete label
space C, information gain can be calculated efficiently via (4). 4) Binarization: After the structured mapping, each image
We first map the label space Y into a intermediate space Z patch x is assigned to a structured label y. Due to the over-
lapping, the result of detection is a map, where each element
Π : Y → Z. (5)
indicates the probability that the corresponding position in the
16·16
Define z = Π(y) in space Z as a 2 = 32640 dimensional original image is on crack region. So we use a threshold α to
vector, which encodes every pair of pixels in the segmentation obtain all the possible regions. A high α value may cause the
mask y. The computational cost of z appears to be significant. incontinuity of cracks and the ignorance of inapparent cracks.
While the dimension of z is still very high, we randomly Therefore, we choose 0.1  α  0.2 in this paper. Fig. 7(a)
select 256 dimension of z to train each split function, using a shows the binarization result when α = 0.1.
distinct reduced mapping function at each node j We conduct the erosion and the dilation operation on the pre-
liminary edge detection results to make the cracks as connective
Πφ : Y → Z. (6) as possible. The inside of the crack is filled and the fragments
Then we apply PCA reduction to map 256 dimensions of z into are connected. Moreover, some of the noises are eliminated.
5 dimensions, with the first dimension being the most signifi- From Fig. 7(b), we can see that small fragments of the detected
cant factor. To obtain the discrete label c ∈ C of each structured region have merged together and the continuity of the crack has
label y, we use the first dimension of each intermediate label z been improved.
to cluster into two sets. Labels in the same cluster are assigned
to the same label c. With the label c, standard information gain
C. Crack Type Characterization and Detection
can be calculated at each node.
After the random structured forests are trained, the structured Each image patch is assigned to a structured label y
labels y are gathered at the leaves of each tree (see Fig. 2). (segmentation mask) after structured learning. Although we
An image patch is routed though each tree based on the split obtain a preliminary result of crack detection so far, a lot of
function until a leaf is reached. The most representative token in noises are generated due to the textured background at the same
the leaf is assigned to the image patch. Fig. 6 shows an intuitive time. Traditional thresholding methods mark small regions as
example. We select the token which has the lowest variance noises according to their sizes. However, in this way, many
with others as the most representative token. inconspicuous cracks may be removed by mistake.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

6 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

Fig. 9. Statistical feature histogram showing the occurrence (in logs) of each
token (sorting in descending order of occurrence). (a) Statistical feature of all
26443 tokens. (b) Only the most representative tokens are shown.

Statistical Neighborhood Histogram: The statistical neigh-

borhood histogram captures the neighborhood information of
two tokens. We calculate the co-occurrence of each pair of
Fig. 8. (a) Original image. (b) One of the detected regions. (c) Statistical tokens only when they are adjacent. There would be 708 2 =
feature histogram of the detected region. (d) Appearance of the ten most 250278 token pairs without reduction. Furthermore, we also
frequent tokens look. find the long tail effect of this distribution. Over 90% occur-
rences of all the token pairs are centered on 956 specific token
pairs. Thus, only these token pairs will be used in the following
Cracks have a series of unique structural properties that section.
differ from noises. Based on this thought, we propose a novel 2) Crack Detection: With the two histograms for each
crack descriptor by using the statistical feature of structured separated region, we can characterized cracks with arbitrary
tokens in this section. This descriptor consists of two statis- topology. In this section, we will introduce how to discriminate
tical histograms, which can characterize cracks with arbitrary the noises from cracks by using the two histograms.
topology. By applying classification method like SVM, we can Vectorization: The distribution of occurrence and co-
discriminate noises from cracks effectively. occurrence are scaled to [0, 1]. Hence, each detected region is
1) Crack Descriptor: Existing crack characterization meth- presented as a long vector with 708 + 956 = 1664 dimensions.
ods categorize cracks into several types, such as longitudinal, Classification: We consider the crack detection procedure
transverse, diagonal, block, and alligator. However, the de- as a classification problem. The crack regions are assigned to
scriptor proposed, which consists of hundreds of dimensions class +1 and the crack free regions are assigned to class −1.
respectively, has greatly broadened the range of representable By applying KNN(k-Nearest Neighbor), SVM (Support Vector
crack. What is more, the crack is no longer limited to a few Machine) with linear kernel and One-Class SVM with linear
types, we extend the types of crack into thousands of kinds. kernel, we obtain the classification model which can discrimi-
We use 26443 structured tokens obtained in the structured nate cracks from noises effectively. The results of our algorithm
learning procedure to characterize the cracks. The statistical using SVM are shown in Fig. 10.
histogram and the neighborhood histogram of these tokens
within a crack can be calculated precisely.
IV. E XPERIMENTS
Statistical Feature Histogram: After the structured learning
procedure, we can obtain the token map. Each point in the In this section, we analyze the performance of our pro-
map indicates the label of token that the 16 × 16 image patch posed method. Part of the Matlab code is supported on Piotr’s
around the corresponding position is assigned to. Statistical Computer Vision Toolbox [46] and Structured Edge Detection
feature histogram in Fig. 8 reflects the composition of the crack Toolbox [27]. All the experiments are conducted on a desktop
comprehensively. Each dimension of this histogram represents with AMD FX(tm)-4300 Quad-Core Processor and 4G RAM.
the number of a certain token. In order to evaluate our method, we compare it with the
The token number from the training result is numerous. traditional method (Canny [47]), and the state-of-the-art road
After plotting the overall occurrence of each token in Fig. 9(a), detection methods (CrackTree [20], CrackIT [48], FFA [25] and
we notice a long tail effect of the token distribution. After MPS [17], [19]).
analyzing the statistical information of appeared tokens, we find We show results on two datasets measuring accuracy perfor-
that over 90% occurrences of all the tokens are centered on mance. We demonstrate the cross dataset generalization of our
708 specific tokens. The occurrences of most tokens make up approach by testing on each dataset using CrackForest learned
only a small percentage of all. Therefore, we only use these on the other.
708 tokens to construct the statistical feature histogram and Unlike other edge detection tasks, the evaluation of crack
the statistical neighborhood histogram. Fig. 9(b) shows the detection performance is difficult. Thereby we define two kinds
occurrence of these tokens. of evaluation indicators for crack detection.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

SHI et al.: AUTOMATIC ROAD CRACK DETECTION USING RANDOM STRUCTURED FORESTS 7

Fig. 10. Part of the results of road crack detection using our proposed method. Notice that our method can eliminate the influence of oil stains, shadows, and
complex background, effectively, and can cope with miscellaneous crack topology.

Crack Detection Accuracy: We use precision, recall and The precision, recall and F1 Score on detect region can be
F1 Score to evaluate the performance of different crack detec- similarly computed by (7) and (8)
tion algorithms.
The precision and recall can be computed on true positive TPr
Prregion = (10)
(TP), false negative (FN) and false positive (FP) TPr + FPr
TP TPr
Prpixel = (7) Reregion = (11)
TP + FP TPr + FNr
TP
Repixel =
TP + FN
(8) 2 × Prregion ×Reregion
F 1region = (12)
2 × Prpixel ×Repixel Prregion +Reregion
F 1pixel = . (9)
Prpixel +Repixel
Crack Continuity Assessment: We define the “Continuity
Assume that the detected pixels which are no more than Index (CI)” as a degree of continuity. It measures how much
five pixels away from the manually labeled pixel are true the detected regions are connected if they belong to the same
positive pixels. crack. Denote M as the number of images in the testing set. Ni
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

8 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

Fig. 11. Results of different algorithms on CFD (from top to bottom: original image, ground truth, Canny, CrackIT, CrackTree, FFA, and CrackForest).

A. CFD Results
as the number of ground truth cracks in the ith image and nij
as the number of true positive regions that cover the jth ground We propose an annotated road crack dataset called CFD. This
truth crack in the ith image dataset is composed of 118 images, which can generally reflect
⎛ ⎞ urban road surface condition in Beijing, China. Each image has
1 ⎝ 1  hand labeled ground truth contours. All the images are taken by
M Ni
1 ⎠
CI = (13) an iPhone5 with focus of 4mm, aperture of f/2.4 and exposure
M i=1 Ni j=1 nij
time of 1/134s. The width of the images ranges from 1 to 3 mm.
From Fig. 10, we can see that these images contain noises such
The continuity is better as CI gets closer to 1. as shadows, oil spots and water stains.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

SHI et al.: AUTOMATIC ROAD CRACK DETECTION USING RANDOM STRUCTURED FORESTS 9

TABLE I
C RACK D ETECTION R ESULTS E VALUATION ON CFD

Fig. 12. Results of different algorithms on AigleRN (from top to bottom: original image, ground truth, CrackIT, FFA, MPS, and CrackForest).

We use the 60%/40% training/testing split with the images detection due to its high sensitivity. CrackIT does not perform
reduced to 480 × 320. Example detections on CFD are shown well on low-resolution and low-contrast images. As a result,
in Fig. 10. The first column lists the original images. The it fails to detect most of the crack pixels in the images. The
corresponding manually labeled cracks are shown in the second accuracy of CrackTree is acceptable. But it may hallucinate a
column as ground truth. The third column shows the prelimi- crack that does not exist. In addition, the width of the crack can
nary detection results after applying random structured forests. not be observed. As for FFA, it may falsely detect landmarks as
Darker color indicates that the pixel is more likely to contain a defects.
crack. After the binarization, crack pixels with less confidence Our method CrackForest performs better than the alterna-
are removed. The use of crack descriptor allows us to transform tives. To be specific, CrackForest (SVM) gives both good
each detected region into a vector. By applying classification precision and recall.
method such as SVM, we can eliminate the noise regions and
keep the crack regions effectively. The final detection results
B. AigleRN Results
are shown in column 5. Our method is robust to noise.
Five methods are conducted on this dataset: Canny, CrackIT, AigleRN dataset [49] contains 38 images with ground truth.
CrackTree, FFA and CrackForest. Results are shown in Fig. 11 We use 60% for training and the rest for testing.
and summary statistics are in Table I. As it can be observed We compare four methods on this dataset: CrackIT, FFA,
intuitively, our method outperforms the alternatives. Traditional MPS and CrackForest. Example AigleRN results are shown
edge detection method Canny is not suitable for road crack in Fig. 12 and Table II. Although CrackIT can detect most
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

10 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

TABLE II
C RACK D ETECTION R ESULTS E VALUATION ON A IGLE RN

TABLE III
C ROSS -D ATASET G ENERALIZATION T EST FOR C RACK F OREST. TRAIN/TEST I NDICATES THE T RAINING /T ESTING D ATASET U SED

of the cracks, a lot of noises are still remained. Besides, the it possible to exploit such structured information and predict
continuity of the detected cracks is not very good. As for FFA, local segmentation masks of the given image patch. Thirdly, a
the precision is acceptable. But when it comes to detecting crack descriptor, which consists of two statistical histograms, is
cracks with complex topology, FFA is less competitive. MPS proposed to characterize the structured information of cracks
performs well on detecting light cracks, but it may hallucinate a and discriminate cracks from noises. In addition, we also
crack that does not exist. CrackForest shows promising results propose an annotated road crack image dataset which can
on most of the indicators. To be specific, CrackForest (SVM) generally reflect the urban road surface condition in China and
still gives both better precision and recall. two indicators to evaluate the performance of crack detection
methods.
C. Cross Dataset Generalization Experimental results prove the effectiveness of our method
To study the ability of our approach to generalize across in suppressing noises compared to several competing methods.
datasets, we ran a final set of experiments. In Table III, we show Our approach yields promising processing speed and state-of-
results on AigleRN using CrackForest trained on CFD and also the-art accuracy.
results on CFD using CrackForest trained on AigleRN. Note Source code is available online: [Link]
that images in the CFD and AigleRN datasets are qualitatively cuilimeng/CrackForest. Our annotated road crack image
quite different, see Figs. 11 and 12, respectively. dataset CFD is also available online: [Link]
In Table III, top, results on AigleRN of the AigleRN and cuilimeng/CrackForest-dataset.
CFD trained models are compared. The precision and recall
do not fluctuate much using two datasets for training. Results VI. L IMITATIONS AND F UTURE W ORK
on CFD of the CFD and AigleRN models, shown in Table III,
bottom, are likewise similar. In our experiments, CrackForest has proven to be quite
The experimental results show that CrackForest could serve promising. However, it does have some limitations:
as a general purpose crack detector without the necessity of • Our method has only performed on static images so far.
retraining. The video streaming is not taken into consideration. In
the future, we will test our method on video datasets.
V. C ONCLUSION • The width of the crack is not measured in our method.
In this paper, we propose an effective and fast automatic road We will focus on the severity level assessment in the
crack detection method, which can suppress noises efficiently future work.
by learning the inherent structured information of cracks. Our
detection framework builds upon representative and discrimina- ACKNOWLEDGMENT
tive integral channel features and combines this representation Thanks professor Qin Zou and professor Manuel Avila for
with random structured forests. This also allows us to train their kind help. And thanks professor Sylvie Chambon and
our framework in a completely supervised manner from a Dr. Rabih Amhaz for their valuable discussions.
small training set. More importantly, we can characterize cracks
and eliminate noises marked as cracks by using two feature R EFERENCES
histograms proposed. [1] H. Oliveira and P. L. Correia, “Automatic road crack segmentation using
Our innovation is shown as follows: Firstly, to capture the entropy and image dynamic thresholding,” in Proc. 17th EUSIPCO,
inherent structure of the road crack, we apply integral channel Aug. 24–28, 2009, pp. 622–626.
[2] H. Cheng, J.-R. Chen, C. Glazier, and Y. Hu, “Novel approach to pave-
features to enrich the feature set of traditional crack detection. ment cracking detection based on fuzzy set theory,” J. Comput. Civil Eng.,
Secondly, the introducing of random decision forests makes vol. 13, no. 4, pp. 270–280, Oct. 1999.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

SHI et al.: AUTOMATIC ROAD CRACK DETECTION USING RANDOM STRUCTURED FORESTS 11

[3] H. Cheng et al., “Novel approach to pavement cracking detection based [30] H. Zhao, G. Qin, and X. Wang, “Improvement of canny algorithm based
on neural network,” Transp. Res. Rec., J. Transp. Res. Board, vol. 1764, on pavement edge detection,” in Proc. 3rd Int. Conf. CISP, 2010, vol. 2,
pp. 119–127, 2001. pp. 964–967.
[4] P. Subirats, J. Dumoulin, V. Legeay, and D. Barba, “Automa- [31] Y.-C. Tsai, V. Kaul, and R. M. Mersereau, “Critical assessment of pave-
tion of pavement surface crack detection using the continuous ment distress segmentation methods,” J. Transp. Eng., vol. 136, no. 1,
wavelet transform,” in Proc. IEEE Int. Conf. Image Process., 2006, pp. 11–19, Jan. 2009.
pp. 3037–3040. [32] S. Chambon and J.-M. Moliard, “Automatic road pavement assessment
[5] T. S. Nguyen, M. Avila, and S. Begot, “Automatic detection and classi- with image processing: Review and comparison,” Int. J. Geophys.,
fication of defect on road pavement using anisotropy measure,” in Proc. vol. 2011, 2011, Art. no. 989354.
Eur. Signal Process. Conf., 2009, pp. 617–621. [33] R. Achanta, F. Estrada, P. Wils, and S. Süsstrunk, “Salient region detection
[6] H. Oliveira and P. L. Correia, “Automatic road crack detection and charac- and segmentation,” in Computer Vision Systems. Berlin, Germany:
terization,” IEEE Trans. Intell. Transp. Syst., vol. 14, no. 1, pp. 155–168, Springer-Verlag, 2008, pp. 66–75.
Mar. 2013. [34] R. Achanta, S. Hemami, F. Estrada, and S. Susstrunk, “Frequency-tuned
[7] H. Oh, N. W. Garrick, and L. E. Achenie, “Segmentation algorithm salient region detection,” in Proc. IEEE CVPR, 2009, pp. 1597–1604.
using iterative clipping for processing noisy pavement images,” in [35] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, “Contour detection
Proc. 2nd Int. Conf. Imaging Technol., Tech. Appl. Civil Eng., 1998, and hierarchical image segmentation,” IEEE Trans. Pattern Anal. Mach.
pp. 138–147. Intell., vol. 33, no. 5, pp. 898–916, May 2011.
[8] M. Petrou, J. Kittler, and K. Song, “Automatic surface crack detection [36] K. Y. Song, M. Petrou, and J. Kittler, “Texture crack detection,” Mach.
on textured materials,” J. Mater. Process. Technol., vol. 56, no. 1–4, Vis. Appl., vol. 8, no. 1, pp. 63–75, Jan. 1995.
pp. 158–167, Jan. 1996. [37] Y. Hu and C.-X. Zhao, “A local binary pattern based methods for pave-
[9] Y. Huang and B. Xu, “Automatic inspection of pavement cracking dis- ment crack detection,” J. Pattern Recognit. Res., vol. 1, no. 20103,
tress,” J. Electron. Imag., vol. 15, no. 1, pp. 013 017–013 017, 2006. pp. 140–147, 2010.
[10] S. Cafiso, A. Di Graziano, and S. Battiato, “Evaluation of pavement [38] J. Zhou, P. S. Huang, and F.-P. Chiang, “Wavelet-based pavement distress
surface distress using digital image collection and analysis,” in Proc. 7th detection and evaluation,” Opt. Eng., vol. 45, no. 2, Feb. 2006,
Int. Congr. Adv. Civil Eng., 2006, pp. 1–10. Art. no. 027007.
[11] M. Gavilán et al., “Adaptive road crack detection system by pavement [39] M. Kass, A. Witkin, and D. Terzopoulos, “Snakes: Active contour
classification,” Sensors, vol. 11, no. 10, pp. 9628–9657, Oct. 2011. models,” Int. J. Comput. Vis., vol. 1, no. 4, pp. 321–331, Jan. 1988.
[12] M. S. Kaseko and S. G. Ritchie, “A neural network-based methodol- [40] V. Kaul, A. Yezzi, and Y. Tsai, “Detecting curves with unknown endpoints
ogy for pavement crack detection and classification,” Transp. Res. C, and arbitrary topology using minimal paths,” IEEE Trans. Pattern Anal.
Emerging Technol., vol. 1, no. 4, pp. 275–291, Dec. 1993. Mach. Intell., vol. 34, no. 10, pp. 1952–1965, Oct. 2012.
[13] Q. Li and X. Liu, “Novel approach to pavement image segmentation [41] P. Delagnes and D. Barba, “A Markov random field for rectilinear struc-
based on neighboring difference histogram method,” in Proc. CISP, 2008, ture extraction in pavement distress image analysis,” in Proc. Int. Conf.
vol. 2, pp. 792–796. Image Process., 1995, vol. 1, pp. 446–449.
[14] N. Tanaka and K. Uematsu, “A crack detection method in road surface [42] B. J. Lee and H. Lee, “Position-invariant neural network for digital
images using morphology,” in Proc. MVA, 1998, vol. 98, pp. 17–19. pavement crack analysis,” Comput.-Aided Civil Infrastruct. Eng., vol. 19,
[15] H. Oliveira and P. L. Correia, “Supervised strategies for crack detection no. 2, pp. 105–118, Jan. 2004.
in images of road pavement flexible surfaces,” in Proc. EUSIPCO, 2008, [43] A. Cord and S. Chambon, “Automatic road defect detection by textural
pp. 25–29. pattern recognition based on AdaBoost,” Comput.-Aided Civil Infrastruct.
[16] C. Koch and I. Brilakis, “Pothole detection in asphalt pavement images,” Eng., vol. 27, no. 4, pp. 244–259, Apr. 2012.
Adv. Eng. Inf., vol. 25, no. 3, pp. 507–515, Aug. 2011. [44] L. Ying and E. Salari, “Beamlet transform-based technique for pavement
[17] R. Amhaz, S. Chambon, J. Idier, and V. Baltazart, “A new minimal path crack detection and classification,” Comput.-Aided Civil Infrastruct. Eng.,
selection algorithm for automatic crack detection on pavement images,” vol. 25, no. 8, pp. 572–580, Nov. 2010.
in Proc. IEEE ICIP, 2014, pp. 788–792. [45] J. J. Lim, C. L. Zitnick, and P. Dollár, “Sketch tokens: A learned mid-level
[18] M. Avila, S. Begot, F. Duculty, and T. S. Nguyen, “2D image based representation for contour and object detection,” in Proc. IEEE CVPR,
road pavement crack detection by calculating minimal paths and dynamic 2013, pp. 3158–3165.
programming,” in Proc. IEEE ICIP, 2014, pp. 783–787. [46] P. Dollár, Piotr’s Computer Vision Matlab Toolbox (PMT). [Online].
[19] R. Amhaz, S. Chambon, J. Idier, and V. Baltazart, “Automatic crack Available: [Link]
detection on 2D pavement images: An algorithm based on minimal path [47] J. Canny, “A computational approach to edge detection,” IEEE Trans.
selection,” IEEE Trans. Intell. Transp. Syst., 2015, 24p, DOI: 10.1109/ Pattern Anal. Mach. Intell., vol. PAMI-8, no. 6, pp. 679–698,
TITS.2015.2477675. hal-01206038. Nov. 1986.
[20] Q. Zou, Y. Cao, Q. Li, Q. Mao, and S. Wang, “CrackTree: Automatic crack [48] H. Oliveira and P. L. Correia, “Crackit—An image processing toolbox
detection from pavement images,” Pattern Recognit. Lett., vol. 33, no. 3, for crack detection and characterization,” in Proc. IEEE ICIP, 2014,
pp. 227–238, Feb. 2012. pp. 798–802.
[21] K. Fernandes and L. Ciobanu, “Pavement pathologies classification using [49] S. Chambon, AigleRN. [Online]. Available: [Link]
graph-based features,” in Proc. IEEE ICIP, 2014, pp. 793–797. Chambon/Crack_Detection_Database.html
[22] Y.-C. J, C. Jiang, and Y. Huang, “Multiscale crack fundamental element
model for real-world pavement crack classification,” J. Comput. Civil
Eng., vol. 28, no. 4, pp. 388–396, Jul. 2012, Art. no. 04014012.
[23] Y. J. Tsai, C. Jiang, and Z. Wang, “Implementation of automatic crack
evaluation using crack fundamental element,” in Proc. IEEE ICIP, 2014,
pp. 773–777. Yong Shi received the Ph.D. degree in management
[24] P. Kontschieder, S. R. Bulo, H. Bischof, and M. Pelillo, “Structured class- science and computer system from The University of
labels in random forests for semantic image labelling,” in Proc. IEEE Kansas, Lawrence, KS, USA.
ICCV, 2011, pp. 2190–2197. He is currently a Professor with the Chinese
[25] T. S. Nguyen, S. Begot, F. Duculty, and M. Avila, “Free-form anisotropy: Academy of Sciences, Beijing, China, where he
A new method for crack detection on pavement surface images,” in Proc. serves as the Director of Research Center on
18th IEEE ICIP, 2011, pp. 1069–1072. Fictitious Economy and Data Science. He is also a
[26] P. Dollár, Z. Tu, P. Perona, and S. Belongie, “Integral channel features,” Professor and a Distinguished Chair of Information
in BMVC, 2009, pp. 91.1–91.11. Technology with the College of Information Science
[27] P. Dollár and C. L. Zitnick, “Structured forests for fast edge detection,” in and Technology, University of Nebraska Omaha,
Proc. IEEE ICCV, 2013, pp. 1841–1848. Omaha, USA. His research interests include data
[28] H.-D. Cheng and M. Miyojim, “Automatic pavement distress detection mining, information overload, optimal system designs, multiple-criteria deci-
system,” Inf. Sci., vol. 108, no. 1–4, pp. 219–240, Jul. 1998. sion making, decision support systems, and information and telecommunica-
[29] A. Ayenu-Prah and N. Attoh-Okine, “Evaluating pavement cracks with tions management.
bidimensional empirical mode decomposition,” EURASIP J. Adv. Signal Dr. Shi is the Editor-in-Chief of International Journal of Information
Process., vol. 2008, no. 1, Mar. 2008, Art. no. 861701. Technology and Decision Making and Annals of Data Science.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

12 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

Limeng Cui is currently working toward the Ph.D. Fan Meng is currently working toward the Ph.D.
degree with University of Chinese Academy of Sci- degree with University of Chinese Academy of Sci-
ences, Beijing, China, with a focus on computer ences, Beijing, China.
vision and machine learning. His research interests include edge detection, im-
Her work lies in image processing, object detec- age segmentation, road crack detection, and propor-
tion and scene recognition. tion learning.

Zhiquan Qi is an Assistant Professor with the Re- Zhensong Chen is currently working toward the
search Center on Fictitious Economy and Data Sci- Ph.D. degree with University of Chinese Academy
ence, Chinese Academy of Sciences, Beijing, China. of Sciences, Beijing, China.
His research interests include object detection, His research is focused on image segmentation
object tracking, change detecting, and machine and proportion learning.
learning.