A Modern Pothole Detection technique using

Deep Learning
Abhishek Kumar Chakrapani Dhruba Jyoti Kalita
Department of Ele. & Electronics Department of Computer Science Department of Computer Science
Engineering and Engineering and Engineering
Gaya College of Engineering Gaya College of Engineering Gaya College of Engineering
Gaya, Bihar, India Gaya-Bihar, India Gaya-Bihar, India
Abhisheksingh73017@[Link] chakrapani3513.a@[Link] dhruba.nitjsr18@[Link]

Vibhav Prakash Singh

Department of Computer Science
and Engineering
National Institute of Technology,
Allahabad
UP, India
vibhav@[Link]

Abstract- Road accident detection and avoidance are a maintenance of roads, potholes get appeared on the
more difficult and challenging problem in India as road which causes road accidents. According to
poor quality of construction materials get used in statistics submitted by the [3] government of India
road drainage system construction. Due to the above from 2013 to 2016 potholes claimed 11,836 lives
problems, roads get damaged early and potholes
appear on the roads which cause accidents. According
and 36,421 people got injured. Pothole problems
to a report submitted by the Ministry of Road cannot be resolved easily because every year
Transport and Highways transport research wing almost all the places suffer from floods, disasters,
New Delhi in 2017, approximately 4,64,910 accidents heavy rainfall, etc. Though we cannot maintain the
happen per year in India. This paper proposed a deep road, we can reduce the number of accidents which
learning-based model that can detect potholes early are getting increased every year. This paperwork
using images and videos which can reduce the has proposed a novel approach to detect potholes
chances of an accident. This model is basically based approximately 100 meters ahead using deep
on Transfer Learning, Faster Region-based learning. Many authors are working on this
Convolutional Neural Network(F-RCNN) and
Inception-V2. There are many models for pothole
problem. In [4] Artis M. [Link]. have proposed an
detection that uses the accelerometer (without using accelerometer-based model that uses Z-THRESH,
images and videos) with machine learning techniques, Z-DIFF, STDEV(Z) and G-ZERO algorithm and
but a less number of pothole detection models can be that can be deployed on any Android OS-based
found which uses only machine learning techniques to devices with limited hardware and software
detect potholes. The results of this work have shown resources. They analyse the performance of the
that our proposed model outperforms other existing model with a 90% true positive value. Lin, J. et.
techniques of potholes detection. al. in [5] have used a non-linear SVM model
classification tool with a Gaussian radial basis
Keywords – Transfer Learning, Deep Learning, Tensor
flow API, Accelerometer, Image Labelling, F-RCNN,
function for the detection of a pothole. In this
inception-v2 model, each image of 64×64 gets converted into
grayscale and used for the experiment. Here for
I. INTRODUCTION each grayscale eigenvalue is calculated and an
average range of eigenvalue lie between 60 to 100
India is the second-largest road network in the is searched before applying the SVM model. After
world [1]. Therefore, the road network plays an this normalization of each image in the range [1,10]
important role in Indian economic development and is done using eigenvector. Pereira, V. et. al. in [6]
social functioning. According to the report [2] in the have used Convolution Neural Network and
last ten years, the Road Transport sector GDP grew compared the performance of their model with
at an annual average rate close to 10% compared to SVM and found that their model out formed SVM
the overall annual GDP growth of 6%. Nowadays with 99.80% accuracy. They deployed the model
road construction is done very rapidly by the using CNN, pooling, ReLU activation function,
government of India. But road maintenance is a Adam Optimizer, and Sigmoid function. Here,
challenging task because of the poor drainage convolution and pooling have been used for feature
system and overloaded vehicles. Due to poor extraction. Adam optimizer is used to reduce the

Authorized licensed use limited to: Cornell University Library. Downloaded on August 26,2020 at [Link] UTC from IEEE Xplore. Restrictions apply.
 cost function and sigmoid function for output external selective search. It consists of External
predicted values between 0 and 1. selective search, CNN with max pooling, ROI
(Region of Interest) pooling layer, fully connected
layers, and output layers. Fast R-CNN takes an
II. MODERN BASED POTHOLE
input image and then with the help of CNN & max-
DETECTION METHOD
pooling layers, a convolutional feature map is
extracted from the image. The ROI pooling layer
The proposed modern based pothole detection
performs a very important task here. We know that
method using the transfer learning technique fully connected layers can accept only certain sizes,
detects potholes in videos/images in real-time. The So ROI pooling layers converts the output of the
method uses common techniques like “use of F-
CNN into certain fixed sizes. When we put Fast R-
RCNN”, “inception v2 model” which are described
CNN with the RPN (Region proposal network), it
below. The main advantage of this method is small
becomes Faster R-CNN. So, basically, the
training time with an easy training process and
difference between Fast R-CNN and Faster R-CNN
higher accuracy. is the Region proposal. In Fast R-CNN, there is an
external selective search whereas in Faster R-CNN
A. Tensorflow Object-detection API RPN is combined with the Architecture.
RPN is the Architecture that makes Fast R-CNN a
Faster R-CNN. In order to reduce the
Tensorflow’s object detection API is a very
computational complexity and requirement, RPN
powerful tool that can quickly enable anyone to
decides where to look in the image. It scans the
build and deploy a powerful image recognition
image and gives k output boxes each with 2 scores
system. It provides many pre-trained models
indicating the probability of availability of an
(trained on different datasets) which can be used to
object. Different size and different aspect ratio of
build customized classifiers/detector/recognizer
boxes are selected in order to accommodate
after fine-tuning. We’ve selected the model named
different types of objects.
“F-RCNN inception v2”.

B. Transfer Learning

Transfer Learning makes use of knowledge gained

while solving one problem and apply it to a
different but related problem. With this technique,
we can save a lot of time. In this technique below
first, we select any pre-trained model (In which all
the parameters are trained), then we perform fine-
tuning. We fetch the new dataset to fine-tune the
pre-trained CNN. If the new dataset is similar to the
original dataset (with which the model has been
trained), the same weights can be used for
extracting the features from the new dataset. In our
case, the dataset is very different from the original
dataset. The Earlier layers of CNN contain more
generic features (edge detector, colour blob
detectors), but the later layers of CNN become
progressively more specific to the details of the
classes contained in the original dataset. So, earlier
layers can help to extract the features of the new
data. We fixed the earlier layers and Re-train the
rest of the layers (because of the small amount of
data).

C. F-RCNN (Faster Region-based Convolutional

Neural Network)

It stands for Faster Region-based Convolutional Fig.1 The Architecture of F-RCNN

Neural Network. Before talking about Faster R- In order to get a trained working Faster R-CNN
CNN we should know about Fast R-CNN. Fast R- architecture, we need to perform the following
CNN is a detector that uses an external proposal or
steps –

Authorized licensed use limited to: Cornell University Library. Downloaded on August 26,2020 at [Link] UTC from IEEE Xplore. Restrictions apply.
  Training of RPN – first of all, we need to
Train the RPN architecture with the
dataset so that it can propose the expected
region.

 Training of Fast R-CNN – As we know,

Faster R-CNN is a combination of RPN
and Fast R-CNN. So, we’ve to train a Fast
R-CNN with the proposals obtained by
RPN (after training) in order to make a
Faster R-CNN.

 Fixing Convolutional layers, fine-tuning

Fig.2b Inception-v1 module with dimension reduction
unique layers to RPN.

 Fixing Convolutional layers, fine-tuning

fully connected layers of Fast R-CNN.

D. Inception – V2

Inception v2 is an upgraded version in the inception

network series. It is a complex architecture of
CNNs. Inception v1 is the first version of this
series. Inception v1 performs convolution on input,
with 3 different sizes of filters (1x1, 3x3, 5x5).
Additionally, max pooling is also performed. The
Outputs are concatenated and sent to the next
inception module. Fig.2a shows a typical inception
v1 module. As the deep learning networks are
computationally expensive, an extra 1x1 convolution is
added before the 3x3 and 5x5 convolutions to limit the
number of input channels (Fig. 2b). Using this
inception module, GoogLeNet neural network Fig.3 Inception-v2 version
architecture (having 9 such inception module) was
built. Furthermore, to improve computational speed
inception v2 architecture is introduced as shown in III. Data Pre-processing
figure 03. In this architecture 5×5 convolution is
factorized to two 3x3 convolution operations. A 5x5 We had the pothole dataset consisting of more than
convolution is 2.78 times more expensive than a 1500 images. This dataset has been compiled at the
3x3 convolution. Moreover, a new method is Electrical and Electronic Department, Stellenbosch
introduced to improve performance. In the new University in 2015. The entire dataset consists of
method, the convolution of filter size n×n is two different sets, one was considered to be simple
factorized to a combination of 1×n and n×1 and the other more complex as shown in figure 04.
convolutions. This method is 33% cheaper than the The pothole detection task was found to be much
single 3×3 convolution. easier if only the region in the images that contained
the road was cropped and then used in detection but
it takes more time to crop 1500 images. We
perform labelling in all images. In the task, we
create a CSV file containing images name, path,
image dimension and coordinate of the pothole in
the image. After that, we create TfRecords for the
dataset for feeding it to the Inception of architecture
and fine-tuning. The first column in the annotation
file contains the path of images. The second
column contains a number of potholes in that image
and consecutive 4 columns indicate the coordinates
of potholes in the image. The size of the image is
Fig.2a Inception-v1 Module, naïve version 3680 × 2760.

Authorized licensed use limited to: Cornell University Library. Downloaded on August 26,2020 at [Link] UTC from IEEE Xplore. Restrictions apply.
 Fig.04 Images from Pothole Dataset

Sample Annotations (csv)

IV. Detection of Pothole in image/Video V. Evaluation in real-time
To Evaluate the system, we perform the following
After pre-processing the data, we created a system set of tasks:
for the detection of potholes in video/images so that  Setting up a laptop and camera on a
the driver can get an early alert about the pothole vehicle (as we’ve not deployed the system
on the road. For the system, we have used inception in any android device or raspberry pi yet).
v2 architecture and perform fine-tuning. We fixed  Region-based
the earlier layers of inception v2 network
 Detecting potholes in real-time using the
(consisting of 22 layers excluding pooling layers)
system in videos captured by the camera.
and re-train only the later layers. For the training, a
special configuration file has been created. There is  Matching the captured data with real data
only one class in our configuration file (pothole). of road.
Training of the model takes some time depending
upon the hardware configuration. After training, the VI. CONCLUSION
testing process has been done and some extra code
has been written to capture video and to detect In this paper, we proposed a system to detect
pothole in the video as well as in image (we potholes in real-time in images/videos captured by
actually need “detection of the pothole in videos”). a camera mounted on the vehicle and to give an
Testing the system shows good performance but it alert to the driver about the pothole on road in front
can be improved further. Figure 05, shows the of the vehicle. Further, our system will detect the
result in the testing process. location of the pothole and upload the same on map
(reflected in android app developed by us) so that
other users who have no camera mounted on their
vehicle can get alert about the pothole using the
app only (however this is our future work). In this
system, we used famous and complex CNN
architectures like Inception v1 (GoogLeNet),
inception v2 and finally select inception v2 in our
system. In the experiment data, the system shows
extremely good results. Our Future works include
experiments for improving the system and making
Fig.05 Performance of the model it simple for common/general use (Read more in
future works section).

Authorized licensed use limited to: Cornell University Library. Downloaded on August 26,2020 at [Link] UTC from IEEE Xplore. Restrictions apply.
 VII. Future Work

Our system successfully detects pothole in

videos/images but there are some works yet to be
done. Those are our future works regarding pothole
detection. We’ll try another good CNN
architectures like the latest versions of inception
(inception v3 and inception ResNet) for improving
the speed and accuracy. Additionally, a GPS
system has to be created for updating the location
of potholes on the map for all users. In near future,
we’ll create a GPS enabled system and an android
app (with GPS and google map enabled) with
which we will update the location of the pothole to
the map when a user finds it on the road so that
other users get the location of potholes earlier
without any detection. We also have to deploy our
system to raspberry pi or Android so that common
people can use that.

REFERENCES
[1] [Link] presentation
[2] [Link]
47/10/10/ [Link]
[3] [Link] ed/fix-the-
pothole-problem/[Link]
[4] Artis Mednis, Girts strazdins, reinholds zviedris,
georgijs kanonirs, leo selavo Digital Signal
processing laboratory, Institute of Electronics and
computer science, “Real time pothole detection using
android smartphones with accelerometers”, 14
December str., Riga, LV 1006, Latvia
[5] Lin, J., & Liu, Y. (2010, August). Potholes detection
based on SVM in the pavement distress image.
In 2010 Ninth International Symposium on
Distributed Computing and Applications to Business,
Engineering and Science (pp. 544-547). IEEE.
[6] Pereira, V., Tamura, S., Hayamizu, S., & Fukai, H.
(2018, July). A Deep Learning-Based Approach for
Road Pothole Detection in Timor Leste. In 2018
IEEE International Conference on Service
Operations and Logistics, and Informatics
(SOLI) (pp. 279-284). IEEE.
[7] S. Nienaber, M.J. Booysen, R.S. Kroon, “Detecting
potholes using simple image processing techniques
and real-world footage”, SATC, July 2015, Pretoria,
South Africa
[8] S. Nienaber, M.J. Booysen, R.S. Kroon, “A
comparison of Low-cost Monocular Vision
Techniques for Pothole Distance Estimation”, IEEE
CIVTS, December 2015, Cape Town, South Africa

Authorized licensed use limited to: Cornell University Library. Downloaded on August 26,2020 at [Link] UTC from IEEE Xplore. Restrictions apply.