2023 International Conference on Integration of Computational Intelligent Systems (ICICIS)

Pune, India. Nov 1-4, 2023

Intelligent Vehicle for Simulated Pothole Detection

using Image Processing
Pranav Garje B.K.Patle
Department of Mechanical EngineeringMIT School of Department of Mechanical EngineeringMIT School of
Engineering, MIT ADT University Pune, India Engineering, MIT ADT University
2023 International Conference on Integration of Computational Intelligent System (ICICIS) | 979-8-3503-1878-4/23/$31.00 ©2023 IEEE | DOI: 10.1109/ICICIS56802.2023.10430301

Pune, India Pune, India

[email protected] [email protected]

Abstract—Road defects such as potholes, cracks, and other appropriate action. This problem is especially important in a
flaws can harm automobiles, impair a driver's ability to drive dynamic outdoor setting. The intriguing topic of finding
safely, and in some circumstances cause serious traffic accidents. potholes has important consequences for maintaining
Therefore, road conditioning methods need to be implemented for roadways. Planning for highway maintenance is hampered by
developing and maintaining roads. Prior until now, detection of the inaccuracy of hand measurements, which might lead to
potholes was done only by visual inspection by human experts incorrect prioritization of remedial work. Every year, many
which is a time-consuming process and can even be inaccurate at lives are lost in traffic accidents. Poor road conditions are a
times. Country’s economy depends on well-maintained roads as major factor in most deaths. Accidents happen more frequently
they are major means of transportation. It becomes essential to
during the monsoon as the condition of the roadways continues
identify pothole and humps in order to avoid accidents and
damages to the vehicles that is caused because of distress to drivers
to deteriorate. Official data, however, seldom reflect the
and also to save fuel consumption. In this regard, this prototype severity of the issue. We may assume that many fatalities are
work presents a simple solution to detect potholes and humps and brought on by bad roads and potholes because the majority of
hence maintaining roads in good quality and avoiding accidents. instances go unreported. It is important to create an automated
Potholes are detected using Image Processing Technique with system for identifying potholes and filling them.
OpenCV and Python are used to detect road defects and rectify The system's main component is the image sensor. It
them by the means of a dispensing mechanism. A Raspberry Pi 4
provides the Raspberry Pi 4 with an input, and it processes the
Model B microprocessor is used on-board the AVPD for
obtained picture results. Using Python and OpenCV, image
controlling it. This project presents an attempt to design and
constructing a prototype of an automated road repair vehicle
processing is done on the findings that were obtained. A
called the Autonomous vehicle for Pothole Detection (AVPD). The Raspberry Pi 4 Model B CPU that is included within the device
AVPD is capable of automatically detecting and filling potholes on controls the AVPD. The Autonomous Vehicle for Pothole
road surfaces without operator assistance. An easy-to-construct Detection (AVPD), a prototype for an automated road repair
mechanical means of pothole detection was employed to reduce vehicle, is created. The AVPD can automatically find and fix
costs and complexity that have thus far been the primary potholes on road surfaces without a human's help. An
disadvantage of automated road repair vehicles. A network autonomous road repair vehicle prototype is being conceived
interface using Wi-Fi was designed based on IOT to enable remote and developed to overcome the primary problem of manual
operability of the AVPD. The vehicle first detects the pothole, pothole identification and doing road repair operations. Based
dispenses the pothole filler material, flattens the filled material on IOT, a network interface using Wi-Fi is developed to enable
andthen goes ahead to detect another pothole. remote control of the AVPD. The pothole is initially spotted by
the vehicle, which then fills it with filler material, eliminates it
Keywords—image processing, opencv, python, microprocessor out, and moves on to notice another one.
I. INTRODUCTION
Worldwide traffic growth has increased road surface
deterioration, leading to the development of potholes, cracks,
and other flaws in the road surface. Roads must be kept in good
shape in order for drivers to travel safely and avoid accidents
brought on by poor road conditions. A pothole is a significant
road defect or a hole on the road surface that is wider than 150
millimeters and deeper than 25 millimeters. Potholes arise for a Fig. 1. Working of AVPD
variety of reasons, such as pressure from moving traffic and
severe weather that causes direct damage to the road surface or The use of smartphone sensors for road surface monitoring
blocked drainage systems [1]. Because of their highly evolved has been studied by Sattar et al. [2]. If smartphones were used
visual systems, humans can quickly process and evaluate vast to detect anomalies in the road surface, the way that
volumes of information. We have highly advanced government agencies track and arrange road repair may
technologies that can decide based on visual data and carry out change. Poor detection accuracy is caused by the low
the required activities. A robot that is really autonomous must frequency of modern smartphone sensors and the under
be able to accurately analyze its environment and take sampled sensor data. The authors examine and contrasts the
most modern mobile technologies for spotting irregularities in

979-8-3503-1878-4/23/$31.00 ©2023 IEEE 1

Authorized licensed use limited to: BMS College of Engineering. Downloaded on October 28,2024 at 17:12:53 UTC from IEEE Xplore. Restrictions apply.
the road surface. Further smartphone research opportunities for reliably identifies potholes but also displays intelligent driving
identifying irregularities in road surfaces are also addressed. In conduct when one is spotted. A vision camera was employed
order to provide vehicles with a smooth and secure road during the development process to test and assess the possible
infrastructure, road surface monitoring is a crucial component. autonomy of the prototype's driving behavior in the
Anomalies such as potholes, cracks, and bumps that impair predetermined environment of the road. Experimental testing
driving comfort and on-road safety must be identified in order of the recommended model on several performance metrics
to effectively monitor the state of the road surface. Research on produced 90% accuracy. Byeong-ho Kang and Su-il Choi's
finding irregularities in the road surface has been substantial. effort to identify potholes using a photo sensor and a LiDAR
Smartphone-based sensing has lately increased in popularity as sensor [7] demonstrates their work and experiment. The first
more integrated smartphone sensors have been more widely step in creating the finest repair strategies for asphalt-surfaced
available. In their study, Chen et al. [3] employed a multi- pavement is automatically identifying potholes. A system is in
sensor system to illustrate the utilization of efficient data- place that uses a camera and 2D LiDAR to detect potholes. The
fusion methods. When measurements are made using a vehicle accuracy of pothole detection is increased by combining a
moving at typical city street speeds, a reliable, affordable road variety of heterogeneous sensor systems. 2D LiDAR is used to
surface monitoring system with enough accuracy to meet collect the road's distance and angle data. An image-based
typical maintenance needs in terms of the detection, pothole identification method is used to improve pothole
localization, and quantification of potholes and similar recognition accuracy and ascertain the shape of the pothole.
qualitative deterioration features can be produced. A number of Trials using a camera and 2D LiDAR show the pothole
automobiles would be outfitted with this relatively priced detection system's efficacy. According to Harshad Sawalakhe
device as part of the suggested method, which is great for and Ramchandran Prakash [8], it is important to consider road
crowdsourcing and allows for more frequent road surface data features, such as road distress including potholes, cracks, and
collecting. To locate potholes and notify the proper authorities, patches of various sizes. The condition of the road has an
Sengottaiyan et al. [4] employed an IoT-based, low-cost, influence on the convenience and safety of everyone travelling
portable gadget. This method assures proper upkeep and in a car. Detecting potholes allows for early warning to the
prevents traffic accidents. The suggested system may be driver and assessment to the organization in charge of road
installed on a moving vehicle, allowing you to take images that repair. The position and design of the pothole can be
are then sent, along with GPS coordinates, to the appropriate communicated to the organizations responsible for road
authorities, further alerting them to take corrective action. They maintenance. An embedded device may use road image
primarily focus on integrating the aforementioned technologies photographs to detect potholes before transmitting the GPS
into motorbikes in a way that is uniquely suited to the user's location to the server of the road maintenance authority staff
requirements. The coordinates of the GPS position will be and uploading it to Google Maps. This prototype system uses
stored on both the Things Board server and the hosting the computer vision library to analyze video input from a
platform of Amazon Web Services. The entire system was camera and recognize any road potholes that may be present. It
successfully integrated with the Raspberry Pi3 Single Board is constructed on the little Raspberry Pi single-board computer.
Computer (SBC), which was used to capture and evaluate the Juyoung Park et al. [9] Here, PotholeEye can automatically
images as well as for email communication protocol. monitor a road's surface and detect pavement damage in real
According to Mehta, J. et al., potholes are a major time via video analysis. PotholeEye preprocesses the images,
inconvenience that result in numerous problems and decreased extracts attributes, and divides the pain into several categories
productivity [5]. Municipal officials claim that one of the rate- while the road manager is driving. PotholeEye was tested using
limiting restrictions for repairs is the geographical localization real hardware, including a sensor that includes a camera, a
of these potholes. PoDAS (Pothole Detection and Analysis small computer, a Global Positioning System receiver, and
System), a novel, reasonably priced, sensor-based, wireless other equipment, on actual roadways every day for over a year.
system that may be put throughout big cities, was presented by As a consequence, 92% of the time, PotholeEye correctly
the creator. They discuss a number of implementation models recognized the pavement damage. Road surface flaw
that may be modified to meet the needs of different cities. By identification using computer vision has been presented by
using validation from a variety of sensors, they have created a Koch et al. [10]. They discuss the current state of practice for
system that is more effective than some of the earlier models visual condition testing. They give a comprehensive overview
that have been released. In order to assess the efficacy and of the method for identifying computer vision faults and
efficiency of the recommended approach, they also provide the assessing the state of concrete roads. Rui Fan et al. [11]
findings of extensive testing carried out in a variety of presented and discussed a strong stereo vision system on an
scenarios. According to Deepak Kumar and Satya Prakash [6], unmanned aerial vehicle (UAV). An evaluation of the status of
advances in vision-based techniques have allowed autonomous the road surfaces is required to guarantee their serviceability
vehicle systems (AVS) to fully perceive driving scenarios. while maintaining the greatest degree of traffic safety. To
Pothole objects might be difficult to recognise due to their lower processing complexity and increase disparity accuracy,
erratic structure in difficult, frantic driving conditions. The the perspective view of the target picture is first converted into
usefulness of currently used methods for correctly identifying the reference view. It has been established that the Markov
potholes has been limited. In the literature, there is little random field model provides a workable solution to the
information on pothole detection and intelligent behavior of functional minimization issue. By minimizing an energy
autonomous driving vehicle systems. Because of this, this function with regard to the roll angle and disparity projection
improved prototype model employing a Raspberry Pi not only model, the disparity maps are ultimately translated. As a result,

2
Authorized licensed use limited to: BMS College of Engineering. Downloaded on October 28,2024 at 17:12:53 UTC from IEEE Xplore. Restrictions apply.
it is easier to tell the damaged portion of the road apart from color objects of different shape, size and depth. In the 1:1
the rest of it. Based on an NVIDIA Jetson TX2 GPU with model, we can easily re-code the program with Canny Edge
CUDA for real- time applications, the suggested solution. Algorithm [12] which extracts the pothole’s edges. The rest of
the actuation of servo would remain the same. In this
The paper is prepared as, Section 1 of the paper prototype, contours are created around the detected red pothole
introduces the title of the paper, Section 2 provides the which actuates the dispensing mechanism.
problem description. A detailed description of system design
and methodology is given in Section 3. In Section 4, the Python is used to write code and image processing
experimental analysis are provided and Section 5 concludes algorithms directly on the Raspberry Pi. The AVPD prototype
this paper with scope for thefuture. uses the HSV color space algorithm. First the BGR colours are
detected in a video frame. Out of which we are selecting only
II. PROBLEM DESCRIPTION Red color using the HSV color frame filter. After this stage,
Potholes can be fixed manually using a variety of contours are created around the detected red object.
techniques. The main purpose of this project is to repair roads Programming is an essential step in the design and
with ease. It takes time and effort to manually fix a road. These development of AVPD as well as the creation of the prototype.
methods are hard and require human efforts. Hence, we have to Python was used for developing the image processing and
automate it. We can save time and money by automating this servo actuation code on the Raspberry Pi since this allowed for
laborious job. There is also a need to have manual controlling quick development and testing. The autonomous part which is
capability with this vehicle. When roads are often loaded and image processing and servo actuation works on the Raspberry
weathered, a pothole may be formed, which might be quite pi whereas the manual controlling capability works on IOT
dangerous for people to go across. The objective of the project platform using NodeMCU. On the Arduino IDE, the
is to prevent accidents on the road and to ensure road safety. NodeMCU code was developed and compiled before being
Due to the vehicle's autonomy, this will lower the cost of road uploaded to the device. Both the Arduino IDE and the
maintenance. Raspberry Pi have the necessary libraries installed. The
following is a description of the used image processing
III. SYSTEM DESIGN AND METHODOLOGY algorithm.
The working methodology of the project is discussed Libraries Opencv and Numpy are imported. PCA9685 is
below. The system's primary component is the camera sensor. imported for running servo motor. Servo motor settings are
Live view of the vehicle is provided via Camera 2, a Logitech configured. Servo position is defined. Camera 1 and Camera 2
C270 camera. The system is based on a Raspberry Pi 4 B 4GB are initiated to accept frames. Live stream is started from
model as the microcontroller. For controlling the Autonomous camera 2. Camera resolution is set. The value of the center
Vehicle for Pothole Detection (AVPD), it is connected to L298 location of the red object is defined. Zero position of the servo
motor driver. When a simulated pothole is detected, the is declared. Red object is detected using hsv. Contours are
Raspberry performs image processing and actuates the servo created around the detected red object. Once the red object is
motor. It is coupled to the servo motor that controls the detected, servo will move zero to ninety degrees which will
dispensing mechanism. dispense the filler material from the dispenser and again close
the dispenser. Shows detected red object with contours. Shows
the masked object in white on black background. Code
Explanation for autonomous working using Image Processing,
Servo Control and For Live View. The red object which
simulates a pothole is detected using the internal on board
camera port and processed. Live stream is started using the
external webcam which is connected on the USB 3.0 port of
the raspberry pi. After detection of the pothole which is the red
object, servo moves 90 degrees and back to original position.
The servo controls the dispensing mechanism. Once pothole is
detected, the filler material is dispensed on the pothole and the
roller presses this filler material eliminating the pothole. To
conclude, image processing is done on Raspberry Pi 4B in
python using OpenCV library. Hue Saturation Value (HSV) is
Fig. 2. Block diagram of AVPD
used for processing the image. Red object is depressed in the
test track which simulates a pothole. Various red objects of
L298 motor driver, to which 2 motors are linked, is different size and shape are placed on the track which
connected to the NodeMCU. M1 is used for direction control, simulates potholes of different size, shape and depth. It creates
and M2 is used for drum control. For steering and IOT control a contour box around the object.
of the vehicle, the NodeMCU is crucial. Which then, gives the output to the dispensing servo. The
Building a 1:1 scale vehicle of AVPD is a big task and it is circuit diagram is presented below.
difficult to achieve by a single individual. Therefore, a
prototype of AVPD is developed on which real potholes cannot
be detected, but potholes are simulated on the track with red

3
Authorized licensed use limited to: BMS College of Engineering. Downloaded on October 28,2024 at 17:12:53 UTC from IEEE Xplore. Restrictions apply.
 Fig. 5. Stage 1: Image processing for pothole detection at interval of 1 second
Fig. 3. Circuit diagram of AVPD

It is essential to calculate the battery runtime of the AVPD.

The power consumption of each component has been taken
into consideration for calculating the run time. Power
consumption for each component has been calculated with the
use of a digital multimeter. Battery life can be calculated by
battery capacity in mAh divided by the total load current drawn
in mA. Table 1 shows the current consumption of each
component.

Fig. 6. Stage 2: Dispensing mechanism is aligned to detected pothole

TABLE I. ESTIMATION OF BATTERY PACK RUNTIME
Total Current
Sr. no. Component Current(A)
(A)
1 30rpm motors 8 8
2 Dispensing mechanism 0.5 8.5
3 Raspberry Pi 1.7 10.2
4 NodeMCU 0.5 10.7 Fig. 7. Stage 3: filler material is dispensed in the pothole
5 Misc. converters 0.7 11.4
IV. EXPERIMENTAL ANALYSIS
The test cases for image processing are displayed below,
It is evident from the above table that the total current simulating various sizes, shapes, and depths of potholes.
requirement for the AVPD is around 11.4A at 12 volts. This Various pothole detection simulations using image processing
comes to around 137 watts. The battery pack that has been are displayed. Stages of simulated pothole detection are
used in the project is a 12Ah LFP battery. The reason for using displayed step by step. Experimental analysis shows the
LFP is that the battery pack has more than 2000 life-cycles on simulated potholes, image processing results at an interval of 1
it. The battery can last easily for more than 5 years. According second and stages of filling the simulated pothole. Results are
to the formula discussed above for calculating the runtime, we shown in tabular form and graphically.
can estimate the running time of AVPD as
12000mAh/11400mA. Therefore, the battery life in hours
comes to 1.052 which is approximately 1hour. The AVPD can
run for approximately 1 hour.

Fig. 8. Stage 4: Dispensed filler material flattened

Trials are taken for pothole detection where total number of

potholes detected are compared with total number of potholes
missed for potholes which are simulated at 3mm, 5mm and
7mm depth on the designed test track. 20 trials are taken for
Fig. 4. Simulated pothole on track of 3mm, 5mm and 7mm depth pothole at 3mm depth in which 19 were detected and 1 was

4
Authorized licensed use limited to: BMS College of Engineering. Downloaded on October 28,2024 at 17:12:53 UTC from IEEE Xplore. Restrictions apply.
missed. 15 trials are taken for pothole at 5mm depth in which artificial intelligence techniques might be used to improve the
18 were detected and 2 were missed and 15 trials are taken for pothole detection accuracy of autonomous vehicles.
pothole at 7mm depth in which 17 were detected and 3 were
missed. A total of 50 trials were taken in which a total of 44 ACKNOWLEDGMENT
potholes were detected and 6 were missed. If we calculate the This work is self-sponsored, and it has no connection with
accuracy, it comes to around 88 percent. The first pothole is any kind of funding.
detected after 8 seconds from the starting position of the track.
The first pothole is 3mm depth. Time taken to fill this REFERENCES
simulated pothole on track is 1.5 seconds. The second pothole [1] Wu, M., Lam, S. K., & Srikanthan, T. (2014). Nonparametric technique
is 5mm depth and detected after 4 seconds. Time taken to fill based high-speed road surface detection. IEEE Transactions on
this simulated pothole on the track is 2 seconds. The third intelligent transportation systems, 16(2), 874-884.
pothole that is detected on the track is 7mm depth. This pothole [2] Strutu, M., Stamatescu, G., & Popescu, D. (2013, October). A mobile
is detected 3 seconds after the previous pothole has been sensor network based road surface monitoring system. In 2013 17th
International Conference on System Theory, Control and Computing
detected. Time taken to fill this simulated pothole on track is (ICSTCC) (pp. 630-634). IEEE.
2.5 seconds. [3] Chen, Y. L., Jahanshahi, M. R., Manjunatha, P., Gan, W., Abdelbarr,
M., Masri, S. F., & Caffrey, J. P. (2016). Inexpensive multimodal sensor
fusion system for autonomous data acquisition of road surface
conditions. IEEE Sensors Journal, 16(21), 7731-7743.
[4] Kamalesh, M. S., Chokkalingam, B., Arumugam, J., Sengottaiyan, G.,
Subramani, S., & Shah, M. A. (2021). An intelligent real time pothole
detection and warning system for automobile applications based on IoT
technology. Journal of Applied Science and Engineering, 24(1), 77-81.
[5] Jinesh Mehta, Vinayak Mathur, Dhruv Agarwal, Atish Sharma, and
Krishna Prakasha. Pothole detection and analysis system (PODAS) for
real time data using sensor networks. Journal of Engineering and
Applied Sciences, 12(12):3090–3097, 2017.
Fig. 9. Pothole detection at various depth [6] Dewangan, D. K., & Sahu, S. P. (2021). PotNet: Pothole detection for
autonomous vehicle system using convolutional neural network.
Electronics Letters, 57(2), 53-56.
[7] Kang, B. H., & Choi, S. I. (2017, July). Pothole detection system using
2D LiDAR and camera. In 2017 ninth international conference on
ubiquitous and future networks (ICUFN) (pp. 744-746). IEEE
[8] Sawalakhe, H., & Prakash, R. (2018). Development of roads pothole
detection system using image processing. In Intelligent Embedded
Systems: Select Proceedings of ICNETS2, Volume II (pp. 187-195).
Springer Singapore.
[9] Kim, Y. R., Kim, T., & Ryu, S. (2014). Pothole detection method in
asphalt pavement. Journal of The Institute of Electronics and
Information Engineers, 51(10), 248-255.
Fig. 10. Time taken in seconds to detect potholes at various depths. [10] Koch, C., Georgieva, K., Kasireddy, V., Akinci, B., & Fieguth, P.
(2015). A review on computer vision based defect detection and
V. CONCLUSION condition assessment of concrete and asphalt civil infrastructure.
Advanced Engineering Informatics, 29(2), 196-210.
An efficient prototype of AVPD has been designed and [11] Fan, R., Jiao, J., Pan, J., Huang, H., Shen, S., & Liu, M. (2019). Real-
developed that can autonomously detect potholes and further time dense stereo embedded in a uav for road inspection. In Proceedings
does the levelling process. Online communication with the of the IEEE/CVF Conference on Computer Vision and Pattern
vehicle for real-time view and manual vehicle controlling Recognition Workshops (pp. 0-1).
capability has been established. The purpose of this project has [12] Murali, G. B., Santosh Kumar, V., Behera, D. N., Mohanta, K. K.,
been achieved which is to repair roads with ease. Repairing Tulankar, O., & Salimath, S. S. (2022). Pothole Detection on Roads
Using Canny Edge Detection Algorithm. In Applications of
road manually is hard and requires human effort. Hence there Computational Methods in Manufacturing and Product Design: Select
was a need to automate this task which would maintain roads, Proceedings of IPDIMS 2020 (pp. 653-661). Singapore: Springer Nature
save time and money. In the future, machine learning or Singapore.

5
Authorized licensed use limited to: BMS College of Engineering. Downloaded on October 28,2024 at 17:12:53 UTC from IEEE Xplore. Restrictions apply.