2023 IEEE International Symposium on Smart Electronic Systems (iSES)

Drone based Potholes detection using Machine

Learning on various Edge AI devices in Real-Time
Aniruddhsinh Parmar Ruchi Gajjar Nagendra Gajjar
Electronics & Communication Electronics & Communication Electronics & Communication
Engineering Department Engineering Department Engineering Department
Institute of Technology Institute of Technology Institute of Technology
Nirma University Nirma University
2023 IEEE International Symposium on Smart Electronic Systems (iSES) | 979-8-3503-8324-9/23/$31.00 ©2023 IEEE | DOI: 10.1109/ISES58672.2023.00016

Nirma University
Ahmedabad, India Ahmedabad, India Ahmedabad, India
aniruddhsinhp2001@[Link] [Link]@[Link] [Link]@[Link]

Abstract— Pothole detection is an important aspect of road vehicles and due to potholes, lots of accidents happened on the
maintenance and safety. The presence of potholes on roads can
roads. So many people get injured in accidents or die due to
cause damage to vehicles, create safety hazards for drivers, and
increase maintenance costs. Timely detection of potholes can these potholes in India every year. The Ministry of Road
alert the driver for better navigation as well as the location of the Transport & Highway have reported that 4775 accidents had
potholes can be shared with the local governing body for taking occurred in 2019 due to potholes, injuring 4012 people and
counteractions. In this paper, we have detected potholes using killing 2140 people. In 2020, there occurred 3,564 accidents
two object detection models, YOLOv5, YOLOv7 & YOLOv8 and where 1471 people got killed and 3064 people got injured due
have employed the trained models on Edge AI Devices for real- to potholes on the roads [1]. The situation worsens during
time inference. We have compared the outcomes of YOLO in monsoon seasons. Sometimes, due to water lodging, these
terms of accuracy, precision and recall. YOLOv8 is giving 78% potholes are not clearly visible and lead to accidents.
of accuracy, YOLOv7 is giving 67% of accuracy and YOLOv5 is
giving 70% of accuracy. These models are deployed on If these potholes can be detected, timely counter measures
Raspberry Pi-4, Nvidia Jetson Nano and Intel UP Squared board. can be taken and these accidents can be avoided. Researchers
The performance of these Edge AI devices is analyzed in terms of and the scientific community are employing various techniques
fps and through experimental results it is observed that the Intel to identify the potholes. Various image processing-based
UP Squared board achieves higher fps. For real-time detection of techniques, machine learning based approaches are reported in
potholes, live videos were collected through drones and were live literature for identification of potholes. The complex object
streamed to the Edge AI devices through the RTMP server for detection architecture like YOLO has been used to detect
real-time inference. Overall, the proposed system is highly pothole [2, 3].
efficient for the reliable detection of potholes under different
road conditions. In this paper we propose a machine learning based
approach to detect the potholes in real time. The pothole is
Keywords – Pothole detection, Machine Learning, Drone detected through live video stream captured via drones and are
Video, Edge AI, YOLO. identified using trained machine learning models ported on
various Edge AI devices. We have created a custom dataset for
I. INTRODUCTION training the machine learning model by capturing images from
various sources such as the internet, manual clicking the photos
One of the basic requirements for development of a Smart of potholes using a camera as well as drone. The trained model
City or a Smart Village is better transportation services. The is then deployed on various embedded devices such as
infrastructure of any nation depends on transportation for better Raspberry Pi, Nvidia Jetson Nano and Intel UP square board,
connectivity, travel and ensuring that necessary requirements for real time inference. A drone sends live video feed to these
timely reach to the people. The growth of economy, education, Edge AI devices, which detect the potholes in real-time. If the
and health sector also rely on the availability of a good GPS coordinates of these potholes are mapped, we can then
transportation system. With the fast-paced life in the modern send them to the governing authorities for taking timely actions
times, the transportation facilities are highly dependent on the against these potholes, in order to avoid accidents.
quality of the roads. Be it a small town or a large city, The main contributions of this work are as follows:
necessary services, emergency services, goods delivery, public
commutation, all can be timely reached to the people if the x Accurate detection of potholes through machine
roads are in good condition. So, the roads must be better and learning, by training the model on custom dataset.
well maintained in order to ensure that these requirements are x Implementation of pothole detection models on Edge
fulfilled. AI devices for performance comparison.
Potholes on roads are causing severe problems with x Real-time surveillance and pothole detection through
transportation connectivity. It is also causing damage to drones.

2832-3602/23/$31.00 ©2023 IEEE 22

DOI 10.1109/iSES58672.2023.00016
Authorized licensed use limited to: VIT University- Chennai Campus. Downloaded on April 24,2025 at [Link] UTC from IEEE Xplore. Restrictions apply.
The rest of the paper is organized as: Section II presents the From Table I, it is evident that YOLO is efficient in
related work done in the area of pothole detection using detecting potholes. However, not many works have deployed
machine learning. Section III describes the proposed this on the embedded platform for real-time inference. In this
methodology. Section IV discusses the experimental results paper, we are using YOLOv5, YOLOv7 and YOLOv8 for the
and the conclusion is presented in Section V. detection of potholes and have deployed the trained models on
three different Edge AI devices and evaluated their
II. RELATED WORK performance. Also, to detect potholes in live videos, we have
With the rise in machine learning approaches for streamed live videos from drones to this hardware.
performing complex detection tasks, various object detection III. METHODOLOGY
models are being employed for the detection of potholes.
YOLO is one such object detector that is employed for various A. Dataset Preparation
object detection tasks in real-time. Different versions of
The dataset was collected from various sources such as the
YOLO have been used in literature for pothole detection [13-
internet, by phone camera and also using the drone. Fig. 1
16].
presents the sample of the images gathered for generating the
dataset. A total of 525 such pothole images were collected and
The authors in [7] used Yolov3 to detect potholes with an
manually annotated by creating the bounding box over the
accuracy of 96%. Multiple YOLO model versions such as
region of interest and labelling the same. Once the dataset was
Yolov1, v2, v3, v4, tiny-v4, v5 were used for pothole
created, it was fed for training in the object detection model.
detection by [13], achieving highest mean average precision of
80%, 85.48%, 95%. Yolov5 was implemented on Nvidia RTX
2070 in [14], achieving a fps of 30 and a detection accuracy of
91%. [15] used Resnet50 for pothole detection with an
accuracy 98%, while [16] achieved an accuracy of 85% on
Yolov7 for pothole detection. Table I presents a summary of
the works reported in the literature for pothole detection.

Table I
Summary of various approaches for pothole detection using machine
learning

Sr. Objective Model Accuracy Embedded Drone

No. and reference Hardware
1 Road Surface YOLOv3 96% No No
Classification
and Fig. 1. Sample images from the dataset
Subsequent
Pothole
Detection
Using Deep The images are preprocessed and resized to 640 x 640. The
Learning [7] complete dataset is divided into 80% for training (424 images),
2 Pothole YOLOv1, 80% No No 10% (53 images) for validation and 10% for testing (48
Detection YOLOv2, 85.48% images). The YOLO models are trained with these images
Using YOLOv3 95%
along with their labels for detection of potholes on images, pre-
YOLOv3
Model [13] recorded videos as well as live videos.
3 Image-based Resnet 98.0% No No
Road Pothole B. Model Training
Detection Three object detection models, YOLOv5, YOLOv7 and
using Deep YOLOv8 were selected for training with the pothole images.
Learning
Model [15]
The hyperparameters used for these models are listed in Table
4 Pothole YOLOv7 85.0% No No II. After training the models with various combinations of the
Detection epochs, image size and batch size, the optimal parameters
using CNN leading to highest accuracy are presented here.
and YOLO
v7 Algorithm Table II
Hyperparameters Details
[16]
Hyperparameters YOLOv5 YOLOv8
5 Pothole YOLOv3 91.0% Yes No
Detection Nvidia Batch size 20 16
and RTX
Epochs 90 100
Avoidance 2070
via Deep Image Size 416 640
Learning on
Edge devices
[14]

23

Authorized licensed use limited to: VIT University- Chennai Campus. Downloaded on April 24,2025 at [Link] UTC from IEEE Xplore. Restrictions apply.
 Once the models are trained, the After the testing is In order to deploy the machine learning models on these
complete, we also do real-time detection of the potholes using Edge AI devices, we need to first check the Edge AI hardware
the drone. It is also detected in the video. We take some online configuration and respective specifications. Deploying DL
videos that help us to show the accuracy of the model that will models on these edge devices needs to have a check on the
be shown in section V. model complexity, frameworks, libraries, computations as well
as memory constraints [8]. Selecting the right device for an
C. Block Diagram of system application depends on the expected outcomes as well as
Figure 2 presents the block diagram of the proposed system evaluation of performance parameters such as inference time,
for pothole detection. The YOLO models are trained with the efficiency, latency [9].
custom created dataset. Once the hyperparameters are tuned To deploy the trained YOLO models on the hardware,
and the models are trained, the trained models are deployed on following steps were taken: Installation of the operating
embedded devices. A drone captures real-time images and system, depending on the choice of device, installation the
videos of the potholes which are then fed to these Edge AI requirements and libraries for the trained model, deployment of
devices. The input gets processed by the trained models and the the trained model and then performing the inference. The
potholes are detected in real-time, as shown in Figure 2. Real- models are tested with images, videos as well as live video
time detection is done by using a drone. However, we can also from camera to compare the outcomes in terms of FPS and the
take the inputs using webcams or a camera on the Edge AI accuracy.
hardware. The drones provide us the advantage of ease of
collection of the data from any location, at any height and E. ML Model with Edge AI Device Using Drone
angle of view. This facilitates detection of potholes on roads,
Drones are used for detection of objects in real-time
irrespective of the traffic, weather or time conditions.
through machine learning models [10-11]. In this work, we are
using a DJI-Mavic Mini 2 drone for real-time pothole
detection. RTMP (Real-Time Messaging Protocol) server is
used to connect the drone to the embedded hardware. For
connectivity, it requires an IP address setup. After completing
the connectivity of the server and the Edge AI device, we
perform inference by the trained model on the live video
stream transmitted by the drone and detect the potholes.

IV. RESULTS
This section is divided into two parts. The first sub-
sections describe the YOLO model training and testing results,
while the last sub-section presents the outcomes of pothole
detection on Edge AI devices in real-time using drone
imaginary.
Fig. 2. Block diagram of the proposed system
The drone controller is connected to the RTMP (Real-time A. Pothole Detection Models Results
Messaging Protocol) server that is connected to the Edge AI The pothole detection models YOLOv5, YOLOv7 and
hardware using the server. RTMP server provide low latency. YOLOv8 are trained on the same custom dataset. The
It is better for Real Time Detection. So, whatever is in the field outcomes are compared in terms of precision, recall and mean
of view of the drone, can be processed on the Edge AI device Average Precision (mAP). mAP is a metric for measuring
and can be viewed on a hardware display. This provides accuracy of object detectors. Table III presents this
portability to the complete system and can be deployed comparison. It can be observed that YOLOv8 has more
anywhere for real-time inference. precision score in comparison to YOLOv5 and YOLOv7 has
less precision score. it is the latest version of YOLO.
D. Pothole Detection model on Edge AI Device
Table III
The capability of object detection models like YOLO and
SSD in detecting multiple objects at a very high accuracy Model comparison in terms of precision, recall and mAP
motivated us to employ YOLO for detection of potholes. We Model Precision Recall mAP
have trained YOLOv5, YOLOv7 and YOLOv8 with the dataset
and then deployed these models on three different hardware - YOLOv5 0.89 0.79 0.70
1) Raspberry Pi-4, 2) Intel UP squared, 3) Nvidia Jetson Nano. YOLOv7 0.70 0.65 0.67
Raspberry Pi-4 is a single board computer with 64-bit quad YOLOv8 0.96 0.81 0.78
core Cortex-A72 processor. It usually employed for IoT, image
and video processing, applications. The Jetson Nano is
specifically designed for AI and ML workloads, providing Fig. 3, Fig. 4 and Fig. 5 presents the pothole detection
excellent performance with its dedicated GPU and CUDA results using YOLOv5, YOLOv7 and YOLOv8. It can be
support. The Intel UP Squared generally offers better CPU observed that both the models are able to efficiently detect the
performance compared to the Raspberry Pi 4 and Jetson Nano, potholes on images as well as videos. The accuracy of the
with its Intel Atom processor. However, the Jetson Nano excels pothole detection in YOLOv8 is 78%, YOLOv7 is 67% and
in GPU performance, making it more suitable for AI and YOLOv5 achieved an accuracy of 70%.
machine learning tasks.

24

Authorized licensed use limited to: VIT University- Chennai Campus. Downloaded on April 24,2025 at [Link] UTC from IEEE Xplore. Restrictions apply.
 Table IV
Comparing the performance on various Edge AI Devices

Devices fps
Raspberry Pi-4 10
Nvidia Jetson Nano 12-15
Intel UP Squared 25-30

(a) (b) (c)

Fig. 3. Pothole detection results in images using YOLOv5. (a)
Detection results on images, (c) Results on video

(a) (b)
(a) (b) (c)
Fig. 4. Pothole detection results in images using YOLOv7. (a)
potholes detection image, (b) video of the detection and (c)
detection of video.

(a) (b) (c) (c) (d)

Fig. 5. Pothole detection results in images using YOLOv8. (a)

and (b) Detection results on images, (c) Results on video

B. Real-time detection Results

To detect the potholes in real-time, live videos from drones
were sent to these edge devices. We have used DJI-Mavic
Mini 2 drone for collecting the data. The videos were sent
through the RTMP server for real-time inference. Table IV
presents the comparison of analysis in terms of frames per (e)
second (fps) obtained from the inference of potholes by Fig 6. Results of pothole detection on hardware through video
Raspberry Pi-4, Nvidia Jetson Nano and Intel UP squared data from drones and pre-recorded videos (a) image of drone,
vision board. It is observed that Intel UP squared board (b) detection of pothole, (c) Edge AI device setup,
achieves the highest fps in comparison to the other two. We (d) pothole detection on videos, (e) Screenshot of pothole
have been able to efficiently detect potholes not only in detection from the video
images and videos but also in live videos captured through
drones.
Fig. 6 presents the detection results on images, videos and V. CONCLUSION
live stream data captured through drone, which are processed
on the Edge AI devices. It is evident from these images that This paper presents a machine learning-based approach for
the proposed system is efficiently able to detect the potholes in pothole detection using YOLO architectures. The YOLOv5
and YOLOv8 are implemented on a custom dataset created by
real-time, thereby providing the facility of taking timely
collecting images from various sources like the internet,
counter measures.
through phone and also by drone. The trained YOLO models
are deployed on Edge AI devices - Raspberry Pi-4, Intel UP
squared and Nvidia Jetson Nano for real-time inference. The
detection results show that YOLOv5 achieves 70%, YOLOv7
achieves 67% and YOLOv8 achieves 78% of accuracy. Live
videos from the drone are sent to these Edge AI devices for

25

Authorized licensed use limited to: VIT University- Chennai Campus. Downloaded on April 24,2025 at [Link] UTC from IEEE Xplore. Restrictions apply.
real-time detection of the potholes. From the detection results, [8] D. Kolosov, V. Kelefouras, P. Kourtessis, P. and I. Mporas, “Anatomy
it can be concluded that the said approach is highly efficient in of Deep Learning Image Classification and Object Detection on
Commercial Edge Devices: A Case Study on Face Mask Detection.”
the detection of potholes in real-time. This can be useful in the IEEE Access, 2022, Vol. 10, pp.109167-109186.
timely detection of potholes before they lead to any serious
[9] C. A. A. Valencia, R. S. S. Suliva and J. F. Villaverde, "Hardware
injuries while driving. Also, drone videos can alert the Performance Evaluation of Different Computing Devices on YOLOv5
governing bodies for road repair and filling up of these Ship Detection Model," 2022 IEEE 14th International Conference on
potholes. Humanoid, Nanotechnology, Information Technology, Communication
and Control, Environment, and Management (HNICEM), Boracay
References Island, Philippines, 2022, pp. 1-5, doi:
10.1109/HNICEM57413.2022.10109499
[10] M. Issame and A. Benyounes, "Real Time Object Detection With
[1] Ministry of Road Transport & Highway Report of 2021: Drone Using Deep Learning Algorithm," 2022 International
[Link] Conference of Advanced Technology in Electronic and Electrical
[2] K. P. Rani, M. Arshad and A. Sangeetha, "Pothole Detection Using Engineering (ICATEEE), M'sila, Algeria, 2022, pp. 1-6, doi:
YOLO (You Only Look Once) Algorithm," 2022 International 10.1109/ICATEEE57445.2022.10093104
Conference on Advancements in Smart, Secure and Intelligent [11] J. Park, D. H. Kim, Y. S. Shin and S. -h. Lee, "A comparison of
Computing (ASSIC), Bhubaneswar, India, 2022, pp. 1-6, doi: convolutional object detectors for real-time drone tracking using a PTZ
10.1109/ASSIC55218.2022.10088357. camera," 2017 17th International Conference on Control, Automation
[3] S. Ajmera, C. A. Kumar, P. Yakaiah, B. Kumar and K. Y. Chowdary, and Systems (ICCAS), Jeju, Korea (South), 2017, pp. 696-699, doi:
"Real-time Pothole Detection using YOLOv5," 2022 International 10.23919/ICCAS.2017.8204318.
Conference on Advancements in Smart, Secure and Intelligent [12] K. Ding, X. Li, W. Guo and L. Wu, "Improved object detection
Computing (ASSIC), Bhubaneswar, India, 2022, pp. 1-5, doi: algorithm for drone-captured dataset based on YOLOv5," 2022 2nd
10.1109/ASSIC55218.2022.10088290. International Conference on Consumer Electronics and Computer
[4] Q. Wu, C. Wang, Y. Han, Q. Kang, J. Li and X. Lu, "Object detection Engineering (ICCECE), Guangzhou, China, 2022, pp. 895-899, doi:
of double-sided copper laminates based on YOLOv5," 2023 IEEE 3 rd 10.1109/ICCECE54139.2022.9712813.
International Conference on Power, Electronics and Computer [13] R. Sathya, B. Saleena and B. Prakash, "Pothole Detection Using
Applications (ICPECA), Shenyang, China, 2023, pp. 171-175, doi: YOLOv3 Model," 2023 IEEE International Students' Conference on
10.1109/ICPECA56706.2023.10075704. Electrical, Electronics and Computer Science (SCEECS), Bhopal,
[5] S. B. Choi, S. -S. Lee, J. Park and S. -J. Jang, "Implementation of post India, 2023, pp. 1-7, doi: 10.1109/SCEECS57921.2023.10063116.
processing hardware for real-time object detection in CNN acceleration [14] C. -W. Kuan, W. -H. Chen and Y. -C. Lin, "Pothole Detection and
system," 2021 36th International Technical Conference on Avoidance via Deep Learning on Edge Devices," 2020 International
Circuits/Systems, Computers and Communications (ITC-CSCC), Jeju, Automatic Control Conference (CACS), Hsinchu, Taiwan, 2020, pp. 1-
Korea (South), 2021, pp. 1-4, doi: 10.1109/ITC- 6, doi: 10.1109/CACS50047.2020.9289701.
CSCC52171.2021.9501433
[15] P. Gupta and M. Dixit, "Image-based Road Pothole Detection using
[6] H. Yanagisawa, T. Yamashita and H. Watanabe, "A study on object Deep Learning Model," 2022 14th International Conference on
detection method from manga images using CNN," 2018 International Computational Intelligence and Communication Networks (CICN), Al-
Workshop on Advanced Image Technology (IWAIT), Chiang Mai, Khobar, Saudi Arabia, 2022, pp. 59-64, doi:
Thailand, 2018, pp. 1-4, doi: 10.1109/IWAIT.2018.8369633. 10.1109/CICN56167.2022.10008278.
[7] R. Agrawal, Y. Chhadva, S. Addagarla and S. Chaudhari, "Road [16] E. S. T. K. Reddy and R. V, "Pothole Detection using CNN and YOLO
Surface Classification and Subsequent Pothole Detection Using Deep v7 Algorithm," 2022 6th International Conference on Electronics,
Learning," 2021 2nd International Conference for Emerging Communication and Aerospace Technology, Coimbatore, India, 2022,
Technology (INCET), Belagavi, India, 2021, pp. 1-6, doi: pp. 1255-1260, doi: 10.1109/ICECA55336.2022.10009324.
10.1109/INCET51464.2021.9456126.

26

Authorized licensed use limited to: VIT University- Chennai Campus. Downloaded on April 24,2025 at [Link] UTC from IEEE Xplore. Restrictions apply.