WEED DETECTION USING CONVOLUTIONAL

NEURAL NETWORK

A PROJECT REPORT

Submitted By

VISHAL.K
JERLIN JINO.J
JIJITH.B
LASWIN.S

in partial fulfilment for the award of the degree

of

BACHELOR OF ENGINEERING
in
COMPUTER SCIENCE AND ENGINEERING

PONJESLY COLLEGE OF ENGINEERING, NAGERCOIL

ANNA UNIVERSITY : CHENNAI 600 025

MAY 2024
 ANNA UNIVERSITY: CHENNAI 600 025
BONAFIDE CERTIFICATE

Certified that this project report “WEED DETECTION USING

CONVOLUTIONAL NEURAL NETWORK” is the bonafide work of
“VISHAL.K (961820104093), JERLIN JINO.J (961820104049), JIJITH.B
(961820104052), LASWIN.S (961820104058)” who carried out the project
work under my supervision.

SIGNATURE SIGNATURE
Mrs. Dr. MANJU C THAYAMMAL, Mrs .N. Shajini., ME, (Ph.D)
M.Tech., Ph.D.,

HEAD OF THE DEPARTMENT, SUPERVISOR,

ASSOCIATE PROFESSOR, ASSISTANT PROFESSOR,
Department of Computer Science Department of Computer
and Engineering, Science and Engineering,
Ponjesly College of Engineering, Ponjesly College of
Engineering,
Nagercoil-629 003. Nagercoil-629 003.

Submitted for the B.E Degree project work viva-voce held on ………… at
PONJESLY COLLEGE OF ENGINEERING.

INTERNAL EXAMINER EXTERNAL EXAMINER

 ACKNOWLEDGEMENT

First of all, we thank GOD almighty for his grace enabling us to

complete this work on time. We would like to extend our deep sense gratitude
to our Chairman Thiru. Pon Robert Singh, M.A., for his encouragement in
accomplishing this project work.

We express our deep sense gratitude and thanks to our beloved Principal
Dr. G. Natarajan, M.E., Ph.D., who is the guiding light of this outstanding
institution for having laid tracks that lead to a bright future. We are very much
thankful to our Director Prof. S. Arulson Daniel, M.Sc., M.Phil., for his
encouragement and construction ideas for my project.

We express our kind gratitude and thanks to our Head of the Department
Mrs. Dr. Manju C Thayammal, M.E., Ph.D.., who has given the source of
inspiration throughout our project. We express our kind gratitude and thanks to
our internal guide Mrs .Shajini., ME, (P.hD) who has given the sound
guidance throughout my project.

We hardly find words to express our gratitude for the help and warm
encouragement that we have received from our parents because, without their
sacrificial help we could not have dream of completing our project successfully.

VISHAL.K
JERLIN JINO.J
JIJITH.B
LASWIN.S
 ABSTRACT
Artificial intelligence, specifically deep learning, is a fast-growing research field today.
One of its various applications is object recognition, making use of computer vision. The
combination no these two technologies leads to then purpose of this thesis. In this project, a
system for the identification of different crops and weeds has been developed as an
alternative to the system present on the Farm Bot company’s robots. This is done by
accessing the images through the FarmBot API, using computer vision for image processing,
and artificial intelligence for the application of transfer learning of an CNN that performs the
plants identification autonomously. The results obtained show that the system works with an
accuracy of 78.10% for the main crop and 53.12% and 44.76% for the two weeds considered.
More over, the coordinates of the weeds are also given as results. The performance of the
resulting system is compared both with similar projects found during research, and with the
current version of the Farm Bot weed detector. From a technological perspective, this study
presents an alternative to traditional weed detectors in agriculture and open the doorste more
intelligent and advanced systems.
 INDEX
S.NO CONTENT PAGE NO

1 INTRODUCTION 1

1.1 BACKGROUND 1

1.2 PROBLEM DESCRIPTION 2

1.3 AIM AND OBJECTIVES 2

1.4 DELIMITATIONS 3

1.5 OVERVIEW 3

2 LITERATURE REVIEW 5

3 SYSTEM ANALYSIS 9

3.1 EXISTING SYSTEM 9

3.1.1 DISADVANTAGES 9

3.2 PROPOSEDSYSTEM 9

3.2.1 QUERIES FROM THE WEB 9

3.2.2. ACCESSING CASE: 9

4 SYSTEM SPECIFICATIONS 10

4.1 HARDWARE SPECIFICATIONS 10

4.2 SOFTWARE SPECIFICATIONS 10

5 SYSTEM IMPLEMENTATION 11

5.1 METHODOLOGY 11

5.2 SUSTAINABILITY 12

5.3ENVIRONMENTAL SUSTAINABILITY 12

5.4ECONOMIC SUSTAINABILITY 13

6 FRAME OF REFERENCE 14
7 INITIAL DESIGN 20

8 REPORT SECTION 21

9 APPENDIX 22

9.1 IMAGECLASSIFICATION 23

9.2 IMAGE PROCESSING 26

9.3 SOURCE CODE 31

10 RESULT 40

11 CONCLUSION 43

12 DISCUSSION 44

REFERENCE 45
 CHAPTER – 1

INTRODUCTION

One of the newest and most researched technologies now a days is deep learning. Deep learning is
a technique used to create intelligent systems as similar as possible to human brains. It has made a big
impact in all types of domains such as video, audio and image processing (Wason, 2018;Sharma, 2019).
On the other hand, agriculture is humanity‟s oldest and most essential act4ity for surv4al. The growth of
population during the last years has led to a higher demand of agricultural products. To meet this demand
without draining the environmental resources the agriculture uses, automation is being introduced into
this field (Mehta, 2016).

The present project aims to merge both concepts by achieving autonomous weed recognition in
agriculture; this goal will be reached by using new technologies such as Matlab, FarmBot and Python
programming, image processing, deep learning and Artificial Neural Networks (ANNs). These concepts
will be explained in more detail throughout this document. This is will be developed for
Naturebruksförvaltningen, a farming school in Töreboda, Sweden.

1.1 BACKGROUND

Robotics and automation have become an emerging subject now a days; substituting and aiding
humans in manual tasks that can become not only tedious and repetit4e, but also difficult due to different
factors such as precision. In order to go in depth on this technology deep learning has been implemented
with the purpose of g4ing these systems intelligence, making them capable of learning. Examples can be
found everywhere, from industries to human kind‟s daily life.

One of these examples is agriculture, where automation has found solution to some of the
challenges faced by farmers on a daily basis such as crop diseases infestations, pesticide control, weed
management, lack of irrigation and drainage facilities and lack of storage management (Jha, et al.,2019).
As a way to bring this new technology to urban orchards, FarmBot Inc. was created. It is a local startup
that is working within advanced precision agriculture through automation and open source technology
(Brown, et al., 2017). FarmBot Inc. has developed a series of robots, called FarmBots, totake care of
these orchards in an autonomous way while respecting the environment.

Naturbruksförvaltningen Sötåsen aims to teach its students how to combine agriculture and
technology. To do so, they intend to introduce a FarmBot into their studies and go a step further, not only
programming it to do the basic agricultural tasks, but also by including deep learning to make the system
capable of differenc in go it own whether there are weed son the orchard or not.
1
1.2 PROBLEM DESCRIPTION

These last years the combination of automation and computer vision has been introduced into
agriculture to reduce human work load. The FarmBot used in this project is one example of that
combination. Its functions range from the automation of basic agricultural act4ities such as watering or
seeding, to more advanced and complex tasks such as differencing between crops and weeds. This weed
detection system is the focus of this project. It is programmed to take pictures of the crop and process
them by a manually act4ated weed-detection software application from FarmBot where the processing is
done based on the colour sand location of the elements of the picture. This weed detector is the starting
point of this these is.

Why does the weed detector have to be improved? Even if this system seems to be fail proof, it is
not. There are three main issues that can be considered: firstly, having to manually act4ate the weed
detector application does not reduce the amount of human labour as much as intended. Secondly, basing
the detection on colours is not accurate due to the possibility of a change of lighting or the similarity of
colours between weed and plants, among other things. Finally, basing the existence of a weed on the
location where the FarmBot has previously planted a seed, does not consider a situation where the
FarmBot does not necessarily know where all the seeds are located. As a way to solve these issues, this
these is will implement a weed detector software based on deep learning which will be explained in
Section 1.3.

1.3 AIM AND OBJECT4ES

The aim of this project is to implement a different type of weed detection system than the explained
in Section 1.2, one that makes use of an ANN to differentiate between crop and weed. Inorder to achieve
this, some object4es need to be set:

1. Image capture using FarmBot

2. Image pre-processing with Matlab

3. ANN training using Matlab

4. ANN testing

5. Use the previous picture store turn weed coordinates

2
 6. Compare ANN performance between the one used by FarmBot and the one used in this project

The Matlab system will implement the pre-processing of images and the training of an ANN in

Order to have a systemable to learn from the already processed images and that processing

autonomously. To take those pictures, the FarmBot will be programmed to capture them every

certain time and forward those images so that Matlab can retrieve them.

1.4 DELIMITATIONS

To establish the boundaries during the development ofthe project, some factors will be taken into
account.The FarmBot programming will not go further than the basic programme needed to take pictures
wherever it is necessary. The ANN to be used will be a previously created network available on
MathWorks community, it will not be created from the beginning in this these is, only the last layers will
be modified, and then the network will be trained. This these is will be developed to differentiate
between crop and weed with Matlab in order to evaluate the differences between Matlab‟s ANN and
Raspberry Pi‟s ANN. The crop and weeds used in this project will be grown in Sötåsen, being the crop
specifically spinach and the weeds dandelions and cleavers.

Some hardware limitations can be found on the development of the project, such as the camera
resolution or the computer characteristics. The camera resolution will limit the quality of the image,
which interferes with the final result of the image processing stage. The computer characteristics will
affect the ANN training speed. These characteristics will determine the accuracy of the project, so the
better they are, the more precise the results will be.

1.5 OVERVIEW

This these is is structured in 11 chapters. Chapter 1 is the introduction to the project and its
background. In Chapter 2, the sustainability of the project can be found. The methodology followed is
explained in Chapter 3. Chapters 4 and 5 are Frame of reference and Literature review, where the
theoretical framework and some similar implementations are analysed. From Chapter 6 to Chapter 8 the
development of the project is explained, Chapter 6 corresponds to the initial design; in Chapter 7 a
research on different image processing techniques is found, and, finally, in Chapter 8 the final
development of the project is done. After this, Chapter 9 shows the results obtained through the these is,
which will be later compared with other technologies in Chapter 10 as the conclusion of the project.
Finally, in Chapter 11 a discussion about the obtained results and their implication in future projects is

3
done. In addition to this, two appendices are included with a detailed explanation of the developed code
in each phase of this is.

4
 CHAPTER – 2

LITERATURE REVIEW

The field this project is based on has been researched many times before; in order to get an
overview of the previously done work, this chapter analyses some of those documents for each part of the
project.

(Marinoudi, et al., 2019). :Agriculture has always been an essential act4ity for surv4al. Over the last
century, and more specific, over the last 15 years, agriculture has started to mechanise and digitise; due to
this evolution and automation, labour flow was almost totally standardised. Nowadays, after introducing
robotics and artificial intelligence into agriculture there is no need of standardization, robots are working
collaborat4ely with humans and learning from them how to realize the basic agriculture tasks such as
weed detection, watering or seeding

(Dankhara, et al., 2019). Weed detection is one of those basic agriculture tasks that are being
automatized and digitised,in this case, because of toxicity related to herbicides; so, reducing human
intervention will make possible a decrease inthe use of herbicides, increasing health care. To achieve this,
robots able to detect plants and classify them into crop or weed are now introduced into agriculture This
implementation has been done in multiples studies such as where Internet of Things (IoT) is applied into
an intelligent robot to differentiate crop and weed remotely; IoT is present in the communication between
a RaspberryPi, wherethe processing is done and the camera and sensors are connected, and the Data
Server, where the Raspberry Pi sends the information obtained. This paper shows an accuracy of 90%-
96% depending on if it is used a Convolutional Neural Network (CNN), a data sheet is being created or it
is being used the training set.

Daman, et al., (2015) and Liang, et al., (2019) both introduce the use of automation into agriculture to
identify weeds, and to do so,they make use of image processing techniques. Daman, etal., (2015)
implement those techniques into an herbicide sprayer robot, capturing images from a Raspberry Pi
camera and extracting pixels‟ colours to process them with d4erse techniques in order to know whether it
is a weed or not. Results were more than successful, after placing plants and weeds randomly, the robot
was tested and weeds were almost totally identified and sprayed, taking the processing stage
approximately 3 seconds. Liang, et al., (2019) implement image processing in drones instead of robots,
that way, they not only detect weeds, but also monitor the growth of crops. By combining image
processing and CNN indrones, they get different accuracies depending on the processing,which is from
98.8% with CNN to 85% using Histograms of Oriented Gradients (HOG).

5
Marzuki Mustafa, et al. (2007), All the previously mentioned processes can be done either in static by
photos or in real-time by videos. have done a research about the implementation of a real-time video
processing. The crop is recorded and processed, offline, using various image processing techniques and a
new developed algorithm that respond correctly to real time conditions. Finally, they achieved an
accuracy over the 80%.

Wafy, et al., (2013), Not only the weed as a plant can be differentiated, more advanced studies such as
differentiate the weeds seeds using Scale-Invariant Feature Transform (SIFT), an algorithm that extracts
the interest points from an image; by using this technique, the minimum accuracy they have is 89.2%.

Olsen, et al., (2015) There is no correct technique to process images in order to obtain the characteristics
needed to identify their elements, and weed detection is not an exception. There are many papers where
different techniques are shown. makes use of segmentation and a rotation variant of HOG in order to
process the images and get the same illumination so they are robust to variations in rotation and scale. By
using these techniques, they got an accuracy of 86.07%.

Samarajeewa (2013) compares two techniques: Local Binary Patterns (LBP) and L*a*b thresholding.
LBP thresholds pixels intensity according to its surroundings; this way, only high intensity values are
visualised, separating plants from the background. L*a*b thresholding selects a threshold value for each
channel in RGB based on histograms. Then, in both techniques, erosion is applied to remove the noise
that can have appeared. This procedure is done with RGB and HSV images; the results obtained show
that LBP has an accuracy of only 48.75% whereas L*a*b thresholding has an accuracy of 89.96%.

Bah, et al., (2017) Another technique usually used is Hough transform; combines the Hough transform
with simple linear iterat4e clustering (SLIC). This method focuses on the detection of crop lines; that
way, what is not located in that line or differs from its neighbours, is supposed to be a weed. Firstly, the
background is segmented and the shadows are eliminated; then the crop line is detected by using some
operations that will end up in obtaining the„skeleton‟of the cropline, from that image,weed can be
differentiated as said before. By following this method, it has been achieved an accuracy of more than
90% and an over-detection inferior to 2%.

Irías Tejeda & Castro Castro (2019) comes up with a generic Matlab algorithm for image processing
of pictures with uniform illumination. The first step is a gray scale conversion with “rgb2gray” and green
pixel subtraction from the converted image,inorder to detect green plants in the images. Then filtering is
done using “medfit2”, which applies a median filter for a neighbourhood of 3x3pixels with the intention
of noise reduction. Image thresholding follows using the Otsu method with the command “graythresh”, in
order to do thresholding segmentation to get the binarized image. Morphological reconstruction comes
6
next, with “imfill” and “bwmorph” to fill the image regions and holes. Next step is labelling and
classification, where connected components are labelled with“bwlabel” and the smaller regions are
removed since they are considered to beweeds. Finally, a threshold based on the classification values of
the area for a crop or a weed is taken for further comparisons.

(Moazzam, etal., 2019).Deep Learning neural networks range from deep neural networks, deep belief
networks, recurrent neural networks and CNNs. The most usually used are CNN, whose layers apply
convolutional filters to the inputs. The networks are rarely created from scratch and most of the ones used
on projects are already existing networks such as LeNet, AlexNet,GoogleNet, SNETor CNET.

Moazzam, et al., (2019) offers a summary of seven different studies, all of the mouse deep learning
convolutional networks approaches forthe weed/crop identification problem, as shown in Table 1. Even if
all the papers mentioned focus on different types of crops, a common element is that most of them only
focus on one crop. Studies using deep learning identification of multiple crops and weeds are not
common.

Starting with Fawakherji, et al., (2019), this study focuses on the classification of sunflower crops and
weeds using pixel-wise segmentation with a CNN. With a training dataset of 500 images, the first step
taken is the pixel-wise classification of soil and vegetation, using UNet semantic segmentation network.
The second step is background removal and extraction of Regions of Interests (ROI) for their later
classification in the third and final step as a crop or weed using a thirteen-layer CNN model. The
accuracy obtained with this method is of a 90%.

Knoll, et al., (2018) and McCool, et al., (2017) both study the usage of image-based CNN for the
detection of carrot crops and weeds. The first paper uses an eleven-layered network to classify three
categories: weed, carrots and background. The network is trained with 500 RGB images taken with a
camera. As for the second paper, it uses GoogleNet pretrained on ImageNet and compresses it creating a
deep CNN which is then trained on an online data set. This method reported an accuracy of 90.5%, mean
while the first paper reported an accuracy of 93%.

Tang, et al., (2017) For soy bean classification uses k-mean classification pre-training prior to the CNN
training. The CNN used consists of a ten layered convolutional network trained with a dataset of
820RGB images to classify between soybean and three different types of weeds.The accuracy of this
process is of a 92.89%. A similar accuracy percentage is found in the classification of maize crops and
weeds,for this, uses approximately 3600 maize and weed images taken by a Raspberry Pi 3 camera, and
performed the testing on four CNNs: LeNet, AlexNet, SNET and CNET. The best accuracy obtained was
with CNET, with a value of 92.08%.

7
Miloto, et al., (2017) focuses on sugar beet and weed classification. With a 94.74% of accuracy, the
training performed on the semantic segmentation-based CNN was done for 48 hours, using nearly 10,000
images. The last paper, Chavan, et al., (2018) is the only one that tries the classification of multiple crops,
creating a hybrid version of AlexNet and VGGNET for weed and crop classification: AgroAVNET,
which is a CNN of f4e layers, trained with 5544 images, with an accuracyof 93.64%.

(Moazzam, et al., 2019). In conclusion, crop and weed detection with the use of deep learning is not yet
a usual topic of research,even if there are more and more attempts.There are still many research gapsnot
considered like the differentiation of different crops and weed combinations. Further more, even some
major essential crops are lacking in this kind of investigation, as there is still a need of creating big data
sets for these crops. Deep learning is still a new a toolfor the autonomous agricultural applications, yet it
seems to be a promising technique and more accurate than other approaches

From these researches the needed knowledge about the necessary pre-processing techniques
that will be used in this project has been acquired; some ofthese are filtering, binarizationand histograms,
a deeper study on them will be done during the development to make sure they suit correctly. Also,
through the study of ANNs, some projects using CNNs have been found, being one of those nets
AlexNet, the one chosen for this project; by this research, a vision on how to work with these nets has
been acquired, as well as the accuracy expected in this kind of projects.

8
 CHAPTER - 3

SYSTEM ANALYSIS

3.1 EXISTING SYSTEM

This project describes one of the most efficient ways for type recognition. It overcomes many of
the drawbacks in the existing solutions to make the weed detection more efficient. It uses natural
language processing to carry out the specified tasks.

This project describes the method to implement a weed detection for desktop using the nural
networks. In this module, the insert photos set are converted to train images by using numpy .and then
classifing the image and it will be check the plants it‟s a weed or not through user interface as output.

3.1.1 DISADVANTAGES

 They propose a new detection scheme that gets two similar results which could cause confusions to
the images is a weed or not in the output.

 Though the efficiency is high of the proposed module, the time consumption for each task to
complete is higher and also the complexity of the algorithms would make it very tough to weak it if
needed in the future.

3.2 PROPOSED SYSTEM

3.2.1 QUERIES FROM THE WEB:

Making queries is an essential part of one‟s life. We have addressed the essential part of a
netizen‟s life by enabling our weed detection to search the image. weed detection supports a plethora of
image train.

3.2.2. ACCESSING CASE:

Being up-to-date in this modern world is very much important. In that way meat a images. And
it‟s one is more explet4e. This application will be assets to a case.

9
 CHAPTER - 4

SYSTEM SPECIFICATIONS

4..1 HARDWARE SPECIFICATIONS

 Processor - Intel Pentium 4

 RAM - 512 MB

 Hardware capacity:80GB

 Monitor type- 15inch colour monitor

 Mouse

 Personal Computer / Laptop

 Nvidia GPU

4.2 SOFTWARE SPECIFICATIONS

 Software :

Jupiter Notebook

 Programming Language :

Python

 Libraries :

OpenCV

Keras

 Dataset :

Kaggle

10
 CHAPTER - 5

SYSTEM IMPLEMENTATION

5.1 METHODOLOGY

As the aim of this project is implement in gaweed detection system, it will be reached by creating a
program able to identify crops and weeds using image processing techniques and deep learning. This
project is not only technical, developing and implementing a program to differentiate crops and weeds
withthe available technologies; but it is also a research project, since it investigates the already existing
knowledge and implementations related to this field of study. AsOates, (2006) says, this type of project
contributes to the creation of knowledge, in this case, by introducing a new technique into the functions
of FarmBot. To develop this kind of project, the methodology explained in this chapter is followed.

This project‟s aim is reached by using deep learning to develop a program capable of identifying
crops and weeds, therefore the strategy followed will be the one denominated as „Design and Creation‟
(Oates, 2006). This strategy focuses on developing new Information Technology (IT) products such as
models,

methodologies, concept or even systems to achieve a contribution to knowledge. In term so this

project, this contribution will be the development of a code from which a computer will autonomously
identify crops and weeds from images taken by a FarmBot.

Following the Design and Creation strategy involves using an iterat4e process involving the
following steps (Figure2),keeping in mind that each step must be finished before moving onto the next
one (Oates, 2006):

 Awareness: recognition of the problem. In the current project, this step englobes the realization of
the Frame of Reference and the Literature Review in order to have a deeper understanding of deep
learning and choosing the most appropriate image processing techniques. This is developed
bydoing a research about this project‟s main concepts, which are robotics and automation, image
processing and deep learning.

Suggestion: design and development of an idea of how the problem could be solved. This will be
addressed as the development ofa proto type ofthe program by considering the techniques chosen in the
previous step. The prototype willbe developed in Matlab,with the functions of downloading the images
taken bythe FarmBot, pre-processing those images and, finally, training and testing the network

11
5.2 SUSTAINABILITY

Nowadays, humanity‟s life style is using up their resources leading to a lack of them in the future.
Sustainable development is the solution to this problem, controlling the usage of actual resources without
compromising the future generation needs (Brundtland, 1987). In order to achieve sustainable
development, as said by Samaan(2020), it is crucial to balance its three pillars economic growth, social
inclusion and environmental protection, as it is shown in Figure 1. This balance tries to be achieved by
accomplishing the 17 goals set by the UN.

Figure1:Sustainable development pillars(Kurry,2011)

As this is focuses mainly on the development of a software that autonomously detects both weeds
and crops, the sustainability will depend on how it is implemented in real life. Nowadays, this
implementation is mainly done with the help of automation. In the case of this study, as mentioned in
Chapter 1, our automaton is a FarmBot.

5.3 ENVIRONMENTAL SUSTAINABILITY

From an environmental point of view, sustainable development aims to balance the actual
consumption of resources and their production rate. When it comes to the usage of energy and CO2
emissions, Nouzil, et al., (2017) states that automation in industry is not environmentally sustainable the
amount of energy used needs to be reduced and so do its emissions, which are approximately the 24% of
the total CO2 emitted. On the other hand, a posit4e aspect of automation is the waste management;
reducing waste by dealing with recycling in industry, including agriculture. Another important aspect is
the reduction of chemicals used, as the precision of a machine surpasses a human worker, the usage of
pesticides is reduced since it will only be used exactly where needed.

12
 As mentioned in Chapter 1, this thesis makes use of a FarmBot, its use is not only helpful for waster
eduction, us in gresources such as water and fertilizers ina more efficient way and pesticides only if
needed, but also with CO2 emissions. According to estimations done by FarmBot Inc.,(2018) the CO2
emitted to produce a FarmBot is between 100 and 150kg, and the yearly CO2 emissions caused by its
use is only of approximately 60kg. Further more, the existing possibility of powering the FarmBot with
solar energy also reduces the CO2 emissions (FarmBot Inc., 2018).

5.4 ECONOMIC SUSTAINABILITY

Economically speaking, sustainability refers to the most optimal use of the existent resources in
order to attain economic growth. Automation has increased gains for the industry by increasing
product4ity and accuracy. The cost of this technology has been reduced since its beginning, but it is still a
high investment for small entrepreneurs. Never the less, economic sustainability is guaranteed.

The economic sustainability of this thesis related to FarmBot can be based on its return on
investment. FarmBot Inc., (2018) estimates a period of three years for its products, comparing the costs
of producing FarmBot grown products against the costs of store bought products.

FarmBot also improves quality of life, by reducing the amount of necessary human supervision
needed on orchards. With this technology no jobs are substituted since it has been developed for personal
use instead of industrial. In this these is, besides FarmBot, social sustainability also focuses on the
reduction of time a person needs to control the weeds; this job will be simplified by the use of a computer
only needing human intervention to take out those detected weeds. In a larger scale if weed elimination
was totally automated this would suppose some job loss, but the workers could be relocated by taking
care of the maintenance and supervision of the robots used to do that elimination.

13
 CHAPTER – 6

FRAME OF REFERENCE

The main concepts of this project are robotics and automation, computer vision and artificial
intelligence. For these last two concepts this thesis will focus on image processing and deep learning.
Hereafter, the theory related to them is explained in detail.

Automation can be defined as “the creation and application of technology to monitor and control
the production and del4ery of products and services” (Gupta & Gupta, 2015). As it is seen, automation is
a versatile technique. Nowadays, it can be found in fields as different as manufacturing, transportation,
utilities, defense, security, and many others. The tasks in which it is involved can vary from installation
and integration of the technology to the marketing and sales of the industry.

Automation and robotics are closely related concepts. Robotics is a science that covers a broad
spectrum of knowledge in order to study and develop systems capable to perform high level tasks, even
resembling as much as possible the performance that a human could have or improving it (Angeles,
2014). Those systems are denominated robots, but what is exactly a robot?Matarić (2007) defines robot
as “an autonomous system which exists in the physical world, can sense its evironment, and cannot on it
to achieve some goals”. Being autonomous means not being directly controlled by a human, therefore a
robot can act on its own as long as it has been programmed to take such decisions. In order to base its
decisions on what is around it, the robot needs to be able to sense its environment this is done by using
sensors, devices capable of capturing disturbances of the environment. The information captured is used
by the robot in order to act on it, following the steps programmed to achieve a pre-established goal.

Nowadays, both robotics and automation are fast-growing fields and are used in a wide range of
applications. Some benefits they have brought to the different industries and sectors are increases in
accuracy, precision and efficiency, product4ity increased due to a faster task execution, quality
improvement and others (Niku, 2020).

There are multiple types of applications where a robot can be used,most of them,as said before,high
level applications resembling human performance. One of those uses is agriculture. Robots introduced to
agriculture can vary their functionality from large fields maintenance to either small back yards. One of
the companies developing these robots is FarmBot, creating systems to carry out the tasks needed to take
care of backyard fields

14
 Figure3:Imaging process overview(Jähne&Haussecker,2000).

A. IMAGEPROCESSING

Image processing is a method to perform operations onan image in orderto improve it for further
analysis. In terms of computer vision,it is called „digital image processing‟ due to the needed of a digital
image to be processed by the computer (Mc Andrew, 2004).

The most common way image processing is performed into a computer vision system is the
following:

i. Image acquisition:the camera and sensors take an image and digitalize it in order to
processit.

ii. Pre-processing:performing some basic processing tasks inorder to have a suitable image to
work with.

iii. Processing: at this point, all the techniques required for the correct modification of the
image are applied.

iv. Representation and description:extracting the most particular features of the objects from
the already processed image, in order to differentiate these objects.

v. Recognition and interpretation:assigning label those objects to completely define them.

As previously mentioned, a digital image is needed in order to be processed. This digital image is
understood as a mathematical matrix where every element or „pixel‟ has its own characteristics
depending on the color, brightness, resolution, etc.; the combination of all the pixels organized in a
certain way will result in a real-world representation (Figure 4) this is usually represented in two ways:
HSV (hue, saturation, value) and RGB (red, green, blue).

15
 Figure4:Pixel representation(ReinaTerol,2019)

By processing each pixel and modifying its characteristics using image processing algorithms,the
image can be improved or even changed completely. McAndrew (2004) d4ides those different algorithms
depending on the tasks they do:

1. Image segmentation:d4ides an image into sub images in order to isolate or identify certain
shapes.

2. Image enhancement:process an image to make it more suitable for anapplication.

3. Image restoration:reverse the damaged one to an image due to a known cause.

There are lots of different algorithms or techniques that canbe used to processan image, here after
the most used techniques are going to be explained (Reina Terol, 2019):

a. Histogram: it shows how many times does a grey level appears in an image (Figure 5).Thanks to
this, it is possible to know if the image is too light, too dark, etc.

(a) (b)

Figure5:Histograms(McAndrew,2004)

b. Filtering: compares a pixel grey levelto its neighbors‟ones, normally to eliminate noise.There
are lotsof different types of noises and so are filters (Figure 6).

16
 (a)Original (b) Gaussian (c) Salt (d)Filtered
&pepper

Figure6:Filters(McAndrew,2004)

The tasks that can be solved using computer vision can be categorized in geometry, positioning,
motion, radiometry, spatial structure and texture, 3D modeling and higher-level tasks such as
segmentation, object identification and classification or recognition and retrieval.

The applications of computer vision are as numerous as the tasks that can be executed. The most
known application and also most developed nowadays is AI; with the aim of training computers for
identification and classification of objects as independently as they could be (Jähne &Haussecker,2000;
SAS, 2019).

c. ArtificialIntelligence: DeepLearning

Artificial Intelligence is an area of computer science that tries to get computers to imitate human-
like intelligent behaviour, such as reasoning, adapting and self-correcting. For a system to be called
artificially intelligent, according to the Turing test, it would have to be able to communicate in anatural
language,to have knowledge and to store it some where, to do reasoning based onthe stored knowledge,
and to be able to learn from its environment (Kok, et al., 2009).

Looking at these requirements, it can be said that one of the most important branches of AI is
machine learning. A system which is in an evolving environment must possess the ability to learn and to
adapt to such changes to be called intelligent,this is done by using ANNs as it will be explained bellow.

In other words, an intelligent system should be able to automatically extract an algorithm for the
execution of a task based on existing accurate data, in order not to replicate this data, but to correctly
predict new cases. That is the aim of machine learning (Ertel, 2017; Alpaydin, 2016).

17
 The way AI and machine learning try to imitate human behaviour is by using ANNs. An ANN is
based on the brain function and its inner communication. It is made up of artificial neurons connected
among themselves and can reinforce or inhibit the act4ation of the neighboring neurons. The ANNs
consist on three basic layers of artificial neurons as shown in Figure 11: An input layer exposed to the
input signals that transmits information to the following layer, the hidden layer. In this layer the
important features are extracted from the information rece4ed, and then transported to the output layer
(Neapolita & Jiang, 2018; Deng & Yu, 2014).
..

..
.

Input layer hidden layer output layer

Figure 11:StructureofanANN

Depending on what kind of task is needed to be executed by the ANN, learning can be done in one
way or another. The two main types of learning methods used in machine learning are the following:
Supervised learning and unsupervised learning (Alpaydin, 2016; Neapolita & Jiang, 2018).

Supervised learning consists on training the system with an existing data set. The training set is
made up of labeled data: inputs with their corresponding output. This kind of method is used with the
goal of learning a mapping function fromthe input to the output, so that g4ennew inputs, the system

can predict the correct output (Shukla, 2017). There are two basic categories in which supervised
learning algorithms can be grouped:

 Classification: When the output variable is a category. For example, classifying emails or spam,
identifying objects in an image, predicting gender based on handwriting.

 Regression: When the output variable is continuous. For example, predicting age,weight or salary.

There are several training methods in which supervised learning can be carried out. One of the most
used due to its fast and easy training is transfer learning. Transfer learning consists on the modification of
the last layers of an existing network with the aim of creating another one capable of working with
different data than the original network.

18
 Unsupervised learning, on the other hand, does not provide predefined labels for the input data,
which means the system will not be g4en any training. The aim of this method is to find hidden
regularities inthe inputs, such as patterns, similarities or differences(Alpaydin, 2016;Bansal, 2017).Itis
mostly used in order to find existing categories in the input data g4en.

Deep learning is a sub-branch of machine learning that uses a specific kind of neural network
architecture. The difference between deep learning networks and others is the number of hidden layers of
the ANN used, since deep learning often uses more hidden layers than the older neural networks (Deng &
Yu, 2014; Neapolita & Jiang, 2018). Some of the advantages it has over other approaches of machine
learning are the large amount of data these neural networks can work on, solving problems directly to end
instead of d4iding them into sub-tasks, and even if it takes a longer time to train these networks, the
testing time is reduced, increasing accuracy. Deep learning is being currently used in applications such as
image and speech recognition, automatic text generation or automatic machine translation, which were
difficult to implement withother types of techniques (Sultana, et al., 2020) .

19
 CHAPTER – 7

INITIAL DESIGN

In this chapter, both an overview of the initial design of the project and the first prototype developed
are explained. The relevant modifications for the design of the final project and the final project itself are
explained in more detail in the following chapters.
Asit has been previously stated,the main focus of this project is the training of a neural network in
order to differentiate crops and weeds. To achieve this, Matlab is used as the principal programming
software forthe image processing and the network training. Other software such as Python and FarmBot
are needed to help with the acquisition of images. To store the pictures, the computer storageis used as
well. The following image (Figure 12) shows a representation of how these software are connected:

 Python is the principal software of the system. It takes images as its input, processes them to
improve their characteristics and finally trains and evaluates an ANN with those images; the ANN
correctly trained is the output of the program. The trained network will be later used to detect crop
and weed from any picture.

 The images used are the ones captured by FarmBot; to make this possible, the first step of the
Matlab programming is the acquisition of those images by executing a Python code.

 Python works as the intermediary between Matlab and FarmBot.Once it is executed, it connects to
FarmBot REST API and requests access to download the pictures taken by the robot. After the
access is granted, the Python code downloads those pictures into the computer where Matlab is
running in order to make them accessible.

 Image acquisition This part is carried out in two programs, the FarmBot Uiand Sublime Text 3.The
first one is used to take the pictures and the second one to download them into the computer. In this
point, anover view of the most important parts of both codes is explained; thefull code explained in
detail can be found in

20
 CHAPTER – 8

REPORT SECTION

21
 CHAPTER - 9

APPENDIX.

The first step is to take pictures of the whole field. This is done with twenty-f4e pictures, f4e for
each column of the f4e rows that the FarmBot workspace consists on. To do so, FarmBot UI has
commands called „Take Photo‟ and „Move To‟, by adding these commands (Figure 13) into a sequence
the robot moves to the specified points and then takes pictures of the crops. Once each picture is taken, it
is automatically sent to its Google Cloud Server, from where all the pictures will be downloaded.

Figure13:Sample of Farm Botcode

After the pictures are stored by FarmBot, it is necessary to access them. By using Sublime Text3 as
programming environment, a Python code has been developed with the aim of downloading and storing
them into the computer so Matlab can later make use of them, as shown in Figure 14.

Figure14:Downloading images code

22
 Previously to Figure 14 code, access has been granted and the images have been requested. In order
to download them, the number of available images is counted and,one by one, their URL and their
coordinates are stored. Finally, a request to download what is addressed on each URL is made, and the
image and coordinates are stored in the computer as files with unique names, as shown in Figure 15.

9.1 IMAGE CLASSIFICATION

Image classification is done in Matlab. The aim of the Matlab code is to train a pre-created network
in order to make it able to perform a classification between spinach and weeds. The full code explained
in detail can be found in Appendix A, while the training and classification processes, the most important
points, are discussed below.

Before starting to work with Matlab, there is one more step to perform. To work with the
downloaded images, Matlab needs to know exactly what is inside of every picture. This has to be done
manually by separating each downloaded picture in folders, named accordingly to the different categories
the net is going to differentiate. Once this is done, Matlab accesses these folders and randomly d4ides the
images in three groups: training, validation and testing, inorder to work with them separately.

The training process is shown in Figure 16.It consists in the modification of AlexNet by changing
its last two layers, and then trains it on the training set of images with some options that will determine
how the network will learn. These steps are followed due to the selection of Transfer Learning as the
network‟s learning method.

Figure16:Network training

With the training options it is possible to visualize the training progress. In the image below two
graphs can be observed, the first one accuracy– iteration and these condone loss – iteration. Here a new
concept is included, the „epochs‟. An epoch is a training cycle with all the images, those images are
d4ided into smaller batches that willbe introduced to the network while training,the number of batches in
an epoch is the number of iterations.

23
 Figure17:Trainingprogress

Figure17 shows the training progress of our network.The training for this prototype consists on 30
epochs with on eiteration each.For each iteration,an ew value is added to the graphs, the accuracy and the
loss. Accuracy is the percentage of correctly classified elements while loss is how badly the network
performs on the training data. Validation can also be seen in this picture, as a black dotted line, at the
beginning and at the end, in order to control that the network keeps improving and stops if it starts getting
worse.

When the network is finally trained, the percentage of accuracy and loss can be seen as the training
results as it is shown in Figure 18. The training done in this prototype has taken about 13 minutes and has
an validation accuracy of 83.33%. This result is good enough to test the network with new images and
get the final accuracy.

Figure18:Training results

The evaluation of the network is the final part of this prototype. By introducing the test images as it
is done in Figure 19, the network is tested and the real accuracy of this classification is obtained. The
final accuracyis of72.41%. Inorder to know where the network fails, a confusion chart is done as it can be
24
seen in Figure 20. From this chart, the number of images correctly and wrongly classified can be
observed. As it is seen, the main problem of this network is the classification of weeds because it
mistakes 6 out of 10 with spinach crops, while the spinach crop is mostly well classified.

Figure19:Classification and evaluation of the network

Figure20:Confusion chart

The perfomance ofthis prototype is good in terms of detection acccuracy as it has a final accuracy
of 72.41%,but it canbe improved. First ofall,to simplify the protoype, two different types of weed have
been labeled as the same one; for the final project both types of weed need to be classified separately.
Another aspect to consider is that, in order to get more precise results, a pre-processing can be done to
the pictures so some ofthe most important features can be emphasized, also adding more images to the

whole process will reflect in better results. Last but not least, image classification implies having
only one kind of plant in each picture,so it is needed to manually crop and separate the weeds and the
spinachs in each picture taken. This can be solved by using „object detection‟ which is done with Regions
with Convolutional Neural Networks (RCNN) instead of a CNN. All the final changes will be taken into
account in the final project,

25
9.2 IMAGE PROCESSING

The main object4e of this these is is training a neural network to perform the differentiation
between certain types of crop and weeds. To do so, the network is trained with images taken using a
FarmBot. In order to have the best accuracy possible, the images are pre-processed so the most
differentiable features are highlighted and the best quality possible is achieved. In this chapter, are search
on some pre-processing methods is done, and the most suitable ones are chosen for the project.

In terms of image quality, not only their processing is important, but also the camera specifications
play a big role. Depending onthe camera, images have different quality. For this project, four different
cameras have been tested in order to use the one where the crops and weeds are more visible. Those
cameras were: FarmBot cameras versions 1.3 (Figure 21) and 1.5 (Figure 22) and „C925e webcam‟ from
Logitech (Figure 23) and „Xiaomi Redmi Note 8 Pro‟ camera from Xiaomi (Figure 24).A comparison of
those cameras specifications and photos is made bellow in Table 2.

The aim of the camera comparison was to choose the one that takes the photos where crops and
weeds are more notice able. By looking at the table above, a picture taken with each camera can be
observed. Figure 21 is a picture taken with FarmBot Genesis camera v1.3, where it is difficult to
differentiate anything because of the brightness. Figure 22 is taken with a different FarmBot version,
v1.5, here the plant is easy to recognize but still it has a lot of brightness making difficult the
differentiation of different plant species. In Figure 23,taken with an external USB camera attached to the
robot, crops are differentiable and less bright, still needing image processing to achieve better quality.
Last but not least, in Figure 24 a picture taken with a Xiaomi Redmi Note 8 Pro mobile phone is shown;
the pictures takenwith this camera will only be used to train the network even though their quality is the
best because the phone cannot be attached to the robot.

This these is is finally done using the pictures taken with Logitech‟s camera attached to the robot
and Xiaomi RedmiNote 8 Pro camera inorder to get a bigger dataset. To get the best accuracy possible
differentiating crops and weeds fromthe picture staken, a pre-processing is done to them. Four different
processing methods, some of them found on this project literature review, are tested with Figure 23 to
find the most optimum pre-processing techniques to implement. Those methods are background removal,
histogram equalization, sharpening and glare removal.

26
 Figure25:Background removal code

The Matlab code for background removal is shown in Figure 25. It takes an image „im‟ and
binarizesit, then it thresholds the saturation and hue of every pixel to extract only the ones close to
greencolour.The final image is conformed only with the pixels that meet the second it ions.An example
of background removal is found in Figures 26, it is easy to observe the difference between the original
picture (Figure 26 (a)) and the processed one (Figure 26 (b)). As it is seen, the performance of this
technique does not work as well as expected in this project because it detects part of the weeds as
background and, sometimes, it does not detect the weed at all.

(a)Originalimage (b) Image without

background

Figure26:Background removal

As said in Section 4.2, a histogram shows how many times a gray level appears in an image. By
using histogram equalization an image is modified to get the optimum usage of its color scale. When a
picture is mostly dark or bright, it is because its histogram is situated either too close tothe left or to the

27
right. Histogram equalization „extends‟the picture‟s histogram inorder to make it possible for the image
to get a higher number of different colour levels.

Figure27:Matlab's command for his to grame qualization

(a)Original image (b)Histogram equalization

applied to the

Original image

Figure28:Histograme qualization

Sharpening is done to improve the content of a blurry picture. It increases the contrast between
different colors of an image, making the edges where these different colors meet standout. This way,the
shapes ofthe figures present inthe image are more defined. InMatlab, this is done with „imsharpen‟, as
shown in Figure 29.

Figure29:Image sharpening inMatlab

This command, „imsharpen‟, takes a colored image and sharpens it, taking into account parameters
such as„Radius‟ and „Amount‟. The „Radius‟ is in control of how much of the region around the edges
should be sharpened, and the „Amount‟controls the contrast increase level, making it softer or stronger
28
depending on the value introduced. Figure 30 shows an example image taken from the pictures captured
by FarmBot (Figure 30 (a)), and the results of applying the sharpening to it with different values of each
parameter Figures 30 (b) and (c).

(a)Original image (b) Sharpenedimage (c) Sharpened image

with radius 1 with radius 2

And amount2 Andamount 2

Figure30:Sharpening

The final processing alternat4e is glare removal which, as the name indicates, aims to reduce the
white bright parts of the pictures caused by over exposure. This is done with a set of commands as shown
below in Figure 31.

Figure31:Glare removal code

29
 The code first loads the image in a variable called „im‟ to then convert the original photo into a
grayscale image (Figure 32 (b)). The next step is creating a binary picture taking only those pixels that
have a value above 225 from the grayscale image, which are the brighter and whiter pixels of the original
image (Figure 32 (c)). This picture is then dilated with „imdilate‟ to fill the small spots near the whiter
area. The three channels of the original RGB image are separated, and each of them is filled with
the„regionfill‟command, using the binary image as a mask. This command fills the regions specified by

The white pixels of the binary image by replacing those pixel values with the neighbouring pixel
values. Finally, the resulting matrices are concatenated with the „cat‟ command in order to recreate the
RGB image. The resulting picture is shown in Figure 32 (d), were it is possible to observe how the white
pixels of the original image (Figure 32 (a)) have now become less bright and are colored As mentioned in
Chapter 1, this thesis makes use of a FarmBot, its use is not only helpful for waster eduction, us in
gresources such as water and fertilizers ina more efficient way and pesticides only if needed, but also
with CO2 emissions. According to estimations done by FarmBot Inc.,(2018) the CO2 emitted to produce
a FarmBot is between 100 and 150kg, and the yearly CO2 emissions caused by its use is only of
approximately 60kg. Further more, the existing possibility of powering the FarmBot with solar energy
also reduces the CO2 emissions (FarmBot Inc., 2018).

according to their surroundings.

(a) Original image (b) Gray scale image

(c) Binary image (d) Resulting image

30
 After studying these four possible ways of processing the images, it was decided that only
sharpening and glare removal would be used. This was done because histogram equalization does not
work well with images with good lighting, and background removal transforms the image too much from
the original picture. Glare removal and sharpening, onthe other hand, are essential for this project since,
dueto camera settings and the surrounding light,the images appear blurry and with over exposure. Figure
33 (b) shows an example of an image that has gone through this processing comparing to the original one
in

(a)Original Image (b) Processed Image

Figure32:Final processing

9.3 SOURCE CODE

1. FARMBOT

To guaranteethe best accuracy possible, the CNN need stobe trained and tested with images taken
from the FarmBot. A code where these images are taken has been developed in the FarmBot UI,an
application fromwhere the user can control the robot remotely. The FarmBot UI is shown in Figure54.

31
 Figure54:FarmBot UI

Not only does the robot have to take the pictures but also moves to the correct place in order to take
photo so fall the crops. There bot‟s work space can be represented with 25 pictures, organised in f4e rows
(in the Y axis) and f4e columns (in the X axis). To do so, different sequences have been programmed:
„Move row‟, „Move X‟, „Sequence‟ and „Move Y and take picture‟. The main code is „Sequence‟. All the
other sequences used in this program are called from here which is why, to run the full code, it is only
necessary to run „Sequence‟.

The first step is shown in Figure 55, it moves the robot to a safe position, which in this case is the
originof the coordinate system {0, 0, 0}. By having this position, it is ensured that the pictures will be
taken in the desired order to later be processed.

Figure55:Move to origin

Once it is in its initial position, it is ready to take pictures.Then ext two commands,which are shown
in Figure 56, make the robot take pictures of a full row. This is made possible by executing the „Move
row‟ sequence until the position of the robot reaches 1.10 meters in the Y axis, which would mean that it

32
has reach the end of the workspace in that axis. After having moved through the whole row, it executes
„Move X‟. This is done four times since the workspace has f4e rows; the final row is done differently and
will be explained later. Both sequences „Move row‟ and „Move X‟ will be explained below.

Figure56:Take pictures of a complete row

„Move row‟ (Figure 57) as its own name says, moves the robot along the rows. This is done by
executing the sequence „Move Y and take picture‟ (Figure 58) four times. It works as follows: once the
robot is in its home position, it takes a picture, waits 5 seconds to ensure the picture was correctly taken
and then moves 0.28 meters in the Y axis in order to take the following picture. It is exactly 0.28meters
because it is the length of Y axis d4ided into the f4e pictures that will the taken there. Once four of those
f4e pictures are taken, the robot‟s Y position will be greater than 1100 so it will exit the IF statement in
Figure 66 and execute „Move X‟.

33
 As said before, only four out f4e pictures in the row are taken in „Move row‟, the last picture is
taken in„MoveX‟sequence as shown in Figure59.This is done because in the sequence above,moving is
the last step but in the fifth picture it won‟t be able to move because it is at the end of the axis; then,the
picture will be taken before changing column instead of row.Therefore, „MoveX‟takes a photo, waits 5
seconds again to ensure it is correctly taken and then moves back to the first column (moving1.1 meters
in the opposite direction of the Y axis) and to the next row (0.385 meters in the X axis, also calculated
taking into account the number of pictures and the length of the axis).

Figure57:'MoveX'sequence

2. PYTHON

The pictures taken by the FarmBot are stored in its Google Cloud Storage; as the access to the
cloud is not g4en, inorder to download them it is necessary to ask for it, and this is done by sending a
request trough the FarmBot API. Todo so, a code in Python language able tosend the request to the API
and download the pictures has been developed.

An API is an interface where the way applications interact and exchange data between them is
specified.It provides the platform, medium and specifications about how the information is structured in
each application in order to make possible the communication among various applications. One type of
structure is REST API, it specifies how data is represented in a format convenient for the client,
including its security, being data exchanged mostly in JSON format. FarmBot uses REST API as its
architecture to control storage, image upload and manipulation and security, and JSON Web Token as
authorization mechanism.

34
 AnAPI token is away to relate an application requesting access to a service to a unique identifier.
The first time an application requests access to your service,some credentials are asked for;if they are
valid, an API token would be generated; that token will be used as a form of authentication, the client
stores that token and sends it with every request so the server verifies it and responds with the data
requested; this process is shown in Figure 61.

Figure61:DataacquisitionusingAPItokens

Figure63:Token generation code

After rece4ing the proper token, the next step is accessing the images stored by FarmBot as
Figure64 shows.. The response to this request is returned as a JSON object that contains d4erse data
related to the images such as the creationdate ,name, URL, and even the camera coordinates where the
pictures were taken,among other data.This information is stored as an ew variable named „images‟.

35
 Figure64:Accessing the images

Having accessed the images, it is now possible to download them to the local storage. The first step
is to determine the number of available images. For every image, its URL and camera coordinates are
stored separately; then the image is downloaded from the URL and saved with a unique name in a
previously specified path, lastly, the same is done with its camera coordinates. The code representing this
process is shown in Figure 65 and Figure 66.

Figure65:Pathdetermination

Figure66:Image download

3. Matlab

An important point of the development of this project is the correct classification between crops
and weeds. A program is developed in Matlab to modify, train, validate and, lastly, test a CNN, with the
goalof it being able to correctly classify the images it is presented. As this is the prototype of the main
program, it is implied that it is not the final version, and as modifications will occur, the final decisions as
well as the changes made are explained accordingly in Chapter 8 of the report.

36
 The first step is to access the images needed. As explained in previous chapters, the images
obtained from the acquisition process are now available in the chosen path. In order to work comfortably
with these pictures, an image Data store is created, as shown in Figure 67. In Matlab, an Image Datastore
object is a variable capable of accessing image files from a specific path stored in a computer, without
importing the data directly into the program. With this variable, it is possible to manage a large collection
of images instead of working with the ind4idual files, without worrying about the memory management.
It is also possible to access properties suchas the labels of each image, which determines whether they are
crops or weeds, based on the folder of their location; this property is fundamental for the training step of
this process. In addition to this, as this type of object is also a valid input for the CNN, it is a good option
for the image management ofthis project.

Figure67:Datastore creation

Once the images have been accessed and the Datastore has been created, it is necessary to
randomly split the images in three lots: one for the training of the CNN, one for its validation and another
for the testing. Each ofthese steps of the programming require a unique set of images to ensure the
network does not memorize the g4en classifications during the training step, and that correctly learn sto
differentiate new images. Figure68 show each new dataset is augmented, modifying some properties
ofthe images such as their size and theircolour. The image dimension mustbe the same as the input
dimensions for the CNN chosen.

AlexNet is the CNN used during this project, and it takes an image input of dimensions 227-by
227-pixels. This convolutional network has been pre-trained on more than a million images and can now
classify into 1000 classes. AlexNet needs to be taught to classify the specific images inthis project with
their correct labels, whichare not included in its known categories. The method chosen, inorder to teach
this network to classify between the crops and weeds of this project, is transfer learning. Transfer
Learning consists on the modification of the layers of a pre-trained network, in this caseAlexNet, and the
training of these layers with a set of known labelled images, the training datastore, with a specific
algorithm.

37
 Figure69:Network modification and training options

As shown in Figure 69, this step starts by loading the pre-trained network into a variable called net.
Examining the layers ofthis network, the ones that need to be modified are the last fully connected layer
and the classification layer. The fully connected layer is where the number of classes is determined;
inAlexNet this layer has 1000 neurons representing the original 1000 categories it was trainedon.In this
project, only two classes are needed: weed and crop. The classification layer is where the output of the
network is g4en. To change the layers accordingly, these modifications are done as shown in lines 17 and
18 of Figure 69.

The train in algorithm is specified in line 19 of Figure 69. The first value of the training options is
„SGDM‟, which stands for Stochastic Gradient Descent with Momentum, it is an algorithm for the
learning approach. Another relevant parameter is the Initial Learn Rate, this value determines the range
in which then network‟sparameters should be modified during the learning. The third and final paramet
introduced in this code as a training option is the validation datastore, it allows the trainingprocess to
verify the progress of the training validating it with a third set of images and labels. These settings are the
training options. In line 20 of Figure 69, the network is trained with these specifications, returning the
trained network and information about accuracy.

The final steps, after the network training, are the image classification and the network‟s
performance evaluation. The command „classify‟ takes the trained network and the test datastore as the
inputs, returning the prediction for each image. The evaluation of the network is then done by obtaining
the percentage of correctly predicted images in line 25 of Figure 70, and lastly, a confusion chart is
drawn with the results obtained from the classification, to visually determine which predictions were
wrong.

38
Figure70:Evaluation

39
 CHAPTER - 10

RESULTS

Once the development of the weed detector is finished, the obtained results are: 54 images with the
corresponding bounding boxes for each plant detected as well as its label, the precision of the weed
detector for each category in this set of images, and a table with the coordinates of the weeds in the
FarmBot bed. In addition to these results, a comparison of the performance of the resulting weed detector
with the original FarmBot weed detector, as well as one of the neural networks shown in the Literature
review (Section 5.3), will be made in this chapter.

The most important result of this thesis is the precision obtained from the network. How well does
it detect each different type of plant it is presented?These results are shown in Figure 51(a),(b) and(c) for
spinach, cleavers and dandelions respect4ely. These percentages come from all the correct and incorrect
guesses the network has made, based on the ground truth table it has been g4en after the detection. At the
beginning of this project a threshold was set to determine what values of accuracy could be considered as
a correctly functioning network. This accuracy was set to be of at least 50%. Compared to this initially
set value, it can be said that the results obtainedare good enough.

(a) Spinach (b) Cleavers (c) Dandelions

accuracy accuracy accuracy
Figure51:Detection accuracy

Another result expected from the weed detector is the visualization of the images with their
respect4e bounding boxes surrounding each plant, and their correct label. These visualizations come from
a filtering done apart from the neural network, to only show those detections with a confidence above
of75%. In Figure 52 some examples ofthis visualization can be found, where itcan be seen that the
resulting images are not too bad. As seen in Figure 52 (a), the number of plants, their position and their
respect4e label is well determined. In other pictures such as Figure52 (b) it is noticeable how the
confidence restriction affectsthe results, asnot allthe present plants are detected.

40
 The last result obtained from the developed weed detector is a table containing the location of all
detected weed plants in the FarmBot bed. This location is stored as X and Y coordinates in millimeters
referred to the home position of the FarmBot: the upper-right corner of the bed.Figure53 shows a part of
the resulting coordinate table from the images used in this evaluation.

(a) (b)

Figure52:Visualization examples

Figure53: Coordinate table

After obtaining these results, the main weed detector system and its performance were supposed to
be compared to the FarmBot weed detector. A summary of this comparison is done in Table 4. The
problem is that the FarmBot system does not implement aneural network for the identification but bases

41
the weed detection solely on the crop localization. Due to this, it can work with multiple type of crops
and unlimited type of weed, since it does not classify the plants according totheir species. It works with
only one picture at a time and can return the coordinates of the weeds of every picture. According to
FarmBot personnel, their weed detector has an accuracy of approximately 95%. Meanwhile the weed
detector developed in this project is basedon deep learning for the crop identification and is capable of
classifying the plants regard less of their location as long as they are spinach, dandelionsor cleavers. It is
able of taking 54 pictures at the same time and is also capable of returning the coordinates of the weeds.

The accuracy obtained is variable for each plant: 78.10% for spinach, 44.76% for dandelions, and
53.12% for cleavers, as previously said.

Then if this these is network accuracy is much lower than FarmBot detection accuracy, why this
thesis is stated as successful? It is because a weed detection system is much more intelligent than
FarmBot‟s built-in detector. This means, the resulting neural network from this thesis detects and
identifies weeds and crops with less information than FarmBot and learns how to do that identification by
its own. FarmBot needs to know where the crops have been planted in advance and its color range,
whereas the developed network just needs to be trained with pictures of what would be detected.
Therefore, the neural network developed in this thesis is a good result.

42
 CHAPTER – 11

CONCLUSION

In conclusion, the project developed in this these is has successfully achieved the principal aimset
in Section 1.3. The principal aim of this these is was the implementation of a system able to identify
crops and weeds using ANNs with images captured by the FarmBot, which will later be compared with
FarmBot built-in weed detector. The accuracy obtained is not of a 100%, but the network differentiates
well enough the different type of plants it has been trained on. Therefore, the main aim of the thesis is
considered as accomplished.

The object4e shave been also achieved. The FarmBot has been used to takes the pictures and then
those pictures have been correctly processed by Matlab, an ANN was trained and tested and then used to
predict the crop and weeds at the pictures and also return the weed coordinates. Finally, a comparison
with other methods was done. Therefore, all the object4es have succeeded.

Through the awareness step of the methodology, enough information was acquired to propose an
initial design of the project. This prototype g4es an initial idea of how to obtain the images with FarmBot
and Python, and an as well as a possible approach to the network training and testing. Once proved
feasible, it was evaluated, and some modifications were proposed. The main changes implemented in the
final version were the addition of image processing, the change from image classification to object
identification by changing the type ofneural network from CNN to RCNN in the training part, and the
addition ofthe coordinate obtention as the final part after the plant identification.

To evaluate the resulting system after the development, the weed detector was tested as it is
supposed to work in a real situation. First the FarmBot programme was tested to obtain a whole set of
pictures from the FarmBot bed. Then the Matlab detection programme was executed, firstly making the
python code execute the image acquisition and then detecting the contents of each image. The results of
this evaluation can be found in Chapter 9 where the performance, accuracy and coordinates obtained are
shown, as well as a comparison of the network with the original FarmBot weed detector and another
similar network found during the research step.

43
 CHAPTER - 12

DISCUSSION

The detection accuracy results achieved in this project are not as high as in other projects with
object identification networks, nevertheless, they are good enough to lead to the implementation of
artificial intelligence in a new field: agricultural robots such as FarmBot. Crop recognition has been used
before in the agriculturalfield, but focusing on FarmBot,theproject developed in this these is entails an
improvement to the existing weed detector system, in spite of the accuracy obtained.

Despite having achieved the object4es set for this thesis, there is still room for improvement. The
main issue that came up during the development was the poor camera characteristics, as the cameras used
were too sensit4e to light.The use of another camera as well as using polarizers over its lens would be
possible improvements to consider. Changing the environment of the picture taking to reduce direct
lighting over the FarmBot bed would also help avoid unwanted glare. This will most certainly improve
the quality of the images, thus improving the network‟s performance.

A. FUTUREWORK

Here, some possible future projects based on this is are proposed. A deeper research on the topic of
this is can be done by considering the creation of the neural network from scratch,instead of using
transfer learning on an existing network. By creating the network architecture there can be more control
over its learning process. Another possible topic to research based on this thesis is the implementation of
a weed detector taking into account a bigger number of crop and weed types, not only spinach, cleavers
and dandelions. Finally, the implementation ofan automatic FarmBot weed removal based on deep
learning detection would be an interesting topic to consider, g4en that the current weed removal system
of FarmBot cannot detect weeds located near a crop.

44
 REFERENCES

Alpaydin,E.,2016.Machine Learning:TheNewAI.Cambridge:TheMITPress.

AlvesVarella, C. A., Marinaldo Gleriani, J. &MedeirosdosSantos, R., 2015. PrecisionAgricultureand Remote

Sensing. In: F. Santos, A. Borém & C. Caldas, eds. Sugarcane: Agricultural Production, Bioenergyand
Ethanol. s.l.:Academic Press, pp. 185-203.

Angeles,J.,2014. FundamentalsofRoboticMechanicalSystems.Fourthed.Switzerland: Springer.

Bah, M. D., Hafiane, A. & Canals, R., 2017. Weeds detection in UAV imagery using SLIC and the hough
transform. 2017 Seventh International Conference on Image Processing Theory, Tools and
Applications(IPTA), pp. 1-6.

Bansal,S.,2017.SupervisedandUnsupervisedlearning. [Online]

Availableat: https://www.geeksforgeeks.org/supervised-unsupervised-learning/[Accessed02032020].

Brown, J. et al., 2017. Polar Coordinate FarmBot Final Project Report, California Polytechnic State
Un4ersity: s.n.

Brundtland, G. H., 1987. Report of the World Comission on Environment and Development: Our Common
Future. Oslo, s.n.

Chavan, R., T. & Nandedkar, A. V., 2018. AgroAVNET for crops and weeds classification: A step
forwardin automatic farming. Computers and Electronics in Agriculture, Issue 154, pp. 361-372.

Córdova-Cruzatty, A. et al., 2017. Precise WeedandMaizeClassificationthroughConvolutionalNeural

Networks, Sangolquí, Ecuador: s.n.

Daman,M.,Aravind,R.&Kariyappa,B.,2015.DesignandDevelopmentofAutomatic WeedDetection and Smart

Herbicide Sprayer Robot. IEEE Recent Advances in Intelligent Computational Systems (RAICS), pp. 257-
261.

Dankhara, F., Patel, K. & Doshi, N., 2019. Analysis of robust weed detection techniques based on the
Internet of Things (IoT). Coimbra, Portugal, Elsevier B.V, pp. 696-701.

De Baerdemaeker, J., 2013. Precission Agriculture Technology and Robotics for Good Agricultural
Practices. IFAC Proceedings Volumes, 46(4), pp. 1-4.

45
Deng, L. & Yu, D., 2014. Deep Learning: Methods and Applications. Foundations and Trends in
SignalProcessing, 7(4), pp. 197-387.

Dumka,A.&Sha,A.,2019.Smartambulancesystemusingconceptofbigdataandinternetofthings.
In:N.Dey,C.Bhatt,A.S.Ashour&S.J.Fong,eds.HealthcareDataAnalticsandManagement. s.l.:Academic Press,
pp. 155-176.

Ertel,W.,2017.IntroductiontoArtificialIntelligence.Seconded.Weingarten, Germany: Springer.

FarmBot Inc., 2018. FarmBot Inc. [Online] Available at: https://farm.bot/pages/series-a[Accessed18 02

2020].

Fawakherji,M.,Youssef,A.B.D.D.P.A.&Nardi,D.,2019.Cropsand Weeds Classification for Precision

Agriculture using context-independent Pixel-Wise Segmentation. s.l., s.n., pp. 146-155.

Gupta,A.&Gupta,P.,2015.ElectricalEnergyEfficientSystemDesignusingAutomationand ArtificialIntelligence
for Classrooms, s.l.: s.n.

IríasTejeda,A.J.&CastroCastro,R.,2019.AlgorithmofWeedDetectioninCropsby ComputationalVision.
Cholula, Mexico, IEEE.

Jähne,B.&Haussecker,H.,2000.Introduction.In:ComputerVisionandApplications.SanDiego: Academic Press,

pp. 1-8.

Jha,K.,Doshi,A.,Patel,P.& Shah,M.,2019. A comprehens4e review on automationin agriculture using

artificial intelligence. Artificial Intelligence in Agriculture, pp. 1-12.

Knoll, F. J. et al., 2018. Improving efficiency oforganic farming byusing a deep learning classification
approach. Computers and Electronics in Agriculture, pp. 347-356.

Kok,J.N.,Boers,E.J.W.,Kosters,W.A.&vanderPutten,P.,2009.ArtificialIntelligence: Definition,trends,
techniques, and cases. s.l.:EOLSS.

Kurry,A.,2011.SustainableDevelopment.[Online]

Available at https://eportfolios.macaulay.cuny.edu/akurry/2011/12/21/sustainabledevelopment/ [Accessed 18

02 2020].

Liang,W.-C.,Yang,Y.-J.&Chao,C.-M.,2019.Low-CostWeedIdentificationSystemUsing Drones.

Candarw,Volume1, pp.260-263.

46
Lu, Y. & Lu, R., 2016. Quality Evaluation of Apples. In: D. Sun, ed. Computer Vision Technology for
FoodQuality Evaluation. s.l.:Academic Press, pp. 273-304.

Marinoudi,V.,Sorensen,C.,Pearson,S.&Bochtis,D.,2019.Roboticsandlabourinagriculture.A context
consideration. Biosystems Engineering, pp. 111-121

Marzuki Mustafa, M., Hussain, A., Hawari Ghazali, K. & Riyadi, S., 2007. Implementation of Image
Processing Technique in Real Time Vision System for Automatic Weeding Strategy. Bangi, Malaysia, IEEE.

Mataric, M. J., 2007. What Is a Robot?. In: The Robotics Primer. Massachusetts:The MIT Press, pp. 1- 6.

McAndrew, A., 2004. An introduction to Digital Processing with Matlab. Notes for scm2511 image
processing.. School of Computer Science and Mathematics, Victoria Un4ersity of technology, 264(1): s.n.

McClelland, C., 2020. What isIoT?- ASimpleExplanationofthe InternetofThings. [Online]

Availableat:https://www.iotforall.com/what-is-iot-simple-explanation/[Accessed14022020].

McCool,C.,Perez,T.&Upcroft,B.,2017.MixturesofLightweightDeepConvolutionalNeuralNetworks: applied
to agricultural robotics. IEEE Robotics and Automation Letters, pp. 1-8.

Mehta,P.,2016.AutomationinAgriculture:AgribottheNextGenerationWeedDetectionand Herbicide Sprayer -

AReview. Journal of Basic and Applied Engineering Research, 3(3), pp. 234-238.

Miloto,A.,Lottes,P.&Stachniss, C.,2017.Real-Time Blob-Wise Sugar BeetsvsWeedsClassification

ForMonitoringFieldsUsingConvolutionalNeuralNetworks.Bonn,Germany, InternationalConference on
Unmanned Aerial Vehicles in Geomatics.

Moazzam, S. I. et al., 2019. A Review of Application of Deep Learning for Weeds and Crops
Classificationin Agriculture. Rawalpindi, Pakistan, s.n., pp. 1-6.

Neapolita,R.E.&Jiang,X.,2018.ArtificialIntelligencewithanintroductiontoMachine Learning. Seconded. s.l.

:CRCPress.

Niku,S.B.,2020.IntroductiontoRobotics:Analysis,Control,Applications.Thirded.California:
Wiley.Nixon,M.&Aguado,A.,2019.FeatureExtractionandImageProcessingforComputerVision.

s.l.:AcademicPress. Nouzil,I., Raza,A.& Pervaiz,S., 2017. Social aspects of automation:Some critical

insights.. Dubai, IOPPublishing.

Oates,J.B.,2006.ResearchingInformationSystemsandComputing. London:SagePublications.

47
Olsen,A.etal.,2015.In SituLeaf ClassificationUsingHistogramsof OrientedGradients.Adelaide, SA,Australia,
s.n.

Pantazi, X. E., Dimitrios, M. & Bochtis, D., 2020. Chapter 1- Sensors in agriculture. In: Intelligent
DataMining and Fusion Systems in Agriculture. s.l.:Academic Press, pp. 1-15.

Pedersen, S. M., Fountas, S., Have, H. & Blackmore, B. S., 2006. Agricultural robots - system analysis and
economic feasibility. Precision Agric, Volume 7, pp. 295-308.

Pramanik, P. K. D., Upadhyaya, B. K. & Pal, T., 2019. Internet of things, smart sensors, and pervas4e
systems: Enabling connected and pervas4e healthcare. In: N.Dey, C. Ghatt, A. S.Ashour &S. J.Fong, eds.
Healthcare Data Analytis and Management. s.l.:Academic Press, pp. 1-58.

Reina Terol, A. J., 2019. Campus Virtual UMA. [Online] Available at:
https://eii.cv.uma.es/course/view.php?id=886[Accessed 02 03 2020].

Samarajeewa, T., 2013. Identification of Lantana Camara Distribution Using Convolutional Neural
Networks, Un4ersity of California Santa Cruz: s.n.

SAS,2019.Computervision:whatis itandwhyitmatters.[Online]

Availableat:https://www.sas.com/en_us/insights/analytics/computer-vision.html[Accessed1502 2020].

Sharma, O., 2019. Deep Challenges Associated with Deep Learning. In: 2019 International Conference
onMachine Learning,BigData,CloudandParallelComputing(COMITCon).Faridabad,India:s.n.,pp. 72-75.

Shukla,S.,2017.RegressionandClassification|SupervisedMachineLearning. [Online]

Available at: https://www.geeksforgeeks.org/regression-classification-supervised-machine-learning/[Accessed

02 03 2020].

Skansi, S., 2018. Introuction to Deep Learning: From logical calculus to artificial intelligence. Zagreb,
Croatia: Springer.

48