A Machine Vision Based Smart Conveyor System

Mengchao Zhang1, Vedang Chauhan2*, Manshan Zhou1

1
Shandong University of Science and Technology, Qingdao, SD 266590 China
2
Western New England University, Springfield, MA 01119 USA
*corresponding author: [email protected]

ABSTRACT

Material flow detection on conveyor based on machine vision is the research topic of this paper. A belt conveyor
system equipped with a camera and micro-controller is used as the test apparatus. The purpose of this experiment is to
obtain the quantity of material on the conveyor belt using machine vision and then develop an intelligent speed
adjustment system for belt conveyor according to the quantity, so as to avoid waste of energy and reduce the wear of the
conveyor. Three image processing algorithms that developed, applied and compared were: 1) Background Subtraction; 2)
Canny edge detection and morphological operations; 3) Particle analysis using. It is observed that all three methods
perform well for material detection on the conveyor belt. However, the particle analysis method resulted in higher
reliability and accuracy with faster processing speed. The research provides new developmental ideas for intelligent
conveyor systems.
Keywords: machine vision, smart belt conveyor, speed regulation, image processing.

1. INTRODUCTION
As one of the most important continuous conveying equipment for bulk materials, a belt conveyor has been
extensively used in mining, metallurgy, chemical industry and other fields. It is developing towards the direction of long
distance, large transportation capacity, high belt speed and intelligence at present [1]. Its power consumption accounts
for a large part of the total in the production process. In the actual production process, especially in the field of coal
mining, the equipment transport capacity of belt conveyor is generally determined according to the peak capacity of
production with certain redundancy coefficients considered. Due to the intermittent nature of production and the non-
uniformity of the belt conveyor transport load, the conveyors usually run at a fixed speed. This results in low power
efficiency of the drive motor and a serious waste of power. For a long belt conveyor system, how to bring down the
energy waste of conveyor and improve the efficiency has gradually become a research hotspot in the industry.
Many scholars have studied and analyzed the energy saving of belt conveyors and pointed out its feasibility. The
influence of speed change of belt conveyor on energy saving effect and its advantages has been studied [2]. Ref. [3]
confirmed that variable speed control of belt conveyors can indeed save energy by means of establishing the energy
model of belt conveyors for optimization. A fuzzy control method has been applied to improve the energy efficiency of
large-scale belt conveying systems [4]. The energy saving is achieved using a speed control system based on the load.
The overall efficiency improved, and the energy consumption was reduced.
At present, three methods are commonly applied in detecting load distribution: 1) Electronic belt scale; 2) The
nuclear scale; 3) Ultrasonic rangefinder. These three methods will be phased out because of its poor accuracy and/or
radiation. With the development of computer processing power and image processing technology, machine vision
solutions are progressively applied in industrial production. Several research papers have been published that focused on
fault detection using machine vision applied to production systems. Machine vision based method for faults detection
and classification was developed and applied to milling machines [5]. Ref. [6] presented a vision-based approach for
fault detection of empty water bottles, and separated bottles with any defects from the mainstream. The results showed
that the good bottle rejection rate was lower than 1% of the total inspections. Ref. [7] researched and developed three
methods of detecting transfer track jams using machine vision for automated assembly ma-chines. Ref. [8] built a
machine vision system with several traditional web cameras and image processing using LabVIEW to verify the
effectiveness of using vision to detect "visual cues" in automated assembly equipment, and the faults like part jams and
feeder jams were easily identified. Researchers have applied advanced machine vision techniques such as template

Thirteenth International Conference on Machine Vision, edited by Wolfgang Osten, Dmitry Nikolaev, Jianhong Zhou
Proc. of SPIE Vol. 11605, 116050C · © 2021 SPIE · CCC code: 0277-786X/21/$21 · doi: 10.1117/12.2586978

Proc. of SPIE Vol. 11605 116050C-1

matching, segmentation, connected component analysis, region growing for object detection and classification from
images and videos [9]-[13].
Some researchers have analyzed the application of machine vision technology for belt conveyors. Ref. [14] proposed
a general machine vision approach for on-line estimation of rock mixture composition and applied it to a very
challenging nickel mineral system with very heterogeneous minerals, similar coloration and rock fragments that can be
dry or wet. The positive results were obtained for dry minerals through a pilot plant conveyor belt application. A kernel-
based method to estimate particle size ranges on a pilot scale conveyor belt was studied [15]. In this work, an edge
detection algorithm, which makes the fines fraction in the coal on the conveyor belt was estimated with the median error
of approximately 24.1%. Ref. [16] developed a color computer vision system to inspect and grade fresh market peaches.
A belt longitudinal tearing inspection system was established based on the virtual instrumentation technology and
machine inspection [17]. A fast image segmentation algorithm to deal belt images on-line for detecting longitudinal rip
and belt deviation was proposed [18]. A new contactless measuring speed system was developed for belt conveyors that
met the measurement accuracy requirements [19]. An improved edge detection algorithm based on canny operator and
morphology processing was proposed and applied for belt position detection [20]. Multispectral visual detection method
for identifying the conveyor belt longitudinal tear and other states of the conveyor belt was proposed and the acceptable
performance was reported [21].
The current research work applied machine vision inspection for detection and classification of the belt conveyor
condition into low, medium and high density based on the amount of material on the conveyor. The output of the
machine vision system signaled a microcontroller that can adjust the speed of the conveyor accordingly. Section Ⅱ
explains the experimental setup, methodology is explained in Section Ⅲ, results are presented and discussed in Section
Ⅳ and the future plan is highlighted in Section Ⅴ.

2. EXPERIMENTAL SETUP
Image data acquisition and processing is carried out on the experimental setup as shown in the Fig.1. The main
elements of the set up are computer, conveyor, camera, lens, lighting, Arduino microcontroller and motor controller.
LEGO bricks are used as the test material on the conveyor belt, and they are fed manually. The Arduino microcontroller
and a photoelectric sensor are used to detect the presence of material on the conveyor belt and signaled a camera to
acquire the images. The acquired images are processed both offline and in real time using MATLAB and LabVIEW
software. The output from the machine vision system is sent to Arduino microcontroller via serial communication. The
microcontroller then adjusts the speed of the conveyor by PWM signal applied to the motor controller.

Figure 1. Experimental setup

The specifications about the equipment used in the experiment is provided in Table 1.

Proc. of SPIE Vol. 11605 116050C-2

 Table 1. Information of the equipment

Name Information

Operating speed: 0-82 ft / min (adjustable); Belt width: 200 mm; Total power: 250W; Overall
Conveyor
dimensions: 59" x 10" x 29.5";
Camera puA1600-60uc - Basler pulse;
Lens Arecont Vision CS-Mount 3.3 to 10.5mm Varifocal Megapixel Lens (Manual Iris)
LUMIENS Brooklyn - LED Lamp; The optical grade lens provides evenly distributed light with no
Light
Hotspots and no flicker.
Arduino Arduino Uno R3 Development Board, based on ATmega328
Sensor MINI-EYE™, PMRP-185, Maximum Sensing range is 8"
Computer CPU: Intel i5-4210M, 2.6GHz; RAM:12GB DDR3; 480G SSD;
Materials LEGO bricks

The amount of material on the conveyor belt was divided into three levels: small amount; medium amount and large
amount. Lego bricks were placed on the conveyor belt manually with a fixed speed conveyor set 40 ft/min. The camera
was set at the resolution of 1280 x 720 with the frame rate of 30 fps. Total 15 videos, 120 seconds each, were recorded
for three different machine conditions: 5 for a small amount of material, 5 for a medium amount of material and 5 for a
large amount of material, as shown in Table 2. All the videos were divided into 5 sets, each with a small amount, a
medium, and a large.
Table 2. Information of acquired video

Details Small amount Medium amount Large amount

Number of the video 5 5 5

Length of each video 120 s 120 s 120 s

Frame rate 30 fps

Resolution 1280x720

3. METHODOLOGY

3.1 Method1: Object detection using Background Subtraction

Background subtraction method is an effective moving object detection algorithm. The method uses the background
parameter model to approximate the pixel value of the background image and compares the current frame with the
background image to achieve the detection of the moving region. The pixel area with a large difference is considered as
the motion area, while the area with a small difference is considered as the background area.
The algorithm steps are as follows:
(1)Get the background image with grayscale processing, 𝐵𝑘
Obtain the current image according to the video sequence with grayscale processing, 𝐹𝑘
Calculate the absolute grayscale image of two frames of image, 𝑑𝑘 (𝑥, 𝑦)

𝑑𝑘 (𝑥, 𝑦) = |𝐹𝑘 (𝑥, 𝑦) − 𝐵𝑘 (𝑥, 𝑦)| (1)

(2)Set the threshold TH and binarize the image to extract the moving target.

Proc. of SPIE Vol. 11605 116050C-3

 0 𝑖𝑓 𝑑𝑘 (𝑥, 𝑦) > 𝑇𝐻
𝑏𝑘 (𝑥, 𝑦) = { (2)
1 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
(3)Apply mathematical morphology filtering the image 𝑏𝑘 (𝑥, 𝑦), and then perform the connected component analysis
of the region. If the area of a connected region is greater than a given threshold, it becomes the target, and that region is
the region of the material on the conveyor.
(4)Calculate the area of the target and the ratio of the area of the material to total area.
(5)Compare the detected material area to preset values and classify the target region into low, medium or high
amount depending on the area. Send the control signal to the microcontroller for the speed control based on the material
amount.

3.2 Method2: Object Detection Using Canny Edge and Morphological Operations
Canny edge detection is a very popular multi-stage edge detection algorithm, which was proposed by John Canny in
1986 [22]. The process of canny edge detection algorithm can be broken down to the following different steps:
(1) Apply Gaussian filter to smooth the image in order to remove the noise.
(2) Find the intensity gradients of the image.
(3) Apply non-maximum suppression to get rid of spurious response to edge detection. It means to find the local
maximum value of the pixel point, and compare the gradient value before and after it along the gradient direction. If the
gradient value is locally maximum, it may be the edge pixel and keep it; Otherwise, it will be suppressed, we could get
candidate edge by this way.
(4) Apply double threshold to determine potential edges. If an edge pixel’s gradient value is higher than the high
threshold value, it is marked as a strong edge pixel; If smaller than the high threshold value and larger than the low
threshold value, it is marked as a weak edge pixel. If smaller than the low threshold value, it will be suppressed. The two
threshold values are empirically determined.
(5) Track edges by hysteresis: Finalize the detection of edges by suppressing all the other edges that are weak and not
connected to strong edges.
(6) Mathematical morphology is used to extract image components that are meaningful for expressing and depicting
the shape of the region from the image, so that subsequent recognition can capture the most essential shape features of
the target object, such as borders and connected areas.
(7) Compare the detected material area to preset values and classify the target region into low, medium or high
amount depending on the area. Send the control signal to the microcontroller for the speed control based on the material
amount.

3.3 Method3: Object Detection Using Particle Analysis

VBAI, machine vision inspection software, fully named Vision Builder for Automated Inspection, is developed by
National Instruments for industrial vision applications.
The steps implemented to perform object detection are as follows:
(1) Image acquisition: A sensor triggers the camera through serial communication to acquire images of the conveyor.
(2) Extraction of "Value" plane from RGB to HSV color space conversion.
RGB color space describes colors in terms of the amount of Red, Green, and Blue planes. HSV color space describes
colors in terms of the Hue, Saturation, and Value. HSV image than RGB image can better reflect the color of the tested
object information, RGB image and HSV image can be converted by the following formula (3, 4, 5):

𝑅 𝐺 𝐵
𝑅′ = 𝐺′ = 𝐵′ = (3)
255 255 255

Proc. of SPIE Vol. 11605 116050C-4

 𝐶𝑚𝑎𝑥 = max(𝑅′ , 𝐺 ′ , 𝐵′ ) 𝐶𝑚𝑎𝑥 = min(𝑅′ , 𝐺 ′ , 𝐵′ ) ∆= 𝐶𝑚𝑎𝑥 − 𝐶𝑚𝑖𝑛 (4)

0° ∆= 0
𝐺 ′ −𝐵′
60° × ( + 0) 𝐶𝑚𝑎𝑥 = 𝑅′ 0 𝐶𝑚𝑎𝑥 = 0
∆
𝐻𝑢𝑒 𝐵′−𝑅 ′ ′ Saturation { ∆ 𝑉𝑎𝑙𝑢𝑒 = 𝐶𝑚𝑎𝑥 (5)
60° × ( + 2) 𝐶𝑚𝑎𝑥 = 𝐺 𝐶𝑚𝑎𝑥 ≠ 0
∆ 𝐶𝑚𝑎𝑥
𝑅 ′ −𝐺 ′
{60° × ( + 4) 𝐶𝑚𝑎𝑥 = 𝐵′
∆

(3)Segmentation using clustering thresholding method.

It is the most common thresholding method, based on minimizing the variances within each of the two clusters. An
optimal threshold is selected that is minimizing the within-class variances of two clusters [23]. Step implemented as
shown in formula (6-12):
M was defined as single channel gray scale classification, M=255;
N1 N2 Sum were defined as the number of pixels in the foreground, background and the total;

𝑁1
(a)Background pixels: 𝜔1 = (6)
𝑆𝑢𝑚

𝑁1 𝑁2
(b)Foreground pixels: 𝜔2 = 1 − 𝜔1 = 1 − = (7)
𝑆𝑢𝑚 𝑆𝑢𝑚

𝜇(𝑡)
(c)Average gray value of background: 𝜇1 = ∑𝑇𝑖=0 𝑖 ∗ Pr(𝑖|𝐶0 ) = ∑𝑇𝑖=0 𝑖 ∗ 𝑃𝑖/ ∑𝑇𝑖=0 𝑃𝑖 = 𝜔1
(8)

𝑃𝑖 𝜇−𝜇(𝑡)
(d)Average gray value prospects: 𝜇2 = ∑𝑀−1 𝑀−1
𝑖=𝑇+1 𝑖 ∗ Pr(𝑖|𝐶1 ) = ∑𝑖=𝑇+1 𝑖 ∗ ∑𝑀−1 = (9)
𝑖=𝑇+1 𝑃𝑖 𝜔2

(e)Cumulative value of gray: 𝜇 = 𝜇1 ∗ 𝜔1 + 𝜇2 ∗ 𝜔2 (10)

(f)Variance between two classes: 𝑔 = 𝜔1 ∗ (𝜇 − 𝜇1 )2 + 𝜔2 ∗ (𝜇 − 𝜇2 )2 (11)

(g)Final simplified formula: 𝑔 = 𝜔1 ∗ 𝜔2 ∗ (𝜇1 − 𝜇2 )2 (12)

(4)Measure intensity: First define the ROI, then calculate average, min and max intensity. Use this information to
classify the conveyor into low, medium and high amount of material.

4. RESULTS AND DISCUSSION

We extracted sample images from each of the three cases randomly, one from small amount, one from medium
amount and one from large amount, the comparative results of three methods are shown in Fig.2.

As shown in Fig.2 (a) (b) (c), these three images are the original images that are randomly selected. Fig.2 (a)
represents a situation where a small amount of material exists on conveyor, Fig.2 (b) represents a medium amount, and
Fig.2 (c) represents a large amount. The results of Fig.2 (a) using Method 1, Method 2, and Method 3 are shown as Fig.2
(d) (h) (k); Fig.2 (e) (i) (m) shows the result of the processing results of Fig.2 (b); Fig.2 (f) (j) (n) shows the results of the
processing results of Fig.2 (c) using these three methods.
The area information has been extracted from processed image and represented as shown in Fig.3. It shows the ratio
of foreground image area to background area and represented as percentage. Fig.3 (a) shows the area of the material as

Proc. of SPIE Vol. 11605 116050C-5

percentage for low material condition. Fig.3 (b) shows the area of the material as percentage for medium material
condition. Fig.3 (c) shows the area of the material as percentage for high material condition for one sample image.
As mentioned earlier, all videos were divided into five sets, each contains a small amount, a medium, and a large
material amount. The first set of videos were processed using three methods, and the area percentage was determined as
shown in Fig.4. Results in Fig. 4 are from all the videos processed using all three methods. Results of Fig. 3 are only
from one sample image processed using each method. It can be concluded that when three algorithms processes the same
video, the variation trend of the results is completely consistent, only a slight difference exits in the value of the result.
The trend in the area can be used to classify material amount into low, medium and high category.

(a) (b) (c)

(d) (e) (f)

(h) (i) (j)

(k) (m) (n)

Figure 2. Comparison of processing results with Method 1, Method 2 and Method 3

Figure 3. Comparison of processing results

Proc. of SPIE Vol. 11605 116050C-6

 Figure 4. Processing result of the first set of videos using 3 methods

Method 1 and Method 2 were good for offline processing due to computational burden, while Method 3 proved to be
the best method for online real time implementation. Hence, Method 3 has been studied in more detail for the further
analysis. The results obtained by Method 3 processing the three set of videos are shown in Fig.5.
Two thresholds were set as the interval setting for conveyor speed regulation, when the area proportion is less than
threshold 1, the load on the conveyor is small and the conveyor should run slowly; When the area proportion is between
threshold 1 and threshold 2, it is considered that the load on the conveyor is in the medium state, and the operating speed
of the conveyor should be appropriately changed; When the area proportion exceeds threshold 2, it is considered that the
load on the conveyor is in a large amount and the conveyor should further improve its speed until it runs at full speed to
prevent the accumulation of materials. The area information has been continuously fed back to the microcontroller that
controls the speed using the PWM signal supplied to the motor controller.

5. CONCLUSION AND FUTURE WORK

Automatic speed regulating system based on the amount of material on the conveyor is an important means to
conserve energy from driving the belt conveyor and hence it is a key part of the development of smart conveyor systems.

Figure 5. Processing result of three sets of videos using method 3

Detecting the load on the conveyor belt is the basis of the automatic speed regulating system. It shows that machine
vision technology can be used to detect the distribution of materials on the conveyor belt precisely with its advanced
features such as simple structure, non-contact measurement and a high-speed processing capacity.
At the same time, the technology will be further applied to the adjustment of the starting mode of the conveyor,
especially in the conveying system composed of multiple conveyors. When the conveyor system consisting of multiple
conveyors starts, it starts against the direction of the flow, that is, the conveyor at the last stage starts first, otherwise it

Proc. of SPIE Vol. 11605 116050C-7

will cause blockage of material accumulation. Then, before the stream of material comes to the last conveyor, it has been
running in the state of no load for a long time. That results in waste of energy and unnecessary wear of the equipment. In
the future, machine vision technology can be combined with a soft start technology to realize start along the flow
direction, reduce no-load running time and help in energy savings and reduce equipment wear.
Future work includes optimization of the methods and customizing them as required by actual coal mine environment
that is dark and dusty. Research will be conducted on how the camera is mounted on a large conveyor and its location,
how to obtain a clear and stable image of the conveyor belt at high speed in low light and noisy environment. After
completing the laboratory tests, the solution will be applied to the actual coal mine to realize the transformation of
scientific and technological achievements. The future work will also include the development of machine learning [24]
and deep learning methods [25] to automate feature extraction for classification of the conveyor load.

REFERENCES

[1] Zhang, M.; Shi, H.; Yu, Y.; Zhou, M. A Computer Vision Based Conveyor Deviation Detection System. Appl. Sci.
2020, 10, 2402.
[2] Kohler U, Sykulla M, Wuschek V. Variable-speed belt conveyors gaining in importance[J]. SURFACE MINING
BRAUNKOHLE AND OTHER MINERALS, 2001, 53: 65-72.
[3] S. Zhang, Y. Tang, Optimal control of operation efficiency of belt conveyor, 2011 Asia-Pacific Power and Energy
Engineering Conference, IEEE, 2011, pp. 1-4.
[4] Pang Y, Lodewijks G, Schott D L. Fuzzy controlled energy saving solution for large-scale belt conveying
systems[C]//Applied Mechanics and Materials. Trans Tech Publications, 2013, 260: 59-64.
[5] Konrad H. Fault detection in milling, using parameter estimation and classification methods[J]. Control Engineering
Practice, 1996, 4(11): 1573-1578.
[6] Shafait F, Imran S M, Klette-Matzat S. Fault detection and localization in empty water bottles through machine
vision[C]//E-tech 2004. IEEE, 2004: 30-34.
[7] Chauhan V, Surgenor B. A comparative study of machine vision-based methods for fault detection in an automated
assembly machine[J]. Procedia Manufacturing, 2015, 1: 416-428.
[8] Szkilnyk G, Hughes K, Surgenor B. Vision based fault detection of automated assembly equipment[C]//ASME 2011
International Design Engineering Technical Conferences and Computers and Information in Engineering
Conference. American Society of Mechanical Engineers, 2011: 691-697.
[9] Jarimopas B, Jaisin N. An experimental machine vision system for sorting sweet tamarind[J]. Journal of food
engineering, 2008, 89(3): 291-297.
[10] BATO, P. M., NAGATA, M., Cao, Q., HIYOSHI, K., & KITAHARA, T. Study on sorting system for strawberry
using machine vision (Part 2)[J]. Journal of the Japanese Society of Agricultural Machinery, 2000, 62(2): 101-110.
[11] Baigvand, M., Banakar, A., Minaei, S., Khodaei, J., & Behroozi-Khazaei, N. Machine vision system for grading of
dried figs[J]. Computers and Electronics in Agriculture, 2015, 119: 158-165.
[12] Arjenaki O O, Moghaddam P A, Motlagh A M. Online tomato sorting based on shape, maturity, size, and surface
defects using machine vision[J]. Turkish Journal of Agriculture and Forestry, 2013, 37(1): 62-68.
[13] Birla R. and Singh A. An enfficient method for quality analysis of rice using machine vision system. Journal of
Advances in Information Technology (JAIT), 2015, 6(3).
[14] Tessier J, Duchesne C, Bartolacci G. A machine vision approach to on-line estimation of run-of-mine ore
composition on conveyor belts[J]. Minerals Engineering, 2007, 20(12): 1129-1144.
[15] Aldrich, C., Jemwa, G. T., Van Dyk, J. C., Keyser, M. J., & Van Heerden, J. H. P. Online analysis of coal on a
conveyor belt by use of machine vision and kernel methods[J]. International Journal of Coal Preparation and
Utilization, 2010, 30(6): 331-348.
[16] Miller B K, Delwiche M J. A color vision system for peach grading[J]. Transactions of the ASAE, 1989, 32(4):
1484-1490.
[17] QI, J. Y., YAN, H. F., TAN, C., & ZHANG, Y. J.Machine vision inspection system of belt tearing base on
LabVIEW [J]. Coal Engineering, 2009, 9.

Proc. of SPIE Vol. 11605 116050C-8

[18] Yang, Y., Miao, C., Li, X., & Mei, X. On-line conveyor belts inspection based on machine vision[J]. Optik-
International Journal for Light and Electron Optics, 2014, 125(19): 5803-5807.
[19] Gao, Y., Qiao, T., Zhang, H., Yang, Y., Pang, Y., & Wei, H.A contactless measuring speed system of belt conveyor
based on machine vision and machine learning[J]. Measurement, 2019, 139: 127-133.
[20] Wang J, Liu Q, Dai M. Belt vision localization algorithm based on machine vision and belt conveyor deviation
detection[C]//2019 34rd Youth Academic Annual Conference of Chinese Association of Automation (YAC). IEEE,
2019: 269-273.
[21] Hou, C., Qiao, T., Zhang, H., Pang, Y., & Xiong, X. Multispectral visual detection method for conveyor belt
longitudinal tear[J]. Measurement, 2019, 143: 246-257.
[22] Wikipedia contributors, 'Canny edge detector', Wikipedia, The Free
Encyclopedia,<https://en.wikipedia.org/w/index.php?title=Canny_edge_detector&oldid=922904879> .
[23] Şahan Ş, Değirmenci A, Çankaya İ. A Study on Clustering Based Image Thresholding Techniques with MATLAB
GUI[J]. International Journal of Engineering Science, 2016, 2152.
[24] Joshi K. D., Chauhan V., Surgenor B. A flexible machine vision system for small part inspection based on a hybrid
SVM/ANN approach. Journal of Intelligent Manufacturing, 2020, 31 (1), 103-125.
[25] Chauhan V., Joshi K. D. and Surgenor B. Image Classification Using Deep Neural Networks: Transfer Learning and
the Handling of Unknown Images. In International Conference on Engineering Applications of Neural Networks
Springer Cham, 2019, 274-285.

Proc. of SPIE Vol. 11605 116050C-9