Proceedings of the 7th
World Congress on Intelligent Control and Automation
June 25 - 27, 2008, Chongqing, China
A New Surface Inspection Method of
TWBS Based on Active Laser-triangulation*
Lei Zhang and Mingyang Zhao Yuanyuan Zou and Shiyi Gao
Shenyang Institute of Automation, Shenyang Institute of Automation,
Chinese Academy of Science Chinese Academy of Science
Graduate School of the Chinese Academy of Sciences Graduate School of the Chinese Academy of Sciences
Shenyang, Liaoning Province, China Shenyang, Liaoning Province, China
{zhanglei1 & myzhao}@sia.cn {zouyy & gshiyi}@sia.cn
Abstract - An vision inspection sensor system for the weld lines, as on door inner and body side aperture panels,
automation of laser welding processes in heavy industries is steels that are more difficult to weld, and non-linear weld
presented. The system consists of PC based vision camera and seams. Quality has become a major issue in modern industry.
stripe type laser diode. For image de-noising, a new surface This trend has forced the manufacturing industry to put an
inspection method of Tailor-welded blanks including in: median
emphasis on quality control to meet more and more
filtering, bi-level thresholding, opening and closing, mid-line of
the laser stripe extraction, has been developed. After extracting demanding quality standards. Product material integrity and
the middle line of the laser stripe, the algorithms of calculating surface appearance are important attributes that will affect
bead width, mismatch, weld slope, convexity, concavity, product operation, reliability and customer confidence.
overthickness and undercut are proposed. Finally, A series of Extensive research and development work has been
experiments show that the proposed method in this paper is undertaken in various aspects of automated inspection,
effective. including component geometric and dimensional checking,
presence verification, and functional testing [2,3]. The laser
Index Terms - image processing. surface inspection. weld welding is faced with the same challenge. Automation of the
defect detection. Laser-triangulation.
welding process has helped achieve more consistent quality
and better productivity, but up until now, the inspection of
I. INTRODUCTION
welds and parts has remained a tedious task often leading to
Tailor-welded blanks (TWBs) offer many advantages to inconsistent results. To cope with this increasing problem,
the automotive industry, including the consolidation of parts many techniques of seam tracking and joint or weld
and the reduction in material wastage and weight. They are recognition based on vision sensors have been suggested and
comprised of sheet metals with either different mechanical developed [4-7].However, most techniques mainly focus on
properties or gauge thicknesses, which are butt-welded the seaming tracking and weld pool detection. Few study
together prior to forming. In this way, the mechanical about a surface inspection of weld defects based on vision are
properties of the part can be locally tailored to the application. reported in detail.
Historically, the use of TWBs was confined to only a few
selected automobiles manufacturers such as Volvo, Toyota,
Nissan, and luxury North American vehicles such as Cadillac.
Today, Mercedes, Volkswagen, and BMW have tailor-welded
blanks door inners with multiple welds. GM, DaimlerChrysler,
and Toyota are leaders in body side aperture tailor-welded
blanks. There are a number of factors behind rapid increase in
TWB usage, the predominant ones being the consolidation of
parts, the increased manufacturing tolerances, and the
lowering of vehicle emissions through reduced vehicle weight
[1]. Shenyang Institute of Automation in Chinese Academy of
Science has developed a new automatic laser welding line for
tailor-welded blanks in 2006 and its layout as shown in Fig.1.
A 4kw YAG laser is used on the line and the maximal welding
speed is 10 meters per minute.
Fig.1 The layout of TWB system
The dominant welding technology was confirmed to be
shifting more toward laser from resistance mash seam, with a This paper deals with a new surface inspection method of
TWBS based on active laser-triangulation. Seven laser
significant focus on YAG laser because of its overall operating
welding defects, bead width, mismatch, weld slope, convexity,
robustness and flexibility for non-linear requirements.
Applications are also becoming more complex with longer
978-1-4244-2114-5/08/$25.00 © 2008 IEEE. 1174
concavity, overthickness and undercut can been detected. includes a Complementary Metal-Oxide-Semiconductor
Experiment results illustrated the method is feasible. (CMOS) camera, lens, a filter and a laser projector. The laser
The organization of the paper is as follows: In section II, projector emits a laser stripe on the weld. On the lateral side,
a welding surface inspection system used on the automatic there located a CMOS camera, which detects the laser stripe.
laser welding line is introduced in detail, include in its The control unit consists of a DSP board and a control board.
structure, basic principle of this system and its Schematic The control board is controls the working state of the
diagram. In section III, the algorithm structure of image mechanic apparatus. The DSP board is an image process board
processing about one profile of laser stripe is proposed, which processes image data from CMOS camera. The control
include in: median filtering, bi-level thresholding, opening and unit extracts the weld bead profile, computes it's geometric
closing, mid-line of the laser stripe extraction. In section IV, features, sends the measured values to a powerful PC, which
the algorithms of calculating bead width, mismatch, weld analyses the results and outputs the inspection result. The
slope, convexity, concavity, overthickness and undercut are mechanic apparatus has an arm on which the optic apparatus is
proposed. Section V and Section VI summarize experimental fixed. The arm is driven by an electrical motor to move along
Results and discusses conclusion respectively. the weld. This can acquire the range data of the whole weld
(length: about 1.6 m).
II. THE WELDING SURFACE INSPECTION SYSTEM BASED ON
ACTIVE LASER-TRIANGULATION
Vision sensors are one of the most powerful forms of
non-contact sensory feedback for monitoring and control of
processes in laser welding. With the rapid growth of the
computer vision and image process, more and more accurate
information can be obtained from the weld pool by a visual
sensor. The inspection system used on the automatic laser
welding line for tailor-welded blanks are based on active laser-
triangulation. The typical principle [8] is shown in Fig.2.
This system consists of four parts: optic apparatus, two
circuit boards, mechanic apparatus and a computer that
controls the whole system (see figure 3). The optic apparatus
Fig 2 The principle of active triangulation
DSP
CMOS
Board
Arm Computer
Laser Control
Board
Control
0echanic $pparatus Unit
Fig.3 The surface inspection system based on vision
gray image is obtained by use of the CMOS camera to capture
it. Its size is 576 × 768 pixels. Fig.4 is the original image of
the weld captured by the CMOS camera. It is shown that
III THE ALGORITHM STRUCTURE OF IMAGE PROCESSING structure light image is greatly affected by strong arc light,
ABOUT ONE PROFILE
smog and splash in the process of welding. Not only the
A 50mw, red diode laser emitter projects visible laser structure light image of welding seam is rough, but also its
radiation (650nm) on the welding seam to form a stripe. The background is noisy. These give rise to difficulty, error and
1175
even failure of the processing of the welding seam image. arithmetic mean of the two values at the centre Where the
Intelligent recognition algorithm, discussed in [9] and [10], size of the neighborhood is 5 × 5.
effectively eliminates these effects, such as using fuzzy 2) Binarization: Binarization is to convert a common
algorithm to recognize weld seam, effectively avoiding the image into a binary image so that objects can be conveniently
noise effect. Besides above algorithms, it is an effective way separated from the background. During the process it is
for the prime gray image to pre-process. The algorithm necessary to set a threshold. In terms of the problem that
structure of image processing about one profile includes resulted in threshold value cannot be well convergent to
image processing, the mid-line of the laser stripe extraction overall optimum when traditional OTSU algorithm[12] cannot
and features extraction. Fig.5 shows the proposed algorithm deal with images with incontinuous gray scale,a modified
structure of image processing. OTSU method is proposed based on the characteristics of a
. series of laser stripe image. Firstly, the gray threshold of the
OTSU method, T, is counted. Secondly, the bi-level
threshold ,R, is given by the equation (2)
R=T −ΔT (2)
where
ΔT = 0.2
The pixels whose grey value are greater than R are
defined as the stripe pixels. ΔT is an amendment value.
3) 2SHQing and Closing: 2pening and closing are two
important morphological operations. The opening of image A
by structuring element B is defined as
Fig.4 The original image of the weld
ADB=(AΘB) ⊕B (3)
Similarly, The closing of image A by structuring element
B is defined as
A• B = (A⊕B)ΘB (4)
Where
B is a 4 × 4 square.
B. The Mid-line of the Laser Stripe Extraction
As only the position and not the intensity of the light
stripe is of interest this position is calculated in the profile
extraction unit resulting in a height profile vector that is the
basis for further computing. Accuracy and reliability are
significantly can be seen as the mid-line of the laser stripe. A
lot of algorithms such as row based maximum search [13],
Fig.5 The algorithm structure of image processing
threshold methods [14], centre of gravity, correlation
techniques [15], usage of the first derivative and many more
A. Image Processing have been suggested for this task. To thin the binary image of
1) Median filtering: The median filtering is the best laser stripe, the average location between the left edge and the
known order-statistics filter. It is necessary because of the right one, which is detected from the binary image, is
strong noise in molten pool images. The main advantages of regarded as the middle line of the laser stripe. The equation (5)
median filtering are that it is simple in calculating that result gives the principle of row based profile extraction that
in higher efficiency, and it has good performances in filtering transforms the CCD-camera frame matrix into the height
white noise and long trail noise. The median filter can also profile vector.
keep the detail information such as edge pieces and sharp
angles. In median filtering the input pixel is replaced by the
median of the pixels contained in the neighborhood[11]. yli + yri
Ci = (5)
Symbolically this can be represented as: 2
v( m, n) = median{ y( m − k , n − l ),( k , l ) ∈ W } . () where, Ci is the y coordinate of the ith point on the" mid-
Where W is suitably chosen neighborhood. The line " and yli, yri are respectively the y coordinate of the ith
algorithm for median filtering requires arranging the pixel left , the ith right edge points of the white stripe in the bimodal
gray values in the neighborhood in increasing or decreasing image.
order and picking up the value at the center of the array.
Generally the size of the neighborhood is chosen as odd
number so that a well-defined centre value exists. If, however,
the size of the neighborhood is even the median is taken as the
1176
IV. FEATURES EXTRACTION extraction.
In this study, in order to calculate the mismatch size and
the bead width of the weld for the butt joint, The method in
the literature [16] are used
A. Weld Slope Detection
Set the initial and end points' coordinate: A(xleft,yleft) ,
B(xright, yright) shown in the Fig.7. So the weld slope is given: a. The result of median filtering b. The binary image
§ yleft − yright ·
θ = arctan ¨ ¸ × 180 / π (6)
¨ xleft − xright ¸
© ¹
B. Convexity and ConcavityDetection
ª yleft 1º ª − xleft º ª s1 º
Let A = «
y 1 » , B = «− x » , S = « » .
¬ right ¼ ¬ right ¼ ¬ s2 ¼
x +s × y +s
di = sgn(s1 ) i 1 i 2 (7)
1 + s22
c. The opening image d. The closing image
where Fig 6 The experimental results
x,y
i
are respectively x and y coordinates of the ith point
i
440
on the profile between A and B.
The convexity and the concavity are respectively zero or 430
the positive maximum and the negative minimum.
420
C. Overthickness and Undercut Detection
Do search near the point A ( B) at the left (right) 410
A
direction until the distance between the point on the profile
line and the point A ( B) along the perpendicular direction. 400
The overthickness is given :
390
max( yi − yleft ), ( yleft > yright , yi − yleft > 0)
°
overthickness = ® max( yi − yright ),( yleft < yright , yi − yrightt > 0) (8) 380
° 0, ( yi − yleft < 0, or , yi − yleft > 0) B
¯
370
where yi is the y coordinate of the point near the point
A(B) on the profile.
360
Similarly, the undercut is given: 0 100 200 300 400 500 600 700 800
max(max( yleft − yi ), max( yright − yi )) Fig 7 The profile of the laser stripe
°
undercut = ® , ( yleft > yi , or , yright > yi ) (9)
° 0, ( yi > yleft , and , yi > yright ) TABLE I
¯
THE RESULTS OF FEATURE EXTRACTION
where yi is the y coordinate of the point near the point Feature Result
A(B) on the profile. Bead Width 0.7380 (mm)
Mismatch 0.0534 (mm)
V. EXPERIMENTAL RESULTS Overthickness 0.0586 (mm)
Undercut 0.0230 (mm)
Experiments are performed using the galvanized steel
Weld Slope 37.7414o
plain sheets. The thickness of one plain sheets is 0.7mm,and Convexity 0.00015 (mm)
the other is 1.6mm.The Trumpf ND: YAG laser's operating Concavity 0.00029 (mm)
power is 1kw. The welding speed is 8 m per minute. The A weld seam about 20 cm on an plain sheet (shown in
defocusing amount is positive 2 mm. Fig 8) with different thickness(0.7mmand1.6mm) welded in
Image processing and mid-line of laser stripe extraction the automatic laser welding line for tailor-welded blanks is
experiments are conducted using the algorithm described inspected. Fig 9 shows an inspection result of convexity,
above. The results are shown in Fig. 6. In detail, Fig.6 (a) is concavity and Weld slope defect about the TWBs used the
the result of median filtering, Fig.6 (b) shows the binary method proposed in this paper. Fig 10 shows an inspection
image, Fig.6 (c) is The opening image , and Fig.6 (d) is the result of Mismatch and Bead width defect. Fig 11 shows an
closing image. Fig.7 is shown the profile of one laser stripe. inspection result of overthickness and undercut defect.
After calibrating the relationship between image coordinate
system and world coordinate system, the real value of
mismatch and bead width in the world coordinate system can
be attained according to the value of them in the image
coordinate system Table 1 shows the results of feature
1177
0.2
mm
0.1
Mismatch
-0.1
-0.2
0 2 4 6 8 10 12 14 16 18 20
cm
1.4
mm
1.2
Bead width
1
Fig 8 a weld seam with different thickness
-4 0.8
x 10
Convexity mm
4
0 2 4 6 8 10 12 14 16 18 20
2 cm
0
Fig.11 The result of Overthickness and Undercut
0 2 4 6 8 10 12 14 16 18 20
cm
-4 VI. CONCLUSIONS
x 10
Concavity mm
-2
-4
This paper has described a new type of weld inspection
system which is based on the active triangulation principle.
-6
0 2 4 6 8 10 12 14 16 18 20 The following conclusions can be drawn from this research
cm 1) The inspection system and its principle used on the
60
automatic laser welding line for tailor-welded blanks are
Weld slope mm
50 introduced.
40 2) Based on active laser-triangulation and computer
vision, the proposed algorithm of image process about one
30
0 2 4 6 8 10 12 14 16 18 20 profile can extract weld seam information and calculate
cm
feature date.
Fig 9 The result of convexity ,concavity and Weld slope 3) The problem of weld slope, convexity, concavity,
overthickness and undercut size inspection can be solved
0.2 using the proposed method. A series of experiments show that
the proposed method in this paper is effective. Surface
mm
0.1
inspection of laser weld can be realized.
0 4) How to make the surface inspection method more
Mismatch
-0.1
efficient and more robust will be studied in the further works.
-0.2
0 2 4 6 8 10 12 14 16 18 20
cm REFERENCES
1.4 [1] Christopher James Bayley, “Incorporating external and internal weld-line
defects into ductility predictions for AA5812 and AA5574 Tailor-welded
mm
1.2
blanks,” Queen's University, Kingston, Ontario, Canada, $ pril 200.
[2] Wu, W., Wang, M. J. and Liu, C. (1996) Automated inspection of printed
Bead width
1
circuit boards through machine vision. Computers in Industry, 28 (2),
0.8 103-1 11.
[3] Newman, T. S. and Jain, A. K. (1995) A survey of automated visual
0 2 4 6 8 10 12 14 16 18 20
inspection. Computer Vision and Image Understanding, 61 (2), 23 1-262
cm [4] Chen S B, Zhao D B, Wu L, Lou Y J, "Intelligent methodology for
sensing, modeling and control of pulsed GEAW part1-butt joint
Fig 10 The result of Mismatch and Bead width welding." Welding J, vol.79,no.6, pp.164–173. 2000.
[5] Jeng-Ywan Jeng, Tzuoh-Fei Mau, Shyeu-Ming Leu, “Computer
integrated vision and neural network technologies for intelligent
automatic laser butt joint welding,” Int. J. Computer integrated
manufacturing, 2000,vol.13,(4):345-357.
[6] Theodore P, Pachids, John N, “Vision-based path generation method for
a robot-based arc welding system,” J Intell Robot Syst,2007, 48:307-331.
[7] Zhang Lei, Zhao Mingyang ,Zou Yuanyuan. Zhao Lihua .A creative
design of robotic visual tracking system in tailed welded blanks based on
TRIZ[J]. China Welding. 2006, Vol.15 (4): 23-25.
[8] Jean-Paul Boillot, Jeffrey Noruk, “Benefits of Vision in Robotic Arc
Welding,” J. Name Stand. Abbrev., in press.
1178
http://www.roboticsonline.com/public/articles/details.cfm?id=749
[9] H. C. Kuo, L.J. Wu, “An image tracking system for welded seams using
fuzzy logic,” Journal of Materials Processing Technology, vol.120, no.2,
pp.169-185, 2002.
[10]J. S. Kim, Y. T. Son, H. S. Cho, “Robust method for vision-based seam
tracking in robotic arc welding,” in Proceedings of fhe 10th IEEE
International Symposium on Intelligent Control, (Monterey, USA),
pp.363- 368,1995.
[11]Rafael C. Gonzalez, Richard E. Woods, "Digital Image Processing,"
Publishing house of Electronics Industry. pp:519-566 ,2004.
[12]FU Zhong Liang Threshold value selection method 0f image [J].
Computer applications,200020(5):37-39
[13]Wu, J., Smith, J.S., Lucas, J., "Weld bead placement system for multipass
welding". IEE Proc. - Science, Measurement and Technology, Vol. 143
Iss. 2, March 1996.
[14]Minge, E., Neue Sensoren und Aktoren in der Laser bear beitung.
Innovationen in der Mikrosystemtechnik, Band 16, 1994.
[15]Kim, J.S., Son Y.T., Cho H.S., Koh, K.I., "A Robust Visual Seam
Tracking System for Robotic Arc Welding," Mechantronics Vol. 6, No.
2, 1996.
[16]Lei Zhang, Mingyang Zhao, Lihua Zhao. Vision-Based Profile
Generation Method of TWB for a New Automatic Laser Welding Line
Proceedings of the IEEE International Conference on Automation and
Logistics August 18 - 21, 2007, Jinan, China pp:1658-1663
1179
