Research on 3D virtual vision matching based on interactive color segmentation

Image segmentation model based on improved 3DUNet

The 3DUNet is a renowned semantic segmentation network that serves as an extension of the UNet architecture. While its fundamental principle is akin to UNet, it distinguishes itself by utilizing 3D convolutions instead of 2D convolutions. This network has found widespread application in image segmentation across various domains (Yu et al., 2022).

Figure 1 illustrates the network structure, featuring convolution templates of sizes 64, 128, 256, 512, and 1,024. The feature map undergoes downsampling via 2 × 2 maximum pooling and applying the ReLU activation function. In the decoding network, the feature map is progressively upsampled and convolved to restore the original image size and channel count. The decoder and encoder are connected through a skip connection, enabling the fusion of feature maps. The fused feature map is further convolved and passed through a 1 × 1 convolution layer with C outputs, where C represents the number of channels or image segmentation categories.

The proposed Res-Swim-UNet model comprises an encoder, decoder, bottleneck layer, and skip connections. The encoder comprises six convolution layers, two residual modules, and four maximum pooling layers. This configuration facilitates extracting image features and downsampling, resulting in five feature maps. The skip connections concatenate the encoder’s multiscale feature maps with the decoder. The decoder exhibits a symmetric structure to the encoder, comprising six convolution layers and two residual modules. The final layer incorporates a 1 × 1 × 1 convolutional layer and sigmoid activation functions to generate prediction probability maps. The bottleneck layer is situated at the lowest point of the U-shaped architecture, where the feature resolution is at its minimum. To optimize the computational cost of the Swim Transformer module, this study incorporates only two modules into the bottleneck layers, considering that the calculation cost of Swim Transformer modules increases linearly with resolution. This approach leads to an improved model with enhanced performance. Figure 2 provides a detailed illustration of the model’s structure.

The conventional Transformer relies on multi-head self-attention modules to establish global interdependencies, enabling the system to extract and analyze global information better. By contrast, the Swim Transformer is designed on the principle of a mobile window and is composed of contiguous sub-modules. Figure 3 shows a detailed illustration of its structure.

Each submodule within the Swim Transformer comprises a normalization layer, a multi-head self-attention module, a residual connection, and a two-layer Multilayer Perceptron (MLP). Window Multi-head Self-Attention (W-MSA) and Shifted Window Multi-head Self-Attention (SW-MSA) are employed for two consecutive submodules of the Swim Transformer. Formulas (1) to (4) calculated the Swim Transformer module in detail.

(1)z ˆ l = W M S A L N z l − 1 + z l − 1 (2)z l = M L P L N z ˆ l + z ˆ l (3)z ˆ l + 1 = S W M S A L N z l + z l (4)z l + 1 = M L P L N z ˆ l + 1 + z ˆ l + 1 where, z ˆ l respectively, represent the W-MSA module and MLP module for the module output, LN represents layer normalization, where the self-attention module can be expressed as Formula (5). (5)Attention Q , K , V = Soft max Q K T d + B V where, Q, K, V ∈ R^M2×d represents three matrices, which are obtained from the input characteristic map through three convolution layers. K^T is the transpose matrix of K, d is the scaling factor set to 64 in the experiment, B is a learnable offset parameter.

Parallax regression

The Res-Swim-UNet model utilizes regression to estimate the segmentation results and employs the fully differentiable Soft Argmin operation (Li, Zhao & Yan, 2022) to yield smooth disparity estimation outcomes. This operation involves calculating the probability of each pixel in the segmented image belonging to various parallax values. The matching cost volume is then processed using 3D CNN and upsampled to obtain the matching cost for each pixel across all parallax values. The prediction cost is negated and regularized using a Softmax operation, resulting in the probability of each pixel belonging to different parallax values. Finally, the parallax values are weighted and summed using the corresponding probability values to determine the parallax value at each pixel, as depicted in Formula (6). (6)d = ∑ d = 0 D max d × σ − C d where d represents the predicted parallax value, Cd represents Matching costs under parallax d , σ . represents the Softmax operation, and its mathematical expression is shown in Formula (7). (7)σ z j = e z j ∑ k = 1 K e z k where j = 1, 2, …, K. The result of the converted output by Softmax is always in the range [0, 1], and the sum of all results is equal to 1, so they show a probability distribution.

Loss function

Given that the smooth L1 loss function demonstrates strong robustness and low sensitivity to outliers (Xu et al., 2022), this study has adopted it as the fundamental loss function, as presented in Formula (8).

(8)L d . d ˆ = 1 N ∑ n = 1 . N L 1 s d n , d ˆ n (9)L 1 s x = 0 . 5 x 2 , x < 1 x − 0 . 5 where N represents the total number of pixels, dn represents the true parallax value, d ˆ n and represents the predicted parallax value. In general, it minimizes the sum L d . d ˆ of the absolute differences between the target value dn and the estimate d ˆ n.

A deep supervision training approach is deployed to supervise the network’s final output and the results obtained from intermediate levels of the network. More precisely, apply a corresponding loss function, such as a cross-entropy loss function, to the output of each layer, compare the loss value of each layer to the real label, perform parallax regression on the volume output of each encoding and decoding structure, and calculate the loss value, backpropagate according to the loss value, update the network parameters, and thus improve the performance of the entire network. The final loss value is the weighted sum of the loss values from each level, as illustrated in Formula (10). (10)L o s s = ∑ i = 1 M w i L i d . d ˆ were wi represents the weight of losses at different levels, M represents the number of supervised levels. After experiments, M = 3 each layer’s corresponding weight parameters are between 0.5–1.0.

Data set and parameter setting

To evaluate the effectiveness and robustness of the proposed algorithm, Comparative experiments were carried out on different stereoscopic image pairs (https://zenodo.org/records/45114). All the comparison experiments were conducted under the same environment and hyperparameter settings.

All experiments were conducted using the Python language, and computation and the OpenCV visual library facilitated compilation. All pixel values were normalized to the range [0, 1.0]. Parameters utilized in the experiments are listed in Table 1. To calculate the mismatching rate, detailed data from mismatching pixels was obtained by comparing each pixel in the resulting parallax map to the corresponding pixel in the standard parallax map, which is calculated by Formula (11): (11)P P B M = 1 N ∑ x , y d c x , y − d t x , y > δ d .

The parameters in the equation have the same meanings as those in Table 1, where N is the number of effective pixels in the image area, dc(x, y) is the disparity map calculated for stereo matching algorithm; dt(x, y) is the true parallax map provided for the dataset, δd is the parallax threshold, which is taken as 1 in the experiment, that is, when the difference between the parallax value calculated by the stereo matching algorithm and the true parallax value is greater than 1, the pixel point is regarded as a mismatched point. The specific step is to give two graphs with different marks, search for the most matched points on the polar line, calculate the similarity of each search window, and finally calculate the parallax.

Comparison of segmentation performance

The segmentation model’s performance plays a vital role in the overall matching effect of the proposed algorithm, given its sequential segmentation and matching approach. To validate the effectiveness of the introduced residual module and Swim Transformer module, a comparison was conducted between the Res-Swim-UNet model proposed in this study and other models, including UNet, UNet++, 3DUNet, and ResUNet. The evaluation of segmentation performance encompassed metrics such as the intersection-over-union ratio (IoU), mean pixel accuracy (mPA), and frames per second (FPS) (Yu et al., 2022; Rahman et al., 2022). The comprehensive comparison results of the five models are presented in Table 2, while Fig. 4 depicts the mPA curve over 600 iterations.

Table 2 and Fig. 5 illustrate that the model proposed in this study outperformed the other models across all three evaluation metrics. Although the improvement in mPA was not particularly significant compared to the other models, the model in this article exhibited superior generalization ability and robustness. By incorporating the residual connection and Swim Transformer, the network in this article demonstrated improvements of 2.9% in IoU and 162% in mPA. Moreover, it achieved an FPS of 65.6/s, representing a 25.4% enhancement over the ResUNet network, which solely utilized the residual module for optimization. These results conclusively demonstrate the effectiveness of the Swim Transformer module proposed in this article, as it successfully enhances the accuracy and efficiency of the image segmentation model.

Comparison of matching performance of different algorithms

To assess the matching performance of the algorithm proposed in this article, a comparison was conducted with the classic Adaptive Weight and RT Census algorithms (Deng et al., 2023; Chen et al., 2023). Table 3 presents the mismatch rates of the various algorithms. Based on the test results, the proposed algorithm achieved an average mismatch rate of 2.04%, surpassing the performance of the other two algorithms. Thus, the proposed model effectively enhances the matching accuracy.

In order to further examine the higher-resolution image matching, we perform stereo matching on standard stereo image pairs of Adirondack (resolution: 718 × 496), Pipes (resolution: 735 × 485), Motorcycle (resolution: 741 × 497), and ArtL (resolution 694 × 554), employing the error matching rate of non-occluded area (Nonocc), all areas (All), and discontinuous area (Disc) as evaluation indices. The experimental results are tabulated in Table 4. Our observations from Table 4 lead to the conclusion that the matching accuracy of the algorithm in this article is high and that it is also suitable for stereo matching of high-resolution images.

Anti-noise test

In addition, 5%, 10%, 15%, and 20% pepper and salt noise and Gaussian noise with standard deviation were added to the standard test images of Tsukuba, Venus, Teddy, and Cones. The two algorithms above, compared in ‘Comparison of matching performance of different algorithms’, were utilized to calculate and obtain the unoptimized disparity map and subsequently compute the mismatch rate. The resulting experimental outcomes are presented in Figs. 5 and 6.

Based on the data presented in Figs. 5 and 6, it can be observed that the Adapt Weight algorithm consistently exhibited a high mismatch rate when subjected to salt and pepper noise, indicating its vulnerability to noise interference. On the other hand, the proposed algorithm showcased superior robustness to such noise. In Gaussian noise, both the proposed algorithm and the RT Census algorithm outperformed the Adapt Weight algorithm. In conclusion, the algorithm proposed in this article demonstrates enhanced noise tolerance without compromising the accuracy of the resulting disparity map, surpassing the performance of both the Adapt Weight and RT Census algorithms.

Discussion

The advancement of convolutional neural network architectures in image segmentation, coupled with the availability of sizeable standardized stereo datasets, has led to extensive research on stereo-matching algorithms based on deep learning. Many of these algorithms now achieve accuracy comparable to that of depth sensors. Compared to traditional stereo matching methods that rely on manually designed features, deep learning-based 3D matching algorithms have significantly improved accuracy. Considering the parallax continuity constraint in stereo matching, which states that there should be no abrupt changes in parallax between the center pixel and its neighboring pixels, except at the image boundary, the image can be divided into segments assuming continuous parallax within each segment. The encoder–decoder network based on 3D convolution offers advantages such as a simple structure, strong scalability, and ease in balancing precision and speed. Therefore, this study combines the classical image segmentation network, 3DUNet, with the stereo-matching algorithm. A new parallax computation network is proposed by incorporating residual and Swim Transformer modules to adjust the network structure and optimize performance. This fusion approach has two advantages in experiments: it ensures adherence to the parallax continuity constraint. It reduces the algorithm’s time complexity by operating at the superpixel level rather than the pixel level. The encoder–decoder network, also known as the hourglass structure, exhibits powerful feature extraction and data dimension reduction capabilities, making it well-suited for stereo-matching tasks. With the continuous progress of social science and technology, three-dimensional matching technology is advancing rapidly. The improvement in the accuracy and speed of matching algorithms has expanded their application scenarios. Against this backdrop, studying the variations in stereo matching is significant. Stereo matching is crucial in obtaining depth information through image matching in 3D reconstruction, stereo navigation, non-contact ranging, and various other technologies. Although stereo matching is widely utilized, numerous unresolved issues remain, making it a challenging and prominent topic in computer vision in recent years.

Although the experimental comparison was not carried out on different data sets in this article, the model could learn more data patterns when there were enough data samples, select meaningful features in feature selection, and reduce overfitting. The pre-trained model was adopted as initialization, and fine-tuning was performed from convergent points to increase the model’s generalization ability.

As an engineering problem, stereo matching involves various factors that affect its accuracy and speed during implementation. No single complex algorithm can address all aspects of stereo matching. This article focuses on the core steps of image pixel segmentation and matching in stereo matching and improves the image segmentation network by integrating it into the stereo matching algorithm. Experimental results have demonstrated the effectiveness of the proposed approach. Therefore, it can be concluded that the algorithm presented in this article can assist researchers in establishing more effective binocular stereo-vision systems and contribute to advancing related fields.