High-resolution imaging in acoustic microscopy using deep learning

The ultrasound images of 17 different coins were included in the data sets. Because a small data set with many similarities is unsuitable for training the DNN, coins from different countries were used to introduce variation into the data set, as coins from the same country have similar patterns. The dimension of each image is $400\times400$ pixel and the pixel size ($50\,\mu$m) is kept constant for both transducers (20 and 50 MHz). All coins have a circular shape. Each image is cropped into multiple images to create diversity and hence more robust training of the network. Cropping is done by starting from the top-left corner of the image and striding by 48 pixels in the x-direction and y-direction. The resulting cropped image is of the dimension $120\times120$. After cropping, the final coin data set is made of 850 images, where 800 images are used for the training of the network and 50 are reserved for validation.

SRGAN [25], ESRGAN [26], IMDN [27, 28], DBPN-RES-MR64-3 [29] and SwinIR [30] models are used for both training and testing. The following section contains information on the training hyperparameters. SwinIR outperforms the others when it comes to generating HR images and comparing them to the ground truth. SwinIR consists of shallow feature extraction, deep feature extraction, and high-quality image reconstruction modules.

Shallow feature extraction

For a low quality input I_LQ , a $3\times3$ convolution layer $H_{SF}(.)$ is used for shallow feature extraction, F₀. This produces stable optimization and maps the input image space to a higher dimensional feature space [30].

Deep feature extraction

Followed by the shallow feature extraction, we have the deep feature extraction, F_DF , using K residual Swin transformer blocks (RSTB) and $3\times3$ convolutional layer. Here, intermediate features F₁, F₂ $,\ldots,$ F_k and output deep feature F_DK are extracted block by block, given by;

$$\begin{align} & F_{i} = H_{\textrm{RSTB}_{i}}\left(F_{i-1}\right), i = 1,2,{\ldots},K \end{align} \tag{ 1 }$$

$$\begin{align} & F_{\textrm{DF}} = H_{\textrm{CONV}}\left(F_{K}\right) \end{align} \tag{ 2 }$$

where the ith RSTB and last convolutional layer are denoted by $H_{\textrm{RSTB}_{i}}(.)$ and $H_{\textrm{CONV}}()$ respectively.

Each RSTB involves dividing images into patches, applying self-attention and feedforward layers, and adhering to core deep learning principles.

Each layer combines self-attention and feedforward neural networks. It starts with patch processing, resulting in X_i , representing patch embeddings at each layer. Multi-head self-attention (MSA), followed by residual connections and layer normalization, produces $Attention_i(X_i)$. Additionally, feedforward neural networks further process the output to capture intricate image features, yielding $FFN_i(LayerNorm_i(Attention_i(X_i) + X_i))$. Thus, we have,

$$\begin{align} {H_{\textrm{RSTB}_{i}}\left(X_i\right) = FFN_i\left(LayerNorm_i\left(Attention_i\left(X_i\right) + X_i\right)\right)}. \end{align} \tag{ 3 }$$

By stacking multiple Swin Transformer blocks, deeper and more complex features are extracted from images. The convolutional layer at the end of feature extraction brings the inductive bias of the convolution operation into the network.

Image reconstruction

The high-quality image I_RHQ is reconstructed by aggregating shallow and deep features as,

$$\begin{align} I_{\textrm{RHQ}} = H_{\textrm{REC}}\left(F_{0}+F_{\textrm{DF}}\right) \end{align} \tag{ 4 }$$

where $H_{\textrm{REC}}(.)$ is the function of the high-quality image reconstruction module. While shallow features mainly focus on low frequencies, deep features recover lost high-frequencies. Due to the long skip connection, the model can transmit low-frequency information directly to the high-quality image reconstruction module. To implement the reconstruction module, the sub-pixel convolutional layer is used to upsample the feature. Also, residual learning is used to reconstruct the residual between the low-quality and high-quality image instead of the high-quality image, given by;

$$\begin{align} I_{\textrm{RHQ}} = H_{\textrm{SwinIR}}\left(I_{\textrm{LQ}}\right)+I_{\textrm{LQ}} \end{align} \tag{ 5 }$$

where $H_{\textrm{SwinIR}}(.)$ is the SwinIR function. The parameters of SwinIR are optimized by minimizing the L₁ pixel loss,

$$\begin{align} L = ||I_{\textrm{RHQ}}-I_{\textrm{HQ}}||_{1} \end{align} \tag{ 6 }$$

where I_RHQ is obtained by taking I_LQ as an input of SwinIR, and I_HQ is the corresponding ground truth high quality image. The RSTB is a residual block with Swin transformer layers (STL) and convolutional layers. For input feature $F_{i,0}$ of the ith RSTB, the intermediate features $F_{i,1}$, $F_{i,2}$ $,\ldots,$ $F_{i,L}$ are extracted as,

$$\begin{align} F_{i,j} = H_{\textrm{STL}_{i,j}}\left(F_{i,j-1}\right), j = 1,2,{\ldots},L \end{align} \tag{ 7 }$$

where $H_{\textrm{STL}_{i,j}}(.)$ is the jth STL in ith RSTB. STL is based on the standard MSA of the original transformer layer. Local attention and shifted window mechanism are the main differences. For a local window feature X, the query, key, and value matrices Q, K, and V are computed as,

$$\begin{align} Q = XP_{Q} \end{align} \tag{ 8 }$$

$$\begin{align} K = XP_{K} \end{align} \tag{ 9 }$$

$$\begin{align} V = XP_{V} \end{align} \tag{ 10 }$$

where P_Q , P_K , and P_V are projection matrices which are shared across windows. Using the self-attention mechanism, the attention matrix is thus computed as,

$$\begin{align} Attention\left(Q,K,V\right) = \textrm{SoftMax}\left(QK^{T}/\surd{d}+B\right)V \end{align} \tag{ 11 }$$

where B is the learnable relative positional encoding. The attention function is performed h times in parallel and the results are concatenated for MSA. Also, a multi-layer perception (MLP) having two fully connected layers with GELU non-linearity between them is used for further feature transformations. Before MSA and MLP, the LayerNorm (LN) layer is added and the residual connection is employed for both modules. Hence,

$$\begin{align} X = MSA\left(LN\left(X\right)\right)+X \end{align} \tag{ 12 }$$

$$\begin{align} X = MLP\left(LN\left(X\right)\right)+X. \end{align} \tag{ 13 }$$

There is no connection across local windows when different layers have fixed partitions. Regular and shifted window partitioning are used alternatively for enabling cross-window connections. Here, shifted window partitioning refers to shifting the features by ($\lfloor M/2 \rfloor$, $\lfloor M/2 \rfloor$) pixels before partitioning.

The model architecture is given in figure 2.

Zoom In Zoom Out Reset image size

The transfer learning approach is adopted to generate HR images in acoustic microscopy. For SRGAN model [25], 2500 epochs are used for training purposes. The Adam optimizer [31] is used with the $learning rate = 10^{-4}$, $\beta_{1} = 0.9$, and, $\beta_{2} = 0.999$. For the ESRGAN model [26], 3000 epochs, Adam optimizer [31], $learning rate = 10^{-4}$, and $\beta_{2} = 0.99$ is used. For IMDN, DBPN-RES-MR64-3, and SwinIR models, Adam optimizer [31], 3500 epochs, $learning rate = 10^{-4}$, $batch size = 8$, and $\beta_{2} = 0.99$ is used for the training purpose. However, a steep decline in SSIM and PSNR values on the validation set can be seen after the 420 epoch. The decline was more evident in SSIM than PSNR. These can be seen in figure 3. PSNR vs epoch is in figure 3, SSIM vs epoch is in figure 3.

Zoom In Zoom Out Reset image size

A similar strategy was followed for the DBPN-RES-MR64-3 model, where the model was trained with $learning rate = 10^{-4}$, and Adam optimizer [31] with $batch size = 16$, $\beta_{1} = 0.9$, and $\beta_{2} = 0.999$; and the IMDN model, where the model was trained with $learning rate = 10^{-4}$, and Adam optimizer [31] with $batch size = 16$, $\beta_{1} = 0.9$, and $\beta_{2} = 0.999$.

Finally, the SwinIR model displayed the best results regarding both PSNR and SSIM scores. The G_loss of the SwinIR training has been displayed in figure 4.

Zoom In Zoom Out Reset image size

There are two operational modes in SAM namely; reflection and transmission mode. A detailed description of the working principles of both modes can be found elsewhere [32]. In this manuscript we consider reflection mode to scan the samples. To focus acoustic energy via a coupling medium (in this case, water), a concave spherical sapphire lens rod is generally utilized. Later on, ultrasound signals are generated from the signal generator and transmitted toward the sample and the reflected waves were recorded. The digitized signal from the samples is called an A-scan or amplitude scan. In order to acquire a C-scan of the sample, repeat this procedure at various points in the XY plane. In another way, a C-scan can be referred to as the summation of A-scans in two dimensions.

For this experiment the data acquisitions were performed on a custom-built SAM (figure 5), integrated with a Standa (8MTF-200-Motorized XY Microscope Stage) high-precision scanning stage controlled by LabVIEW [33] program. A similar experimental setup was employed earlier by our group to determine and correct the inclined sample [34, 35]. The acoustic microscopic features were implemented employing National Instruments’ PXIe FPGA modules and FlexRIO hardware. It was enclosed in a PXIe chassis (PXIe-1082) which consists of an arbitrary waveform generator (AT-1212). For acoustic imaging, the transducer was excited with signals (Mexican hat) and delivered into an RF amplifier (AMP018032-T) for further amplification of the ultrasonic signals [36]. The acoustical reflections caused by the transmitted signal on the surface of the sample were picked up and fed into a custom-designed amplifier. The role of such an amplifier is to amplify the currents into an output potential. These signals were then amplified with a custom-designed pre-amplifier and digitized with a 12-bit high-speed (1.6 GS s⁻¹) digitizer (NI-5772).

Zoom In Zoom Out Reset image size

For ground truth, an Olympus 50 MHz focused transducer having an aperture of 6.35 mm and a focal length of 12 mm was used to scan all the coins and also the biological specimen. By focusing the acoustic energy on the coin’s top surface, the sample was scanned in the x and y directions with $50\ \mu$m steps. On the other-hand low resolution images were acquired with a 20 MHz transducer (focal length 50 mm). All experiments were performed in distilled water and the room temperature was kept constant at around 22 °C during the experiments. For testing the models a biological sample was employed in SAM for imaging. The sample used for this experiment was a discarded reindeer antler that was collected from the jungle of Tromsø (Norway). The moss on the antler was first removed by cleaning it with lukewarm water and 96% ethanol. After cleaning, the sample was diced and boiled in distilled water at 100 °C in order to remove any unwanted biological substance from the antler. The sample was thereafter put on the sample holder and allowed to dry before being scanned.

UiT The Arctic University of Norway has a focus on research on topics and concerns related to Arctic life such as reindeer. The local Arctic tribes consider that Reindeer’s antlers indicate the health and well-being of reindeer. Therefore, investigating them is a long-term interest for us. Now, to study such biological samples, above mentioned preparation step is generally required which includes slicing the antler to view it under the microscope. Potentially, in the long term, our method can provide a more scientifically rooted study of this conjecture.

Initially, 17 coins from different countries and with different dimensions were scanned for 20 MHz (low resolution) and 50 MHz (ground truth). Later on, they were cropped and several images were created, and eventually, a training data set of 800 images were generated, the models were prone to overfitting and low PSNR and SSIM scores (7.92 and 0.064 respectively). Hence, transfer learning was used in training the models. This significantly increased the PSNR and SSIM scores and the overall results of the models. Hence, pre-trained models trained using data sets like celebA data-set [37] and DIV2K data-set [38–41] were used and they were fine-tuned using the coin data set, which improved the model performance to a large extend.

Figure 7 shows some more examples of outputs we got across models without using the transfer learning approach.

Zoom In Zoom Out Reset image size

Popular digital resolution enhancement techniques include the nearest neighbor interpolation algorithm, bi-linear interpolation, and cubic convolution interpolation. However, when applied to images, all of these algorithms have some drawbacks. Errors occur when the picture is overly expanded. The nearest neighbor interpolation algorithm results in significant image quality loss, as well as visible mosaic and jagged phenomena. Because of the poor design of the interpolation function, the output image of the bi-linear interpolation algorithm suffers from quality damage and low calculation accuracy. The cubic convolution interpolation algorithm requires a significant amount of calculation and is also complicated and time-consuming. Interpolation-based algorithms also have issues with computational complexity, noise amplification, and blurry images. Deep learning methods and techniques, on the other hand, have advanced in recent years, and thus deep learning-based SR models are used. These methods frequently achieve cutting-edge performance on various resolution enhancement benchmarks.

A biological sample (Reindeer Antler) of unknown size, shape, and surface morphology was employed to validate the reproducibility of the suggested model. This experiment was conducted with a discarded reindeer antler gathered from the local jungle. To remove the moss, the antler was cleaned with lukewarm water and ethanol. Following cleaning, the antler was diced and boiled at 100 °C in distilled water for 30 minutes to eliminate any undesired biological substances. The sample was then dried and mounted on the sample holder in preparation for scanning.

The figure 8 shows the result obtained after testing the unknown biological sample (Reindeer Antler). The original image was taken to be the ground truth and a LR image was generated from the original image (figure 8). The low-resolution image was given as input to the final model to get the generated output image using the SwinIR model. The result showed a PSNR value of 31.88 and an SSIM value of 0.8406. Although the PSNR and SSIM values are comparatively less than the average SSIM and PSNR scores obtained in the test data set, it is because a data set consisting of only coin images was used to train the model. Also, it is to be noted that the image is evaluated only quantitatively.

Zoom In Zoom Out Reset image size