Chest X-ray Inpainting with Deep Generative Models

The context encoder trains a convolutional neural network to generate the contents of an arbitrary image region conditioned on its surroundings [7]. Specifically, we feed the input image with a mask to an autoencoder, and it outputs the generated content under the mask. Both the encoder and the decoder in the autoencoder are convolutional networks, where the encoder has six layers and the decoder has five. Furthermore, the discriminator is also a convolutional network with five layers. The last layer outputs a scalar value between 0 and 1 indicating how real its input is.

The model applies two loss functions to the output of the autoencoder, one is the pixel-wise reconstruction loss, and the other is the adversarial loss. This combined loss produces sharp and semantically meaningful results, and the loss has the form $L=\lambda_{rec}L_{rec}+\lambda_{adv}L_{adv}$. We assign $\lambda_{rec}=0.998$ for the reconstruction loss and $\lambda_{rec}=0.002$ for the adversarial loss. Our implementation is based on the publicly available PyTorch [18] code ¹¹ 1 https://github.com/BoyuanJiang/context_encoder_pytorch, but we modify their code by doubling the initial channels from 64 to 128 for both the autoencoder and the discriminator. This modification results in better image quality.

The semantic image inpainting [6] model proposed by Yeh et al. is trained in two steps. First, we train a deep generative model DCGAN [19] on real images. After this generative training, we obtain a mapping between a latent code and its corresponding image produced by DCGAN. Then we fix the weights in DCGAN, and iteratively find the latent code $z$ whose mapped image matches the best with the original image outside a mask under a weighted context loss. This means, the generated pixels closer to the missing patch need to match more closely to the original image than distant pixels. In addition, they have a prior loss which penalizes unrealistic images. The loss is different than the context loss as the prior loss uses GAN to discriminate high-level image features; whereas the context loss is a weighted $l1$ loss. Our implementation is based on the TensorFlow [20] implementation from the blog written by Brandon Amos [21], but we modified the code to have the exact implementation of the original paper.

Yu et al. recently proposed the contextual attention model that is able to synthesize novel image structures as well as blend in textures from surrounding regions [9]. The training of this model consists of two stages, which goes through the coarse-to-fine transition in the quality of the generated patch. In the first stage, a dilated convolutional network is used to produce an initial coarse prediction. In the second stage, the coarse prediction from the first stage becomes the input and it is split into 2 paths. One is through another dilated convolutional network, the other is through a contextual attention layer. The results from these two paths are concatenated together and fed to another neural network to predict refined results. This model was shown to improve upon the method [8] for inpainting in natural images. We use the TensorFlow implementation provided by the authors ²² 2 https://github.com/JiahuiYu/generative_inpainting.

To obtain the input training images for the generative models, we first use a popular medical image segmentation tool U-Net [22] to outline the lung locations. Then we randomly crop out 20 image patches of size 128 $\times$ 128 that have an overlap with the lung fields from each chest x-ray. This provides 1.2M patches for training. For each image patch, we mask out the center 64 $\times$ 64 patch and let our model to generate it.

Training was performed on a variety of modern GPUs in a cluster, using one GPU per model. The GPU models we used include Nvidia 1080 Ti, Nvidia Titan Xp and Nvidia Titan V. Training took approximately 5 days for context encoders to reach 70 epochs, 6 days for semantic image inpainting to reach 80 epochs, and 7 days for the contextual attention model for 55 epochs.

Figure 1 displays five chest x-ray patches randomly chosen from the 880 healthy ones in the test set. We compare the generative results of the three models. The right most column shows the original five patches, and the left most column is the input patch to the algorithms, with the masked out region to be inpainted in the center. Ideally, the generative models would produce a result identical to the original patches. The middle three columns are the results from the three generative models, namely, the context encoder, the semantic inpainting model, and the contextual attention model.

In general, we find that the context encoder is able to generate sharp and semantically meaningful patches; however, there are some discontinuities at the borders of the generated patch. On the other hand, the contextual attention model has better global consistency and therefore the borders of the inpainted region are often hardly noticeable. However, the content inside the patch tends to be a bit blurrier than the context encoder. The semantic inpainting model can sometimes produce good results, but its performance in general falls short to the other two. Nevertheless, all models can produce realistic looking results. Table 1 provides the means and standard deviations of the peak signal-to-noise ratio (PSNR) over all 880 healthy images in the test set. The PSNR value is computed with respect to the center 64 $\times$ 64 missing patch rather than the entire patch. We note the semantic inpainting model achieves the highest average PSNR in this case; however, this quantitative assessment doesn’t fully correlate with the visual results.

In contrast, we want to understand how the generative patches would look like if we present the model with abnormal chest x-rays. The ChestXray14 dataset is a comprehensive dataset with many images and sufficient annotations. However, sometimes those annotations are less accurate. To ensure high quality annotations, we randomly choose 33 abnormal chest x-rays labeled as containing lung nodules from the 880 abnormal ones in the test set, and asked an experienced human reader to provide a 64 $\times$ 64 bounding box around a nodule in each x-ray. These 33 x-rays contain a variety of nodular abnormalities. We then add the neighboring 32 pixels on each side of this bounding box to provide some context. Now we have 33 image patches of size 128 $\times$ 128, where each central 64 $\times$ 64 patch contains the abnormality. Then we apply the three generative models to these 33 patches, and the results are demonstrated in Figure 2. The layout of Figure 2 is the same as Figure 1, and we show the average PSNR values for the 33 abnormal regions in Table 2.

As expected, we obtain much less agreement between the generated patches and the real ones, because our generative models have been trained solely on patches extracted from normal images (according to the ChestXray14 labels) so we can expect them to generate healthy looking patches. However, if part of the abnormality shows up outside the mask, such as the third example in Figure 2, then some generative models can produce wrong results by providing continuity with respect to the surrounding abnormal structures. We conclude in this case, the context encoder and the contextual attention model produce better results than the semantic inpainting model.

Lastly, we randomly select a few healthy and abnormal chest x-ray patches, and subtract our generated patches from the original ones on the pixel-level. As shown in Figure 3 and 4, if the original patch does not have an abnormality, the subtraction produces a region of white noise. This means the differences between the generated and the real ones are small. On the other hand, the subtraction from an abnormal image leaves us a patch with a spherical image structure, which highlights the region of the abnormality. Thus we have demonstrated that the inpainting methods have the potential to be used for enhancement of abnormalities via subtraction.

To better quantify the visual quality of the generated patches, we conducted a two-alternative forced choice (2AFC) observer study. We prepared 160 random pairs of chest x-rays. Both images of the pair are complete 1024 $\times$ 1024 chest x-ray images from the test set, either from the 880 normal or the 880 abnormal images. In one of the two images, a 128 $\times$ 128 patch opverlapping with the segmented lung fields is selected and the central 64 $\times$ 64 region is inpainted by one of the three models in this study. The two images are presented to the observer side by side and the observer has to indicate which of the two images is the real, unaltered, chest x-ray³³ 3 The images of this observer study can be found here: https://www.dropbox.com/sh/s2mo1fwuvjvl2mg/AACgg0x_DOl1W6SN8hsesmxsa?preview=0.PNG.

The accuracy obtained in these three mixed 2AFC experiments (one for each type of inpainting method) corresponds to the area under the ROC curve [23] for distinguishing a real chest x-ray from one where a 64 $\times$ patch overlapping with the lung fields has been inpainted by one of the 3 methods. The observer was a medical researcher with $>20$ years of experience in reading chest radiographs. Table 3 presents the results of this study. The contextual attention model is the most successful one to fool the expert, because more than half of the time the expert would select the image with an inpainted region as the real chest x-ray. On the other hand, the semantic inpainting model has a more difficult time to fool the expert, where only one third of the time its results are considered ‘real’. The context encoder’s results are usually also not detectable with observer accuracy close to chance levels. These findings are consistent with the results of the previous sections.