Steered Mixture-of-Experts Autoencoder Design for Real-Time Image Modelling and Denoising

Abstract

Research in the past years introduced Steered Mixture-of-Experts (SMoE) as a framework to form sparse, edge-aware models for 2D- and higher dimensional pixel data, applicable to compression, denoising, and beyond, and capable to compete with state-of-the-art compression methods. To circumvent the computationally demanding, iterative optimization method used in prior works an autoencoder design is introduced that reduces the run-time drastically while simultaneously improving reconstruction quality for block-based SMoE approaches. Coupling a deep encoder network with a shallow, parameter-free SMoE decoder enforces an efficent and explainable latent representation. Our initial work on the autoencoder design presented a simple model, with limited applicability to compression and beyond.
In this paper, we build on the foundation of the first autoencoder design and improve the reconstruction quality by expanding it to models of higher complexity and different block sizes. Furthermore, we improve the noise robustness of the autoencoder for SMoE denoising applications. Our results reveal that the newly adapted autoencoders allow ultra-fast estimation of parameters for complex SMoE models with excellent reconstruction quality, both for noise free input and under severe noise. This enables the SMoE image model framework for a wide range of image processing applications, including compression, noise reduction, and super-resolution.

1 Introduction

The Steered Mixture-of-Experts (SMoE) framework has been first introduced for coding images [1] with a sparse regression model and has since been further investigated not only for still images but expanded to video [2] and higher dimensional data such as light field images and video[3]. Application fields of the framework are in compression, [4] [5] [6] [7] [8] and recently extended to denoising and super-resolution [9].
The SMoE framework describes in-stationarities like edges and smooth transitions in an image through a sparse representation, forming an edge-aware model. Steered kernels model correlations between pixels, following a divide-and-conquer-principle[10]. Utilizing its steering properties, excellent edge representations can be obtained. For compression, SMoE parameters are quantized and coded and can be used directly to reconstruct the image, thus maintaining a high interpretability of its parameters. In contrast to other block-based compression approaches like JPEG, SMoE models describe images in the spatial rather than the transformation domain.
To optimize the model parameters, Expectation-Maximization (EM) or Gradient-Descent (GD) algorithms can be employed, albeit GD optimization results in higher reconstruction quality [4] [5]. A further advantage of GD optimization is the ability to apply well-known training techniques explored in neural network based approaches [11] to the SMoE framework, enabling the use of cost functions like mean-squared-error (MSE) or SSIM. Impressive results rivaling JPEG2000 are presented in the work of Jongebloed et al. [7] and demonstrate the powerful abilities of the regression model.
The basis for the SMoE image model is an edge-aware, continuous non-linear regression function, capable of modeling smooth and sharp transitions in an image without blocking or ringing artifacts, which can be seen in JPEG-like encoders. Fig. 1 depicts the functionality of the model and illustrates the reconstruction results in contrast to other image compression standards. The SMoE reconstruction displays outstanding edge quality, simultaneously preserving smooth transitions in the image. At the same bit rate, both PSNR and SSIM are improved compared to JPEG, JPEG2000, and HEVC-Intra. By creating an edge-aware model of the image, light noise in the original image is also removed, showing its native denoising capabilities.
The foundation of our model are gaussian kernels, displayed in Fig. 1(f). The locations and steering properties harvest the correlation between pixels and form the gating function, depicted in Fig. 1(g), which support the edge-aware properties of the model. Sharp edges are modeled with minimal overlapping gates, and smooth transitions by vastly overlapping ones. SMoE thus forms a description of edges and smooth transitions emphasizing important features in an image.
Describing images through a SMoE model allows for modeling beyond the regular N-dimensional pixel grid and can be applied to arbitrarily shaped data like point clouds or irregularly sampled imagery. Additionally it can be extended to N-dimensional signals including light-fields [12] and video data [2]. Once built, the continuous SMoE model allows resampling imagery to any resolution in time and space, which includes edge-aware super-resolution and motion-interpolation.
A major drawback of the SMoE model is the iterative nature of GD optimization, resulting in high processing times to obtain well-optimized SMoE parameters which make this approach unfeasible for real-time applications. To circumvent the interative training method, an autoencoder design was introduced in our previous work[13] enabling the estimation of SMoE parameters with run-time savings by a factor of $500-1,000$ compared to the work of Tok et al.[4], while achieving comparable reconstruction qualities. Combining a deep encoder network with a shallow SMoE decoder enforces the encoder to provide SMoE parameters in the latent space, ready for compression and coding, simultaneously maintaining full compatibility with the established SMoE framework. However, the initial autoencoder design[13] only supports radial kernels without steering capabilities, which provides narrow applicability for further SMoE applications.
In this work, we present an improved framework of the autoencoder design extended to model complex image patterns and adapted to serve promising SMoE denoising applications. First, steering properties are enabled resulting in the ability to model image structures of higher complexity and consequently improve reconstruction quality. Additionally, the effect of smaller block sizes on both the reconstruction quality in terms of PSNR and SSIM and run time is examined, leading to a higher kernel density to further improve the reconstruction quality. In previous work SMoE models showed promising results for image denoising applications [9], unfortunately with high computational demands. In this paper, we investigate SMoE-AE models trained on noisy data for applications in SMoE denoising algorithms.

2 Edge-aware Steered Mixture-of-Experts Model

(a) Original

(b) JPEG
PSNR:$26.33dB$
SSIM: $0.82$

(c) HEVC
PSNR:$26.05dB$
SSIM: $0.77$

(d) JPEG2000
PSNR:$29.43dB$
SSIM: $0.87$

(e) SMoE
PSNR:$31.66dB$
SSIM: $0.9$

(f) SMoE Kernels

(g) SMoE Gates

Figure 1: Compression performance at 0.43 bpp[13]

The scope of Steered Mixture-of-Experts regression is to optimize the underlying regression function Eq. 1 to determine the luminance value $y_{p}(\underline{x})$ of each position $\underline{x}$ in an image. The main components of this function are the experts $m_{i}(\underline{x})$ and gating function $w_{i}(\underline{x})$, which form a weighted sum over all $K$ kernels:

$$y_{p}(\underline{x})=\sum_{i=1}^{K}m_{i}(\underline{x})\cdot w_{i}(\underline{x})$$ (1)

While this work focuses on constant experts $m_{i}(\underline{x})=m_{i}$, hyperplanes [1] or polynomials are equally possible. In Eq. 1 the experts are weighted by the gating functions $w_{i}(\underline{x})$.
In our work, each gating function $w_{i}(\underline{x})$ is defined as a soft-max function consisting of $K$ gaussian kernels $\mathcal{K}$:

$$w_{i}(\underline{x})=\frac{\mathcal{K}(\underline{x},\underline{\mu}_{i},\underline{\Sigma}_{i})}{\sum_{j=1}^{K}\mathcal{K}(\underline{x},\underline{\mu}_{j},\underline{\Sigma}_{j})}$$ (2)

More specifically:

$$\mathcal{K}(\underline{x},\underline{\mu}_{i},\underline{A}_{i})=exp[-\frac{1}{2}(\underline{x}-\underline{\mu}_{i})^{T}\underline{A}_{i}\underline{A}_{i}^{T}(\underline{x}-\underline{\mu}_{i})]$$ (3)

with $\underline{\mu}_{i}$ being the center position of each kernel and $\underline{A}_{i}$ the steering properties. The covariance matrix of the gaussian kernel $\underline{\Sigma}_{i}$ is thus given as the inverse Cholesky decomposition, to stabilize training by omitting matrix inversion and ensure positive semidefiniteness. $\underline{A}_{i}$ is given as a lower triangular matrix with the following dimensions:

$$\underline{A}_{i}=\begin{pmatrix}a_{11}&0\\ a_{21}&a_{22}\end{pmatrix}$$ (4)

However, the dimension of $\underline{A}_{i}$ can be extended accordingly for higher dimensional data.
Together with the experts $m_{i}$, the center positions $\underline{\mu}_{i}$ in each dimension of the image and the steering parameters in $\underline{A}_{i}$ need to be estimated from the pixel data, resulting in 6 parameters per kernel.
A simpler representation was used in prior works [4] [13] using radial kernels instead of steered kernels. This drastically reduces the number of required parameters, while still maintaining a good representation of edges and smooth transitions in images [4]. When assigning the same bandwidth for all kernels $B$, the regression function Eq. 1 simplifies to:

$$y(\underline{x})=\sum_{i=1}^{K}m_{i}\cdot\frac{exp(-\mathcal{B}||\underline{x}-\underline{\mu}_{i}||^{2})}{\sum_{j=1}^{K}exp(-\mathcal{B}||\underline{x}-\underline{\mu}_{j}||^{2})}$$ (5)

Employing this reconstruction equation the needed SMoE parameters reduce to 3 per kernel. The SMoE model parameters can be effectively optimized using the gradient descent (GD) method as proposed in [4] [5], which outperforms the EM Algorithm in reconstruction quality[1]. Optimization using GD defines a loss function $\mathcal{L}$ to calculate e.g. the mean squared error over all pixels $N$.

$$\mathcal{L}:=\frac{1}{N}\sum_{n=1}^{N}(y_{n}(\underline{x})-\sum_{i=1}^{K}m_{i}(\underline{x}_{n})\cdot w_{i}(\underline{x}_{n}))^{2}$$ (6)

$$\underset{m_{i},\underline{\mu}_{i},\underline{A}_{i}}{\mathrm{arg\,min}}\{\mathcal{L}\}$$ (7)

The drawback of models trained using GD, to which we refer as SMoE-GD, is the high dependency on the initialization of the center position $\underline{\mu}_{i}$ which can lead to subpar results by finding only suboptimal minima. Furthermore, to achieve satisfactory results many training iterations are necessary causing a high computational cost. This makes SMoE-GD unfeasible for many real-time applications.

3 Autoencoder Design for Steered Mixture-Of-Experts

Previous work [13] introduced the autoencoder design for radial kernels and showed its efficiency in comparison to SMoE-GD. In this work, we further develop SMoE-AE, to be applicable to the aforementioned SMoE applications, to reduce the run time, and make the SMoE model feasible for real-time applications.

Figure 2: SMoE Autoencoder

The basic concept of SMoE-AE was presented in Fleig et al.[13]. Fig. 2 illustrates the approach and depicts the principal structure of the autoencoder design. It deviates from the conventional autoencoder design [14][15], by combining a deep encoder network with a shallow untrainable SMoE decoder, which performs the SMoE reconstruction based on Eq. 1. The deep encoder network is trained on ground truth images in an end-to-end process, coupled with the SMoE decoder. By combining the SMoE decoder in an autoencoder architecture, we ensure the encoder predicts feasible SMoE parameters in the latent space and simultaneously enables loss functions like MSE or SSIM to optimize rate-distortion. The first SMoE-AE was designed to be comparable directly with the work of Tok et al.[4] and is the basis for this work. In [13] the bottleneck layer was restricted to yield parameters for $K=4$ radial kernels, while the Bandwidth $B$ was provided as a hyperparameter. The input data was comprised of $16\times 16$ pixel greyscale blocks and was designed to reconstruct blocks of the same dimensions. The reader is referred to [13] to see the architectural structure of the basic encoder network, in addition to training specifications. This work aims to build upon the foundation of the basic SMoE-AE design and expands the framework to reconstruct higher complexity image data and improve noise robustness.
The great advantage of SMoE is the aforementioned ability to form an edge-aware model in the spatial domain, which is best utilized by employing steering properties to the kernels. In this work we enable steering properties via the Cholesky decomposition in Eq. 3 in the SMoE decoder, which additionally ensures $\underline{A}_{i}$ to be a lower triangular matrix. By expanding the number of parameters needed to satisfy Eq. 3 with three additional parameters/kernel, the latent space output of the encoder network is increased accordingly. We restrict the number of Kernels to $K=4$, thus resulting in a 24-dimensional output. To account for the increased complexity of the latent space output, additional layers in the encoder network are added. The full structure is depicted in Tab. 1.

Table 1: Architectural structure of SMoE Encoder

Convolutional Layer		Dense Layer
Quantity	7	Quantity 6
Filter	16,32,64,128,256,512,1024	Output Dimension	1024,512,256,128,64,24
Padding	Same
Kernel Size	3x3

To improve the reconstruction quality and explore the applicability of SMoE-AE to various use cases of SMoE we train our encoder network on a smaller block size of $8\times 8$ pixel/block. The general framework of the encoder remains equal, maintaining the complexity demands on the network, and simultaneously quadrupling the kernel density over the whole image. Smaller block sizes of dimension $8\times 8$ are beneficial to perform denoising on low to medium levels of noise [16] and are i.e. used in BM-SMoE and S-SMoE algorithm[9] for denoising.

Figure 3: Principal functionality of S-SMoE, for a window of $2x2$ resulting in four hypotheses for the yellow pixel point.

Additionally BM-SMoE and S-SMoE have great potential in super-resolution applications [9]. Its basic concept is depicted in Fig. 3. To employ S-SMoE a sliding window with dimensions $N\times N$ pixels and a step size $S$ pixels is moved across the image. For each extracted block SMoE parameters are predicted. Using a step size $S$ smaller than the dimensions of the sliding window $N$ results in overlapping blocks, thus receiving multiple hypotheses of reconstructions at every single pixel point of the image, depicted in yellow in Fig. 3. SMoE regression is performed for every extracted block and repeating this process for every overlapping block results in numerous hypotheses for each pixel of the image, each deviating slightly due to the sparse representation of the model. Averaging all possible blocks results in a reliable representation for each pixel of the image. With a growing number of hypotheses, a higher sampling of all edges in an image is performed. Unfortunately, optimizing SMoE parameters through SMoE-GD for a small step size is computationally unfeasible, posing an optimal use-case for SMoE-AE run-time acceleration.

Table 2: Comparison of SMoE-GD and SMoE-AE for radial and steered Kernels and 16x16 blocks

	SMoE-GD				SMoE-AE
	Radial Kernels		Steered Kernels		Radial Kernels		Steered Kernels
Sequence	PSNR [dB]	SSIM	PSNR [dB]	SSIM	PSNR [dB]	SSIM	PSNR [dB]	SSIM
Baboon	22.91	0.42	23.63	0.63	22.95	0.56	23.59	0.63
Boats	25.46	0.69	26.28	0.72	25.65	0.70	26.64	0.74
Bridge	22.80	0.54	23.30	0.59	22.91	0.55	23.56	0.60
Cameraman	27.08	0.85	28.48	0.86	27.79	0.85	29.17	0.88
Elaine	30.63	0.75	30.86	0.86	30.34	0.73	31.02	0.75
Lena	28.83	0.82	29.80	0.84	29.05	0.82	29.91	0.84
Livingroom	25.19	0.67	25.97	0.71	25.55	0.69	26.46	0.73
Peppers	29.50	0.79	29.94	0.79	29.56	0.78	30.52	0.80
Kodak	25.77	0.70	26.08	0.71	25.94	0.70	26.92	0.74
Encoding Time [s]	296.29		347.36		0.25		0.44
Decoding Time [s]	0.02		0.22		0.02		0.22
Encoding RTG	1185		789

4 Experiments

The final encoder network comprises over $74M$ trainable parameters in total for $8\times 8$ blocks and over $275M$ for $16\times 16$ input blocks. The increased amount of trainable parameters for bigger block sizes result in higher training times which consumed around $20h$ for $8\times 8$ blocks and $60h$ for $16\times 16$ blocks on an Nvidia GTX 1070 Ti. Regardless of block size, the innermost bottleneck layer of the encoder network yields the expert values $m_{i}$, the kernel center values $\underline{\mu}_{i}$ and the steering parameters $a_{1}$ to $a_{3}$ for every kernel and can be used directly to solve the regression function in Eq. 1. To train the encoder network the mobile subset of the clic dataset [17] was used. The subset contains $1048$ training images and $61$ validation images, the testing was made on well-known $512\times 512$ test images[18] and all $24$ images of the Kodak PhotoCD dataset [19]. Therefore all images were converted to grey levels and the pixel values were normalized to the $[0...1]$ domain. Then the images were cropped to $1024\times 1024$ and divided into $16\times 16$ pixel or $8\times 8$ pixel blocks respectively, resulting in a total of over $4$ million for $16\times 16$ pixel blocks and over $17$ million for $8\times 8$ pixel blocks.
It is necessary to train the network for every block size separately since SMoE-AE adapts to model more detailed structures in smaller blocks, and in larger blocks to more coarse structures. This results in a loss of quality when predicting SMoE parameters for $8\times 8$ pixel blocks with a network trained on $16\times 16$ pixel blocks.
Test images were preprocessed equally, except the dimensions of the images remained unchanged. For training with noise, noise was added to each block, regardless of train or test data. The original block served as the training target. During training, the blocks were shuffled after every training epoch to encourage the best possible generalization of the encoder network.
Regarding the innermost layer, the parameters were split into expert values $m_{i}$ and kernel center values $\underline{\mu}_{i}$, which were restricted to $[0...1]$, while the steering parameters $a_{1}$ to $a_{3}$ remained unrestricted to achieve full steering capabilities. These parameters ranged for most images between around $-50$ and $+50$.
The training was performed over $30$ epochs using Adam Optimizer with default parameters, a batch size of $64$, and a constant learning rate of $l=5\cdot 10^{-5}$. The loss was calculated as the MSE of the original block to the reconstruction. Rectified linear unit (ReLU) was used as the activation function, except in the innermost layer a linear activation function was used to enable an unrestricted range of the steering parameters.
After training the encoder and decoder were uncoupled and the pre-trained encoder network was used to predict SMoE parameters for any image block.
The primary goal of SMoE-AE is to reduce the run-time in estimating SMoE parameters while maintaining the reconstruction quality in terms of PSNR and SSIM of SMoE-GD. Consequently, we used SMoE-GD as a benchmark to compare the capabilities of SMoE-AE. To objectively compare run-time gains (RTG) of SMoE-AE to SMoE-GD, we ensure convergence of SMoE-GD by performing its optimization for $5,000$ iterations.

(a) Radial kernel
$16\times 16$ blocks

(b) Steered kernel
$16\times 16$ blocks

(c) Steered kernel
$8\times 8$ blocks

(d) S-SMoE with $S=1$
$8\times 8$ blocks

Figure 4: Crop of test image Cameraman reconstructed with radial and steered kernels and different block sizes

4.1 Supporting variable block-sizes and model complexities

Tab. 2 provides results comparing SMoE-GD and SMoE-AE on $16\times 16$ blocks. For radial kernels both SMoE-GD and SMoE-AE are comparable in results regarding PSNR and SSIM, achieving only marginal differences in reconstruction quality. SMoE-GD achieves better results for the image Elaine, with $0.29$dB higher PSNR and $0.2$ higher SSIM, while the test image Cameraman converges to a PSNR $0.71$dB lower than SMoE-AE. This observation affirms the assumption, that SMoE-GD is highly dependable on the initialization of the parameters, which can lead to difficulties in finding global optima but can occasionally form excellent results. Even though the results of SMoE-AE and SMoE-GD are very similar, the encoding time differs drastically. On average approx. $300s$ per image are necessary for the SMoE-GD encoder to converge and to achieve comparable results to SMoE-AE, which predicts SMoE parameters in $0.25s$/image, resulting in an encoding RTG by a factor of $>1,000$. This tremendous gain in run time is remarkable, given the preserved reconstruction quality. By maintaining the shallow properties at the decoder network the reconstruction of the image from SMoE Parameters is as equally fast as SMoE-GD.
Tab. 2 depicts additionally the reconstruction results for $K=4$ steered kernels. Adding steering capabilities leads to the doubling of parameters in the latent space. However, the autoencoder framework can easily adapt to these increased demands and is trained to predict the additional parameters with a high reconstruction quality. Regardless of the optimization procedure, the PSNR improved substantially, grounded in the improved ability of kernels to steer along edges and model more complex patterns in a block. This is depicted in Fig.4(a) and Fig. 4(b) for a crop of the test image Cameraman. The struts of the tripod are more detailed and appear sharper using steered kernels compared to radial kernels. Such fine details can be better represented with steered kernels, as they can steer in a narrow line, thus creating a sharp line in the image, with sharp transitions on both sides. Compared to SMoE-GD the adapted SMoE-AE autoencoder achieves $0.8$dB reconstruction gain on the Kodak data set on average while speeding up the process by a RTG factor of $750$. The average time for reconstructing a $512\times 512$ image reduced from $>5$ minutes remarkably to $<1$ second.

Table 3: Reconstruction Quality comparison on 16x16 pixel blocks and 8x8 pixel blocks

	SMoE-GD				SMoE-AE
	16x16		8x8		16x16		8x8
Sequence	PSNR [dB]	SSIM	PSNR [dB]	SSIM	PSNR [dB]	SSIM	PSNR [dB]	SSIM
Baboon	23.63	0.63	28.50	0.89	23.69	0.63	29.26	0.91
Boats	26.28	0.72	30.84	0.87	26.64	0.74	32.04	0.89
Bridge	23.30	0.59	26.49	0.81	23.56	0.60	27.52	0.86
Cameraman	28.48	0.86	35.39	0.96	29.17	0.88	35.43	0.97
Elaine	30.86	0.86	32.79	0.98	31.02	0.75	33.25	0.83
Lena	29.80	0.84	34.40	0.92	29.91	0.84	35.37	0.93
Livingroom	25.97	0.71	30.18	0.87	26.46	0.73	32.06	0.91
Peppers	29.94	0.79	33.24	0.86	30.52	0.80	34.07	0.88
Kodak	26.08	0.71	29.35	0.85	26.92	0.74	31.68	0.91
Encoding Time [s]	347.36		346.56		0.44		0.7
Decoding Time [s]	0.22		0.24		0.22		0.24
Encoding RTG	789		495

Reducing the block size to $8\times 8$ pixel blocks leads to a four times higher density of kernels per block while maintaining the general framework of the encoder network and parameters per block. The results depicted in Tab. 3 compare the reconstruction quality of $16\times 16$ and $8\times 8$ pixel blocks for SMoE-GD and SMoE-AE with steered kernels. It is expected to achieve higher reconstruction quality with smaller blocks, as is true for both optimization approaches. The quality gain for SMoE-AE is higher compared to SMoE-GD since the network was able to optimize the kernel positions to the smaller kernel size and higher density. The increased number of parameters in total required to model all blocks in the image results in a slower convergence for SMoE-GD, thus leading to a lower gain in reconstruction quality for all test images. On the Kodak data set with $8\times 8$ blocks the SMoE-AE improves the reconstruction quality by a remarkable $2$dB with again drastically reduced run time. For SMoE-GD to achieve comparable results to SMoE-AE around $7,500$ iterations are necessary, leading to an encoding RTG of $>1,000$. A visual comparison of $16\times 16$ to $8\times 8$ pixel blocks is depicted in Fig. 4(b) and Fig. 4(c). Blocking artifacts appear reduced and edges stretching over multiple blocks are continuous. Small features are displayed in high detail due to the high density of the kernels, leading to an overall higher-quality reconstruction.

Table 4: S-SMoE with shift of $S=1$,$S=4$ and $S=8$

	S-SMoE with SMoE-AE
	$S=1$		$S=4$		$S=8$
Sequence	PSNR [dB]	SSIM	PSNR [dB]	SSIM	PSNR [dB]	SSIM
Baboon	32.54	0.96	31.35	0.94	29.26	0.91
Boats	34.43	0.92	33.69	0.91	32.04	0.89
Bridge	29.37	0.9	28.79	0.89	27.52	0.86
Cameraman	40.23	0.98	38.80	0.98	36.43	0.97
Elaine	34.11	0.85	33.86	0.85	33.25	0.83
Lena	37.76	0.95	37.01	0.95	35.37	0.93
Livingroom	34.56	0.94	33.73	0.93	32.06	0.91
Peppers	35.18	0.89	34.83	0.89	34.07	0.88
Kodak	34.03	0.94	33.30	0.93	31.68	0.91
Encoding Time [s]	41.60		3.00		0.70
Decoding Time [s]	13.95		0.88		0.24

Using SMoE-GD in S-SMoE requires computational run-times of 6-7 hours on one greyscale $512\times 512$ pixel image with $S=1$, while parameter prediction with SMoE-AE allows for full image results in a viable time as depicted in Tab. 4. Using a shift of $S=1$ results in $255k$ blocks for a $512\times 512$ pixel image, a computationally intensive task even combined with the fast prediction times of SMoE-AE. Incorporating SMoE-AE results in an encoding RTG of around $500$ in comparison to SMoE-GD. Apart from run-time, the advantages of S-SMoE are reported in Tab. 4. $S=8$ equates to non-overlapping blocks and is equal to native SMoE-AE. A step size of $S=4$ results in an overlap of a half block with each step, therefore four hypotheses can be made for every pixel, except for pixels at the border of the image, since no additional padding is implemented. Building the average over four hypotheses already leads to great improvements of $1.5$dB on average in PSNR and $0.2$ in SSIM. Additionally, averaging over a step size smaller than the block size leads to the reduction of blocking artifacts, that are visible otherwise. This effect is displayed in Fig. 4(d) for $S=1$. With a step size of $S=1$ a total of $64$ hypotheses are established for each pixel point in an image, thus improving the result further by $0.7$dB in PSNR and $0.1$ in SSIM compared to $S=4$, without blocking artifacts. These capabilities can be utilized in applications like denoising.

4.2 Noise Robustness

Table 5: Comparison of SMoE-GD, SMoE-AE, and SMoE-AE_noisy with added speckle noise of variance $\delta^{2}=0.01$

	Noisy Original		SMoE-GD		SMoE-AE		SMoE-AEnoisy
Sequence	PSNR [dB]	SSIM	PSNR [dB]	SSIM	PSNR [dB]	SSIM	PSNR [dB]	SSIM
Baboon	25.53	0.73	27.16	0.82	27.13	0.81	27.10	0.82
Boats	25.38	0.62	28.81	0.77	28.67	0.75	29.60	0.81
Bridge	26.35	0.81	25.80	0.77	26.24	0.79	26.07	0.78
Cameraman	25.64	0.56	31.26	0.78	30.32	0.74	32.60	0.90
Elaine	25.09	0.56	29.94	0.78	28.98	0.66	30.97	0.75
Lena	25.69	0.56	31.02	0.78	30.09	0.73	32.16	0.81
Livingroom	25.95	0.67	28.61	0.79	28.99	0.79	29.51	0.83
Peppers	25.97	0.61	30.56	0.76	29.86	0.73	31.49	0.81
Kodak	26.98	0.67	28.10	0.77	28.81	0.77	29.03	0.80

(a) Noisy Original
PSNR:25.97dB SSIM:0.67

(b) SMoE-GD
PSNR:30.56dB SSIM:0.76

(c) SMoE-AE
PSNR:29.86dB SSIM:0.73

(d) SMoE-AE_noisy
PSNR:30.94dB SSIM:0.77

Figure 5: Test image Peppers with added speckle noise of variance $\delta^{2}=0.01$

The robustness to noise is examined and the autoencoder is trained on noisy images. Each $8\times 8$ block is overlaid with additive speckle noise of variance $\delta^{2}=0.01$ before serving as input data to the encoder network and trained to calculate the loss over the original block. By adding noise to each block individually in every training iteration, the noise pattern changes moderately in each training epoch and successfully models real noise occurrences. The encoder network can adapt to the noise pattern to model the underlying edges in images, and subsequently denoise the images. The framework needs to be trained for every noise pattern separately.
Tab. 5 compares SMoE-GD and SMoE-AE for added speckle noise with a variance of $\delta^{2}=0.01$, as well as the effect of training SMoE-AE_noisy on noisy images. SMoE is capable of performing light denoising tasks natively by creating an edge-aware model. Training the autoencoder on noise improves reconstruction quality for images with long and distinct high-contrast edges, like for the test image Cameraman or Boats. Images with a lot of high-frequency data like Baboon or Bridge do not benefit from SMoE-AE_noisy to the same extent, since the encoder produces overly smooth blocks and therefore erases primary high-frequency data. Overall on the Kodak data set the noise trained SMoE-AE_noisy outperforms SMoE-GD by almost $1$dB. In Fig. 5 the effect of additional smoothing is visible in the smooth pepper skin, which is notably more even in the reconstruction of SMoE-AE_noisy compared to SMoE-GD and SMoE-AE.
Denoising results using the S-SMoE approach are provided in Tab. 6 employing SMoE-AE_noisy, omitting the SMoE-GD comparison due to excessive run-time demands. Combining the edge-aware SMoE model with averaging over multiple hypotheses for each pixel in an image, results in great denoising improvements. For $S=1$ an improved PSNR of $1.98$dB and $0.07$ SSIM can be reported on the Kodak data set in comparison to $S=8$. Test image Kodim23 from the Kodak data set is depicted in Fig. 6(a) for $S=1$ and $S=8$ and compared to the BM3D result. (best viewed electronically and zoomed in) For $S=1$ blocking artifacts are invisible and the noise is drastically reduced. The denoising results are comparable to state-of-the-art BM3D [16] approach.

(a) Noisy Original
PSNR:26.78dB SSIM:0.53

(b) S-SMoE $S=1$
PSNR:35.12dB SSIM:0.93

(c) S-SMoE $S=8$
PSNR:32.43dB SSIM:0.84

(d) BM3D Reconstruction
PSNR:35.1dB SSIM:0.92

Figure 6: Test image Kodim23 with added speckle noise of variance $\delta^{2}=0.01$

Table 6: S-SMoE denoising results with SMoE-AE_noisy

			S-SMoE with SMoE-AEnoisy
	Speckle noise $\delta^{2}=0.1$		$S=1$		$S=4$		$S=8$
Sequence	PSNR [dB]	SSIM	PSNR [dB]	SSIM	PSNR [dB]	SSIM	PSNR [dB]	SSIM
Baboon	25.53	0.73	29.27	0.87	28.54	0.86	27.10	0.82
Boats	25.38	0.62	31.38	0.85	30.83	0.84	29.60	0.81
Bridge	26.35	0.81	27.47	0.83	27.04	0.82	26.07	0.78
Cameraman	25.64	0.56	35.15	0.93	34.33	0.92	32.60	0.90
Elaine	25.09	0.56	32.04	0.78	31.70	0.77	30.97	0.75
Lena	25.69	0.56	34.15	0.90	33.52	0.89	32.16	0.86
Livingroom	25.95	0.67	31.36	0.87	30.77	0.86	29.51	0.83
Peppers	25.97	0.61	32.89	0.92	32.46	0.83	31.49	0.81
Kodak	26.90	0.67	31.01	0.87	30.50	0.86	29.03	0.80
Encoding Time [s]	n/a		41.60		3.00		0.70
Decoding Time [s]	n/a		13.95		0.88		0.24

5 Conclusion

In this paper, we proposed an advanced autoencoder framework to predict Steered Mixture-of-Experts parameters for improving reconstruction quality for complex and noisy images, while reducing run time demands drastically compared to gradient descent optimized SMoE models. Prior autoencoder implementations were restricted to radial kernels, consequently providing narrow applicability with other SMoE applications and limiting the complexity of the reconstructed images. This work utilizes steering capabilities of kernels and reduced block size and achieved higher reconstruction qualities in both objective and subjective measures, simultaneously maintaining run-time savings in contrast to iterative gradient descent optimization. Furthermore, the robustness to noise by training the encoder network on noisy ground truth images, makes it applicable to denoising applications using SMoE models.
Further research is necessary to explore adaptive block sizes and kernel numbers, to fully benefit of SMoEs ability to model long-range correlation in images.