GAMIVAL: Video Quality Prediction on Mobile Cloud Gaming Content

Previous spatial-temporal bandpass statistics-based video quality models, like RAPIQUE [10], have been shown to efficiently capture the perceptual impacts of complex real-world distortions. In these models, bandpass and divisive normalization processes yield mean-subtracted, contrast-normalized (MSCN) coefficients applied on the input image (or on a previously computed feature map) $I(i,j)$:

where $(i,j)$ are spatial indices, $C$ = 1 is a saturation constant that prevents instabilities, and $\mu$ and $\sigma$ are weighted local means and standard deviations [2, 3] within a gaussian-weighted spatial window centered at location $(i,j)$.

It has been widely observed that the bandpass MSCN coefficients of natural image or video frames reveal an underlying statistical regularity. However, visual contents rendered by computer graphics, like gaming videos, typically contain fewer details, and are generally smoother, hence their bandpass statistics differ from those of naturalistic videos or images. Feature computations on these regions can be less stable. Following [30], we introduce a neural noise model:

by which we add white Gaussian noise $W_{s}\sim N(0,\sigma^{2}_{W_{s}})$ to the image before computing the MSCN coefficients.

As in [30], we have observed that the distributions of the MSCN coefficients of high quality gaming content video frames tends towards Gaussianity. To visualize this, we selected one natural image from the LIVE IQA database [29]: building2, and two pristine video frames (720p) from two games in the LIVE-Meta MCG database [24]: Plants vs. Zombies and Sonic, as shown in Fig. 2. Fig. 3 top portion shows the histograms of the MSCN coefficients of each of these images before and after adding the simulated neural noise ($\sigma_{W_{s}}$=1.5). It may be observed that histograms of the gaming video frames contain singular spikes, but after the noise was added to the gaming video frames, their MSCN histograms became very similar to those of the natural image, with a Gaussian appearance. Another gaming frame (State of Survival) was chosen to further visualize how these regularities are affected by distortion. Fig. 3 lower portion shows the MSCN coefficients of this frames after resampling to different resolutions and bitrates. Quantifying these deviations to predict perceived quality is effectively accomplished using the basic statistical feature extraction module NSS-34 in RAPIQUE [10]. We apply this module to the luma and chromatic feature maps at two scales (the original scale resized to 540p and 270p) as in RAPIQUE [10], by uniformly sampling 2 frames per second, thus producing 680 spatial features.

Since temporal bandpass statistics have been shown to be predictive of frame rate-dependent video quality [31], the authors of RAPIQUE [10] proposed a temporal model utilizing temporal bandpass coefficients and subsequentially applying spatial MSCN transforms, as in Eq. (1). The MSCN coefficients of temporal bandpass coefficients of gaming videos also exhibit discontinuous histograms, as shown in Fig. 4 top portion. Hence again apply additive neural noise before computing the spatial MSCN coefficients

as before, where ${Y_{k}}(x,t)$ are the temporal bandpass coefficients, $k=1,...,7$ denotes subband indices, $\textbf{x}=(x,y)$ and t are spatial and temporal coordinates, and $W_{t}\sim N(0,\sigma^{2}_{W_{t}})$ is the noise added to the temporal model.

As shown in Fig. 4 lower portion, after adding the noise ($\sigma_{W_{t}}$=1.5), the histograms of the MSCN coefficients of temporal bandpass coefficients of gaming videos also present Gaussian appearances. These regularities are modified by the presence of distortions, which provides a way of quantifying deviations, and therefore, to predict video quality scores. Towards this end, we deploy the NSS-34 operator set in RAPIQUE [10] on the temporal bandpass coefficients of each analyzed gaming video at two scales (the original scale resized to 540p and 270p), sampling at 8 frames per second and then applying a temporal Haar filter as in [10] to extract 7 bandpass responses, yielding 34$\times$7$\times$2 = 476 temporal features at each time sample. Section V-A provides additional information on the selection of the parameters $\sigma_{W_{s}}$ and $\sigma_{W_{t}}$ of the additive noise elements ${W_{s}}$ and ${W_{t}}$ in equations (2) and (3), respectively. It is also shown that model performance significantly improves by the addition of noise, and remains robust over even orders of magnitude of the parameters.

Since existing gaming video content datasets contain too few videos to allow training of CNN feature extractors from scratch, we instead applied NDNetGaming model [14]. Unlike the CNN branch in RAPIQUE [10], (a pre-trained ResNet-50 model [32]), the authors of [14] instead fine-tuned a DenseNet-121 model [33]) in two steps: first, they retrained the last 57 convolutional layers of a DenseNet-121 by replacing the fully connected (FC) layer with a dense layer, using VMAF values as proxy subjective quality labels. Second, they fine-tuned the resulting pre-trained model by retraining the last 36 layers based on human subjective image quality ratings. Finally, they assigned a different weight to each frame, based on its temporal complexity. Without being retrained on different cloud game video databases, this model might not deliver robust and generalized performance. The authors of [34] showed that, without fine-tuning, the simple feature vector of an FC layer could be a useful quality indicator if a shallow regressor is trained on top. Therefore, we discarded the pooling layer of the original NDNetGaming model, thereby yielding 1024 activation features. By also taking into account the time complexity, we also devised a temporal sampling strategy. Instead of extracting features on every frame, the CNN backbone (NDNetGaming) operates at 2 frame per second.

After obtaining all of the spatial, temporal, and CNN-based features within each one-second chunk, they are concatenated into a 2180-dimensional feature vector. By average-pooling the vectors within each video chunk, a vector of features is obtained for the entire video and then trained with a shallow or deep regressor head. In our implementation, we used a support vector machine regressor (SVR) to map the features to predicted video quality scores [3, 7, 6, 35].

To evaluate the performance of our model against other methods, we used the recently created LIVE-Meta MCG database [24]. This database contains mean opinion scores (MOS) of 600 landscape and portrait gaming videos generated from 30 pristine source sequences obtained from 16 different games using 20 different resolution-bitrate pairs to compress each pristine video. The resolution ranged from 360p to 720p, while the bitrates range from 250 kbps to 2 mbps.

The subjective human study was conducted on a Google Pixel 5 mobile device. All of the videos were upscaled to fit the mobile screen size (1080p) using FFMPEG’s default bicubic interpolation function following decoding to native videos. To map features to the MOS that was obtained from users viewing a 1080p Pixel 5 display, we applied all of the compared algorithms on the display resolution.

Table I shows the model performances on the LIVE-Meta MCG database. It may be observerd that GAMIVAL achieved the best performance, while VSFA and RAPIQUE ranked second and third respectively. It is worth noting that the two gaming VQA models (NDNet-Gaming and GAME-VQP) also yielding fairly good fair performance. For better visualization, Fig. 5 shows box plots of the SRCC, KRCC, and PLCC values. It may be noticed that the GAMIVAL values are more tightly grouped, indicating its stable performance and low variance.

Section V-B includes an ablation study that analyzes the contributions of each GAMIVAL feature set. We further summarize the complexity and runtime comparison of the NR-VQA models in Section V-C.

To study the parameters $\sigma_{W_{s}}$ and $\sigma_{W_{t}}$ of the noise elements $W_{s}$ and $W_{t}$ in (2) and (3), we evaluted GAMIVAL on the LIVE-Meta MCG Database while varying these parameters. As may be seen from Table II, the performance of GAMIVAL was remarkably robust over a very wide range of the two noise parameters, which we held equal, even over several orders of magnitude. We selected the best-performing values $\sigma_{W_{s}}=\sigma_{W_{t}}=1.5$, but GAMIVAL significantly outperformed all other models for all nonzero parameter values.

To analyze the contribution of each GAMIVAL feature set, we conducted an ablation study. Fig. 7 shows the increase in performance obtained as each feature combination is added. It may be observed that the temporal “additive noise” features improved performance more than did the spatial “additive noise” features. The NDNetGaming component contributes significantly to the performance, likely because it adds semantic significance to the quality predictions.

The experiments were performed in MATLAB R2021a and Python 3.6.10 under Ubuntu 20.04.4 LTS on a Desktop with an Intel Xeon CPU E5-2620 v4@2.10GHz processor, and 64GB RAM. For a fair comparison, all algorithms were carried out on the CPU. We used one 1080x2160 video from the LIVE-Meta MCG database to study the computational efficiency of the NR-VQA models. Table III lists the execution time and floating point operations, while Fig. 8 shows visual representations of the trade-off between performance and complexity. It may be observed that although VSFA delivers very good performance, it has the highest computational complexity. The top-performing algorithms, RAPIQUE and GAMIVAL, are both computationally efficient, making them promising options for real-time video quality prediction applications.