Improving generalization of vocal tract feature reconstruction: from augmented acoustic inversion to articulatory feature reconstruction without articulatory data

An autoencoder (AE) is an artificial neural network architecture that attempts to reconstruct its input through a latent representation (encoding). It consists of two parts: a mapping from the input to the latent representation (encoder, $e$), and the input reconstruction starting from the encoding (decoder, $d$).

In this approach a deep neural network with one residual layer (ResDNN) takes articulatory prior vectors $\mathbf{z}$ as input features and targets acoustic features (Figure 1). The residual layer [20] modulates the input $\mathbf{z}$ with its left and right context weighted by a learned parameter, thus returning a coarticulation-modulated version of the $\mathbf{z}$.

Formally,

the output of each i-th element of the residual layer $\mathbf{\hat{z}}_{t}$ is defined as:

$R_{t}$ is the residual at time $t$, and the $w^{R}_{sg}$’s are the learning parameters of the residual network. The sums taken over time and features model co-articulation effects. The network is trained to minimize the following loss function:

where $\mathbf{\hat{x}}$ is the reconstructed audio, $\lambda_{w}$ controls the penalization term, and $\mathbf{w}^{R}\in\mathbb{R}^{G\cdot(2T+1)}$.

All the experiments were carried out on the 47 American English speaker subset of XRMB used in [11, 12], with the only exception that we discarded speaker JW33 (used for validation in [11, 12]), as we discovered some corrupted audio (while we kept speaker JW58 which was removed in [12] and only removed some corrupted utterances).

Articulatory data consists of x-y trajectories of: upper and lower lips, $4$ tongue points, one mandible molar and one mandible incisor. For the training-testing matched condition we split the dataset into disjoint sets of 35/7/4 speakers for training/validation/testing respectively.

For the training-testing mismatched condition we split the dataset by gender. We refer to the so-obtained subsets as Male and Female , with 22 and 24 speakers respectively. For supervised methods, when Female was used as testing dataset, Male was split into 18/4 speakers for training/validation respectively. In the opposite case, Female was split into training and validation, with 19 and 5 speakers respectively.

Articulatory features were preprocessed as in [11], while acoustic features are the first 13 MFCCs, computed every 10ms from 25ms Hamming windows, plus deltas and delta-deltas. Both acoustic and articulatory features are per-speaker z-normalized.

Supervised methods are based on bidirectional LSTMs (BLSTMs). The networks have 5 layers each containing 250 memory blocks, with peephole connections and hyperbolic tangent activation function. All experiments were carried out using Adaptive Momentum Optimizer [24], a piecewise constant learning rate with initial value set to 0.1, a 0.9 momentum, $\epsilon=e^{-8}$ and initial decay rates of first and second moments 0.9 and 0.999, respectively. Weights were initialized with Xavier initialization [25]. Early stopping was applied to determine the number of training epochs.

In all weakly supervised methods, the network input consists of the central vector plus $T=12$ context vectors per side. Training was performed with stochastic gradient descent. Learning rate exponentially decayed every 10000 steps, with initial value 0.01 and 0.96 decay rate. Training was performed for 50 epochs or stopped earlier if the acoustic feature reconstruction error did not decrease.

Both AE types have a hourglass shape, symmetric w.r.t. the encoding layer. Each encoder (as well as the decoder) has 3 layers with 200, 130, 70 nodes respectively, decreasing towards the encoding layer which has $G=10$ nodes in AE1 and 39 nodes in AE2. Again we used Xavier initialization.

ResDNNs have 4 layers with 1000 nodes each, while the residual layer has $G=10$ nodes. We fixed $\lambda_{w}=0.01$ and grid-searched the remaining hyper-parameters, based on the audio reconstruction.

We evaluated all methods by computing the average (over features) root mean squared error (RMSE) and the average Pearson’s correlation coefficient ($r$) between per-speaker z-normalized reconstructed and measured AFs (so RMSE is a normalized RMSE).

In Table 1, we compare the average RMSE and correlation for PT and VTV reconstruction of different BLTSM inputs. BLTSM training and evaluation were repeated twice, with different random initialization. To keep tables more readable we only report the mean, the std. dev. is always lesser than 0.01.

Results suggest that phonological features (phone labels, LFs and SFs) can outperform MFCCs, and, surprisingly, MFCCs slightly improve reconstruction when combined with phonological features, despite MFCCs containing much more detailed information than phonological features. LFs and SFs do not produce relevant improvement w.r.t. phone labels. Table 1 also shows AI results when only one speaker is used for training in order to quantify the gap w.r.t. to multi-speaker training data and to compare with weakly supervised methods in a limited articulatory data setting.

Results of weakly supervised methods in the matching conditions setting are summarized in Table 2. We compare them with the Baseline case, where the phonological features are directly compared with measured AFs. Again, all experiments were carried out twice (std. dev. $<$ 0.01). Although LFs and SFs have a similar number of quantization levels, SFs largely outperform LFs in all methods. Most importantly, the generated AFs $\mathbf{\hat{z}}$ always correlate more with actual AFs than the priors $\mathbf{z}$, with the exception of method AE1. That means that AE2 and ResDNN successfully transform the original prior articulatory information into articulatory features that are closer to the actual AFs. AE2 is the most effective method.

To show that SFs well generalize across speakers, we re-computed the SFs based on only one or two training speakers (SF1s and SF2s) and repeated the weakly supervised experiments. Interestingly, results obtained with SF1s and SF2s do not significantly differ from SFs. This implies that the statistical representations calculated on few speakers (or just one!) well characterize the vocal tract of any other speaker. Importantly, in this limited data setting, ResDNN and AE2 outperform the best supervised method (e.g., $r=0.537$ and $r=0.563$ vs. $r=0.517$). Note that Table 2 shows the best AE1 and AE2 performances on the validation set, achieved by fixing $\lambda_{z}$ and $\lambda_{x}$ at 2 and 0.5, respectively. We did not report the RMSE for the LFs, as they do not reflect the real measurements of the articulatory data. More detailed results can be found in Table 3, where the best AE2 performance is reported for two test speakers and for each VTV.

Table 4 shows the results of the supervised methods in the training-testing mismatched conditions. The most striking result is that MFCCs not only perform significantly worse than SFs but even worsen SFs performance when combined with them. This is due to the strong speaker dependency of MFCCs (despite their per-speaker normalization), that may be alleviated through speaker adaptation.

Regarding weakly supervised methods, in this case AE2 only, we computed SFs on one set and used them as prior information for training AE2 on the other dataset. Note that, in this case, AE2 is trained and tested on the same speakers (e.g., Female), while priors are computed on other speakers (e.g., Male). Indeed, since AE2 is only trained on acoustic data, which are always available, there is no need to generalize to new speakers. Results in Table 5 show that (i) AE2 almost matches the supervised method with MFCC; (ii) even in the mismatched case, AE2 reconstruction is not affected by a reduction of articulatory data to a single speaker.