Speaker Adaptation Using Spectro-Temporal Deep Features for Dysarthric and Elderly Speech RecognitionThanks: Mengzhe Geng, Zi Ye, Tianzi Wang, Guinan Li and Shujie Hu are with the Chinese University of Hong Kong, China (email: {mzgeng,zye,twang,gnli,sjhu}@se.cuhk.edu.hk).
Xurong Xie is with Institute of Software, Chinese Academy of Sciences, Beijing, China (email: xurong@iscas.ac.cn).
Xunying Liu is with the Chinese University of Hong Kong, China and the corresponding author (email: xyliu@se.cuhk.edu.hk).
Helen Meng is with the Chinese University of Hong Kong, China (email: hmmeng@se.cuhk.edu.hk).

Speaker adaptation techniques adopted by current deep neural networks (DNNs) based ASR systems targeting normal speech can be divided into three major categories: 1) auxiliary speaker embedding feature based methods that represent speaker dependent (SD) features via compact vectors [55, 56, 57, 58], 2) feature transformation based approaches that produce speaker independent (SI) canonical features at the acoustic front-ends [59, 60, 61, 62, 63] and 3) model based adaptation techniques that compensate the speaker-level variability by often incorporating additional SD transformations that are applied to DNN parameters or hidden layer outputs [64, 65, 66, 67, 68, 69].

In the auxiliary speaker embedding feature based approaches, speaker dependent (SD) features such as speaker codes [55] and i-Vectors [56, 57] are concatenated with acoustic features to facilitate speaker adaptation during both ASR system training and evaluation. The estimation of SD auxiliary features can be performed independently of the remaining recognition system components. For example, i-Vectors [56, 57] are learned from Gaussian mixture model (GMM) based universal background models (UBMs). The SD auxiliary features can also be jointly estimated with the back-end acoustic models, for example, via an alternating update between them and the remaining SI DNN parameters in speaker codes [55]. Auxiliary feature based speaker adaptation methods benefit from both their low complexity in terms of the small number of SD feature parameters to be estimated, and their flexibility allowing them to be incorporated into a wide range of ASR systems including both hybrid DNN-HMM systems and recent end-to-end approaches [70].

In feature transformation based speaker adaptation, feature transforms are applied to acoustic front-ends to produce canonical, speaker invariant inputs. These are then fed into the back-end DNN based ASR systems to model the remaining sources of variability, for example, phonetic and phonological context dependency in speech. Feature-space maximum likelihood linear regression (f-MLLR) transforms [63] estimated at speaker-level from GMM-HMM based ASR systems [59, 61] are commonly used. In order to account for the vocal tract length differences between speakers, physiologically motivated vocal tract length normalization (VTLN) can also be used as feature transformation [60, 62]. Speaker-level VTLN normalized features can be obtained using either piecewise linear frequency warping factors directly applied to the spectrum, or affine linear transformations akin to f-MLLR.

In model based adaptation approaches, separately designed speaker-dependent DNN model parameters are used to account for speaker-level variability. In order to ensure good generalization and reduce the risk of overfitting to limited speaker-level data, a particular focus of prior researches has been on deriving compact forms of SD parameter representations. These are largely based on linear transforms that are incorporated into various parts of DNN acoustic models. These include the use of SD linear input networks (LIN) [59, 67], linear output networks (LON) [21], linear hidden networks (LHN) [66], learning hidden unit contributions (LHUC) [71, 68, 69], parameterized activation functions (PAct) [72, 69], speaker-independent (SI) and SD factored affine transformations [73], and adaptive interpolation of outputs of basis sub-networks [74, 75]. In addition to only modelling speaker-level variability in the test data during recognition, the estimation of SD parameters in both the system training and evaluation stages leads to more powerful speaker adaptive training (SAT) [65] approaches, allowing a joint optimization of both the SD and SI parameters during system training.

In contrast, only limited research on speaker adaptation techniques targeting dysarthric and elderly speech recognition has been conducted so far. Earlier works in this direction were mainly conducted in the context of traditional GMM-HMM acoustic models. The application of maximum likelihood linear regression (MLLR) and maximum a posterior (MAP) adaptation to such systems were investigated in [76, 77, 9, 78]. MLLR was further combined with MAP adaptation in speaker adaptive training (SAT) of SI GMM-HMM in [11]. F-MLLR based SAT training of GMM-HMM systems was investigated in [79]. Regularized speaker adaptation using Kullback-Leibler (KL) divergence was studied for GMM-HMM systems in [80].

More recent researches applied model adaptation techniques to a range of state-of-the-art DNN based dysarthric and elderly speech recognition systems. Normal to dysarthric speech domain adaptation approaches using direct parameter fine-tuning were investigated in both lattice-free maximum mutual information (LF-MMI) trained time delay neural network (TDNN) [40, 43] based hybrid ASR systems and end-to-end recurrent neural network transducer (RNN-T) [36, 48] systems. In order to mitigate the risk of overfitting to limited speaker-level data during model based adaptation, more compact learning hidden unit contributions (LHUC) based dysarthric speaker adaptation was studied in [12, 37, 41] while Bayesian learning inspired domain speaker adaptation approaches have also been proposed in [81].

One main issue associated with previous researches on dysarthric and elderly speaker adaptation is that the systematic, fine-grained speaker-level diversity attributed to speech impairment severity and aging is not considered. Such diversity systematically manifests itself in a range of spectro-temporal characteristics including articulatory imprecision, decreased volume and clarity, breathy and hoarse voice, increased dysfluencies as well as slower speaking rate.

In order to address this issue, novel deep spectro-temporal embedding features are proposed in this paper to facilitate auxiliary speaker embedding feature based adaptation for dysarthric and elderly speech recognition. Spectral and temporal basis vectors derived by singular value decomposition (SVD) of dysarthric or elderly speech spectra were used to structurally and intuitively represent the spectro-temporal key attributes found in such data, for example, an overall decrease in speaking rate and speech volume as well as changes in the spectral envelope. These two sets of basis vectors were then used to construct DNN based speech pathology severity or age classifiers. More compact, lower dimensional speaker specific spectral and temporal embedding features were then extracted from the resulting DNN classifiers’ bottleneck layers, before being further utilized as auxiliary speaker embedding features to adapt start-of-the-art hybrid DNN [41], hybrid TDNN [3] and end-to-end (E2E) Conformer [6] ASR systems.

Experiments were conducted on four tasks: the English UASpeech [82] and TORGO [83] dysarthric speech corpora; the English DementiaBank Pitt [84] and Cantonese JCCOCC MoCA [85] elderly speech datasets. Among these, UASpeech is by far the largest available and widely used dysarthric speech database, while DementiaBank Pitt is the largest publicly available elderly speech corpus. The proposed spectro-temporal deep feature adapted systems outperformed baseline i-Vector [56] and x-Vector [86] adapted systems by up to $2.63\%$ absolute ($8.63\%$ relative) reduction in word error rate (WER). Consistent performance improvements were retained after model based speaker adaptation using learning hidden unit contributions (LHUC) was further applied. The best speaker adapted system using the proposed spectral basis embedding features produced the lowest published WER of $25.05\%$ on the UASpeech test set of $16$ dysarthric speakers. Speech pathology severity and age prediction performance as well as further visualization using t-distributed stochastic neighbor embedding (t-SNE) [87] indicate that our proposed spectro-temporal deep features can more effectively learn the speaker-level variability attributed to speech impairment severity and age than conventional i-Vector [56] and x-Vector [86]. The main contributions of this paper are summarized below:

1) To the best of our knowledge, this paper presents the first use of spectro-temporal deep embedding features to facilitate speaker adaptation for dysarthric and elderly speech recognition. In contrast, there were no prior researches published to date on auxiliary features based speaker adaptation targeting such data. Existing speaker adaptation methods for dysarthric and elderly speech use mainly model based approaches [76, 77, 9, 78, 79, 11, 12, 36, 40, 37]. Speaker embedding features were previously only studied for speech impairment assessment [88, 89, 90].

2) The proposed spectro-temporal deep features are inspired and intuitively related to the latent variability of dysarthric and elderly speech. The spectral basis embedding features are designed to learn characteristics such as volume reduction, changes of spectral envelope, imprecise articulation as well as breathy and hoarse voice, while the temporal basis embedding features to capture patterns such as increased dysfluencies and pauses. The resulting fine-grained, factorized representation of diverse impaired speech characteristics serves to facilitate more powerful personalized user adaptation for dysarthric and elderly speech recognition.

3) The proposed spectro-temporal deep feature adapted systems achieve statistically significant performance improvements over baseline i-Vector or x-Vector adapted hybrid DNN/TDNN and end-to-end (E2E) Conformer systems by up to $2.63\%$ absolute ($8.63\%$ relative) word error rate (WER) reduction on four dysarthric or elderly speech recognition tasks across two languages. These findings serve to demonstrate the efficacy and genericity of our proposed spectro-temporal deep features for dysarthric and elderly speaker adaptation.

The rest of this paper is organized as follows. The derivation of spectro-temporal basis vectors using SVD speech spectrum decomposition is presented in Section II. The extraction of spectro-temporal deep embedding features and their incorporation into hybrid DNN/TDNN and end-to-end Conformer based ASR systems for speaker adaptation are proposed in Section III. A set of implementation issues affecting the learning of spectro-temporal deep embedding features are discussed in Section IV. Section V presents the experimental results and analysis. Section VI draws the conclusion and discusses possible future works.

When training the speech impairment severity or age classification DNNs to extract deep spectro-temporoal embedding features, the top-$d$ principal spectral or temporal basis are used as input features to train the respective DNNs sharing the same model architecture shown in Fig. 3, where either speech pathology severity based on, for example, the speech intelligibility metrics provided by the UASpeech [82] corpus, or the binary aged v.s. non-aged speaker annotation of the DementiaBank Pitt [84] dataset, are used as the output targets.

The DNN classifier architecture is a fully-connected neural network containing four hidden layers, the first three of which are of $2000$ dimensions, while the last layer contains $25$ dimensions. Each of these hidden layers contains a set of neural operations performed in sequence. These include affine transformation (in green), rectified linear unit (ReLU) activation (in yellow) and batch normalization (in orange), while the outputs of the first layer are connected to those of the third layer via a skip connection. Linear bottleneck projection (in light green) is also applied to the inputs of the middle two hidden layers while dropout operation (in grey) is used on the outputs of the first three hidden layers. Softmax activation (in dark green) is used in the last layer. Further fine-grained speaker-level information can be incorporated into the training cost via a multitask learning (MTL) [74] interpolation between the cross-entropy over speech intelligibility level or age, and that computed over speaker IDs. The outputs of the $25$-dimensional bottleneck (BTN) layer are extracted as compact neural embedding representations of the spectral or temporal basis vectors (bottom right in Fig. 3).

When training the DNN speech impairment severity or age classifier using the SVD temporal basis vectors as the input, a frame-level sliding window of $25$ dimensions was applied to the top-$d$ selected temporal basis vectors. Their corresponding $25$-dimensional mean and standard deviation vectors were then computed to serve as the “average” temporal basis representations of fixed dimensionality. This within utterance windowed averaging of temporal basis vectors allows dysarthric or elderly speakers who speak of different word contents but exhibit similar patterns of temporal context characteristics such as slower speaking rate and increased pauses to be mapped consistently to the same speech impairment severity or age label. This flexible design is in contrast to conventional speech intelligibility assessment approaches that often require the contents spoken by different speakers to be the same [96, 97, 90]. It not only facilitates a more practical speech pathology assessment scheme to be applied to unrestricted speech contents of unknown duration, but also the extraction of fixed size temporal embedding features for ASR system adaptation.

The speaker-level speech impairment severity or age information can be then captured by the resulting DNN embedding features. For example, visualization using t-distributed stochastic neighbour embedding (t-SNE) [87] reveals the speaker-level spectral basis neural embedding features averaged over those obtained over all utterances of the same non-aged clinical investigator (in red) or elderly participant (in green) of the DementiaBank Pitt [84] corpus shown in Fig. 4c demonstrate much clearer age discrimination than the comparable speaker-level i-Vectors and x-Vectors shown in Fig 4a and Fig. 4b respectively. Similar trends can also be found on the Cantonese JCCOCC MoCA [85] corpus designed by a similar data collection protocol based on neuro-physiological interviews comparable to the English DementiaBank Pitt corpus¹¹ 1 Due to the relatively smaller number of speakers included in the UASpeech [82] and TORGO [83] dysarthric speech corpora (29 and 15 respectively), t-SNE visualization of spectral embedding features are performed on the English DementiaBank Pitt and Cantonese JCCOCC MoCA datasets containing $688$ and $369$ speakers each..

The compact $25$-dimensional spectral and temporal basis embedding features extracted from the DNN speech impairment severity or age classifiers’ bottleneck layers presented above in Section III-A are concatenated to the acoustic features at the front-end to facilitate auxiliary feature based speaker adaptation of state-of-the-art ASR systems based on hybrid DNN [41], hybrid lattice-free maximum mutual information (LF-MMI) trained time delay neural network (TDNN) [3] or end-to-end (E2E) Conformer models [6], as shown in Fig 5. For hybrid DNN and TDNN systems, model based adaptation using learning unit contributions (LHUC)  [68] can optionally be further applied on top of auxiliary feature based speaker adaptation, as shown in Fig. 5a and Fig. 5b respectively.

In the two dysarthric speech corpora, speech pathology assessment measures are provided for each speaker. In the UASpeech data, the speakers are divided into several speech intelligibility subgroups: “very low”, “low”, “mid” and “high” [82]. In the TORGO corpus, speech impairment severity measures based on “severe”, “moderate” and “mild” are provided [83]. In the two elderly speech corpora, the role of each speaker during neuro-physiological interview for cognitive impairment assessment is annotated. Each interview is based on a two-speaker conversation involving a non-aged investigator and another aged, elderly participant [84, 85].

By default, the speech intelligibility metrics provided by the UASpeech corpus, or the binary aged v.s. non-aged speaker annotation of the DementiaBank Pitt dataset, are used as the output targets in the following ablation study over embedding target choices. In order to incorporate further speaker-level information, a multitask learning (MTL) [74] style cost function featuring interpolation between the cross-entropy error computed over the speech intelligibility level or age labels, and that computed over speaker IDs can be used.

As is shown in the results obtained on the UASpeech [82] data in Table I, using both the speech intelligibility and speaker ID labels as the embedding targets in multi-task training produced lower word error rates (WERs) across all severity subgroups than using speech intelligibility output targets only (Sys.7 v.s. Sys.6 in Table I). The results obtained on the DementiaBank Pitt [84] data in Table II suggest that there is no additional benefit in adding the speaker information during the embedding process (Sys.7 v.s. Sys.6 in Table II). Based on these trends, in the main experiments of the following Section V, the embedding network output targets exclusively use both speech severity measures and speaker IDs on the UASpeech and TORGO [83] dysarthric speech datasets, while only binary aged v.s. non-aged labels are used on the DementiaBank Pitt and Cantonese JCCOCC MoCA [85] elderly speech datasets.

For auxiliary feature based adaptation techniques including the spectral and temporal basis deep embedding representations considered in this paper, it is vital to ensure the speaker-level homogeneity to be consistently encoded in these features. As both forms of embedding features are computed on individual utterances, additional smoothing is required to ensure such homogeneity, for example, an overall reduction of speech volume of a dysarthric or elderly speaker’s data, to be consistently retained in the resulting speaker embedding representations. To this end, two forms of speaker embedding smoothing are considered in this paper. The first is based on a simple averaging of all utterance-level spectral or temporal embedding features for each speaker. The second smoothing method is based on Latent Dirichlet allocation (LDA) [98] based clustering of utterance-level spectral or temporal embedding features. Following earlier researches [99], a $100$-component Gaussian Mixture Model (GMM) is trained first to quantize the utterance-level spectral or temporal embedding features of the same speaker, before LDA clustering is applied to produce speaker-level features of varying dimensions ranging from $10$, $25$ to $50$.

A general trend observed in the results of both Table I and II is that using spectral embedding feature smoothing, whether via a simple speaker-level averaging (Sys.6 in Table I and II) or LDA clustering (Sys.3-5 in Table I and II), produced better performance than directly using non-smoothed spectral embedding features (Sys.2 in Table I and II). Across both the UASpeech and DementiaBank Pitt tasks, the simpler speaker-level averaging based smoothing (Sys.6 in Table I and II) consistently outperform LDA clustering (Sys.3-5 in Table I and II), and is subsequently used in all experiments of the following Section V.

In this part of the ablation study on implementation details, the effect of the number of principal spectral or temporal basis vectors on system complexity and performance is analyzed. Consider selecting the top-$d$ principal SVD spectral and temporal basis components, the input feature dimensionality of the spectral basis embedding (SBE) DNN network is then expressed as $40\times d$, for example, $80$ dimensions when $d=2$. The temporal basis embedding (TBE) network is $50\times d$ including both the $25$ dimensional mean and the $25$ dimensional standard deviation vectors both computed over a frame-level sliding window of $25$ dimensions for each of the selected top-$d$ principal temporal basis vector, for example, $250$ dimensions when $d=5$. The input dimensionality of the comparable spectro-temporal basis embedding (STBE) network modelling both forms of bases is then $40\times d_{s}+50\times d_{t}$, if further allowing the number of principal spectral components $d_{s}$ and that of the temporal components $d_{t}$ to be separately adjusted.

In the experiments of this section, $d_{s}$ and $d_{t}$ are empirically adjusted to be $2$ and $5$ for dysarthric speech (Sys.2-9 in Table I) while $4$ and $10$ for elderly speech (Sys.2-9 in Table II). These settings were found to produce the best adaptation performance when the corresponding set of top principal spectral or temporal basis vectors were used to produce the speaker embedding features. For example, as the results shown in both Table I and II for the UASpeech and DemmentiaBank Pitt datasets, varying the number of principal spectral components from $1$ to $40$ (the corresponding input feature dimensionality ranging from $40$ to $1600$, Sys.10-16 in Table I and II) suggests the optimal number of spectral basis vectors is generally set to be $2$ for the dysarthric speech data (Sys.7 in Table I) and $4$ for the elderly speech data (Sys.6 in Table II) when considering both word error rate (WER) and model complexity.