MALEFA: Multi-grAnularity Learning and Effective False Alarm Suppression for Zero-Shot Keyword Spotting

As schematically depicted in Fig. 2, MALEFA employs a two-stream encoder with separate audio and text encoders. Both audio and text modalities are processed independently and later aligned in the pattern extractor.

As illustrated in Fig. 2, the pattern extractor employs cross-attention to align audio and text embeddings. The text embedding $\mathbf{E}_{t}$ serves as the query ($Q$), while the audio embedding $\mathbf{E}_{a}$ provides both keys and values ($K,V$). This allows each phoneme to attend to the most relevant audio frames, yielding a joint representation:

$$\mathbf{E}_{\text{joint}}=\text{CrossAttention}(Q=\mathbf{E}_{t},K=\mathbf{E}_{a},V=\mathbf{E}_{a}),$$

(1)

where $\mathbf{E}_{a}\in\mathbb{R}^{T_{a}\times 128}$ and $\mathbf{E}_{t}\in\mathbb{R}^{T_{t}\times 128}$.

The joint embedding $\mathbf{E}_{\text{joint}}$ is passed through a Gated Recurrent Unit (GRU), followed by two classification heads. One head predicts utterance-level matching probability score between audio and text, denoted as $q_{\text{utt}}$. While the other operates on temporal segments of $\mathbf{E}_{\text{joint}}$ to capture phoneme-level alignment sequence $\mathbf{q}_{\text{phon}}$.

Phoneme-level Contrastive Learning (PCL). The audio encoder outputs frame-level CTC logits $\mathbf{z}\in\mathbb{R}^{T_{a}\times V}$, supervised by the standard CTC loss [21]:

$$\mathcal{L}_{\text{CTC}}=-\log q_{\text{CTC}}(\mathbf{y}\mid\mathbf{z}),$$

(2)

where $q_{\text{CTC}}$ marginalizes over valid frame–phoneme alignments. With the aid of Viterbi decoding, we obtain alignment confidences $s_{i}$ for each audio-text pair. The corresponding PCL loss is

$$\mathcal{L}_{\text{PCL}}=\tfrac{1}{N}\sum_{i=1}^{N}\big[m_{i}(1-s_{i})^{2}+(1-m_{i})s_{i}^{2}\big],$$

(3)

where $m_{i}\in\{0,1\}$ indicates whether the $i$-th pair is matched. This encourages high alignment confidence for positives and penalizes spurious overlaps instead.

False alarms (FA), i.e., false keyword detections on non-target audio, remain a major challenge facing KWS. Conventional BCE training maximizes overall accuracy but does not explicitly penalize FA, often requiring post-hoc threshold tuning. We therefore introduce a precision-constrained objective:

$$\mathcal{L}_{\text{FA}}=-\log(\text{Precision})+\lambda\cdot\max(0,\alpha-\text{Precision}),$$

(6)

where the first term discourages low-precision predictions and the second enforces a margin constraint if precision falls below $\alpha$ (scaled by $\lambda$)  [22]. For differentiability, true positives (TP) and false positives (FP) are approximated by smooth sigmoid bounds:

	TP	$\displaystyle=\sum(1+\gamma\delta)\,\sigma(\gamma x-\delta)\,x_{\text{true}},$		(7)
	FP	$\displaystyle=\sum(1+\gamma\delta)\,\sigma(\gamma x+\delta)\,(1-x_{\text{true}}),$		(8)

with $x_{\text{true}}\in\{0,1\}$, sigmoid $\sigma(\cdot)$, steepness $\gamma=7.0$, and offset $\delta=0.035$. We set $\alpha=0.9$ and $\lambda=10.0$. This auxiliary loss is combined with BCE to improve FA suppression during training.

Both utterance- and phoneme-level predictions are supervised using BCE losses, denoted by $\mathcal{L}_{\text{utt}}$ and $\mathcal{L}_{\text{phon}}$, respectively. The overall training objective combines all loss terms:

$$\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{utt}}+\mathcal{L}_{\text{phon}}+\mathcal{L}_{\text{CTC}}+\mathcal{L}_{\text{PCL}}+\mathcal{L}_{\text{UCL}}+\mathcal{L}_{\text{FA}},$$

(9)

where $\mathcal{L}_{\text{CTC}}$ (Eq. 2), $\mathcal{L}_{\text{PCL}}$ (Eq. 3), $\mathcal{L}_{\text{UCL}}$ (Eq. 4), and $\mathcal{L}_{\text{FA}}$ (Eq. 6) correspond to the CTC, phoneme-level contrastive, utterance-level, and FA-aware losses, respectively, all of which are assigned an equal weight of 1. To maintain the focus of this study, the exploration of alternative weighting strategies is left beyond the scope of this work.

We use the LibriPhrase train-clean-100 and train-clean-360 sets with MUSAN noise [23] for training. Evaluation is conducted on four benchmarks: LibriPhrase Easy/Hard (L_E/L_H) from train-other-500 (low/high phonetic confusion), Google Speech Commands V2 (G) [24] (35 commands under diverse conditions), Qualcomm Keyword Speech (Q) [25] (accented/domain-specific keywords), and AMI [26] (12h meeting recordings segmented into 2s clips for FA evaluation).

We employ Google speech embeddings [19] as the pre-trained audio encoder. All models are trained for 50 epochs using Adam with a fixed learning rate of $10^{-3}$, batch size $N=1000$, and mini-batch size $M=5$ for UCL (Section 2.4). Experiments are conducted on an NVIDIA RTX 4090 GPU using TensorFlow.

Table 1 compares MALEFA with prior ZSKWS models and presents an ablation study. While CED [12] achieves strong accuracy, its Conformer-based encoder [13] incurs much higher complexity, limiting on-device usage. Compared with PhonMatchNet [8], on LP_H, it suffers a significant drop (AUC $=88.52$, EER $=18.82$), whereas Ours maintains higher robustness (AUC $=93.58$, EER $=13.91$). Ablation results show that removing FA-aware loss (w/o FA) sharply increases EER, excluding UCL degrades robustness to phonetic ambiguities, and discarding PCL reduces fine-grained alignment on LP_H. In contrast, the full MALEFA, integrating all three components, achieving state-of-the-art performance across benchmarks with only 0.7M parameters. This confirms that FA-aware loss, UCL and PCL are jointly essential for reliable ZSKWS.

Table 2 reports the FAR across test sets. Compared with [8], which suffers from high FARs (17.9% on AMI), our MALEFA reduces FAR to below 0.01% on all benchmarks. Ablation further confirms the contribution of each component: removing FA-aware loss (w/o FA) causes the largest degradation (14.5% on AMI), excluding UCL increases FAR to 1.3%, and discarding PCL still yields higher FAR (0.085% on AMI). Overall, the complete MALEFA achieves the lowest FAR across all datasets, highlighting that PCL, UCL, and FA-aware learning are complementary and jointly indispensable.

To illustrate the impact of contrastive objectives, we visualize cosine similarity matrices for five representative keywords from the G dataset in Fig. 3. The baseline model (Original) exhibits high similarity not only on correct matches but also across confusable pairs (e.g., “bed” vs. “three”), indicating risk of FA. Introducing UCL enforces stronger inter-class separation, effectively suppressing non-matching similarities and yielding cleaner diagonal patterns. Further adding PCL sharpens the alignment, driving non-matching scores close to zero while preserving near-perfect self-matches. These results qualitatively confirm that UCL improves global discrimination and PCL complements it with fine-grained alignment, together enhancing robustness against phonetically similar triggers.

Figure 4 compares phoneme-to-frame attention maps for the keyword “hey android” with and without PCL. Without PCL, the attention is diffused, yielding imprecise phonetic boundaries that may cause false alarm. With PCL, alignments become sharper and more localized, indicating that the model learns more discriminative frame-level representations and improves robustness against acoustically similar distractors.