Speaker Identity in Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts Approach

The article presents a novel framework for speaker verification (SV) that addresses the challenges posed by non-verbal vocalizations (NVVs) in text-to-speech (TTS) and voice conversion (VC) systems. It combines frozen Data2Vec features with ECAPA-TDNN architecture and incorporates a Mixture of Experts (MoE) module for domain-aware routing. The approach achieves a significant reduction in equal error rate (EER) for both NVVs and speech, improving from 38.93% to 22.66% for NVVs and from 13.17% to 9.24% for speech, demonstrating its effectiveness for practitioners working with multimodal audio data.