Repurposing a Speech Classifier for Guided Diffusion-Based Speech Generation

The paper introduces a method for repurposing a conventional speech classifier to serve as the backbone for guided diffusion-based speech generation, eliminating the need for a separate classifier and diffusion model. By utilizing a frozen noise-conditioned classifier in log-Mel space and attaching a lightweight subnetwork trained under a Denoising Score Matching objective, the approach achieves high speech quality while significantly reducing memory and computational costs. This advancement is significant for practitioners as it streamlines the model architecture for conditional speech synthesis, enhancing efficiency without compromising performance.