Audio-visual Contrastive Alignment for Diffusion-based Visual-conditioned Speech Enhancement

The paper presents a novel approach to audio-visual speech enhancement (AVSE) by integrating a contrastive audio-visual loss into a diffusion-based model that utilizes cross-attention for visual conditioning. This method enhances the model's ability to leverage visual cues, resulting in improved interference suppression and signal reconstruction, particularly in low signal-to-noise ratio (SNR) scenarios. The findings are significant for practitioners as they demonstrate a method to enhance speech recovery in challenging auditory environments, with the code made available for further exploration.