Interleaved Speech Language Models Latently Work In Text

This paper analyzes interleaved speech-text language models (SLMs) to understand the interaction between speech and text modalities in latent space. The study finds that these models undergo an implicit transcription phase, where spoken words are decoded as text tokens in intermediate layers, achieving a high accuracy of 77% for identifying the correct transcription. These insights into the internal mechanisms of SLMs could inform future optimization strategies for practitioners working with multimodal AI systems.