Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering

The paper presents a novel approach to enhancing full-duplex spoken language models (FD-SLMs) by addressing the issue of state inertia, which occurs when models fail to promptly transition from a generative to a perceptive state during user interruptions. The authors introduce the Zero-Buffer Benchmark (ZBB) to evaluate the models' comprehension in these scenarios and propose a technique called activation steering using a perception vector, which improves interruption handling in multiple state-of-the-art FD-SLMs, such as PersonaPlex, achieving a correctness increase from 28% to 45% and an initial-word occurrence rate improvement from 40% to 72% without requiring fine-tuning. This advancement is significant for practitioners as it enhances the responsiveness of conversational AI systems, allowing for more fluid interactions in real-time applications.