PRISM: Prosody-Integrated Multi-Agent Reasoning Framework for Empathetic Spoken Dialogue

PRISM is a new multi-agent framework designed for empathetic spoken dialogue systems, addressing limitations in traditional cascade pipelines and end-to-end models by decoupling speech perception, response generation, and speech synthesis. It introduces a prosody-to-language translation mechanism that enhances the reasoning of large language models and allows for the integration of external knowledge tools, resulting in improved metrics for empathy and prosodic alignment in dialogue generation. This framework is significant for practitioners as it offers a structured approach to incorporating emotional and contextual nuances in spoken dialogue systems, enhancing user interaction quality.