Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources

The article introduces ST-AudioQA, a spatio-temporal audio question-answering dataset and benchmark that utilizes first-order ambisonic (FOA) renderings to enhance semantic understanding of dynamic sound sources. It presents the ST-Audio Encoder, which integrates event semantics with source trajectory data, and ST-AudioLM, which connects these audio representations to a language model for improved reasoning about sound events in terms of identity, location, and motion. This development is significant for practitioners as it enhances the capability of audio-language models to reason about complex sound environments, improving applications in fields such as robotics and interactive AI systems.