Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encoding

Spatial-Omni introduces a novel method for integrating First-Order Ambisonics (FOA) spatial audio into existing multimodal large language models (LLMs) using the SO-Encoder, which allows for enhanced spatial audio understanding without altering original audio encoders. The approach includes the creation of the SO-Dataset, SO-QA, and SO-Bench, comprising 400K FOA spatial audio clips and 2.1M spatial question-answer pairs, covering 16 subtasks in spatial audio understanding. This development is significant for practitioners as it enhances the capability of LLMs to process spatial audio cues, improving applications in sound localization and spatial reasoning.