Afrispeech Semantics: Evaluating Audio Semantic Reasoning in Spoken Language Models Across Domains and Accents

The paper presents a comprehensive evaluation of audio language models (ALMs) for semantic reasoning across five tasks: entailment, consistency, plausibility, accent drift, and accent restraint. It highlights the limitations of current ALMs in handling accent variation and domain shifts, emphasizing the need for improved benchmarks that assess models' reasoning capabilities using spoken audio as the primary evidence. This research is significant for practitioners as it underscores the importance of robust evaluation metrics in developing more equitable and effective ALMs.