Scaling Audio Models Efficiently: A Joint Study of Compute Constraints and Optimization Behavior

This study presents a unified framework for optimizing compute allocation in speech processing tasks, specifically Automatic Speech Recognition (ASR) and Speech Emotion Recognition (SER), by analyzing model size, input length, and representation resolution. Experiments on the LibriSpeech and CREMA-D datasets reveal that scaling model size from Tiny (39M) to Small (244M) significantly reduces word error rate (WER) by 8.22%, but further scaling to Medium (769M) yields diminishing returns of only 2.35%. The findings indicate an optimal audio duration for SER and suggest that reducing encoder token resolution can effectively lower inference costs with minimal impact on performance, providing valuable guidelines for practitioners designing efficient speech models.