Fast Speech Foundation Model Distillation Using Interleaved Stacking

The paper introduces a novel method called interleaved stacking for distilling large speech foundation models (SFM) into efficient student models, aiming to enhance training efficiency and reduce deployment latency. This approach maintains consistent layer positioning during the stacking process, addressing performance degradation issues seen in traditional stacking methods. The effectiveness of interleaved stacking is validated using the SUPERB benchmark, which is significant for practitioners looking to optimize model training in low-resource environments.