Mind the Gap... or Not? How Translation Errors and Evaluation Details Skew Multilingual Results

This paper examines the performance discrepancies of large language models (LLMs) across multiple languages, revealing a consistent performance gap even for high-resource languages, attributed to translation errors and evaluation inconsistencies in existing benchmarks. The authors propose a semi-automatic quality assurance method to rectify these issues, demonstrating that addressing data quality can significantly alter conclusions about multilingual capabilities. They also release a corrected version of the multilingual math benchmark dataset (MGSM-Rev2) to facilitate improved evaluation practices in cross-lingual research.