Are Multilingual Models Actually Improving? Isolating True Cross-Lingual Transfer

The paper introduces the Hardness Adjusted Transfer (HAT) Score as a new metric to more accurately assess cross-lingual transfer in multilingual models, addressing issues with existing evaluation methods. An analysis of twenty language models across three benchmarks reveals that while smaller models are capable of transfer, the overall progress in cross-lingual transfer with increasing model size has been slower than anticipated, despite clear improvements over time. This insight is crucial for practitioners as it highlights the need for more nuanced evaluation metrics and suggests that scaling models alone may not be sufficient for enhancing cross-lingual capabilities.