CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data

CommonLID is a newly introduced, community-driven benchmark for language identification (LID) that encompasses 109 languages, specifically designed to address the challenges posed by noisy web data. The benchmark reveals that existing LID models overestimate their accuracy for many languages, emphasizing the need for improved evaluation metrics in multilingual contexts. This resource is crucial for practitioners aiming to develop more representative and high-quality multilingual corpora for training language models.