What Language is This? Ask Your Tokenizer

UniLID is a novel language identification (LID) method that utilizes the UnigramLM tokenization algorithm, focusing on probabilistic language modeling to determine the most likely language of a given string. It is designed to be data- and compute-efficient, allowing for the incremental addition of new languages without retraining, and integrates seamlessly into existing language model pipelines. Empirical results indicate that UniLID achieves approximately 70% accuracy with only five labeled samples per language, outperforming traditional baselines like fastText and CLD3, particularly in low-resource and dialect identification scenarios, making it a valuable tool for enhancing multilingual NLP applications.