Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insights

This study presents an empirical analysis of multilingual tokenization for 17 Indic languages, evaluating subword algorithms like BPE and Unigram LM, as well as factors such as script normalization and vocabulary construction strategies. Key findings indicate that script-specific normalization enhances tokenization quality, Unigram LM better maintains morphological boundaries compared to BPE, and cluster-based vocabulary construction improves downstream task performance. These insights are critical for practitioners developing language models for low-resource, morphologically rich languages, emphasizing the need for linguistically informed tokenization strategies.