Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet

The paper presents a novel approach to multilingual tokenization using the International Phonetic Alphabet (IPA) to create language-agnostic input representations. By training matched pairs of text and IPA subword tokenizers across 24 languages and 14 scripts, the authors demonstrate that IPA tokenizers enhance tokenization quality, particularly for non-Latin scripts, and improve generalization to unseen languages. This advancement is significant for practitioners as it addresses performance disparities in multilingual models at the tokenization stage, potentially leading to better model performance across diverse languages.