Transcribing Bengali Text with Regional Dialects to IPA using District Guided Tokens

The paper introduces the District Guided Tokens (DGT) technique for transcribing Bengali text with regional dialects into the International Phonetic Alphabet (IPA). By prepending district tokens to input sequences, the approach fine-tunes transformer-based models, notably achieving superior results with the ByT5 model compared to mT5, BanglaT5, and umT5, particularly in handling out-of-vocabulary words. This work emphasizes the necessity of integrating regional dialect information in natural language processing systems to address the phonological diversity in languages like Bengali.