ROMEVA: Geometry-Preserving Vocabulary Expansion for Roman Urdu Language Models

The article introduces ROMEVA (Roman Urdu Embedding-preserving Vocabulary Adaptation), a method designed to enhance multilingual models like mBERT for the Roman Urdu language, which faces challenges due to spelling variations leading to sub-word fragmentation. ROMEVA employs a combination of sub-word-average initialization and PCA-guided anchor loss to stabilize embeddings while expanding the vocabulary with 500 new tokens derived from a 36,130-comment corpus. The results indicate that while ROMEVA preserves embedding stability, naive fine-tuning yields better performance in sentiment classification, highlighting a potential trade-off between embedding preservation and model adaptability in morphologically inconsistent languages.