LangMAP: A Language-Adaptive Approach to Tokenization

LangMAP introduces a language-adaptive tokenization method that enhances the UnigramLM algorithm for multilingual applications, allowing for language-specific tokenization using a single shared vocabulary. This approach enables the adaptation of pretrained model tokenizers without altering their vocabulary and performs language-specific tokenization at inference without prior language knowledge. The method demonstrates improved morphological boundary alignment across various languages and coding contexts, although its effectiveness in knowledge-related tasks shows variability.