Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models

A new study examines the UTF-8 validity of byte-aware language models, specifically analyzing a 355M parameter model trained on 80 billion tokens from a multilingual corpus. The research reveals that while perplexity stabilizes after 2.1 billion tokens, UTF-8 validity requires 4.2 billion tokens to reach similar reliability, highlighting that rare characters can generate higher structural validity than common ones. This finding emphasizes the need for practitioners to assess UTF-8 generation separately from traditional language modeling metrics, as reliable UTF-8 output is crucial for handling diverse Unicode inputs effectively.