The African Language Tax: Quantifying the Cost, Latency, and Context Penalty of Tokenizing African Languages in Frontier LLMs

The article quantifies the tokenization penalties faced by speakers of African languages when using frontier large language models (LLMs), revealing that these languages incur a median tokenization premium of 1.88x compared to English, with certain scripts like N'Ko experiencing up to 8.92x. The study evaluates 20 African languages across various tokenizers, finding that the best tokenizer, Gemma 4, still leaves a significant premium of 2.38x. This disparity leads to increased inference costs and reduced context windows for African language users, highlighting a critical digital divide that affects language accessibility in AI applications.