The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs

The study presents a framework for evaluating local-language access to cultural knowledge in large language models (LLMs), addressing limitations in existing methodologies that conflate language proficiency with knowledge access. By analyzing 80 models across 13 locales, the research reveals a consistent advantage for English in culture-agnostic questions, while local languages show improved access to cultural knowledge when proficiency is accounted for. This finding underscores the importance of considering language-specific knowledge access in model evaluation, suggesting that practitioners may overlook the potential of local languages in cultural contexts despite apparent performance disparities.