Beyond 'One Language, One Script': Quantifying Orthographic Bias in Multilingual VLMs with PuMVR

The paper introduces PuMVR (Punjabi Multimodal Visual Reasoning), a benchmark designed to assess orthographic bias in Vision-Language Models (VLMs) by evaluating 375 image-reasoning tasks across three scripts of Punjabi. The study reveals significant discrepancies in model performance, with accuracy differences of up to 16% between scripts and Script Consistency Rates (SCR) as low as 24.8%, highlighting the limited transferability of reasoning across scripts. This work emphasizes the need for script-agnostic evaluation metrics, challenging existing multilingual assessment methods and advocating for more equitable AI solutions.