Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguity

The paper introduces VOIR DIRE, a multimodal benchmark comprising 626 culturally paired image-prompt artifacts to evaluate MLLM-as-a-Judge performance across U.S. and Chinese contexts. It identifies two key failures in model calibration: a positivity-floor calibration failure and an orientation failure, with implications for cross-cultural evaluation metrics. This research highlights the necessity for practitioners to report model alignment against specific cultural reference pools and to recognize cross-pool divergence as an inherent property of the models, thus informing more accurate assessments of MLLM performance in diverse cultural settings.