BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories

BabelJudge is an open-source benchmark and reliability audit framework designed to evaluate biases in LLM-as-a-judge models, addressing issues such as position bias, verbosity bias, order inconsistency, and cross-lingual degradation without requiring human preference labels. It employs a novel gold-labeling method through controlled perturbations, and evaluation of the Qwen2.5-7B-Instruct-4bit model across multiple languages revealed significant reliability drops, particularly in lower-resource languages like Swahili. This framework is crucial for practitioners as it provides a comprehensive tool for assessing model reliability in diverse linguistic contexts, enhancing the robustness of NLP evaluations.