CulMind: Benchmarking Multimodal Understanding and Reasoning in Chinese Cultural Heritage

CulMind and CulMind-R are newly introduced benchmarks for evaluating Multimodal Large Language Models (MLLMs) in the context of Chinese Cultural Heritage (CCH), encompassing 50 tasks from over 100 museums and a 24-task reasoning subset. The benchmarks utilize ReaScore, a task-adaptive metric for assessing reasoning quality by weighting task-specific dimensions, revealing significant discrepancies between model answers and reasoning quality, particularly on complex tasks. This resource enables practitioners to conduct more nuanced evaluations of MLLMs' understanding of cultural heritage, fostering advancements in multimodal reasoning capabilities.