Surprise-Guided MergeSort: Budget-Efficient Human-in-the-Loop Ranking via Adaptive Comparison Scheduling

The article introduces the Surprise-Guided MergeSort (SGS) framework, which optimizes human-in-the-loop ranking by prioritizing comparisons that require human judgment while automating others. SGS utilizes a bottom-up MergeSort scheduler, a composite Surprise Scorer that integrates VLM confidence and other metrics, and an adaptive budget allocator, achieving a significant reduction in non-informative comparisons and improving Kendall's τ by 6 to 12 points over Active Elo within the same budget. This method enhances annotation efficiency in subjective ranking tasks, making it valuable for practitioners seeking to reduce human effort while maintaining accuracy in model training.