AlgoSimBench: Identifying Algorithmically Similar Problems for Competitive Programming

AlgoSimBench is a newly introduced benchmark consisting of 402 multiple-choice questions designed to evaluate the ability of Large Language Models (LLMs) to identify algorithmically similar problems (ASPs) in competitive programming. The benchmark's unique setup pairs each reference problem with one ASP and three distractors, promoting reliance on algorithmic reasoning over superficial cues. The evaluation reveals that LLMs struggle with this task, but the proposed Attempted Solution Matching (ASM) technique, which assesses similarity based on LLM-generated solutions, improves accuracy by 9%, and when combined with BM25, achieves an additional 11.8% gain over existing embedding models. This benchmark is significant for advancing research on LLM capabilities and retrieval methods in algorithmic contexts.