Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently

The paper presents a theoretical analysis comparing reinforcement learning with verifiable rewards (RLVR) to supervised fine-tuning (SFT) in enhancing reasoning capabilities of large language models. It establishes that SFT, when trained solely on optimal paths, fails to enable efficient backtracking, while RLVR facilitates learning to backtrack from dead ends using outcome rewards, resulting in significant computational efficiency during inference. This research is crucial for practitioners as it suggests that integrating RLVR can improve reasoning performance and optimize resource allocation in LLM applications.