Verifiable Counterfactual Supervision for Process Reward Models

The paper introduces a method for verifiable counterfactual supervision in process reward models (PRMs), which involves generating paired correct and erroneous reasoning trajectories to identify the first unsupported transition. This approach utilizes a verified symbolic reasoning chain, injecting controlled errors at intermediate steps, and ensures coherence in subsequent reasoning. Experimental results demonstrate that this method enhances performance on logical reasoning benchmarks, improving Best-of-8 reranking and indicating potential for transfer to mathematical evaluations, which is significant for practitioners aiming to develop more robust PRMs.