Beyond Penalizing Mistakes: Stabilizing Efficiency Training in Large Reasoning Models via Adaptive Correct-Only Rewards

The paper introduces ACOER (Adaptive Correct-Only Efficiency Reward), a new training method for large language models aimed at improving reasoning efficiency by applying rewards only to correct answers. This approach mitigates issues of reward collapse and over-compression that arise from traditional length-penalizing strategies in Group Relative Policy Optimization (GRPO). Evaluations on mathematical reasoning benchmarks demonstrate that ACOER enhances accuracy while reducing token generation by over 60%, providing a stable framework for optimizing efficiency in LLM training.