Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoning

The paper introduces Strategy-Guided Policy Optimization (SGPO), a novel framework that enhances reasoning capabilities in language models by distilling reusable strategies instead of merely imitating specific solution trajectories. SGPO employs a token-level forward-KL objective to transfer strategic guidance into unguided policies and utilizes adaptive instance-level weighting to optimize the distillation process based on model competence. Experimental results demonstrate that SGPO significantly outperforms traditional methods, including supervised fine-tuning and reinforcement learning approaches, achieving an average score improvement of 2.2 points on the Qwen2.5-7B-Instruct model across four mathematical benchmarks, highlighting its potential for enhancing generalization in AI applications.