ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillation

The article introduces ReNIO (Reweighting Negative trajectory Importance for LLM On-Policy Distillation), a method that enhances on-policy distillation (OPD) by assigning greater importance to student-generated outputs (SGOs) that lead to incorrect reasoning, thereby preserving exploratory reasoning capabilities. By leveraging the student-to-teacher probability ratio, ReNIO effectively identifies and weights pivotal tokens from negative trajectories, improving performance on mathematical reasoning and code generation tasks, with reported gains of up to 10.00% for models like R1-Distill-Qwen-7B. This approach maintains the advantages of prefix-conditioned training while addressing the limitations of traditional OPD, making it a significant advancement for practitioners in LLM optimization.