On the Position Bias of On-Policy Distillation

The paper introduces Importance-Weighted On-Policy Distillation (IW-OPD), a refinement of On-Policy Distillation (OPD) that addresses the position bias in token-level supervision. By adjusting token weights based on the discrepancy between student and teacher distributions, IW-OPD significantly enhances learning efficiency and convergence speed, outperforming standard OPD by up to 6.9 points on the AIME-2025 benchmark. This advancement is crucial for practitioners as it suggests a more effective approach to leveraging teacher-student dynamics in reinforcement learning, particularly in scenarios with long rollouts.