A Formula-Driven Survey and Research Agenda for On-Policy Distillation

The article presents a comprehensive survey on on-policy distillation (OPD) for large language models (LLMs), proposing a formula-driven taxonomy that categorizes methods based on direct distributional losses and policy-gradient log-ratio updates. It highlights key factors influencing OPD effectiveness, such as state compatibility and probability routing, and introduces two mechanisms—temporal credit and vocabulary routing—that are crucial for understanding stability in sampled-token OPD. This research is significant for AI practitioners as it provides a structured framework for optimizing OPD techniques, addressing failure modes, and guiding future implementations in training LLMs.