HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation

HERO (Hindsight-Enhanced Reflection from Environment Observations) is a new self-distillation framework designed to improve reinforcement learning in multi-turn settings by using next environment observations for locally aligned feedback. It addresses the issue of credit assignment by providing compact turn-level diagnostics that evaluate the necessity and validity of actions taken, leading to improved task success and reduced unnecessary actions on benchmarks like TauBench and WebShop. This approach is particularly beneficial in scenarios with limited training turn budgets, enhancing performance where traditional methods struggle to provide effective reward signals.