The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL

The paper introduces the Optimal Token Baseline (OTB) for improving Reinforcement Learning (RL) in Large Language Models (LLMs) by addressing the issue of exploding gradient variance in long-horizon tasks. The OTB is derived from first principles, proposing a method where gradient updates are inversely weighted by their cumulative gradient norm, and utilizes a Logit-Gradient Proxy to efficiently estimate this norm with only forward-pass probabilities. This approach enhances training stability and reduces token consumption by over 65% while achieving comparable performance to larger group sizes, making it significant for practitioners seeking efficient RL training methods in LLM applications.