Rethinking the Trust Region in LLM Reinforcement Learning

The paper introduces Divergence Proximal Policy Optimization (DPPO), a novel approach to reinforcement learning for fine-tuning Large Language Models (LLMs), addressing the limitations of the traditional Proximal Policy Optimization (PPO) algorithm. DPPO replaces the heuristic clipping mechanism of PPO with a principled constraint based on direct estimates of policy divergence, such as Total Variation or KL divergence, while employing efficient Binary and Top-K approximations to minimize memory overhead. Empirical evaluations indicate that DPPO enhances training stability and efficiency, making it a significant advancement for practitioners in the RL-based fine-tuning of LLMs.