VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Training

The article introduces VRPO, a novel framework designed to enhance value modeling for robust reinforcement learning (RL) in the post-training phase of large language models (LLMs) under noisy supervision. VRPO improves stability and generalization by integrating auxiliary losses from a frozen language model and employing a variational information bottleneck to filter noise, transforming the value model into an active regulator of reward noise. Experimental results demonstrate that VRPO outperforms traditional methods like PPO and GRPO across various tasks, emphasizing the importance of robust value modeling in RL applications.