Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning

The article introduces Group-Graph Policy Optimization (G2PO), a novel reinforcement learning algorithm designed to address challenges in long-horizon agentic tasks by transforming linear interaction trajectories into a global state-transition graph. G2PO enhances credit assignment through group-aggregation state-value estimation, reducing sampling variance and bias, and employs an edge-centric advantage estimation strategy to prioritize critical transitions. Experimental results on benchmarks like WebShop, ALFWorld, and AppWorld show that G2PO outperforms existing models, achieving up to a 22.2% improvement in success rates compared to the previous state-of-the-art, GRPO, making it a significant advancement for practitioners in multi-turn agentic RL scenarios.