GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control

The paper introduces Gradient Alignment Control (GAC), a method designed to stabilize asynchronous reinforcement learning (RL) training for large language models (LLMs) by addressing the instability caused by high cosine similarity in policy gradients during asynchronous updates. GAC employs gradient projection to regulate training dynamics, ensuring convergence even under conditions of bounded staleness. This approach allows practitioners to leverage asynchronous execution in RL without sacrificing training stability, potentially enhancing the efficiency of LLM training processes.