Accelerating Disaggregated RL for Visual Generative LLMs with Diffusion-Based Parallelism and Trainer-Assisted Generation

The article introduces DigenRL, a disaggregated reinforcement learning framework designed for diffusion-based generative large language models (LLMs). Key innovations include a generation-axis pipeline (GAP) and time-step parallelism (TSP) for enhanced pipelining, an elastic trainer-assisted generation (TAG) approach for dynamic resource allocation, and an asynchronous strategy to optimize pipeline utilization. Experimental results demonstrate that DigenRL achieves throughput improvements of 1.56 to 2.10 times over existing systems like veRL-Omni and GenRL, making it a significant advancement for practitioners working on efficient RL systems in generative AI.