Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning

The article presents a comprehensive survey on the application of reinforcement learning (RL) in training large language models (LLMs), emphasizing the need for a structured examination of RL algorithms beyond the commonly used PPO and GRPO methods. It categorizes the RL process into three stages: MDP creation, exploration techniques, and learning strategies, highlighting underexplored areas such as off-policy actor-critic training and bootstrapping methods, which could enhance LLM training. This framework aims to guide researchers in both RL and LLMs towards more effective methodologies and identifies key opportunities for integrating established RL techniques into LLM development.