MAGNIFIED: RL Fine-tuning of Multimodal Large Language Models for Motion Planning

The article presents MAGNIFIED, a reinforcement learning fine-tuning (RLFT) approach for multimodal large language models (MLLMs) aimed at improving motion planning in autonomous driving. By utilizing token-level rewards and mapping predicted tokens to vehicle trajectories, MAGNIFIED enhances planning performance, achieving over a 10.5% reduction in overlap rate and a 38.9% reduction in off-road rate compared to a supervised fine-tuning baseline on the Waymo Open Motion Dataset. This approach highlights the importance of aligning MLLM objectives with real-world planning considerations, making it a significant advancement for practitioners in autonomous vehicle development.