Weight-Space Geometry of Offline Reasoning Training

The paper analyzes six offline reinforcement-learning training methods (SFT, RFT, DFT, RIFT, Offline GRPO, DPO) using a shared model (Qwen3-4B) and attention-only LoRA to compare their weight-space geometry and performance on downstream tasks. The findings indicate that SFT, RFT, and RIFT yield similar weight updates and comparable accuracy (87-88% on GSM8K), while DPO achieves the highest accuracy (93.5%) but requires a significantly smaller learning rate, suggesting that optimizer and loss function choices critically influence performance. This research provides insights into the mechanistic differences between methods, which is essential for practitioners to optimize training strategies in offline reinforcement learning.