Bridging Modal Isolation in Interleaved Thinking: Supervising Modality Transitions via Stepwise Reinforcement

The paper presents MoTiF (Modality Transition Fidelity), a two-stage training framework designed to address Modal Isolation in interleaved thinking models, which alternately process text and images. It introduces a modality transition loss to quantify issues like cross-modal hallucination and visual utilization deficits, enhancing model coherence and accuracy across four visual puzzle benchmarks. This approach emphasizes the need for explicit supervision at modality boundaries, rather than relying solely on end-task optimization, which is crucial for practitioners developing multimodal AI systems.