Where's the Plan? Locating Latent Planning in Language Models with Lightweight Mechanistic Interventions

The study investigates latent planning in language models, focusing on the formation of internal representations that influence token generation. Using rhyming-couplet completion, the authors apply linear probing and activation patching across models Qwen3, Gemma-3, and Llama-3, revealing that only Gemma-3-27B causally utilizes future-rhyme information, with a significant shift in causal influence occurring around layer 30. This research is critical for practitioners as it highlights the importance of specific architectural features and attention mechanisms in enhancing the planning capabilities of language models.