Where's the Plan? Locating Latent Planning in Language Models with Lightweight Mechanistic Interventions

A new study finds that only one of three tested large language models — Gemma-3-27B — causally relies on internal representations of a future rhyming word when completing couplets, despite all models showing they can encode that information [1]. The work, posted to arXiv by researcher Nick Rui, examines what the authors call “planning site formation” — the point during a forward pass where a model builds a representation of a structurally required future token and whether that representation actually drives generation [1]. The team used rhyming-couplet completion as a constrained test bed, applying linear probing and activation patching across the Qwen3, Gemma-3, and Llama-3 families at more than ten scales [1]. Linear probing showed that future-rhyme information is decodable at the line boundary in all three model families, with the signal growing stronger as model size increases [1]. Activation patching told a different story. Only Gemma-3-27B exhibited a causal reliance on the boundary encoding; every other model conditioned on the rhyme word throughout generation, showing near-zero causal effect at the boundary despite the strong probe signal [1]. In Gemma-3-27B, the causal driver migrated from the rhyme word to the line boundary around layer 30, a transition the authors call a “handoff” [1]. Two-stage path patching localized that handoff to five attention heads, recovering roughly 90% of the rhyme-routing capacity at the newline token [1]. The findings arrive as the broader ecosystem of tools for running such models locally continues to mature. Libraries like llama.cpp have become a de facto standard for on-device inference, underpinning platforms such as Ollama and LM Studio [4]. Ollama itself provides a command-line interface, a local REST API, and model-management tools that let developers run open-weight models — including the Llama and Gemma families — on consumer hardware [6]. Separately, researchers have explored token-pruning techniques that strip out vocabulary for irrelevant languages, showing that compressed versions of Qwen3, Gemma-3, and Llama-3 can improve generation stability and, in machine translation, sometimes boost performance on domain-specific tasks while sharply reducing memory footprint [3]. Large language models are typically defined as models with many parameters trained via self-supervised learning on vast text corpora [5]. The new mechanistic study suggests that even when such models appear to “know” a future constraint, that knowledge is not always wired into the generation pathway — a distinction with implications for how developers interpret probing results when evaluating model reasoning [1].