Gaze Heads: How VLMs Look at What They Describe

Researchers have identified a small set of attention heads inside vision-language models that act as a gaze mechanism, tracking and describing specific image regions. The finding, posted to arXiv on June 12, offers a new lever for steering multimodal model outputs without retraining. The study, led by Rohit Gandikota, examines how a vision-language model internally solves the task of describing an image [1]. The authors find that a specific mechanism emerges: a small set of attention heads in the model's language-model backbone, which they call gaze heads, whose attention tracks the image region the model is currently describing [2]. The team used comic strips as a controlled testbed where narrative order is laid out spatially, identifying the heads with a simple correlation score from a few forward passes [2]. These gaze heads do not merely track the image tokens being described. Redirecting their attention to a chosen region forces the VLM to describe that region instead [2]. A single attention-mask intervention on the top-100 gaze heads, which constitute fewer than 9% of all heads, steers the model's answer to any chosen comic panel with 83.1% accuracy [2]. The same intervention on random heads fails to redirect the answer, and intervening on all heads destroys generation [2]. The mechanism also supports continuous control: switching the gaze target mid-generation makes the model wrap up its current panel description and move to the new one within a few tokens [2]. Beyond comics, the same intervention redirects answers to chosen regions in natural COCO images [2]. The mechanism further recurs across model sizes from 2B to 32B parameters and across other VLM architectures, although some frozen-encoder families show no comparable head set [2]. The paper, weighing in at 12,636 KB, was submitted through arXivLabs, a framework that allows collaborators to develop and share new arXiv features directly on the website [1]. The authors have released code, an interactive demo, and datasets alongside the paper [2]. The work arrives amid broader efforts to understand the internal mechanisms of large language models, which are trained with self-supervised learning on vast amounts of text [8]. Recent research has also probed how generative systems handle structured artifacts such as quantum circuits, though that work notes a persistent gap between generated outputs and practical deployment on hardware [3]. The gaze-heads finding demonstrates that targeted edits identified through mechanistic analysis can serve as practical inference-time levers for steering multimodal model behavior, without any retraining [2]. The paper is available on arXiv and linked through the Hugging Face Hub, where paper pages allow the community to find associated models, datasets, and interactive demos [4].