Attention Alignment Between Humans and Vision-Language Models

A new study finds that the decoder architecture of vision-language models, more than the encoder, determines how closely a model’s spatial attention aligns with human gaze patterns recorded during image-viewing tasks. The research, posted to the arXiv preprint repository on June 16, 2026, compared spatial attention maps from six vision-language models against human fixation heatmaps collected on 200 images across two tasks: general description and social captioning [1][2]. Vision-language models are AI systems that jointly process images and text, extending large language models into the visual domain [3]. The six models spanned a 2×2 factorial design of CNN versus ViT encoders crossed with LSTM versus Transformer decoders, plus two additional open-source models, Molmo 7B-D and Qwen3.5 9B [1][2]. The authors, led by Isaac Christian, found that decoder choice dominated alignment outcomes. Switching from a Transformer to an LSTM decoder increased alignment by 40 to 50 percentage points, reaching 80 to 87 percent of the human noise ceiling, compared with 40 to 59 percent for Transformer-based decoders [1][2]. Encoder architecture played a secondary role, contributing a 5-to-20-point advantage depending on the decoder family. The CNN-LSTM combination was the most aligned model overall, scoring 85 to 87 percent [1][2]. Attention mechanisms, which assign importance weights to different parts of an input sequence, are central to how these models process information [5]. The study revealed a trade-off: LSTM-decoder attention maps, despite their higher alignment with human fixations, were spatially diffuse and showed minimal differentiation between tasks. In contrast, the ViT-Transformer model, which posted the weakest alignment scores, exhibited the sharpest spatial concentration and the strongest task differentiation [1][2]. A hemispatial-neglect simulation further demonstrated that removing attention harmed LSTM decoders more than Transformer decoders [1][2]. In an exploratory extension using TRIBE-simulated synthetic neural responses, the researchers found that fixation alignment and neural relevance did not track together. CNN-Transformer attention maps better predicted synthetic brain activity, particularly in early visual cortex, even though they showed lower fixation alignment [1][2]. The findings suggest that top-down and bottom-up components in these models trade off what they predict in behavioral versus synthetic neural data [1][2].