Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models

A research team has proposed a self-evolving framework for large multimodal models that directly regularizes visual attention, addressing a failure mode where models rely on language patterns instead of image content during generation [1]. The framework, called Visual Invariance Self-Evolution (VISE), targets what its creators term "visual under-conditioning" — a persistent problem in existing self-evolving large multimodal models (LMMs) where the decoder attends insufficiently to visual tokens and leans on statistical language priors to produce outputs [1]. Prior self-evolving approaches use multi-role self-play and self-consistency reward schemes that optimize answer agreement without verifying whether the model actually references the image [2]. As a result, these models struggle on vision-language understanding tasks such as image captioning and visual question answering [1]. VISE operates within a single model, without specialist roles, external reward models, or annotations, and trains on raw unlabeled images [3]. The framework introduces two complementary invariance-based rewards: a geometric invariance reward that enforces spatial consistency under known transformations, and a semantic invariance reward that penalizes evidence-agnostic generation by requiring the model to recognize the absence of evidence when predicted regions are perturbed [4]. The researchers, including Shravan Venkatraman, found that a well-pretrained LMM already possesses sufficient knowledge to formulate meaningful queries about its visual content, eliminating the need for multi-role architectures [3]. Using Qwen3-VL-2B as the base model, VISE achieved gains of +16.85 CIDEr on COCO and +19.66 CIDEr on TextCaps, while reducing object hallucination by 5.0 Chair-I points [1]. The improvements generalized across four model families and scales, and extended to additional benchmarks including +14.7 CIDEr on TextCaps, +16.55 on Flickr30k, with gains the authors describe as 4 to 7 times larger than prior self-evolving methods [3]. Mechanistically, the team measured generation-time visual attention — the fraction of attention each generated token assigns to image tokens across transformer layers — averaged over 100 random COCO images [4]. VISE consistently assigned more attention to visual tokens across mid-to-late decoder layers, indicating that the semantic invariance reward encourages the decoder to maintain visual conditioning throughout generation rather than reverting to language priors after initial image encoding [4]. This finding aligns with separate research on hallucination in large vision-language models, which has shown that hallucinations often arise from the progressive weakening of attention weight to visual tokens in deeper transformer layers [5]. The work arrives amid broader efforts to improve reliability in generative AI systems, which have seen rapid deployment since the AI boom of the 2020s driven by transformer architectures and large language models [6]. Major research laboratories including Google DeepMind have contributed foundational advances in multimodal models and reinforcement learning techniques that inform the self-evolving paradigm [7]. The VISE code and models are publicly available [1].