MLLMs Get It Right, Then Get It Wrong: Tracing and Correcting Late-Layer Textual Bias

Multimodal large language models consistently favor text over images when the two conflict, but new research shows they often form correct vision-based predictions in intermediate layers before overriding them in the final output, a phenomenon researchers call “late-layer textual override” [1][2]. The finding, posted to the arXiv preprint server on June 16, 2026, challenges assumptions that these models simply fail to process visual information when it contradicts text [1][2]. Instead, the visual information is encoded — it simply does not survive to the output [1][2]. The paper examines how prediction trajectories within the model reveal whether the final answer will be correct: 85% of failures shift toward text, while 89% of successes shift toward vision [1][2]. This directional signature enabled the researchers to develop a training-free intervention called CALRD, or Conflict-Aware Layer Reference Decoding [1][2]. The method detects when a confident visual prediction is being suppressed and restores it at inference time, without requiring additional training or external knowledge [1][2]. Experiments across five MLLMs of varying architectures demonstrated up to 9.4% absolute improvements on conflict benchmarks while largely preserving standard performance [1][2]. The work addresses a documented vulnerability in multimodal systems. As large language models have grown in scale and capability — trained with self-supervised learning on vast amounts of text — their integration with vision components has created new failure modes when modalities disagree [8]. The arXiv repository, which hosts the paper, has served as a primary distribution channel for such computer science research since its founding in 1991, now receiving approximately 24,000 submissions per month as of late 2024 [6]. The authors frame the contribution as recovering what the model already knew but failed to preserve [1][2]. By tracing internal prediction shifts, the method identifies moments when a model abandons a correct visual interpretation in favor of contradictory text, then intervenes to restore the earlier, vision-aligned prediction [1][2].