The Hidden Evolution of Disguised Visual Context inside the VLM

A new study from Wish Suharitdamrong examines how visual tokens are reshaped inside large language models, finding that the integration architecture determines what visual features a vision-language model can use and how those features align with language space [1]. Visual tokens enter large language models as raw signals devoid of linguistic structure, and their transformation into meaningful representations depends entirely on the integration architecture [1]. The paper, submitted on 18 June 2026, provides a controlled comparison of two dominant vision-language model paradigms: in-context injection, where visual tokens are treated as prompts within the input sequence, and layer-wise injection, where they are inserted directly into intermediate layers [1]. The submission totals 23,665 KB [1]. The research evaluates both approaches under identical training conditions across single-image, multi-image, and video benchmarks [1]. The authors uncover what they describe as a hidden evolution: visual tokens enter the LLM as disguised visual context and are progressively reshaped, with each paradigm capturing fundamentally different frequency characteristics of the visual signal [1]. This internal evolution determines which visual features the model can utilize effectively and how visual representations align with the language space [1]. The study further demonstrates that attention allocation alone is insufficient for strong performance; instead, performance is driven by the quality of visual representations at each layer [1]. The work addresses a gap in the literature, as a controlled comparison of how these architectural choices affect visual information and its internal transformation has remained underexplored [1]. The findings carry implications for the design of future multimodal systems, suggesting that engineering how visual information is integrated may be as consequential as scaling model size or data volume [1].