The Vision Encoder as a Privacy Boundary: Visual-Token Side Channels in Encoder-Free Vision-Language Models

Encoder-free vision-language models expose a previously undocumented architectural privacy vulnerability, according to a study posted to arXiv on 10 Jun 2026. Researchers demonstrated that intermediate visual tokens in models such as Gemma4 and Fuyu can be inverted to recover recognizable image structure and readable text, including held-out access codes [1][2]. The paper, titled "The Vision Encoder as a Privacy Boundary: Visual-Token Side Channels in Encoder-Free Vision-Language Models," argues that traditional vision-language models (VLMs) benefit from a vision encoder that compresses image pixels into semantic embeddings, implicitly attenuating pixel-local detail required for exact text recovery [1][2]. Encoder-free VLMs remove this component, routing image patches directly into the language-model token stream and thereby creating a pre-output side channel [2]. Under a token-access adversary scenario, the authors built decoders that inverted visual-token streams from two encoder-free models, Gemma4 and Fuyu. The decoders recovered recognizable image structure and readable held-out access codes. In contrast, matched encoder-based control models localized target regions but recovered no exact strings [1][2]. Within-model ablations identified spatial sampling fidelity of the visual-token grid as the operative factor enabling the leakage, with character-direction sampling density proving more consequential than token or value count [2]. The vulnerability extends beyond exported tokens: Gemma4 layer-0 key-value cache tensors were directly invertible, situating the side channel inside KV caches that production serving stacks commonly persist for decoding efficiency [1][2]. The attack withstood clutter, realistic document degradation, and zero-shot transfer to public document images. Value-level defenses such as additive noise and quantization failed to block the inversion [2]. The researchers conclude that effective mitigation requires reducing spatial sampling, and that the decision to remove a vision encoder should be treated as a first-class privacy consideration in VLM deployment [1][2]. The study was submitted to arXiv, an open-access repository of electronic preprints that, as of November 2024, receives about 24,000 submissions per month [8]. arXiv hosts papers across mathematics, physics, computer science, and related fields, and its content is moderated but not peer-reviewed [8].