ViT-Up: Faithful Feature Upsampling for Vision Transformers

A new framework called ViT-Up improves how Vision Transformers produce high-resolution feature maps for tasks like segmentation, outperforming existing methods that rely on external image guidance, according to research posted to arXiv on June 12, 2026 [1]. Vision Transformers, or ViTs, have become a leading architecture for visual representation learning, but their reliance on relatively small patch-token grids — a consequence of the quadratic cost of global self-attention — creates a bottleneck for dense prediction tasks such as semantic segmentation and depth estimation [2]. This limitation has driven the development of task-agnostic feature upsamplers [2]. Current state-of-the-art methods can produce visually sharp dense representations, yet their dependence on shallow image encoders for guided upsampling can introduce feature leakage, fragmentation, and blur [2]. ViT-Up addresses these issues by replacing external image guidance with layer-wise query construction drawn from intermediate ViT hidden states [1]. The approach enables feature prediction at arbitrary continuous image coordinates while preserving alignment with the backbone feature space [2]. In experiments, ViT-Up consistently outperformed state-of-the-art image-guided upsamplers across dense prediction and semantic correspondence benchmarks [1]. Using the DINOv3-S+ backbone, ViT-Up improved over prior methods by up to +2.07 mIoU on the Cityscapes dataset and +4.17 [email protected] on SPair-71k [2]. When scaled to the larger DINOv3-B backbone, gains increased to +3.36 mIoU and +8.09 [email protected], indicating the framework scales favorably with backbone capacity [2]. The paper was submitted under the Computer Vision and Pattern Recognition category on arXiv, an open-access repository of electronic preprints that, as of late 2024, receives approximately 24,000 new articles per month [6].