RepSAM: Bridging Foundation Models to Robotic Vision via Representation-Guided Adaptation

A new parameter-efficient fine-tuning framework called RepSAM adapts foundation models for robotic vision, achieving near-parity with full fine-tuning while using a fraction of the computational resources, according to research submitted to arXiv on May 25, 2026 [1]. The framework targets a known weakness in applying general-purpose vision models to robotics: performance drops when models encounter unstructured environments. The researchers attribute this degradation to non-uniform representation shifts across transformer layers. Shallow layers exhibit substantial domain gaps, with Centered Kernel Alignment scores below 0.5, while deep layers transfer more effectively, registering CKA scores above 0.7 [1]. Deep learning architectures, including the transformer models underpinning this work, rely on stacking artificial neurons into multiple layers to process data for tasks such as classification and representation learning [3]. RepSAM addresses the layer-specific gap through a CKA-guided rank allocation strategy paired with a multi-modal fusion module, designed to handle challenging scenarios that include transparent objects and cluttered scenes [1]. On six benchmarks and robotic manipulation tasks, RepSAM achieved 89.0 percent mean Intersection over Union, reaching 97.9 percent of the 90.9 percent mIoU posted by full fine-tuning [1]. The parameter reduction is stark: trainable parameters drop from 632 million to 4.0 million, a 158-fold decrease [1]. Training completes in four hours on a single A100 GPU, compared with 384 GPU-hours for the full fine-tuning baseline, a 96-fold reduction in compute [1]. Against the DoRA adaptation method, RepSAM outperformed by 7.9 percentage points mIoU, and the gains were statistically significant at p < 0.01 [1]. When deployed in robotic manipulation, the efficiency improvements translated into a 12.0 percent absolute increase in success rates over a LoRA baseline using only RGB input [1]. The work was posted on arXiv under the Computer Science and Robotics category and is available through the arXivLabs experimental projects framework, which allows community collaborators to develop and share new features on the platform [1].