Revisiting Model Stitching In the Foundation Model Era

A new study finds that heterogeneous Vision Foundation Models — trained on different datasets with different objectives — can be reliably stitched together, challenging prior assumptions that model stitching only works between models trained on the same data. The research, led by Zheda Mai and colleagues, revisits the technique of model stitching, which connects the early layers of one model to the later layers of another through a lightweight stitch layer [1]. Earlier work had shown that small models trained on identical datasets, such as ResNet-18 on CIFAR-10, remained stitchable despite different initializations or training objectives [3]. The new study extends this question to large Vision Foundation Models (VFMs) including CLIP, DINOv2, and SigLIP 2 — models that vary substantially in their training data, objectives, and whether they process vision alone or vision with language [2]. The authors introduce a systematic protocol that examines stitch points, stitch layer families, training losses, and downstream tasks [1]. They found that two conventional approaches to training the stitch layer — matching intermediate features at the stitch point or optimizing the task loss end-to-end — struggle to preserve accuracy in the VFM setting, with failures most pronounced at shallow stitch points [5]. A closer analysis revealed why these methods falter. Low feature-matching error at the stitch point did not guarantee aligned final representations, particularly for shallow stitches. Meanwhile, task-loss training faced an optimization challenge: gradients from the prediction head had to traverse frozen target-model layers to reach the stitch layer, weakening the training signal [5]. The team identified a more effective approach: applying a feature-matching loss at the target model's penultimate layer, rather than at the stitch point itself [2]. This simple modification made heterogeneous VFMs reliably stitchable across vision tasks. For deep stitch points, the resulting stitched model could outperform either constituent model while incurring only a small inference overhead from the stitch layer [1]. Building on these results, the researchers propose the VFM Stitch Tree, an architecture that shares early layers across multiple VFMs while retaining each model's later layers [4]. The design offers a controllable trade-off between accuracy and latency, which could benefit multimodal large language models that rely on several vision encoders simultaneously [2]. The study reframes stitching from a diagnostic tool for probing representational similarity into a practical method for combining complementary strengths of different foundation models [1].