ActiveSAM: Image-Conditional Class Pruning for Fast and Accurate Open-Vocabulary Segmentation

A new inference framework called ActiveSAM accelerates open-vocabulary semantic segmentation without requiring any target-dataset training, according to a preprint posted to arXiv on June 15 [1]. The method builds on Meta’s Segment Anything Model 3 and prunes irrelevant class vocabularies before full-resolution decoding [2]. ActiveSAM addresses a core inefficiency in applying SAM 3 to open-vocabulary semantic segmentation: the standard approach runs full-resolution decoding over the entire dataset vocabulary, even though each image typically contains only a small subset of classes [2]. The framework instead estimates an image-conditioned active set from a low-resolution presence preview, then decodes only the retained classes at full resolution using bucketed prompt multiplexing with the frozen SAM 3 decoder [2]. The preview stage relies solely on class-presence evidence and skips unnecessary segmentation-head computation, while a final margin-aware background calibration step suppresses low-confidence pixels [2]. The authors report that ActiveSAM requires no weight updates and no oracle class-presence labels [2]. Across eight open-vocabulary semantic segmentation benchmarks, the framework outperforms the current state-of-the-art SegEarth-OV3 by approximately +1.4 mIoU on average while running up to 5.5x faster on large-vocabulary datasets [2]. The paper also notes that ActiveSAM exhibits the strongest robustness under image corruption designed to simulate real-world distribution shift, which the authors suggest makes it suitable for noisy-input domains such as autonomous driving and embodied AI [2]. The preprint was submitted to arXiv’s Computer Vision and Pattern Recognition section [1]. arXiv, founded in 1991, is an open-access repository of electronic preprints that are moderated but not peer-reviewed; as of November 2024, the platform receives about 24,000 new articles per month [6]. The repository hosts papers across mathematics, physics, computer science, and related fields, and has grown from half a million articles in 2008 to over two million by the end of 2021 [6]. Code for ActiveSAM is available on GitHub through the VILA-Lab organization [2].