SAMTok: Representing Any Mask with Two Words

A new method called SAMTok compresses any image region mask into just two discrete tokens, allowing large vision-language models to perform pixel-level tasks using standard next-token prediction and reinforcement learning, according to a paper published on arXiv [1]. The approach, detailed by Yikang Zhou and collaborators, addresses a persistent bottleneck in multi-modal large language models (MLLMs). While these models handle text and images, adding precise pixel-wise capabilities has typically required complex region encoders and specialized decoders that complicate scaling [2]. SAMTok sidesteps this by treating masks as a new language. It converts any region mask into two special tokens and reconstructs the mask from them with high fidelity, enabling base MLLMs to learn segmentation and grounding without architectural changes [2]. The tokenizer builds on the SAM2 architecture and was trained on 209 million diverse masks using a mask encoder and a residual vector quantizer [2]. The researchers formatted 5 million data samples for mask understanding and generation [2]. When integrated with the QwenVL model series, the resulting system, QwenVL-SAMTok, achieved state-of-the-art or comparable performance across a suite of benchmarks: region captioning, region visual question answering, grounded conversation, referring segmentation, scene graph parsing, and multi-round interactive segmentation [2]. The team also introduced a textual answer-matching reward to enable efficient reinforcement learning for mask generation, reporting substantial gains on the GRES and GCG benchmarks [2]. The work lands as the broader AI community continues to push for more accessible and reproducible research. arXiv has integrated with Hugging Face Spaces to let authors and the community attach interactive demos directly to paper pages, allowing anyone to test models in a browser without writing code [3][4]. The SAMTok paper’s arXiv page includes a Demos tab, part of the arXivLabs framework that lets collaborators build features on top of the repository [1][4]. Hugging Face Spaces, launched in October 2021, now hosts over 12,000 such demos, built with tools like Gradio and Streamlit [3]. The release also coincides with a period of intense competition in open-weight AI development. Chinese firm DeepSeek, founded in July 2023, drew global attention in January 2025 when its DeepSeek-R1 model matched leading proprietary systems at a fraction of the reported training cost—$6 million versus the estimated $100 million for OpenAI’s GPT-4 [6]. DeepSeek’s models are released under permissive open-source licenses, primarily the MIT License, though training data is not openly licensed [6]. The SAMTok authors have similarly made their code and models publicly available [2].