DataComp-VLM: Improved Open Datasets for Vision-Language Models

A new benchmark called DataComp-VLM aims to standardize how researchers evaluate the curation of large-scale training datasets for Vision-Language Models, addressing a gap in systematic comparison tools for the field [1]. The benchmark, introduced in a paper posted to arXiv on June 26, 2026, collects 160 datasets spanning four data types: image-caption pairs, multimodal interleaved documents, text-only, and instruction-tuning data [1][2]. The assembled corpus, named DCVLM, contains 6 trillion multimodal tokens [1][2]. It allows participants to test curation strategies — including filtering, mixing, formatting, and sampling — across model sizes ranging from 1 billion to 8 billion parameters and token budgets from 6.25 billion to 200 billion [1][2]. Models trained under the benchmark are evaluated on a suite of up to 52 downstream benchmarks covering nine domains [1][2]. The researchers conducted extensive experiments and reported that data mixing, rather than filtering, proved central to building a high-quality training dataset [1][2]. Instruction-heavy mixtures scaled better than caption-heavy ones, with performance gains widening at larger scales [1][2]. The resulting dataset, DCVLM-Baseline, trained an 8-billion-parameter VLM to 63.6% accuracy on a 33-task core suite using 200 billion training tokens [1][2]. That result represents an improvement of 5.4 percentage points over FineVision, which the authors describe as the state-of-the-art open VLM training dataset [1][2]. The release of DataComp-VLM contributes to a broader push for reproducible, data-centric artificial intelligence research. The field of vision-language modeling has expanded rapidly, yet systematic methods for comparing data curation pipelines have remained limited [1][2]. The benchmark and its accompanying artifacts will be made publicly available, according to the paper [1][2].