CogCanvas: A Benchmark for Evaluating Multi-Subject Reference-Based Image Generation

A new benchmark called CogCanvas exposes a steep performance drop in multi-subject image generation, showing that current diffusion models fail to reliably bind objects and fashion items to specific people when group sizes exceed three subjects [1]. The benchmark, detailed in a paper submitted to arXiv on 14 June 2026, comprises 1,952 curated reference images drawn from 100 celebrity identities, 115 distinctive objects and fashion items, and 29 real-world background scenes, including landmarks [1]. From this pool, the authors constructed 1,361 compositional prompts covering group sizes of two to five people [1]. The curation pipeline relied on DINOv2-based deduplication, a two-stage aesthetic filter, and automated derivation of structured interaction and position graphs that serve as ground-truth supervision [2]. CogCanvas supports three evaluation tasks: reference-based multi-human-object generation, text-to-image compositional generation, and reference retrieval, all assessed under a unified six-axis protocol [2]. The authors introduced two new metrics tailored to the multi-reference setting. BG-Sim scores background fidelity on SAM 3-masked regions via DINOv3 feature similarity, while Attr-VQA uses a multimodal large language model to verify per-subject attribute binding and inter-person interactions against the structured graphs [2]. Large language models are machine learning systems with many parameters trained on vast amounts of text for natural language processing tasks [8]. Benchmarking five state-of-the-art methods revealed that every model degrades substantially as group size grows from two to five, with near-complete failure on object and fashion binding beyond three subjects [1]. The work addresses a gap in existing evaluation suites, which the authors say test only one axis at a time and do not jointly capture multi-identity composition with human-object interaction, background grounding, and spatial plausibility [2]. The paper appears on arXiv, a preprint server that accounts for roughly 95 percent of the paper URLs Hugging Face users have linked in their repositories [4]. Hugging Face and arXiv have collaborated to embed interactive demos directly alongside papers on arXiv abstract pages, allowing users to try state-of-the-art research without writing code [5]. The CogCanvas paper page on Hugging Face can link to associated models, datasets, or demo Spaces, and the authors can claim authorship to have the page verified [4].