Can AI Draw Science? A Benchmark for Evaluating Scientific Figure Generation by Text-to-Image and Multimodal Models

A new benchmark called SciDraw-Bench has been introduced to evaluate how well AI models generate scientific figures, addressing a gap left by existing image-generation tests that focus on natural images rather than the specialized demands of diagrams, schematics, and graphical abstracts [1][2]. The benchmark consists of 32 structured scientific-figure generation tasks spanning eight figure types and ten disciplines [1][2]. Each task pairs a natural-language prompt with a machine-checkable specification that defines required labels, relations, components, conventions, and negative constraints [2]. The evaluation protocol measures four dimensions: Text Fidelity, which uses optical character recognition to assess label recall and character error rate; Semantic Correctness, judged by a vision-language model against the specification; Structural Quality; and Convention Adherence [2]. A meta-evaluation protocol and preliminary inter-judge reliability analysis are included, though human-rating validation remains ongoing [2]. Existing benchmarks such as GenEval, T2I-CompBench, and DPG-Bench evaluate compositionality, object counting, or photorealism in natural images, but none measure whether a generated scientific figure is usable in practice [1][2]. The researchers argue that usable scientific figures require correct and legible text labels, faithful depiction of entities and their relationships, coherent diagrammatic structure, and adherence to disciplinary drawing conventions [2]. In a pilot evaluation covering all eight figure types, a domain-specific system called SciDraw AI substantially outperformed representative general-purpose text-to-image models on every dimension and figure type [1][2]. The largest performance gaps appeared on semantic correctness and convention adherence, while text fidelity remained the most difficult dimension for all systems tested [1][2]. The authors also outline a planned code-to-figure baseline as a future extension [2]. The work arrives amid broader growth in generative AI, which since the 2020s has made text-to-image, audio, and video generation widely available [3]. The transformer architecture, introduced in 2017, accelerated advances in the field, contributing to an AI boom in the current decade [3]. As generative models are increasingly applied to specialized domains such as scientific communication, benchmarks like SciDraw-Bench aim to provide structured evaluation criteria that go beyond general image quality.