Do VLMs Reason Like Engineers? A Benchmark and a Stage-wise Evaluation

A newly introduced benchmark, EngVQA, tests whether vision-language models can reason through engineering problems that demand interpreting technical diagrams and applying physical principles, according to a paper posted to the arXiv preprint server [1]. The benchmark contains 696 problems spanning five engineering subjects and pairs them with an eight-stage automated evaluation framework that scores each step of a model’s solution rather than only the final answer [1]. The framework’s scores closely track human judgment, recording a Pearson correlation of 0.975 and a mean absolute error of 0.67 on a 10-point grading scale [1]. Existing multimodal benchmarks have focused heavily on end-answer accuracy, offering little visibility into whether a model’s intermediate reasoning is physically consistent [1]. The authors argue that engineering tasks require a different capability: selecting the correct governing equations, reading force diagrams, and sustaining multi-step logic without producing answers that look plausible but violate physical laws [1]. When the researchers applied the framework to several open and closed-source vision-language models, they found “substantial limitations in current engineering reasoning capabilities” [1]. The paper does not name the specific models tested, but the results indicate that even state-of-the-art systems struggle with the structured, process-heavy demands of engineering problem solving [1]. Large language models, the text-processing backbone of many vision-language systems, are typically evaluated on benchmarks that measure factual accuracy, alignment, and reasoning, though biased or inaccurate training data can undermine reliability [8]. The EngVQA framework adds a layer of scrutiny by requiring models to demonstrate correct reasoning at each stage, a design the authors say is essential for applications in engineering education and technical decision-making where errors can propagate silently [1]. The paper was submitted to arXiv on June 9, 2026, by Mohamad Tawseeq Syed and collaborators [1]. arXiv, which was founded in 1991 and now receives roughly 24,000 submissions per month, hosts preprints that are moderated but not peer-reviewed [6]. The repository has long served as a primary distribution channel in computer science and physics, and its associated arXivLabs program allows third-party developers to build discovery tools on top of the article corpus [5][6].