EngTrace: A Symbolic Benchmark for Verifiable Process Supervision of Engineering Reasoning

A new benchmark called EngTrace aims to test whether large language models can reason through safety-critical engineering problems, moving beyond simple answer-checking to verify the step-by-step logic behind their solutions [1]. The benchmark, detailed in a paper posted to the arXiv preprint repository, was built from 90 parameterized templates that each generate unique problem instances [1]. Those templates span three major engineering branches, nine core domains, and 20 distinct areas, producing 1,350 test cases designed to stress-test generalization across different physical scenarios [1]. The work was submitted by Ayesha Gull and revised most recently in June 2026 [1]. Existing evaluation tools such as MMLU, MATH, and HumanEval assess isolated cognitive skills but do not capture the physically grounded reasoning that engineering demands, where scientific principles, quantitative modeling, and practical constraints must converge [1]. EngTrace addresses that gap with a two-stage evaluation framework. The first stage uses automated procedural checks to validate intermediate reasoning traces, and the second stage employs a heterogeneous AI Tribunal to assess trace fidelity alongside final answers [1]. The authors evaluated 27 leading large language models and found a trade-off between numeric precision and the faithfulness of the reasoning traces [1]. They also identified what they describe as a complexity cliff, a point at which abstract mathematical pre-training no longer translates into the integrative reasoning required for advanced engineering tasks [1]. Large language models are a type of machine learning system trained on vast amounts of text for tasks such as language generation [8]. The arXiv repository where the EngTrace paper appears was founded in 1991 and hosts preprints across physics, mathematics, computer science, and other fields; as of late 2024, it was receiving about 24,000 submissions per month [6]. Papers on arXiv are moderated but not peer-reviewed [6].