GRACE: Step-Level Benchmark for Faithful Reasoning over Context

A new benchmark called GRACE aims to catch subtle reasoning failures in large language models by evaluating the faithfulness of each step in a chain-of-thought response, rather than just the final answer [1]. The benchmark, introduced in a paper posted to the arXiv preprint server, is described as the first human-annotated, step-level faithfulness benchmark with a data-driven error taxonomy for context-grounded textual reasoning [1][2]. It covers chain-of-thought traces from 10 models across 4 source datasets, with each step annotated for faithfulness, error category, and a natural language explanation [1][2]. The project's taxonomy, discovered through unsupervised clustering, organizes failures into two tracks: GRACE-Inference for deductive errors and GRACE-Grounding for factual grounding errors, each containing four categories [1][2]. The evaluation set is human-annotated and designed to be challenging [1][2]. The researchers report that current models show substantial room for improvement [1][2]. In a practical application, integrating step-level faithfulness signals into reinforcement learning pipelines improved both downstream accuracy and reasoning reliability [1][2]. The work addresses a known limitation of chain-of-thought prompting, where individual reasoning steps can silently deviate from source evidence even when a final answer is correct [1][2]. The paper was submitted on 15 June 2026 and is hosted on arXiv, an open-access repository for electronic preprints that, as of late 2024, receives about 24,000 submissions per month [9]. The research appears in the Computation and Language category, a field closely tied to the development of large language models, which are machine learning models with many parameters trained on vast amounts of text for tasks such as language generation [11][4].