Precision Is Not Faithfulness: Coverage-Aware Evaluation of Grounded Generation with a Complete Oracle

Reference-free faithfulness metrics used to evaluate AI-generated text share a blind spot: they measure only precision and therefore reward models that say almost nothing, according to new research that proposes a coverage-aware evaluation method [1]. The study, led by Juan Salas and submitted to arXiv on June 8, 2026, argues that current faithfulness benchmarks cannot detect when a model abstains from stating relevant facts because they lack a complete, enumerable set of ground-truth facts to measure against [1][3]. The researchers turned to Formula 1 telemetry, a domain where strategic ground truth — pit laps, tire compounds, undercuts, and outcomes — can be derived deterministically and exhaustively from public timing data [3][4]. This completeness, absent in open-domain settings, allowed the team to measure recall, or coverage of relevant facts, alongside precision for the first time [3]. The benchmark comprises 7,253 decision instances across 157 races in English, Spanish, and Portuguese [1][4]. Results showed that the frontier model with the highest precision, grok-4.3 at 0.89, covered only 0.46 of the facts that mattered and ranked last by F1 score [6]. More verbose models, such as DeepSeek-V3.2, were slightly less precise but far more complete and rose sharply under the combined metric [6]. “The ranking by precision and the ranking by F1 disagree in every language,” the paper states, calling this its central finding [6]. A prompt ablation confirmed the low coverage was not an artifact of under-prompting; explicitly asking models to be thorough did not close the gap [5]. The same effect was replicated in a second complete-oracle domain using NOAA weather forecasts [1][2]. The researchers then fine-tuned small models ranging from 1 billion to 7 billion parameters on the complete oracle, which closed the precision-recall gap entirely, achieving an F1 score of approximately 0.98 and outperforming every zero-shot frontier system regardless of scale [1][2]. The proposed metric was validated through controlled perturbation and showed system-level Spearman agreement of 1.0 between a model-free regex extractor and a cross-family LLM extractor [1][4]. The team also developed a verifier-guided generation method that improves both precision and recall without requiring references [1][5]. The benchmark, structured annotations, metric, baselines, and an interactive demo have been released publicly [1][5].