Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extraction

A new confidence engine called ExtractConf aims to address a critical weakness in large language model document processing: the inability to reliably flag when an extraction is wrong, a failure researchers say is more dangerous than a missing value in high-stakes workflows [1]. The system, detailed in a paper submitted to arXiv on June 23, 2026, targets pipelines in financial reconciliation, compliance verification, and procurement automation where a silently incorrect extraction can cause cascading errors [1]. The core problem is not extraction accuracy but confidence estimation—knowing field by field whether a result can be trusted or should be deferred to a human reviewer [1]. Standard approaches, including token-level log-probabilities, verbalized confidence scores, and multi-sample self-consistency checks, all collapse toward all-positive behavior at practical thresholds, offering no reliable separation between trustworthy and untrustworthy extractions [1]. ExtractConf generates confidence estimates by comparing two structurally different readings of the same document [1]. A field-guided “Hunter” call extracts each field under schema-slot completion pressure, while a document-guided “Mapper” call scans holistically and surfaces values grounded in document content [1]. This asymmetry produces different failure modes: Hunter hallucinates values for absent fields, while Mapper misses visually non-salient ones, making their disagreement independently informative [1]. The engine fuses cross-call disagreement with LLM-internal uncertainty signals, OCR quality, image quality, and spatial layout into a classifier that requires no domain-specific rules or retraining [1]. On the DocILE dataset, which comprises 55-field invoices with a 26% failure rate, ExtractConf achieved a 0.928 ROC AUC and reduced selective prediction risk by 70% over a logprob-mean baseline [1]. At 80% coverage, accuracy reached 99.1%, a threshold the authors describe as enabling a practical human-in-the-loop workflow [1]. In a zero-shot transfer test on CORD receipts, the system attained a 0.858 AUC, and lightweight Lasso recalibration reduced expected calibration error by 89% and the Brier score by 43%, confirming the signals generalize across document domains [1]. The research addresses a growing need as organizations deploy LLMs for document understanding at scale. While the primary paper focuses on invoices and receipts, the broader challenge of reliable confidence estimation in machine learning systems has been studied across domains, including catalysis informatics, where transfer learning between datasets has been used to improve model performance on smaller, specialized corpora [4]. The ExtractConf approach, by grounding confidence in structural disagreement rather than a single model’s internal probabilities, offers a distinct path toward safer automation in document-intensive industries [1].