RedactionBench

A new benchmark called RedactionBench aims to measure how well artificial intelligence systems can redact personally identifiable information, revealing that even the most advanced models struggle when context determines what should be hidden, according to research submitted on 17 Jun 2026 [1]. The benchmark comprises 200 manually annotated documents spanning 11 domains, mostly drawn from real-world sources [1]. The researchers argue that existing evaluations treat redaction as a simple entity-recognition task, ignoring the principle of contextual integrity — the idea that a phone number in a public directory carries different privacy implications than the same number in a medical record [1]. To address this, they introduce R-Score, a character-level metric designed to treat semantically similar redactions equally and ignore superficial formatting differences, such as how a phone number is masked [1]. Tests across 35 model families, including Named Entity Recognition models, entity-extraction Small Language Models, and frontier models equipped with agentic tools, showed that contextual redaction remains unsolved [1]. A human evaluation involving more than 80 users underscored the difficulty: annotators agreed with target labels on mandatory redactions 89.4 percent of the time and on safe text preservations 94.1 percent of the time, but agreement dropped to 47.7 percent for contextual redactions [1]. This split illustrates the subjective nature of contextual privacy and motivated the design of R-Score, which separates contextual ambiguity from strict precision [1]. The release of RedactionBench provides a standardized baseline for future privacy-preserving systems [1]. The work arrives as large language models are deployed in sensitive domains where data cleaning is a prerequisite, yet the tools for measuring redaction quality have lagged behind the capabilities of the models themselves [1]. The benchmark and metric are intended to spur more efficient model design and consistent evaluation practices [1].