TRAP: Benchmark for Task-completion and Resistance to Active Privacy-extraction

A new benchmark called TRAP reveals that large language models cannot simultaneously achieve high task accuracy and zero privacy leakage when handling sensitive documents, according to research posted on arXiv. The study evaluated 22 models and found that all exhibited non-trivial leakage of private information [1][2]. The benchmark, formally named Task-completion and Resistance to Active Privacy-extraction, was introduced in a paper submitted to the arXiv preprint repository on June 17, 2026 [1][2]. arXiv, which began in 1991 and now hosts over two million articles, serves as a primary distribution channel for research in computer science and other fields before formal peer review [6]. The TRAP framework addresses a growing tension in agentic AI systems: models deployed in document-intensive workflows, such as booking flights, must use private data like passport numbers to complete tasks but should never expose that data in responses because they cannot authenticate the end user [2]. Each TRAP scenario consists of a document containing private information, a task query requiring the model to invoke a tool using private fields, and an attack query designed to extract the same information in natural language [1][2]. The authors tested 22 models, including frontier proprietary systems and open-source models at multiple scales [1][2]. Every model family displayed measurable leakage, and the researchers found that stronger instruction-following capability correlated with higher leakage rates [1][2]. Prompt-based defenses reduced leakage but imposed a significant penalty on task accuracy. The study further demonstrated that prompt optimization cannot escape this trade-off [1][2]. The authors proved that for any softmax-based model, no soft-constraint defense—such as prompt engineering—can jointly deliver high task success and zero leakage probability [2]. Motivated by this impossibility result, the researchers proposed a structural mitigation called private field isolation, which replaces sensitive fields with hash keys before they reach the model. This approach largely prevented leakage while preserving task accuracy [2]. The paper was made available through arXiv's standard distribution channels, which include experimental community tools developed under the arXivLabs framework [4][5]. arXivLabs, launched in 2020, allows third-party collaborators to build features on top of the repository while adhering to arXiv's values of openness, community, and user data privacy [4].