IDP-Bench: Benchmarking ability of LLMs to protect personal information in interdependent privacy contexts

Researchers have introduced IDP-Bench, a benchmark designed to test how well large language models handle interdependent privacy — situations where one person’s data can be exposed by someone else’s actions, according to a paper posted to arXiv [1]. The benchmark is grounded in the Contextual Integrity (CI) framework, a theory that evaluates privacy based on the flow of information within specific social contexts [1][2]. The authors note that while LLMs are increasingly deployed as personal AI assistants with access to sensitive user data, prior privacy evaluations have focused almost exclusively on individual-level risks, overlooking interdependent privacy scenarios [2]. IDP-Bench is the first benchmark to target this gap [1][2]. Eight open-source LLMs were evaluated across three levels of interdependent privacy reasoning using two LLM judges [1][2]. The results showed strong co-ownership recognition: six of the eight models exceeded 90% accuracy in identifying when information belonged to multiple people [1][2]. However, the models consistently struggled with other tasks. Seven of the eight models scored below 74% on identifying IDP-specific parameters, such as secondary subjects — individuals who are affected by a data disclosure even if they are not the primary subject [1][2]. Models also had difficulty judging the appropriateness of sharing information. Five of the eight models scored below 77% on this measure [1][2]. The paper reports that while the ability to judge sharing appropriateness improved with model scale, performance tended to decline in smaller models [2]. Prompt sensitivity remained high on IDP-specific questions, meaning small changes in how a question was phrased could produce different answers [2]. The CI framework, which underpins the benchmark, defines privacy not as a simple binary of public versus private but as a function of information attributes, actors, and transmission principles within a given context [2]. The benchmark’s design reflects this complexity, testing whether models can parse these nuanced parameters when multiple people’s data is at stake. The researchers have made the data and code for IDP-Bench publicly available on GitHub [1][2].