CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models

A new benchmark called CASE-Bench aims to improve the safety evaluation of large language models by incorporating the context in which a query is made, according to a preprint posted on arXiv [1]. The framework challenges existing methods that assess model safety by examining individual questions in isolation. The benchmark, formally introduced in a paper submitted on 24 Jan 2025, assigns distinct, formally described contexts to categorized queries based on Contextual Integrity theory [1][2]. The authors argue that current safety benchmarks often focus solely on the refusal of problematic queries, overlooking the importance of context and potentially causing models to refuse queries in safe settings, which diminishes user experience [2]. To build the benchmark, the researchers recruited a sufficient number of annotators to ensure the detection of statistically significant differences among experimental conditions, a departure from previous studies that mainly relied on majority voting from just a few annotators [2]. An analysis using CASE-Bench on various open-source and commercial LLMs revealed a substantial and significant influence of context on human judgments, with a p-value of less than 0.0001 from a z-test [2]. The study also identified notable mismatches between human judgments and LLM responses, particularly in commercial models within safe contexts [2]. The paper, authored by Xiao Zhan and colleagues, has been revised several times, with the fourth version posted on 28 Jun 2026 [1]. The research was shared on arXiv, an open-access repository for electronic preprints in fields such as computer science that is not peer-reviewed [6]. As of late 2024, the repository receives about 24,000 articles per month [6].