ConflictScore: Identifying and Measuring How Language Models Handle Conflicting Evidence

A new metric called ConflictScore aims to quantify how large language models handle situations where their source documents contain both supporting and contradictory evidence, a scenario existing factuality tests overlook [1]. The framework, introduced in a preprint on arXiv, addresses a gap in current evaluation methods. Standard metrics for factuality and faithfulness determine if a model’s answer is supported or contradicted by its grounding documents, but they fail when both types of evidence are present simultaneously [2]. ConflictScore decomposes a model’s response into atomic claims and labels each one against every grounding document [1]. It then produces two scores: ConflictScore-Count (CS-C), which measures the proportion of claims that exhibit conflicts, and ConflictScore-Ratio (CS-R), which captures the balance between supporting and contradicting evidence [2]. The researchers also built ConflictBench, a benchmark designed to test models on diverse forms of conflict, including ambiguity, contradiction, and divergent opinions [1]. In experiments, ConflictScore detected overconfident claims across different domains and functioned as a corrective feedback mechanism that improved truthfulness on the TruthfulQA dataset [2]. The work arrives as the field of AI safety grapples with the reliability of increasingly capable models. AI safety is an interdisciplinary field concerned with preventing accidents and misuse, and it encompasses AI alignment, which seeks to ensure systems behave as intended [4]. The ability to recognize and appropriately express uncertainty when faced with conflicting information is a component of robust behavior. The psychological concept of cognitive dissonance describes the mental discomfort people experience when holding inconsistent beliefs, which they are motivated to resolve, often through rationalization or selective perception [5]. A language model that confidently asserts one side of a contested issue without acknowledging the dispute fails to mirror this human awareness of inconsistency. The preprint was posted on arXiv, an open-access repository for scientific papers that are moderated but not peer-reviewed [7]. The platform, launched in 1991, now receives about 24,000 submissions per month and serves as a primary distribution channel for research in computer science, physics, and mathematics [7]. The ConflictScore paper’s authors propose that their metric can serve as a diagnostic tool for model developers seeking to reduce overconfident or one-sided outputs when source material is genuinely divided [1].