A Two-Stage Statistical Framework for Evaluating Associative Interference in Large Language Models

A new statistical framework separates response compliance from task-consistent classification when evaluating associative interference in large language models, according to a paper submitted June 12. The two-stage method adapts the Implicit Association Test to a controlled, forced-choice format and was applied to three contemporary systems: Claude Sonnet-4, Gemini 2.5 Pro, and GPT-5 [1]. The study defines associative interference as reduced task-consistency in incongruent relative to congruent conditions [1]. While all three models showed uniformly high compliance with the structured response format, interference effects varied substantially across models and domains [1]. Claude Sonnet-4 exhibited strong interference in the Gender–Career domain, with a DeltaP of 0.086 and a 95% credible interval of [0.026, 0.173] [1]. The model also showed smaller but credible effects in the Gender–Science domain [1]. Gemini 2.5 Pro displayed attenuated interference, and GPT-5 exhibited minimal or no detectable interference across the domains tested [1]. The framework addresses a known methodological limitation in LLM bias evaluation: the conflation of refusal behavior with task performance [1]. By modeling item-level variability and isolating interference from compliance, the authors argue their approach provides a principled alternative to prior adaptations of human psychological paradigms [1]. The Implicit Association Test, originally developed for human subjects, measures the strength of automatic associations between concepts by comparing response times in congruent and incongruent pairing tasks [5]. Claude Sonnet-4 is part of a series of large language models developed by Anthropic, a San Francisco-based AI company founded in 2021 by former OpenAI members [7]. Anthropic trains its models using a technique called “constitutional AI” to improve ethical and legal compliance [6]. The company was privately valued at an estimated $965 billion in May 2026 [7]. GPT-5 and Gemini 2.5 Pro are competing frontier models from OpenAI and Google DeepMind, respectively. All three systems belong to the transformer family of neural network architectures, which rely on multi-head attention mechanisms to contextualize tokens within a context window [3]. The paper’s findings suggest that IAT-style associative asymmetries are not a universal property of LLMs but depend on model-specific characteristics [1]. The authors conclude that associative interference can be substantially mitigated in modern systems [1]. The study appears on arXiv under the statistics methodology category and has not yet been peer-reviewed [1].