On the Reliability of Cue Conflict and Beyond

A new dataset and evaluation framework called REFINED-BIAS aims to correct instability and ambiguity in how neural networks are diagnosed for shape-texture bias, according to a paper posted on arXiv [1]. The work identifies key flaws in the widely used cue-conflict benchmark and proposes a method for more reliable cross-model comparisons [1]. The cue-conflict benchmark has been influential in probing whether vision models prefer shape or texture, and in motivating the insight that a stronger, human-like shape bias is often associated with improved in-domain performance [1]. However, the paper's authors find that the current stylization-based instantiation of the benchmark can yield unstable and ambiguous bias estimates [1]. Stylization may not reliably instantiate perceptually valid and separable cues, nor does it control their relative informativeness [3]. Ratio-based bias scores can obscure absolute cue sensitivity, making models with vastly different cue utilization appear similar, and restricting evaluation to a small subset of labels can distort model predictions by ignoring the full decision space [3]. The authors note that an increase in a model's shape-bias score does not necessarily mean the model has become more shape-sensitive; it may simply indicate that texture sensitivity has deteriorated more severely [3]. Therefore, relative preference is informative only when interpreted alongside cue-specific sensitivity, rather than as a standalone measure of cue utilization [3]. To address these limitations, the paper introduces REFINED-BIAS, an integrated dataset and evaluation framework for reliable and interpretable shape-texture bias diagnosis [1]. The framework constructs balanced, human- and model-recognizable cue pairs using explicit definitions of shape and texture [1]. It then measures cue-specific sensitivity over the full label space via a ranking-based metric, enabling fairer cross-model comparisons [1]. The paper, authored by Pum Jun Kim, was initially submitted on 11 March 2026 and revised on 11 June 2026 [1]. Across diverse training regimes and architectures, REFINED-BIAS enabled fairer cross-model comparison, more faithful diagnosis of shape and texture biases, and clearer empirical conclusions, resolving inconsistencies that prior cue-conflict evaluations could not reliably disambiguate [2]. The code for the framework is publicly available [3].