A Diagnostic Framework and Multi-Evaluator Audit of Evaluator-Driven Preference Dynamics in Self-Adapting LLM Agents

A new diagnostic framework called EPC reveals that measurements from proprietary large language model evaluators can become invalid within weeks, according to a preprint study. The framework detected version-conditional instability that makes single-snapshot evaluator studies unreliable [1]. The EPC framework comprises the Multimodal Preference Collapse Index (MPCI), an evaluator-indexed coupling matrix, and Jensen-Shannon divergence (JSD). Researchers applied it across eight experimental conditions, totaling 122 unique repetitions [1]. Coupling coefficients ranged from 0.00 to 1.18 across per-condition means, with a coefficient of variation of approximately 0.9 [1]. Four conditions showed strong coupling, while four collapsed to near-zero values [1]. The study's most informative measurement involved a drift in GPT-4o between May and June. An eight-run re-replication inverted the study's conclusion, demonstrating that the diagnostic instrument detected its own instability [1]. Self-evaluation consistently collapsed, with 97 percent of cases registering zero JSD and a JSD value of 0.003, though the authors note that floor effects are possible [1]. Machine learning, the broader field encompassing these models, relies on statistical algorithms that learn from data and generalize to unseen tasks without explicit programming [2]. The instability documented in the EPC study underscores a challenge inherent to such systems: performance can shift as underlying model versions change, even when the evaluation protocol remains identical [1]. An output-format confound analysis found a per-strategy aggregate correlation coefficient of 0.89, but the per-instance correlation dropped to 0.219 with a p-value of 0.093 [1]. The authors report the Preference Convergence Index as a metric for preference convergence. The finding is not any single coupling magnitude but the pattern of version-conditional instability that undermines single-snapshot evaluator studies [1]. The preprint, submitted on June 29, 2026, includes all data and the EPC framework for public release [1].