On the Stability of Prompt Ranking in Large Language Model Evaluation

A new study finds that the top-ranked prompt for large language models frequently changes under minor evaluation variations, undermining the reliability of standard prompt-selection workflows. The research, submitted to arXiv on 23 Jun 2026, examined prompt ranking stability across three open-weight LLMs and two benchmark tasks [1]. The authors introduced common sources of variability, including random seeds and limited evaluation subsets, and found that while overall rank correlations were often moderate to high, the identity of the top-performing prompt frequently shifted [2]. This instability leads to unreliable selection decisions when practitioners choose a single prompt for downstream use [2]. Prompt-based interaction has become the dominant way to use large language models. In a typical workflow, multiple candidate prompts are evaluated and the highest-ranked one is selected [2]. The study challenges the implicit assumption that these rankings remain stable under minor changes in evaluation conditions [1]. To address the problem, the researchers proposed a stability-aware selection strategy based on a lower confidence bound that accounts for both performance and variance [2]. Their results showed the approach improves robustness in unstable settings while staying competitive in more stable regimes [1]. The findings underscore the importance of accounting for evaluation uncertainty in both prompt selection and LLM benchmarking [2]. High-quality evaluation datasets are a cornerstone of machine-learning research, yet producing labeled training data remains difficult and expensive [4]. The study's focus on evaluation variability adds to a broader conversation about reproducibility in ML, where minor changes in data splits or random seeds can alter conclusions [2]. The work also arrives as LLM-based tools proliferate; for instance, the chatbot Grok, launched in November 2023, has been integrated into social platforms and hardware products, illustrating how quickly models move from research to deployment [3]. The paper does not include external quotes, relying instead on quantitative evidence from its experiments [1]. The authors argue that ignoring evaluation uncertainty can lead to overconfident prompt choices, a concern that parallels discussions in other fields where small perturbations in input data can shift model rankings [2][4].