Large Language Model Selection with Limited Annotations

Researchers have introduced SELECT-LLM, a framework designed to identify the best large language model for a specific task while dramatically reducing the need for costly human annotations, according to a paper published on arXiv [1]. The framework, developed by Yavuz Durmazkeser, is described as the first active model selection system for large language models [1]. It operates by pinpointing a small set of queries whose annotations provide the most information for determining the top-performing model from a pool of candidates [2]. The selection rule relies on expected information gain, calculated from pairwise similarities between the text outputs generated by different models [2]. Because the method only requires access to these generated responses, it can be applied to both open-weight and black-box models without needing to examine internal architectures or model weights [2]. This approach addresses a practical bottleneck in the field. Large language models are neural networks trained on vast text corpora to handle tasks such as generation, summarization, and translation, but their evaluation is often resource-intensive [3]. Standard benchmarking attempts to measure reasoning, factual accuracy, and safety, yet fixed evaluation sets can be expensive to annotate [3]. The SELECT-LLM framework was tested across 23 datasets and 156 evaluated models, spanning diverse task families and multiple text evaluation metrics [2]. In every experimental setting, it outperformed the strongest baseline method [2]. The paper reports annotation cost reductions of up to 81.8% for selecting the single best model and up to 84.78% for identifying a near-best model [2]. These results suggest that active selection strategies, which draw on principles from statistical machine learning, can make model evaluation more efficient [5]. The work was submitted to arXiv on May 24, 2026 [1].