Searching the Internet for Challenging Benchmarks at Scale

A new computational framework can automatically mine the Internet to build challenging benchmarks for artificial intelligence models, bypassing the need for human curation. The approach, described in a paper posted to arXiv, treats the web as a search space and uses a multi-armed bandit algorithm to locate the hardest test topics at a fraction of the cost of exhaustive evaluation [1]. The framework addresses a growing problem in AI evaluation: static benchmarks are saturating. As language models improve, they achieve near-perfect scores on fixed test sets, leaving little room to expose genuine weaknesses. Even expert-curated challenge sets quickly lose their diagnostic power after repeated use and targeted optimization [1][2]. Language model benchmarks are standardized tests that pair datasets with evaluation metrics to measure capabilities such as language understanding, generation, and reasoning [3]. Wenda Xu and collaborators model the Internet as a vast space of topics and formalize the search for difficult material as a multi-armed bandit problem. In this setup, each topic’s difficulty is unknown until the system performs an expensive sample-and-evaluate query. The researchers deploy an epsilon-greedy strategy that balances exploration of new topics with exploitation of known difficult ones. The method identifies the most challenging topics while exploring only 6% of the search space, yielding a 100 times cost reduction over exhaustive evaluation [1][2]. The team validated the framework on machine translation and knowledge question answering. The difficulty rankings it produced held steady across independent evaluation metrics, including GEMBA-SQA and MetricX, as well as across different languages and model architectures [1][2]. The paper was submitted to arXiv on September 30, 2025, and revised through May 2026 [1]. The concept of using the Internet as a distributed resource for large-scale computational problems has precedent. For more than two decades, the SETI@home project harnessed volunteer computing to analyze radio signals for signs of extraterrestrial intelligence, processing data from millions of personal computers before shifting to analysis of its accumulated dataset in 2020 [4]. The new benchmarking framework applies a different model — algorithmic search rather than distributed computing — but shares the principle of treating the web as an infrastructure for scientific inquiry.