Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results

A new community effort aims to standardize the fragmented landscape of artificial intelligence evaluation by introducing a unified schema and repository called Every Eval Ever, according to a paper submitted to arXiv on June 12 [1]. The project addresses a core problem in AI research: evaluation results are stored in incompatible formats and scattered across leaderboards, papers, blog posts, and custom repositories, making comparison difficult [2]. Different evaluation frameworks can also produce divergent scores for nominally identical tests and record metadata inconsistently [2]. Every Eval Ever proposes a single JSON document schema that is source-agnostic, capable of ingesting results from evaluation harnesses and academic papers alike [2]. The schema optionally stores per-instance outputs to enable fine-grained analysis [2]. The accompanying community-crowdsourced database is hosted on Hugging Face, a platform that already provides infrastructure for sharing machine learning models, datasets, and interactive demos [4][5]. The repository currently catalogs 22,235 models, 2,273 unique benchmarks, and 31 evaluation formats [1][2]. The project also contributes automatic converters from popular formats, evaluation harnesses, and leaderboards to the unified schema [2]. Hugging Face has previously collaborated with arXiv to integrate machine learning demos directly alongside papers, allowing readers to try models without writing code [4][5][6]. The Every Eval Ever repository extends that ecosystem toward systematic evaluation tracking. The proliferation of large language models from organizations such as DeepSeek, which released its R1 model in January 2025, has intensified the need for consistent evaluation benchmarks [7]. DeepSeek's models are described as open-weight, with parameters shared openly but training data not openly licensed [7]. The broader field of large language models encompasses systems trained with self-supervised learning on vast text corpora [8]. Researchers including Douwe Kiela, who previously led the team at Meta AI that introduced retrieval-augmented generation and later served as Head of Research at Hugging Face, have contributed to the infrastructure that supports such evaluation efforts [9].