When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation

Nearly half of 60 language model benchmarks analyzed have reached saturation, according to a recent study^[1]. Benchmark saturation occurs when models achieve near-perfect scores, limiting their ability to differentiate between models.

The study, which examined 14 properties related to saturation, found that rates of saturation increase with age^[1]. Expert curation, rather than public test data, was found to impact resilience to saturation. Design choices can extend benchmark longevity and inform more durable evaluation approaches. A separate study on large language models found that human-preference alignment constrains their ability to express feelings^[2]. Researchers used a rubric-based self-rewarding training scheme with Group Relative Policy Optimization (GRPO) to train models to exhibit human-like behavior. The trained models showed robustness to sycophancy-inducing questions and bias in disambiguated conditions, but degradation in truthful question-answering capability was observed. The findings suggest that benchmark saturation is a significant issue in AI development, and that new approaches are needed to create more durable benchmarks.