KCSAT-ML: Probing Reasoning Models with Nationwide-Cohort Human Difficulty

A new benchmark called KCSAT-ML uses a decade of Korean college-entrance math problems to test reasoning models against per-item difficulty signals derived from hundreds of thousands of human test-takers, according to research published June 9 on arXiv [1]. The dataset spans the 2014 through 2025 administrations of the Korean College Scholastic Ability Test, known as Suneung, and contains 664 problems. A 339-item core set carries official per-item error rates from the nationwide cohorts that sit the exam each year [1][2]. The authors pair the benchmark with a metric they call Difficulty-aligned Reasoning Gain, or DRG, which measures whether a model’s errors cluster on items humans found difficult or on items humans found easy [1]. Across a range of vision-language models, and large language models accessed via optical character recognition, the study reports three patterns. First, low-budget accuracy collapses on the high-human-error tail at every model size. Second, test-time scaling raises token use roughly linearly with cohort error rate, while accuracy gains follow a non-monotonic curve. Third, within a single model family, test-time scaling flips between anti-scaling on the hardest items and overthinking on easier ones [1][2]. On the DRG metric, models with near-identical aggregate accuracy can sit at near-opposite values. One model gets wrong what humans also find hard, while another solves the hardest items yet fails on items humans find easy — a contrast that aggregate accuracy hides [1][2]. The researchers note that most existing math reasoning benchmarks lack a per-item difficulty signal grounded in actual human performance, which KCSAT-ML is designed to supply [1]. The code and a dataset builder will be open-sourced on GitHub under the naver-ai organization [1][2]. The work was submitted to arXiv on June 9, 2026, and is available in both PDF and HTML formats [1].