Creating and Evaluating K-12 GenAI Assessment Graders Through Context Engineering

A new study finds that large language models can score K-12 math and science assessments with substantial agreement to human raters, though their performance varies in English language arts and they are better suited to providing formative feedback than final grades. The paper, submitted to arXiv in 2026, evaluates an LLM grader that uses commercially available foundation models with context and prompt engineering to score student work against a rubric [1]. The researchers drew on data from the Massachusetts Comprehensive Assessment System (MCAS) and tested models including Claude Sonnet 4, Haiku 4.5, GPT-5, and GPT-5 Mini [2]. They measured agreement using Quadratic Weighted Kappa (QWK) and Proportional Reduction in Mean-Squared Error (PRMSE) [2]. The results showed that LLM graders, especially those based on foundational models with more parameters, achieve substantial agreement with human raters in mathematics and science assessments, while performances vary in ELA [2]. This suggests generic foundation models can be effective at scoring in given contexts [2]. Automated scoring systems and machine learning techniques have existed for decades, but generative AI now enables educators to implement standards-based grading with greater efficiency and scale [2]. Educational technology, or edtech, encompasses computer hardware, software, and educational theories used to facilitate learning, and the industry consists largely of privately owned companies producing and distributing these technologies for commercial purposes [5]. The study's findings arrive as AI development accelerates globally. The AI market in India, for instance, is projected to reach $8 billion by 2025, growing at a 40% compound annual growth rate from 2020 to 2025, with applications in education among other sectors [3]. Language model benchmarks are standardized tests designed to evaluate performance on natural language processing tasks, using datasets and corresponding evaluation metrics to measure capabilities in areas such as language understanding, generation, and reasoning [4]. High-quality labeled training datasets for supervised machine-learning algorithms are usually difficult and expensive to produce because of the large amount of time needed to label the data [6]. The MCAS data used in this study represents one such dataset applied to an educational context [2]. Beyond the numerical scores, the study's analysis of teacher and student feedback revealed strong acceptance of AI-generated narrative feedback but skepticism toward numerical scores [2]. This indicates that LLMs function most effectively as formative tools rather than summative evaluators [2]. The authors conclude that thoughtfully designed hybrid models combining AI efficiency with teacher judgment can reduce workload, enhance feedback quality, and support equitable assessment practices without displacing professional expertise [2].