Simulating Students' Java Programming Errors with Large Language Models

A new study examines whether large language models can generate synthetic Java programming errors that mimic real student mistakes, using a dataset of more than 74,000 authentic submissions to benchmark five models across three prompting strategies [1]. The research, posted to arXiv on June 12, 2026, draws on the CodeWorkout dataset, which contains 74,000+ unique student Java submissions spanning 37 distinct programming problems [1]. The authors evaluated five large language models under three prompting approaches: Input-Output, Chain-of-Thought, and iterative Self-Refine [1]. Performance was measured along two axes — diversity of error patterns and alignment with authentic student mistakes — and the study also tracked how results shifted across problems of varying difficulty [2]. Quantitative results showed a divergence: all models produced a wide range of errors, but their fidelity to real student submissions varied considerably [3]. Claude Sonnet 4 delivered the most balanced performance, maintaining relatively low edit distances from human code even as problem difficulty increased [4]. Other models, including GPT-5 and Gemini 2.5 Pro, exhibited substantially larger distances on medium- and high-struggle problems, meaning their synthetic errors drifted further from authentic student behavior [5]. To test whether the generated errors could pass as human, the researchers ran a blinded expert annotation study with 401 participants [1]. The annotators compared synthetic and authentic errors and found them functionally indistinguishable [1]. This qualitative finding bolsters the quantitative benchmarks, suggesting that under the right conditions, LLM-generated mistakes can serve as plausible stand-ins for real student submissions [2]. The study also identified a trade-off tied to problem difficulty. Higher-struggle-level problems elicited more diverse errors from the models, but those errors were less student-like [3]. The authors note that this pattern has implications for the design of intelligent tutoring systems and teachable agents, where synthetic errors could supplement scarce classroom data during early deployment of new programming tasks [1]. Obtaining representative student errors for newly designed assignments has historically been slow and expensive, since authentic submissions accumulate only after extensive classroom use [2]. The paper frames LLM-based simulation as a potential shortcut for learning-analytics pipelines and adaptive feedback systems, while cautioning that model choice and prompting strategy must be calibrated to the difficulty of the target problems [4].