Measuring Whether LLM Tutors Teach or Solve: A Diagnostic for Educational Impact

A new diagnostic framework challenges the assumption that large language models strong at solving math problems are equally effective as educational tutors, finding only a moderate correlation between the two capabilities across publicly reported models [1]. The study, submitted to the arXiv preprint repository on 15 Jun 2026, examines whether public LLM tutoring benchmarks can distinguish learning-supportive behavior from mere answer production [1]. The authors propose a lightweight diagnostic based on the gap between solving-oriented and pedagogy-oriented benchmark performance [1]. Using public MathTutorBench leaderboard results, they found that these dimensions are only partially aligned: across eight publicly reported models, the correlation between solving and pedagogy composites is 0.421, and several models shift meaningfully in rank when evaluation moves from solving to pedagogy [1]. arXiv, which began on August 14, 1991, serves as an open-access repository of electronic preprints in fields including computer science and now receives about 24,000 article submissions per month [6]. The platform hosts arXivLabs, a framework enabling community collaborators to develop experimental tools that appear on article record pages [4]. The study appeared under the Computer Science > Artificial Intelligence category and was accessible through standard arXiv abstract-page features, including the Bibliographic Explorer and CORE Recommender [5]. The researchers further analyzed the public TutorBench sample and showed that agency-relevant behaviors are explicitly encoded in benchmark rubrics, especially in active-learning settings that reward guiding questions, calibrated hints, and non-disclosive scaffolding [1]. These findings suggest that educational-impact evaluation should not treat task success as a sufficient proxy for learning support [1]. The authors argue that public tutoring benchmarks can better support positive-impact evaluation by reporting solving-oriented and pedagogy-oriented scores separately and by making disclosure-sensitive, student-agency-preserving criteria more explicit [1]. Large language models are a type of machine learning model designed for natural language processing tasks, trained with self-supervised learning on vast amounts of text [8]. Their increasing proposal as educational tutors has prompted calls to measure the social impact of NLP systems in practice, motivating the diagnostic approach [1].