A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions

A tertiary review of 30 secondary studies on large language model-based code generation finds that while benchmark accuracy is strong, real-world generalization remains weakly supported and robustness is fragile across tasks, according to a preprint posted to arXiv in May 2026 [1][2]. The study, which followed systematic review guidelines and used backward-and-forward snowballing, identified 30 secondary studies published between 2017 and 2025, with rapid growth in the number of publications since 2023 [1][2]. Evidence was synthesized using the SWEBOK knowledge areas and the HELM framework [1][2]. The authors report that accuracy appears strong on benchmarks but is weakly supported for real-world generalization, while robustness proves fragile across different tasks and configurations [1][2]. Efficiency constraints were found to be pervasive, and issues of toxicity and bias remain under-reported in the existing literature [1][2]. The review identifies dominant challenges concerning economic feasibility, evaluation validity, and socio-technical integration [1][2]. Future research directions point toward domain-aware model improvement and the need for holistic, standardized evaluation frameworks [1][2]. The authors conclude that LLM-based code-generating tasks represent a fast-maturing yet unevenly evaluated research area, highlighting the need for domain-aware model improvements and holistic, standardized evaluation that addresses efficiency and associated costs [1][2]. Inter-rater agreement statistics were used to audit extraction reliability, and study quality was formally assessed as part of the review methodology [1][2]. The search strategy encompassed multiple digital libraries, supplemented by screening steps designed to capture relevant secondary evidence [1][2]. The paper notes that existing tertiary studies have provided little coverage of the broader effects and real-world integration of LLM-based code generation, motivating the consolidation of secondary evidence in this review [1][2]. The findings arrive as the broader field of artificial intelligence integration faces parallel challenges in other domains. In medical imaging informatics, for instance, AI and machine learning technologies are being developed for automation methods, disease classification, and improvements in diagnostic accuracy, yet integration faces challenges with data management and security [3]. Similarly, the concept of memory as an information processing system—encompassing encoding, storage, and retrieval—has long recognized that normal functioning, decay over time, and damage all affect accuracy and capacity, a parallel to the fragility observed in LLM-based code generation systems [4]. Preventive healthcare frameworks categorize interventions into primal, primary, secondary, and tertiary prevention levels, with tertiary prevention focusing on managing established disease and reducing complications [5]. The tertiary review methodology applied in the arXiv study mirrors this layered approach to evidence synthesis, consolidating secondary studies to guide future primary research and practical deployment [1][2][5].