Beyond Correctness: Enhancing Architectural Reasoning in Code LLMs via Scalable Labeling with Agentic Judgment

A team of researchers has introduced an agentic judging pipeline designed to improve how large language models handle architectural reasoning in software engineering tasks, a capability that has lagged behind basic code-generation benchmarks. The pipeline, described in a paper submitted to arXiv on June 12, 2026, uses a strong LLM as a scalable proxy for expert architectural evaluation. It comprises two components: the Architecture Complexity Judge (ACJ), which estimates the architectural understanding a task demands, and the Architecture Quality Judge (AQJ), which evaluates whether a patch conforms to repository-specific architectural conventions through source-grounded rubrics [1][2]. The authors note that such architectural understanding is prohibitively expensive to label manually and cannot be verified through tests alone [2]. Large language models are neural networks trained on vast amounts of text for natural language processing tasks, typically based on transformer architectures [3][5]. While they have substantially improved software engineering, real-world development requires a deeper grasp of system architecture that standard correctness benchmarks do not capture [1][2]. Language model benchmarks are standardized tests designed to evaluate performance on tasks such as language understanding, generation, and reasoning, and are maintained by academic institutions, research organizations, and industry players to track progress [4]. The researchers fine-tuned Qwen3 models at three scales — 8B, 14B, and 32B parameters — on a curated set of 3,360 instances [1][2]. On the SWE-bench Verified benchmark, the fine-tuned models achieved resolved rates of up to 27.2 percent, representing an improvement of up to 540 percent over the base model and 256 percent over unfiltered fine-tuning [1][2]. The trained models also showed strong cross-language generalization and consistent improvements in architectural patch quality, according to the paper [2]. The work arrives amid intensifying competition in the LLM landscape. Chinese firm DeepSeek, founded in July 2023, has drawn attention for producing models comparable to OpenAI’s GPT-4 and o1 at a fraction of the training cost, reportedly using approximately one-tenth the computing power consumed by Meta’s comparable Llama 3.1 model [9]. DeepSeek’s models are described as open-weight, meaning the exact parameters are openly shared but the training data is not openly licensed [9]. The new paper’s approach to scalable labeling via agentic judgment addresses a persistent bottleneck in AI development: the cost and difficulty of obtaining high-quality training signals for nuanced skills. The integration of demos and code repositories with research papers, such as the collaboration between Hugging Face and arXiv that allows users to find open-source demos directly on a paper’s abstract page, reflects a broader push toward reproducibility and accessibility in machine learning research [6][7][8].