Beyond the Autoregressive Horizon: A Comprehensive Survey of Diffusion Models, World Modelling, and State Space Models for Code

A survey submitted to arXiv on 9 Apr 2026 argues that autoregressive language models, despite driving progress in automated software engineering, face structural limits in code reasoning that emerging paradigms like diffusion models and world models could address [1]. The paper, titled "Beyond the Autoregressive Horizon," contends that the next-token prediction mechanism central to current large language models introduces bottlenecks in global planning, long-range dependency maintenance, and grounding in program execution semantics [1]. The authors note a heavy skew in existing literature toward autoregressive approaches and instead survey three alternative architectures. Diffusion Models generate code through a holistic denoising process, which the survey states can capture long-range syntactic constraints often missed by autoregressive systems [1]. Code World Models simulate execution states to support reasoning, while State Space Models offer linear-time efficiency for processing massive contexts [1]. The discussion of diffusion models draws on principles with a long computational history. Monte Carlo methods, which rely on repeated random sampling to obtain numerical results, were conceptualized by Polish mathematician Stanisław Ulam and are widely used for optimization, numerical integration, and modeling phenomena with significant input uncertainties [2]. These stochastic foundations underpin the denoising diffusion probabilistic models now being adapted for structured outputs such as source code. By connecting these architectural developments with findings from cognitive neuroscience, the survey outlines directions for building what it calls "System 2" code generation agents — systems capable of more deliberate, multi-step reasoning [1]. The work arrives as the broader machine-learning community grapples with dataset complementarity and transfer learning. Recent research has explored how models trained on large datasets, such as OC20, can improve performance on smaller, related datasets through joint training or fine-tuning, a technique that has proven useful in catalysis informatics and drug discovery [4]. The survey does not include direct experimental results but provides a comprehensive review of the theoretical underpinnings and potential integration of these non-autoregressive paradigms [1]. The authors frame the shift as a necessary step to overcome logic and scaling bottlenecks inherent in next-token prediction, though they acknowledge that the existing literature remains dominated by autoregressive methods [1].