From Tokens to States: LLMs as a Special Case of World Models and the Continuous Path Beyond

A new preprint argues that large language models and world models are not opposing approaches but points on a single continuous spectrum, challenging a prominent 2022 position by Yann LeCun that general intelligence requires abandoning autoregressive token prediction [1][2]. The paper, posted to the arXiv preprint repository on June 26, contends that the AI community has adopted an unnecessarily binary framing of the relationship between large language models (LLMs) and world models [1][2]. LLMs are trained to predict the next token in a sequence, while world models aim to simulate aspects of reality to enable planning and reasoning [2]. Yann LeCun argued in 2022 that achieving general intelligence requires moving away from autoregressive token prediction toward architectures that operate in latent spaces [1][2]. The new work pushes back, asserting that LLMs are a degenerate special case of world models in which the state space is the set of all token sequences and the only available action is appending a single token [1][2]. From this perspective, world models represent a strict generalization of LLMs rather than a replacement [2]. The authors map a natural continuum from next-token prediction to Joint Embedding Predictive Architectures, with intermediate stations that include multi-token prediction, future-summary prediction, and next-latent prediction [1][2]. Each step along this spectrum relaxes one of the constraints that define an LLM [2]. The paper notes that moving away from pure token prediction also progressively surrenders two practical advantages that have made LLMs trainable at scale: access to internet-scale self-supervised text data and a transformer architecture co-designed for discrete token prediction [1][2]. Transformers, which use attention mechanisms to model long-range dependencies in data, form the backbone of modern LLMs [4][8]. The preprint examines whether the transformer can generalize to continuous-state prediction or whether a fundamentally new architectural primitive will be required [2]. It also frames the data question as a cliff between self-supervised text and the instrumented, action-labeled environments needed for richer world models [2]. The preprint has not yet undergone peer review, consistent with arXiv's role as an open-access repository for scholarly e-prints that are moderated but not formally refereed [6]. The submission rate to arXiv was approximately 24,000 articles per month as of November 2024 [6]. The paper appears in the Computation and Language category, which covers natural language processing research [1].