From Simulation to Enaction: Post-trained language models recognize and react to their own generations

Post-trained language models exhibit a measurable shift in behavior when generating their own responses, producing output distributions with entropy three to four times lower than when processing externally provided text, according to a new study [1]. The paper, posted to arXiv on May 25, examines how models change after post-training, the process that follows initial pretraining on vast text corpora. During pretraining, models act as passive predictors with no incentive to model the consequences of their own outputs [1]. Post-training alters this dynamic. A model producing its own responses can benefit from recognizing that it is on-policy, and the researchers present evidence that post-trained models do exactly that [1]. This recognition is implicitly encoded in their output distributions. Across model families and size classes, on-policy output distribution entropy is 3--4$\times$ lower than off-policy entropy [1]. The authors trace part of this effect to an internal representation of input surprise. The model tracks the unlikeliness of the most recent input token according to its prior predictions, and this tracking causally modulates output entropy [1]. One manifestation appears with open-ended prompts. Post-trained models, unlike their pretrained counterparts, collapse their uncertainty over the topic of their upcoming response before emitting the first output token. Violating this cached intention with a different-topic prefill results in higher output entropy [1]. The study also tested whether models can distinguish on-policy contexts from prefills via explicit verbal report. The models can, but the researchers note that this explicit recognition routes through a different mechanism than the implicit recognition seen in the entropy measurements [1]. The findings arrive amid broader discussions about how artificial systems might develop forms of self-modeling. The concept of embodied cognition, which investigates how cognition is shaped by the bodily state and capacities of an organism, has long argued that perceptual systems and bodily interactions with the environment are essential to cognitive functions such as memory recall and meaning attribution [3]. While language models lack physical bodies, the paper's framing of post-training as a shift from passive prediction to a form of enaction suggests a computational parallel to the situatedness that embodied cognition theorists emphasize [3]. Military strategists have used simulation-based training for centuries. Modern wargaming was invented in Prussia in the early 19th century, and the Prussian military eventually adopted it as a tool for training officers and developing doctrine [5]. The paper's distinction between models that merely predict and models that must operate within the consequences of their own outputs echoes the difference between studying a battle plan and executing one while under fire.