Post-training is (Massive) Supervised Learning

A new position paper argues that the dominant method for training large language models—a massive post-training phase of supervised fine-tuning and reinforcement learning—represents a reversion to the “pre-train then fine-tune” paradigm of the BERT era, explicitly tailoring models to specific benchmarks [1]. The paper, posted to arXiv, contends that current post-training methodologies function primarily as a distribution-fitting mechanism, optimizing models for predefined behaviors and in-distribution datasets rather than fostering general capability [1]. The authors draw a direct historical parallel to the early days of LLMs, when task performance was heavily dependent on fitting models to narrow evaluation sets [1]. To support this claim, the researchers conducted an empirical comparison, fine-tuning both pre-trained and randomly initialized models on modern reasoning datasets. They found that models post-trained from scratch yielded “highly non-trivial performance,” underscoring the power of the post-training process itself [1]. The post-training phase under scrutiny typically combines supervised fine-tuning (SFT) with reinforcement learning from human feedback (RLHF). RLHF is a technique designed to align an AI agent with human preferences by first training a reward model on human ranking data, then using that model to optimize the agent’s policy through algorithms like proximal policy optimization [3]. While effective, the process is expensive; sourcing high-quality human preference data remains a challenge, and if the data is not drawn from a representative sample, the resulting model can exhibit unwanted biases [3]. The paper’s authors argue that the field must move beyond this extensive post-training for predefined behaviors. They advocate for a shift toward training procedures where models “learn how to learn,” a departure from the current cycle of fitting to static benchmarks [1]. This call for change is situated within the broader context of machine learning, a field built on statistical algorithms that learn from data and generalize to unseen examples, often through empirical risk minimization [4]. Modern LLMs are built on the transformer architecture, which uses attention mechanisms to model long-range dependencies in data and has become the foundation for large-scale text generation [5]. The paper suggests that the next evolution in training must break from the distribution-fitting mold to achieve more generally capable systems [1].