Weight-Space Geometry of Offline Reasoning Training

A geometric analysis of six offline reasoning-training methods reveals that popular loss functions drive models into fundamentally different regions of weight space, even when trained on identical data, according to a preprint posted to arXiv on June 21, 2026 [1]. The study, led by Aleksandr Nikolich, compared supervised fine-tuning (SFT), rejection fine-tuning (RFT), direct fine-tuning (DFT), reward-weighted iterative fine-tuning (RIFT), offline Group Relative Policy Optimization (GRPO), and Direct Preference Optimization (DPO) on math reasoning tasks [1]. All methods were trained on identical rollouts from a single Qwen3-4B base model using attention-only Low-Rank Adaptation (LoRA) [1]. The researchers then probed the resulting weight deltas with cosine similarity, principal-angle subspace analysis, linear mode connectivity, and centered kernel alignment (CKA) [1]. Three of the methods — SFT, RFT, and RIFT — produced nearly colinear weight updates. Their cosine similarity measured at least 0.97, and the median top-1 principal angle across 144 modules was roughly 7 degrees [1]. Accuracy on the GSM8K benchmark was statistically indistinguishable among the three, ranging from 87 to 88 percent across 1,319 samples, with pairwise McNemar test p-values at or above 0.15 [1]. DFT diverged further in direction than any reward-weighted method, despite using the same training data [1]. Offline GRPO introduced a substantial component orthogonal to the SFT direction — approximately 67 percent globally and up to 86 percent in late layers — while remaining within the SFT loss basin [1]. DPO occupied a near-orthogonal subspace, exhibited a mode-connectivity barrier, and collapsed late-layer CKA to roughly 0.46 [1]. It also posted the highest accuracy scores: 93.5 percent on GSM8K, with a McNemar p-value below 10^-9 against every other method, and 30.0 percent on AIME26, compared with 3.3 to 10.0 percent for the other losses [1]. The authors note that DPO was trained with a learning rate 10 times smaller than the other methods, following standard convention, so the observed accuracy and update-norm gaps reflect the joint effect of loss-function choice and optimizer settings [1]. A learning-rate-matched comparison of DPO is left for future work [1]. The preprint appeared on arXiv, an open-access repository that hosts e-prints across physics, mathematics, computer science, and related fields, and which has served as a primary distribution channel for machine-learning research since its founding in 1991 [9].