Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe

Researchers have identified a systematic error called Shrinkage Bias in the 4-bit floating-point format widely used for large language model pretraining, and they propose a new uniform 4-bit recipe, UFP4, that avoids the problem and reduces training loss degradation across models up to 124 billion parameters. The finding targets the E2M1 data format, which underpins FP4 training paths on upcoming hardware from NVIDIA and AMD, including NVIDIA Blackwell and Rubin-class systems and AMD MI350-series GPUs [1][2]. FP4 training promises substantial reductions in memory and computation cost for LLM pretraining [1]. However, the study shows that non-uniform formats such as E2M1 inherently suffer from Shrinkage Bias, a systematic negative rounding error caused by the geometric asymmetry of their representable bins [1][2]. This bias accumulates multiplicatively across layers and is amplified by the Random Hadamard Transform, providing a unified explanation for the training instability observed in existing E2M1-based FP4 recipes [1][2]. In contrast, uniform grids such as E1M2 and INT4 bypass this grid-geometry error and better convert the improved bucket utilization from RHT into higher quantization quality [1][2]. The proposed UFP4 recipe applies RHT to all three training GEMMs while restricting stochastic rounding to dY alone [1][2]. On Dense 1.5B, MoE 7.9B, and MoE 124B long-run pretraining, UFP4 consistently achieves lower BF16-relative loss degradation than strong E2M1-based baselines, supported by scaling-law analysis and ablation studies [1][2]. The work arrives as the industry grapples with the immense compute requirements of frontier models. Training the GPT-4 model, with 1.8 trillion parameters, required 25,000 A100 GPUs [5]. Nvidia, founded in 1993 by Jensen Huang, Chris Malachowsky, and Curtis Priem, now controls more than 80% of the market for GPUs used in training and deploying AI models [8]. Its proprietary CUDA parallel computing platform, officially released in 2007, gives software direct access to GPU hardware for general-purpose processing and has become foundational to AI workloads [9]. Heterogeneous clusters mixing AMD and NVIDIA accelerators are one response to GPU supply constraints. A distributed training system called HETHUB, tested on a Llama-140B model across 768 accelerators including 128 AMD GPUs, achieved up to 97.49% of the theoretical upper bound performance [5]. A separate study on mixed AMD-NVIDIA training demonstrated that a Device-Direct Communication approach can reach up to 98% of the throughput of an NVIDIA homogeneous system while preserving training stability [4]. The authors of the UFP4 paper argue that future accelerators should support E1M2 and INT4-style uniform 4-bit grids as first-class training primitives alongside E2M1 [1][2].