Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention

A new study identifies the mechanism behind catastrophic training failures in low-precision transformer models, attributing the instability to biased rounding errors and low-rank attention representations that compound during flash attention computations [1]. The paper, posted on the arXiv preprint server, provides what its authors call the first mechanistic explanation for a persistent failure case in machine learning: the sudden and total collapse of the training loss when using low-precision arithmetic with flash attention [1]. The work was submitted by Haiquan Qiu and has undergone several revisions, with the latest version dated June 14, 2026 [1]. The analysis shows the failure is not a random artifact. Instead, it is driven by two intertwined phenomena. First, the attention mechanism begins to produce similar low-rank representations. Second, the inherent biased rounding errors in low-precision arithmetic compound on themselves. Together, these factors create a cycle of error accumulation that corrupts weight updates and derails the training process [1]. Low-precision formats are widely pursued for their computational efficiency, but training instabilities have long hindered their adoption [1]. The preprint repository arXiv, where the paper appears, was founded in 1991 and now hosts over two million articles, with a submission rate of about 24,000 papers per month as of late 2024 [10]. To validate their findings, the researchers introduced a minimal modification to the flash attention algorithm that mitigates the bias in rounding errors. This change stabilized the training process, confirming the analysis and offering a practical solution [1]. The code associated with the paper has been made publicly available on GitHub [2]. The paper's submission history shows the first version was uploaded on October 5, 2025, with a file size of 24,667 KB. A second version followed five days later at 24,668 KB. The third and fourth revisions, submitted in April and June of 2026, both measured 36,292 KB [1].