MegaFold: Efficient Training of Next-Generation 3D Attention Protein Models on Cross-Platform GPUs

A new training system called MegaFold sharply reduces the computational cost of next-generation protein-structure models that use 3D attention, according to a preprint posted to the arXiv repository [1][2]. The work targets a bottleneck that makes AlphaFold3-style architectures far more expensive to train than comparably sized transformers [1][2]. The preprint, authored by Ahan Gupta and colleagues, was submitted on 24 June 2025 and revised on 13 June 2026 [1]. It describes how models such as AlphaFold3 compute 3D attention over 2D pairwise representations, producing tensors whose memory and computation costs scale cubically with sequence length [1][2]. The authors write that this workload creates massive 3D attention maps, complex inter-operator dependencies, kernel fragmentation, and heavy host-side data pipelines that differ substantially from large-language-model training and lead to poor GPU utilization [1][2]. MegaFold addresses these problems with four components: a memory-efficient 3D-attention kernel, a communication-efficient sharding strategy for quadratic representations, fused operator implementations for critical execution paths, and a determinism-aware host-device pipeline that eliminates preprocessing stalls [1][2]. The system was evaluated on both NVIDIA H200 and AMD MI250 GPUs [1][2]. On 32 GPUs, MegaFold enabled training with up to 3.36× longer sequence lengths while cutting end-to-end execution time by up to 1.73× on NVIDIA hardware and 1.62× on AMD hardware [1][2]. The paper appears on arXiv, an open-access repository that hosts electronic preprints across physics, mathematics, computer science, quantitative biology, and other fields [6]. Founded in 1991, arXiv passed the two-million-article milestone by the end of 2021 and now receives roughly 24,000 submissions per month [6]. Preprints on the platform are moderated but not peer-reviewed [6]. The MegaFold manuscript is listed under the Quantitative Biology > Biomolecules category and is available in PDF and experimental HTML formats [1]. arXiv also supports community-built tools through its arXivLabs framework, which allows third-party developers to create features that appear on article abstract pages [4][5]. These tools include citation explorers, code-and-data linkers, and recommender systems, all operating under guidelines that require adherence to arXiv’s values of openness, community, excellence, and user-data privacy [4][5]. The MegaFold abstract page surfaces several of these integrations, including Bibliographic Explorer, Connected Papers, and the CORE Recommender [1][5].