Pre-Training for Simulation-Based Science: A Study on Jet Foundation Model Training Objectives

A new study systematically compares pre-training strategies for foundation models in simulation-based science, revealing that combining supervised classification with self-supervised masked particle modeling is particularly effective when fine-tuning labels are scarce [1][2]. The research, submitted in 2026, uses the OmniLearned High Energy Physics FM framework to evaluate three pre-training methods: supervised classification, flow-matching generation, and self-supervised masked particle modeling (MPM) [1][2]. All models were pre-trained on the JetClass dataset and then fine-tuned on two downstream tasks—top jet classification and JetNet conditional generation [1][2]. The work addresses a shift in scientific machine learning, where accurate simulations can produce large, labeled datasets, unlike industrial settings that typically rely on self-supervision with masking due to a lack of labels [1][2]. For classification tasks, the study finds that pure classifier pre-training performs best when downstream labels and model capacity are abundant [1][2]. However, in the low-finetuning label regime, combining classifier pre-training with MPM proves uniquely powerful [1][2]. Flow matching-based generative pre-training, by contrast, provides little benefit for downstream classification [1][2]. The researchers also observed a notable separation between task types: for downstream generation, flow matching must be included in the pre-training objective to achieve a significant fine-tuning advantage, suggesting that classification and generation tasks are orthogonal [1][2]. A model must be pre-trained on both objectives to transfer effectively to both kinds of downstream work [1][2]. The study contributes a template for controlled scaling analysis of pre-training objectives, a framework that could extend to other simulation-heavy scientific domains [1][2]. The broader context of transfer learning in the sciences has seen related efforts. For example, the Open Catalyst 2020 (OC20) dataset enabled the catalysis community to use transfer learning to improve model performance on smaller datasets, and researchers have explored whether OC20 can aid the newer OC22 dataset through joint training or transfer learning [7]. Such approaches mirror the small-molecule and drug-discovery fields, where transfer learning has been used to bridge varying levels of electronic structure calculations and related tasks [7]. The current study’s systematic comparison of pre-training objectives adds a structured methodology to this growing area of investigation [1][2].