An Open-Source Training Dataset for Foundation Models for Black-box Optimization

Researchers have released BBO-Pile, an open-source dataset containing over 500,000 optimization trajectories across 3,095 distinct black-box problems, aiming to accelerate the development of foundation models for black-box optimization [1][2]. The dataset, described as the largest publicly available resource for this task, was introduced in a paper submitted to arXiv on 22 May 2026 [1][2]. Prior efforts to build foundation models for black-box optimization were constrained by reliance on non-public or purely synthetic data, which limited reproducibility and the ability to generalize to real-world problems [2]. Foundation models are machine learning models trained on vast datasets so they can be applied across a wide range of use cases, with large language models being a common example [3]. BBO-Pile is designed to address that gap. The authors used the dataset to train a family of foundation models at multiple scales, with parameter counts ranging from 2 million to 80 million and training token counts from 200 million to 2 billion [1][2]. The paper studies how these models' performance scales with compute resources [2]. The release follows a broader trend in machine learning where the availability of high-quality training datasets has been a key driver of progress, alongside advances in algorithms and hardware [5]. High-quality labeled datasets for supervised learning are often difficult and expensive to produce because of the time required to label the data [5]. By making BBO-Pile open source, the researchers provide a resource that can be freely used, modified, and shared under open-source licensing principles [4]. The results indicate that large-scale pre-training is a viable method for imitating black-box optimization methods [1][2]. The work lays a foundation for future research into foundation models that can learn optimization principles from diverse problem classes without the extensive manual hyperparameter tuning typically required by conventional methods [2].