Surrogate modeling for Bayesian optimization beyond a single Gaussian process

A new approach to Bayesian optimization sidesteps the need to manually choose a surrogate model by deploying an ensemble of Gaussian processes that adaptively selects the best fit during the optimization run, according to a paper posted on arXiv [1]. Bayesian optimization is a technique for tuning black-box functions that are costly to evaluate, such as hyperparameter tuning in machine learning, drug discovery, and robotics [2]. The method relies on a surrogate model to sequentially choose query points, balancing exploration of unknown areas with exploitation of promising regions [2]. Traditionally, practitioners must preselect a single Gaussian process model with a specific kernel function, a choice that often requires domain expertise [2]. The new method, detailed in a paper by Qin Lu, replaces that single model with an ensemble of Gaussian processes [1]. The ensemble adaptively selects the surrogate model on-the-fly, creating a Gaussian process mixture posterior that the authors describe as having "enhanced expressiveness" for the target function [2]. Gaussian processes, also known as kriging, are a method of interpolation that provides the best linear unbiased prediction under suitable prior assumptions [3]. The technique originated in geostatistics, where it was used to estimate gold grades from borehole samples, and is now widely applied in computer experiments [3]. To decide where to sample next, the new framework uses Thompson sampling, a strategy that requires no additional design parameters [2]. Scalability is addressed through random feature-based kernel approximation for each Gaussian process model [2]. The paper also provides a convergence analysis based on Bayesian regret for both sequential and parallel settings [2]. The first version of the paper was submitted on 27 May 2022, with a file size of 3,092 KB [1]. A revised version followed on 26 May 2026, weighing in at 17,480 KB [1]. The work was developed within the arXivLabs framework, which supports experimental projects from community collaborators [1].