Accelerating Reinforcement Learning Training Using Simulation Surrogate Models

A new study proposes using simulation surrogate models to speed up reinforcement learning training, addressing the computational burden of high-fidelity simulations used to model complex stochastic systems [1][2]. High-fidelity simulation models are standard tools for analyzing complex stochastic systems, but their computational cost can be prohibitive [2]. Researchers are now exploring cheaper surrogate models that approximate the input-output relationship of these simulations [2]. In a paper submitted on 26 May 2026, Mohammadmahdi Ghasemloo and colleagues investigate a class of these surrogates designed to accelerate reinforcement learning (RL) training [1][2]. RL has become a prominent framework for making online decisions in stochastic environments, and simulations are increasingly used as training grounds for RL models [2]. The work focuses on settings where the reward structure, model parameters, or system dynamics change over time, requiring models to be retrained [2]. The researchers examined the interactions between simulation models, surrogate models, and RL models [2]. Through numerical experiments on a stochastic service system modeled via discrete-event simulation, they demonstrated that leveraging surrogate models can substantially accelerate both initial RL training and subsequent re-training [2]. Reinforcement learning is a branch of machine learning, a field concerned with developing statistical algorithms that learn from data and generalize to unseen tasks [3]. Modern RL often employs deep learning, which utilizes multilayered neural networks to process data [5]. These networks consist of connected units called artificial neurons, loosely inspired by biological neural networks, that transmit signals and adjust connection weights during training [4]. Training such networks is a compute-intensive process, often accelerated by graphics processing units (GPUs) and large datasets [4]. Architectural innovations like transformers, which use attention mechanisms to model long-range dependencies in data, have become the basis for large language models and other advanced systems [4]. The study's approach aims to reduce the computational load by substituting expensive simulation calls with faster surrogate approximations during the RL training loop [2]. The paper was submitted to arXiv on 26 May 2026 at 18:23:42 UTC and is available as a 369 KB PDF [1].