Finding the Time to Think: Learning Planning Budgets in Real-Time RL

A new machine-learning framework forces artificial agents to decide not just what to do, but how long to think about it while the clock ticks. The work, posted to arXiv on 24 Jun 2026, formalizes variable-delay real-time reinforcement learning, where an environment advances even as an agent deliberates [1]. Standard reinforcement learning assumes the environment pauses indefinitely for an agent’s next move. The authors discard that assumption, studying settings where the world progresses during deliberation [1]. They call the formulation variable-delay real-time RL, building on earlier real-time formalizations. Under this model, an agent must choose a planning duration at each decision point, knowing that waiting too long can be as damaging as acting rashly [1]. The researchers found that naively planning how long to plan can paralyze an agent. Instead, they trained a lightweight gating policy atop a planner to select state-dependent planning budgets [1]. The gating policy learns when a situation demands deeper search and when a fast, shallow response suffices. This approach echoes the broader trend of reasoning models that allocate extra computation at inference time to improve performance on multi-step logical tasks [2]. The team tested the method across five real-time game environments: Pac-Man, Tetris, Snake, Speed Hex, and Speed Go [1]. In each domain, the gating policy outperformed fixed-budget and heuristic baselines. The policy also transferred to a hardware-separated setup where the environment and the agent ran on two different GPUs, demonstrating that the learned timing behavior survives a shift in physical compute conditions [1]. The paper appears on arXiv, the open-access repository that hosts preprints across physics, computer science, and related fields and now receives roughly 24,000 submissions per month [10]. The work was posted under the Machine Learning category and is accessible through the site’s abstract page, which also surfaces community-built tools such as the Bibliographic Explorer and CORE Recommender through the arXivLabs framework [1][8][9]. By making deliberation time an explicit choice rather than a hidden cost, the variable-delay formulation adds a dimension to agent design that mirrors real-world constraints, where decisions must balance speed against accuracy. The authors frame the gating policy as a practical mechanism for navigating that trade-off without hand-tuned heuristics [1].