ForecastBench-Sim: A Simulated-World Forecasting Benchmark

A team of researchers has released ForecastBench-Sim, a new benchmark that uses simulated game worlds to test how well AI systems make probabilistic forecasts, sidestepping the slow resolution and data scarcity that hamper real-world evaluation [1]. The benchmark, described in a paper posted to arXiv on June 17, 2026, is built on game rollouts from Freeciv, an open-source turn-based strategy game modeled on the Civilization series [1]. Forecasters receive a structured snapshot of the current game state — called a world report — and must answer questions about hidden future states. The simulation then continues and scores the forecasts against what actually occurs [2]. Real-world forecasting benchmarks carry inherent constraints: outcomes can take months or years to resolve, tail events are rare, and counterfactual questions are nearly impossible to score with certainty [3]. ForecastBench-Sim is designed to complement those benchmarks by providing controlled, immediately resolvable tasks [1]. Because the world is simulated, researchers can generate binary or continuous forecasting questions at arbitrary time horizons and even create paired intervention worlds by mutating the savegame state before rollout, enabling conditional or causal questions [5]. The benchmark supports two question formats. Binary questions ask whether an event or relation will hold at a later turn — for example, whether a civilization’s treasury will exceed a threshold. Continuous questions ask for a future value of a world variable, such as the treasury value itself, and elicit five quantiles: p10, p25, p50, p75, and p90 [3]. Question families cover city counts, technologies, treasury values, and a broader set of comparison and threshold events [5]. Forward-looking tasks use horizons labeled H1 through H7, corresponding to 30-turn increments past a turn-60 snapshot, spanning turns 90 through 270 — roughly the second half of a typical game. H0 questions ask about the snapshot state itself and serve as comprehension checks rather than forecasting tasks [3]. The paper, authored by Jaeho Lee, Nick Merrill, and Ezra Karger, was presented as a Spotlight at the Forecast@ICML26 workshop [4]. It reports validation slices from model evaluations and an anonymized human pilot, though full comparative results remain compact in the initial release [2]. The authors frame the work as a tool for studying probabilistic reasoning, calibration, and causal updating under dynamic world states — capabilities that are difficult to isolate in real-world settings where ground truth arrives slowly, if at all [4].