acceptodds
Under review as a conference paper at ICLR 2027

ForecastBench-Sim: Forecasting Simulated Worlds

Abstract

Predicting how a complex system will evolve is among the most general tests of reasoning under uncertainty, and among the hardest to benchmark. Real-world forecasting benchmarks often resolve slowly and provide sparse evidence of accuracy in forecasting rare or counterfactual events. We introduce ForecastBench-Sim, a forecasting benchmark that asks models to forecast the outcomes of simulated worlds. We generate questions about rare events, conditional outcomes, and responses to causal interventions. Across 24 models, three simulations, and nine distinct tasks, model scores closely track general measures of model capabilities ( []). ForecastBench-Sim provides tasks whose question sets can be refreshed, varied in difficulty, and immediately scored. We discuss potential mechanisms to prevent saturation and avoid benchmark gaming, focusing on the automatable extensibility of our framework to new unseen questions.

Primary area:
datasets and benchmarks
Keywords:
forecasting, benchmarks, evaluations, simulations, calibration, counterfactual reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_AKgoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ForecastBench-Sim, a benchmark for evaluating LLM-based forecasting using simulated worlds. The framework uses three simulators, Micropolis, FreeCiv, and Starsim, to generate forecasting tasks covering rare events, continuous outcomes, conditional predictions, and causal interventions. The benchmark estimates outcome probabilities and evaluates model predictions using excess scoring rules. Experiments involving 24 LLMs show that the benchmark scores are generally correlated with ECI. The paper also provides additional analyses of forecasting performance, model rankings, and the effects of different evaluation settings.

Critical strengths
  1. The paper addresses several limitations of existing real-world forecasting benchmarks, particularly the long waiting time for event resolution and the difficulty of evaluating rare events and counterfactual outcomes.
  2. The benchmark provides a broader evaluation of forecasting capabilities by incorporates three different simulation environments and evaluates multiple types of forecasting tasks, including rare events, conditional forecasting, and interventional predictions.
  3. The benchmark evaluated 24 LLMs and provides detailed analyses of model performance, correlations with general model capabilities, prompt sensitivity, and forecasting horizons.
Critical weaknesses
  1. ECI primarily measures general model capabilities rather than forecasting ability. In Appendix C.4–C.5, the correlation between the combined ForecastBench-Sim score and the real-world ForecastBench score is only 0.374, with a 95% confidence interval of [-0.216, 0.815]. This raises concerns about how well the proposed benchmark reflects real-world forecasting capabilities.

  2. 15 cities in Micropolis, 8 games in FreeCiv, and 4 epidemic scenarios in Starsim were used for large number of simulation replays. However, repeated simulations of the same worlds do not necessarily provide independent evaluation settings, it remains unclear whether the reported results can generalize to unseen world configurations or different simulation environments.

  3. Appendix E.1 identifies missing artifacts required for exact reproduction, including uncommitted changes to the Micropolis simulation code and missing original model responses or predicted quantiles. Some additional supporting materials are also retained privately. These limitations make it difficult to independently reproduce the complete benchmark and verify all reported results.

Questions for authors
  1. Despite the large number of simulation replays, the evaluation relies on relatively few underlying worlds. Can the authors explain more about whether the reported results can generalize to unseen world configurations or different simulation environments?
  2. Could the authors further clarify what evidence supports the claim that ForecastBench-Sim measures forecasting ability rather than general reasoning or simulation-specific knowledge?
  3. Have the authors considered evaluating the benchmark on additional unseen world configurations to determine whether the reported model rankings remain stable?
  4. In the Starsim experiments, the models only receive textual world reports, while the ground-truth distributions are generated using the complete simulation states. Could the authors clarify whether this information mismatch introduces unavoidable prediction errors and how it may affect the fairness of the evaluation?
  5. Will all the missing artifacts identified in Appendix E.1, including the original simulation code changes and model outputs, be made available to ensure full reproducibility?
3 comments

Thank you for your thoughtful review. We are glad you appreciate how ForecastBench-Sim addresses limitations of real-world forecasting benchmarks, the breadth of its forecasting tasks, and the detail of our analysis across models and settings.

Question 1 — “...whether the reported results can generalize to unseen world configurations or different simulation environments?”

We agree that repeated replays do not provide additional independent world configurations. To test whether rankings depend on which worlds we chose, we split the observed worlds into two disjoint halves and fit model scores on each half separately (new Appendix C.6, summarized in Appendix E.1). Model rankings agree highly across these halves (ρ = 0.94 on average), and at ρ = 0.79–0.84 when comparing scores derived from two simulators with scores from the held-out third. These results support ranking stability across subsets of our current worlds and simulators; however, they do not establish generalization to newly generated configurations or additional simulation environments.

Question 2 — “...what evidence supports the claim that ForecastBench-Sim measures forecasting ability rather than general reasoning or simulation-specific knowledge?”

The original ECI analysis showed an association with general capability. We now compare ForecastBench-Sim with forecasting-specific benchmarks (Appendix E.2). Its combined score correlates strongly with FutureEval (ρ = 0.90) and Prophet Arena (ρ = 0.86), while agreement with ForecastBench remains weaker and uncertain (ρ = 0.37). After controlling for ECI, correlations with FutureEval and Prophet Arena remain positive (ρ = 0.59 and 0.67), although both 95% confidence intervals include zero. These results offer suggestive, though not conclusive, evidence of forecasting information beyond that captured by ECI.

In addition, a new main-text section (Section 6) shows that a systematic forecasting error observed in our simulations also appears when models forecast historical measles cases. This demonstrates that the benchmark can diagnose failures relevant to real-world forecasting, although it does not isolate the contribution of simulator-specific knowledge.

Question 3 — “Have the authors considered evaluating the benchmark on additional unseen world configurations...?”

Yes. Over time, adding new world configurations, and new worlds, will demonstrate robustness, and help benchmark maintainers adjust difficulty to avoid saturation. Our Discussion identifies fresh worlds and varied configurations as future work.

Question 4 — “...whether this information mismatch introduces unavoidable prediction errors and how it may affect the fairness of the evaluation?”

We agree that the full-state oracle has an information advantage over models receiving textual reports. We have updated the Discussion to clarify that excess loss can reflect forecasting error, alongside a potentially unavoidable gap due to limited observability. Future work could supply models with full simulator states, potentially allowing them to explore the state with an agentic harness.

In the meantime, we have added a Starsim SIR baseline that uses only information supplied in the report (Section 5; Appendix C.1). It outperforms most evaluated models on unconditional continuous questions, demonstrating that better forecasts were possible with the available information. We will try to find analogous baselines for the other two worlds before publication.

Question 5 — “Will all the missing artifacts...be made available to ensure full reproducibility?”

Yes. The MicropolisCore engine changes are now vendored in the benchmark repository, and Appendix A identifies the commit that reproduces the Micropolis trunk runs and replays. The original model responses, including predicted quantiles, will be packaged and released online before publication and linked from the paper.

We are still assembling additional supporting inputs needed for complete independent reproduction, including the FreeCiv candidate export. Appendix F.1 describes the remaining artifact dependencies. We are working to complete this package before publication (and, where possible, during this discussion window).

Thanks for the authors' reply. Unfortunately I will maintain my score. The additional correlations support the benchmark’s relevance to forecasting, but it remains unclear whether they reflect forecasting-specific ability beyond general model capability, as both ECI-controlled correlations have confidence intervals that include zero. The split-world analyses support ranking stability within the existing settings but do not establish generalization to unseen configurations. The report-based SIR baseline demonstrates room for improvement, but it does not separate forecasting errors from the unavoidable information gap relative to the full-state oracle. Full reproducibility also remains pending the release of the remaining artifacts.

AuthorsOpenReview ↗

Re: Capability & full reproducibility

Thank you for the follow-up. We agree that the ECI-controlled correlations leave open how much forecasting-specific ability the benchmark captures, and that the split-world analysis establishes ranking stability only within the settings we evaluated. The Starsim SIR baseline shows that better forecasts were possible from the same report. How much of the gap to the full-state oracle is unavoidable remains open.

Regarding reproducibility: The updated supplement now includes the FreeCiv candidate export and the archived inputs needed to rebuild the paper’s tables and figures. As promised, we will release the original model responses separately before publication and link them from Appendix F.1.

Reviewer_ntpPOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a benchmark to evaluate forecasting based on three game simulators: Micropolis, FreeCiv and Starsim. A range of open weights and closed models are evaluated on the proposed benchmark.

Critical strengths

I find the idea of using game simulators for this research appealing, as one can obtain genuine uncertainty estimates, answer questions regarding interventions, etc. I believe this would be enabling for other researchers to build on. The benchmark construction itself is sound.

Critical weaknesses

The main weakness is acknowledged by the authors in the discussion section: it is unclear how the findings from evaluations using this benchmark relate to model performance in the real world. I appreciate the correlation analysis against ECI, but not sure how interpret it. It would be more interesting to look at how findings regarding different models might diverge when using different models.

Questions for authors

None

3 comments

Thank you for your thoughtful review. We are glad you find the use of simulators appealing, and very heartened to hear that you think our work would be enabling for other researchers to build on.

We agree that our original submission was unclear about how findings from evaluations relate to model performance in the real world, and that correlation with ECI alone does not establish relevance to real-world forecasting. We have added two analyses to address these concerns.

First, on model performance in the real-world, a new main-text section (Section 6) shows that a systematic forecasting error we noticed during analysis also appears when models forecast real-world data (historical measles cases). This analysis provides a concrete example of the benchmark diagnosing a failure relevant to real-world forecasting. The fact that a shared failure mode exists makes ForecastBench-Sim a promising setting for reinforcement learning with verifiable rewards (RLVR), a point we now discuss in the Discussion. (Whether improvements learned in simulation transfer to real-world forecasting remains to be tested.)

Second, we now show correlations with forecasting-specific benchmarks (Appendix E.2). ForecastBench-Sim’s combined score correlates strongly with FutureEval (ρ = 0.90) and Prophet Arena (ρ = 0.86), while agreement with ForecastBench remains weaker and uncertain (ρ = 0.37). After controlling for ECI, correlations with FutureEval and Prophet Arena remain positive (ρ = 0.59 and 0.67), although both 95% confidence intervals include zero. These results offer suggestive, though not conclusive, evidence that we measure forecasting ability beyond general model capability captured by ECI.

Finally, we would appreciate clarification about your final suggestion (“It would be more interesting to look at how findings regarding different models might diverge when using different models.”). If your suggestion concerns differences in model rankings across worlds or simulators, we now examine this in Appendix C.6. Rankings correlate at ρ = 0.94 on average across disjoint halves of the observed worlds, and at ρ = 0.79–0.84 when comparing scores from two simulators with the held-out third. We would welcome clarification if you intended a different comparison.

Reviewer_ntpPOpenReview ↗

Appreciate the correction

I appreciate the additional points from the authors. The scale of acceptance/rejection on the review form doesn't allow for a finer judgement, but I think the new correlation analysis strengthen the paper. And apologies for the lack of clarity in my comment, I rephrase here:

"It would be more interesting to look at how findings regarding different models might diverge when using different benchmarks.”

You kind of address this, but it would be better for elaborate on what are the differences in model rankings between say ForecastBench-Sim and ForecastBench.

AuthorsOpenReview ↗

Re: Differences in model rankings

Thank you for clarifying. Among the 17 models covered by both benchmarks, Gemini 3.1 Flash Lite and GPT-5.4 Nano rank 4th and 6th on ForecastBench-Sim but 17th and 16th on ForecastBench. Two models move the other way: o4-mini ranks 10th on ForecastBench-Sim and 2nd on ForecastBench, while Haiku 4.5 ranks 12th on ForecastBench-Sim and is tied for 6th on ForecastBench. Together, these four models account for 71% of the sum of squared rank differences between the two benchmarks.

As we now report in Appendix E.2, differences in model coverage alone do not explain the weaker agreement with ForecastBench. Across the same 14 models, ForecastBench-Sim’s ranking correlates more strongly with FutureEval (ρ = 0.85) than with ForecastBench (ρ = 0.46). We do not yet know why these particular models change rank.

Reviewer_oFHLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ForecastBench-Sim, a benchmark that constructs forecasting questions from three simulated worlds. Repeated simulation rollouts estimate event probabilities and outcome distributions, enabling immediate scoring and evaluation of rare events, conditional outcomes, and responses to causal interventions. Experiments across 24 models examine forecasting performance and its relationship to general model capabilities.

Critical strengths
  1. Simulated worlds address several inherent limitations of live forecasting benchmarks: they allow outcomes to be resolved quickly and enable controlled, repeatable evaluation of rare events, conditional outcomes, and causal interventions that are difficult to study in the real world live forecasting benchmark.

  2. Repeated rollouts with different random seeds provide estimates of event probabilities, offering a more informative reference for evaluating probabilistic forecasts than a single binary outcome.

Critical weaknesses
  1. The practical value of the benchmark is not sufficiently demonstrated. The experiments mainly show that stronger models perform better on simulated forecasting tasks. However, correlation with general capability does not establish forecasting-specific value. It remains unclear how the benchmark helps diagnose forecasting failures, improve forecasting systems, or predict real-world forecasting performance.

  2. Replay probabilities are conditioned on the full simulator state, while models receive only a partial textual report. Consequently, some excess loss may reflect missing information rather than forecasting errors. This does not invalidate comparisons between models, but makes absolute scores harder to interpret. A stronger analysis of information sufficiency would help.

  3. I agree that live forecasting benchmarks involve long waits and many uncontrolled factors. However, resolved real-world events paired with carefully filtered pre-resolution evidence also allow immediate evaluation while preserving real-world context. The paper should clarify what simulated tasks add beyond such evaluations and whether simulator performance tells us anything useful about real-world forecasting ability.

Questions for authors

N/A

2 comments

Thank you for your thoughtful review. We’re glad you appreciate how our approach enables fast, controlled evaluation of otherwise difficult-to-study forecasting tasks, and how repeated rollouts provide a more informative reference for evaluating probabilistic forecasts.

Practical and forecasting-specific value. We agree that correlation with general capability alone does not establish forecasting-specific value. You asked (1) about how the benchmark helps diagnose forecasting failures, and (2) how well it predicts real-world forecasting performance.

On (1), whether our benchmark can help to diagnose forecasting failures, a new main-text section (Section 6) shows that a systematic forecasting error we noticed during analysis also appears when models forecast real-world data (historical measles cases). This analysis provides a concrete example of the benchmark diagnosing a failure relevant to real-world forecasting. The fact that a shared failure mode exists makes ForecastBench-Sim a promising setting for reinforcement learning with verifiable rewards (RLVR), a point we now discuss in the Discussion.

On (2), predicting real-world forecasting performance, we now compare ForecastBench-Sim with three real-world forecasting benchmarks (Appendix E.2). We find it correlates strongly with FutureEval (ρ = 0.90) and Prophet Arena (ρ = 0.86), while agreement with ForecastBench remains weaker (ρ = 0.37). After controlling for ECI, correlations with FutureEval and Prophet Arena remain positive (ρ = 0.59 and 0.67), although both 95% confidence intervals include zero. These correlations offer suggestive, though not conclusive, evidence that we measure forecasting ability beyond general model capability captured by ECI.

Information sufficiency and score interpretation. We agree that the full-state oracle has an information advantage over the evaluated models. A future version of this benchmark might allow the models to inspect the world state directly (in practice, this would likely require agents to run in an agentic harness). We might vary available information to study how observability affects forecasting performance. We have updated our Discussion to note this limitation and future work with these extensions.

In the meantime, to help us interpret excess loss due to missing information, we have added a Starsim SIR baseline that uses only information supplied in the report (Section 5; Appendix C.1). It outperforms most evaluated models on Starsim’s unconditional continuous questions, demonstrating that better forecasts were possible using the information available in the reports.

What simulation adds beyond historical evaluation. While “resolved real-world event” allow immediate resolution, we cannot always verify whether a real-world event is encoded in a models weights, even if we filter pre-resolution evidence. In a competitive benchmark, reported knowledge cutoffs may be incomplete, inaccurate, or impossible to independently verify.

ForecastBench-Sim enables immediate evaluation without relying on these claims: each benchmark step could rely on fresh, privately generated simulations, which cannot be leaked in any training data (by construction). (Simulation also permits replay-based estimates of outcome distributions, counterfactual forecasting questions, and targeted evaluation of rare events, as you note in your review). We now better clarify this motivation in our introduction.

I appreciate the authors’ responses and additional experiments. The shared forecasting failure observed in simulated and real-world data addresses my main concerns about the benchmark’s practical value. I will raise my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.