- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ForecastBench-Sim, a benchmark for evaluating LLM-based forecasting using simulated worlds. The framework uses three simulators, Micropolis, FreeCiv, and Starsim, to generate forecasting tasks covering rare events, continuous outcomes, conditional predictions, and causal interventions. The benchmark estimates outcome probabilities and evaluates model predictions using excess scoring rules. Experiments involving 24 LLMs show that the benchmark scores are generally correlated with ECI. The paper also provides additional analyses of forecasting performance, model rankings, and the effects of different evaluation settings.
Critical strengths
- The paper addresses several limitations of existing real-world forecasting benchmarks, particularly the long waiting time for event resolution and the difficulty of evaluating rare events and counterfactual outcomes.
- The benchmark provides a broader evaluation of forecasting capabilities by incorporates three different simulation environments and evaluates multiple types of forecasting tasks, including rare events, conditional forecasting, and interventional predictions.
- The benchmark evaluated 24 LLMs and provides detailed analyses of model performance, correlations with general model capabilities, prompt sensitivity, and forecasting horizons.
Critical weaknesses
-
ECI primarily measures general model capabilities rather than forecasting ability. In Appendix C.4–C.5, the correlation between the combined ForecastBench-Sim score and the real-world ForecastBench score is only 0.374, with a 95% confidence interval of [-0.216, 0.815]. This raises concerns about how well the proposed benchmark reflects real-world forecasting capabilities.
-
15 cities in Micropolis, 8 games in FreeCiv, and 4 epidemic scenarios in Starsim were used for large number of simulation replays. However, repeated simulations of the same worlds do not necessarily provide independent evaluation settings, it remains unclear whether the reported results can generalize to unseen world configurations or different simulation environments.
-
Appendix E.1 identifies missing artifacts required for exact reproduction, including uncommitted changes to the Micropolis simulation code and missing original model responses or predicted quantiles. Some additional supporting materials are also retained privately. These limitations make it difficult to independently reproduce the complete benchmark and verify all reported results.
Questions for authors
- Despite the large number of simulation replays, the evaluation relies on relatively few underlying worlds. Can the authors explain more about whether the reported results can generalize to unseen world configurations or different simulation environments?
- Could the authors further clarify what evidence supports the claim that ForecastBench-Sim measures forecasting ability rather than general reasoning or simulation-specific knowledge?
- Have the authors considered evaluating the benchmark on additional unseen world configurations to determine whether the reported model rankings remain stable?
- In the Starsim experiments, the models only receive textual world reports, while the ground-truth distributions are generated using the complete simulation states. Could the authors clarify whether this information mismatch introduces unavoidable prediction errors and how it may affect the fairness of the evaluation?
- Will all the missing artifacts identified in Appendix E.1, including the original simulation code changes and model outputs, be made available to ensure full reproducibility?
3 comments
Thanks for the authors' reply. Unfortunately I will maintain my score. The additional correlations support the benchmark’s relevance to forecasting, but it remains unclear whether they reflect forecasting-specific ability beyond general model capability, as both ECI-controlled correlations have confidence intervals that include zero. The split-world analyses support ranking stability within the existing settings but do not establish generalization to unseen configurations. The report-based SIR baseline demonstrates room for improvement, but it does not separate forecasting errors from the unavoidable information gap relative to the full-state oracle. Full reproducibility also remains pending the release of the remaining artifacts.
Re: Capability & full reproducibility
Thank you for the follow-up. We agree that the ECI-controlled correlations leave open how much forecasting-specific ability the benchmark captures, and that the split-world analysis establishes ranking stability only within the settings we evaluated. The Starsim SIR baseline shows that better forecasts were possible from the same report. How much of the gap to the full-state oracle is unavoidable remains open.
Regarding reproducibility: The updated supplement now includes the FreeCiv candidate export and the archived inputs needed to rebuild the paper’s tables and figures. As promised, we will release the original model responses separately before publication and link them from Appendix F.1.
Thank you for your thoughtful review. We are glad you appreciate how ForecastBench-Sim addresses limitations of real-world forecasting benchmarks, the breadth of its forecasting tasks, and the detail of our analysis across models and settings.
Question 1 — “...whether the reported results can generalize to unseen world configurations or different simulation environments?”
We agree that repeated replays do not provide additional independent world configurations. To test whether rankings depend on which worlds we chose, we split the observed worlds into two disjoint halves and fit model scores on each half separately (new Appendix C.6, summarized in Appendix E.1). Model rankings agree highly across these halves (ρ = 0.94 on average), and at ρ = 0.79–0.84 when comparing scores derived from two simulators with scores from the held-out third. These results support ranking stability across subsets of our current worlds and simulators; however, they do not establish generalization to newly generated configurations or additional simulation environments.
Question 2 — “...what evidence supports the claim that ForecastBench-Sim measures forecasting ability rather than general reasoning or simulation-specific knowledge?”
The original ECI analysis showed an association with general capability. We now compare ForecastBench-Sim with forecasting-specific benchmarks (Appendix E.2). Its combined score correlates strongly with FutureEval (ρ = 0.90) and Prophet Arena (ρ = 0.86), while agreement with ForecastBench remains weaker and uncertain (ρ = 0.37). After controlling for ECI, correlations with FutureEval and Prophet Arena remain positive (ρ = 0.59 and 0.67), although both 95% confidence intervals include zero. These results offer suggestive, though not conclusive, evidence of forecasting information beyond that captured by ECI.
In addition, a new main-text section (Section 6) shows that a systematic forecasting error observed in our simulations also appears when models forecast historical measles cases. This demonstrates that the benchmark can diagnose failures relevant to real-world forecasting, although it does not isolate the contribution of simulator-specific knowledge.
Question 3 — “Have the authors considered evaluating the benchmark on additional unseen world configurations...?”
Yes. Over time, adding new world configurations, and new worlds, will demonstrate robustness, and help benchmark maintainers adjust difficulty to avoid saturation. Our Discussion identifies fresh worlds and varied configurations as future work.
Question 4 — “...whether this information mismatch introduces unavoidable prediction errors and how it may affect the fairness of the evaluation?”
We agree that the full-state oracle has an information advantage over models receiving textual reports. We have updated the Discussion to clarify that excess loss can reflect forecasting error, alongside a potentially unavoidable gap due to limited observability. Future work could supply models with full simulator states, potentially allowing them to explore the state with an agentic harness.
In the meantime, we have added a Starsim SIR baseline that uses only information supplied in the report (Section 5; Appendix C.1). It outperforms most evaluated models on unconditional continuous questions, demonstrating that better forecasts were possible with the available information. We will try to find analogous baselines for the other two worlds before publication.
Question 5 — “Will all the missing artifacts...be made available to ensure full reproducibility?”
Yes. The MicropolisCore engine changes are now vendored in the benchmark repository, and Appendix A identifies the commit that reproduces the Micropolis trunk runs and replays. The original model responses, including predicted quantiles, will be packaged and released online before publication and linked from the paper.
We are still assembling additional supporting inputs needed for complete independent reproduction, including the FreeCiv candidate export. Appendix F.1 describes the remaining artifact dependencies. We are working to complete this package before publication (and, where possible, during this discussion window).