- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the evaluation of Large Language Models (LLMs) in the context of event forecasting—specifically, whether metrics used in existing benchmarks accurately reflect an LLM's performance within an actual event forecasting system. The authors survey key tasks, methodologies, and benchmarks in the field and propose an "Evaluation Contract" framework, which formalizes parameters such as forecasting objectives, time horizons, available information, event determination criteria, and evaluation metrics. Furthermore, the authors construct an "Evidence Ledger" containing 53 entries to verify and compare experimental results from existing studies. They also analyze how model rankings on the ForecastBench leaderboard vary based on the weighting schemes used to aggregate different data sources.
Critical strengths
The problem addressed in this paper is of significant practical importance. As LLMs are increasingly applied to event forecasting, it is crucial to accurately measure a model's intrinsic predictive capability—while disentangling the effects of model reasoning, access to information, and evaluation biases.
Additionally, the authors conduct a meticulous verification of results and provide some useful analyses. For instance, the sensitivity analysis regarding weighting schemes on ForecastBench demonstrates that different methods of aggregating metrics can alter model rankings. These observations offer valuable insights for the design of future benchmarks.
Critical weaknesses
1. The paper’s core contributions overlap substantially with existing work, resulting in a lack of novelty.
Many of the key issues addressed—such as temporal information leakage, the reliability of historical backtesting, model dependence on external information, and the extent to which benchmark scores reflect genuine predictive ability—have already been systematically discussed in existing research. For instance, Paleka et al. [1] not only explicitly raised these evaluation issues but also analyzed—through specific case studies—problems such as temporal leakage, post-hoc test construction, and the limitations of extrapolating from historical evaluations to real-world predictive performance.
Although the authors acknowledge the overlap with existing work in Table 1 and frame their contribution as "result-level evidence auditing," I find the incremental contribution to be limited.
While the paper does compile, verify, and analyze experimental results from prior research—and uses local sensitivity analysis to uncover certain evaluation limitations in existing benchmarks—these analyses remain somewhat fragmented. The paper fails to offer sufficiently impactful empirical findings or a systematic quantification of how prevalent these evaluation issues are. Consequently, I believe the new contributions presented here are insufficient to substantiate the level of innovation required for an independent research paper at ICLR.
2. The paper’s primary quantitative analysis holds some value, but its scientific significance and generalizability remain limited.
In Section 6.3 and Appendix D, the authors conduct a weight-sensitivity analysis of "ForecastBench." By recalculating rankings using existing scores from 130 publicly available model configurations, they demonstrate that altering the aggregation weights assigned to "Dataset" versus "Market" data can lead to reversals in the relative rankings of certain configurations.
While this result illustrates that leaderboards are sensitive to aggregation rules—and the authors further examine the impact of rounding errors in the original scores—the experiment is essentially a deterministic re-weighting of previously published aggregate scores. It does not assess statistical uncertainty based on raw, event-level predictions, nor does it adequately address whether these rank reversals carry any practical significance.
More importantly, when different models excel on different subsets of the data, rank reversals resulting from changes in aggregation weights are to be expected. For example, as shown in Appendix D, Claude 3.5 Sonnet (in a "scratchpad" configuration) performs better on the "Dataset" portion, whereas GPT-4 Turbo (in a "zero-shot" configuration) performs better on the "Market" portion; thus, it is natural for their relative rankings to shift as the aggregation weights are adjusted. While the authors have further quantified the scale of this phenomenon, they have not sufficiently explained why the observed rate of reversals constitutes a significant new finding, nor have they clarified the implications of these shifts for the practical design of benchmarks.
Furthermore, the paper’s sensitivity analysis regarding weighting schemes is conducted exclusively on ForecastBench; it remains unclear whether a similar phenomenon would be equally pronounced in other benchmarks that aggregate data from multiple sources. Additionally, while the paper demonstrates that model rankings depend on evaluation weights, it does not establish that this dependency is inherently a flaw in a benchmark, nor does it discuss how specific aggregation weights should be aligned with actual forecasting capabilities or application-specific objectives.
3. The readability of the paper could be improved; the core findings lack clear organization and presentation.
Some parts of the paper are phrased with excessive caution, frequently emphasizing what stronger conclusions cannot be drawn from the existing experimental results. While such qualifiers help ensure the rigor of the arguments, their high density disrupts the flow of the text and makes it difficult for readers to quickly grasp the key findings of each section.
For instance, in Sections 4.1 and 6.3—where the limitations of various benchmarks are discussed—there is a lack of a concise, centralized summary of the core findings.
Moreover, although the paper focuses on evaluation discrepancies across different methods and benchmarks, a significant portion of these cross-study comparisons is presented solely through lengthy descriptive text. While the authors have included some figures, I believe that incorporating more systematic comparison tables, summaries categorized by the quality of evidence, and visualizations of key results would greatly enhance the paper’s readability and help readers more intuitively grasp its primary contributions.
References
[1] Paleka et al. Pitfalls in Evaluating Language Model Forecasters. ICLR 2026.
Questions for authors
- Does the analysis of the "Evidence Ledger" reveal any significant conclusions that have not been identified in previous research? What is the incremental value of these findings relative to the existing literature?
- Is the sensitivity to weighting schemes observed in ForecastBench also present in other benchmarks? Could the authors conduct extended experiments to verify the generality of this phenomenon?