acceptodds
Withdrawn from ICLR 2027

Predict the Future: A Review of Language Model Event Forecasting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

What does it mean for a language model to predict the future? Reported forecasting performance can reflect different sources: historical outcome recall, timely evidence retrieval, reasoning about unresolved events, or access to human and market estimates. This review examines real-world event forecasting through the relationship between prediction targets, forecasting mechanisms, and the evidence used to evaluate them. We distinguish probabilistic forecasts, structured event predictions, and open-ended future generation, clarifying their temporal and resolution requirements. We synthesize retrieval, reasoning, aggregation, calibration, training, and agentic updating as interventions in the forecasting process, and examine the comparisons needed to attribute their benefits. We map existing benchmarks and selected performance results to explicit evaluation conditions, including information availability, forecast horizon, outcome resolution, scoring rules, and aggregation populations. A traceable result-level evidence map preserves the experimental context needed to interpret reported progress. This perspective separates access to predictive information from the ability to use it, and distinguishes probability quality from scenario coverage and decision utility. The review develops a research agenda for temporally valid evaluation, calibrated uncertainty, and reliable forecast updating, connecting methodological advances to the capabilities they can substantiate.

Primary area:
foundation or frontier models, including LLMs
Keywords:
event forecasting, large language models, probabilistic forecasting, benchmarks, evaluation, survey

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_BSiUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the evaluation of Large Language Models (LLMs) in the context of event forecasting—specifically, whether metrics used in existing benchmarks accurately reflect an LLM's performance within an actual event forecasting system. The authors survey key tasks, methodologies, and benchmarks in the field and propose an "Evaluation Contract" framework, which formalizes parameters such as forecasting objectives, time horizons, available information, event determination criteria, and evaluation metrics. Furthermore, the authors construct an "Evidence Ledger" containing 53 entries to verify and compare experimental results from existing studies. They also analyze how model rankings on the ForecastBench leaderboard vary based on the weighting schemes used to aggregate different data sources.

Critical strengths

The problem addressed in this paper is of significant practical importance. As LLMs are increasingly applied to event forecasting, it is crucial to accurately measure a model's intrinsic predictive capability—while disentangling the effects of model reasoning, access to information, and evaluation biases.

Additionally, the authors conduct a meticulous verification of results and provide some useful analyses. For instance, the sensitivity analysis regarding weighting schemes on ForecastBench demonstrates that different methods of aggregating metrics can alter model rankings. These observations offer valuable insights for the design of future benchmarks.

Critical weaknesses

1. The paper’s core contributions overlap substantially with existing work, resulting in a lack of novelty.

Many of the key issues addressed—such as temporal information leakage, the reliability of historical backtesting, model dependence on external information, and the extent to which benchmark scores reflect genuine predictive ability—have already been systematically discussed in existing research. For instance, Paleka et al. [1] not only explicitly raised these evaluation issues but also analyzed—through specific case studies—problems such as temporal leakage, post-hoc test construction, and the limitations of extrapolating from historical evaluations to real-world predictive performance.

Although the authors acknowledge the overlap with existing work in Table 1 and frame their contribution as "result-level evidence auditing," I find the incremental contribution to be limited.

While the paper does compile, verify, and analyze experimental results from prior research—and uses local sensitivity analysis to uncover certain evaluation limitations in existing benchmarks—these analyses remain somewhat fragmented. The paper fails to offer sufficiently impactful empirical findings or a systematic quantification of how prevalent these evaluation issues are. Consequently, I believe the new contributions presented here are insufficient to substantiate the level of innovation required for an independent research paper at ICLR.

2. The paper’s primary quantitative analysis holds some value, but its scientific significance and generalizability remain limited.

In Section 6.3 and Appendix D, the authors conduct a weight-sensitivity analysis of "ForecastBench." By recalculating rankings using existing scores from 130 publicly available model configurations, they demonstrate that altering the aggregation weights assigned to "Dataset" versus "Market" data can lead to reversals in the relative rankings of certain configurations.

While this result illustrates that leaderboards are sensitive to aggregation rules—and the authors further examine the impact of rounding errors in the original scores—the experiment is essentially a deterministic re-weighting of previously published aggregate scores. It does not assess statistical uncertainty based on raw, event-level predictions, nor does it adequately address whether these rank reversals carry any practical significance.

More importantly, when different models excel on different subsets of the data, rank reversals resulting from changes in aggregation weights are to be expected. For example, as shown in Appendix D, Claude 3.5 Sonnet (in a "scratchpad" configuration) performs better on the "Dataset" portion, whereas GPT-4 Turbo (in a "zero-shot" configuration) performs better on the "Market" portion; thus, it is natural for their relative rankings to shift as the aggregation weights are adjusted. While the authors have further quantified the scale of this phenomenon, they have not sufficiently explained why the observed rate of reversals constitutes a significant new finding, nor have they clarified the implications of these shifts for the practical design of benchmarks.

Furthermore, the paper’s sensitivity analysis regarding weighting schemes is conducted exclusively on ForecastBench; it remains unclear whether a similar phenomenon would be equally pronounced in other benchmarks that aggregate data from multiple sources. Additionally, while the paper demonstrates that model rankings depend on evaluation weights, it does not establish that this dependency is inherently a flaw in a benchmark, nor does it discuss how specific aggregation weights should be aligned with actual forecasting capabilities or application-specific objectives.

3. The readability of the paper could be improved; the core findings lack clear organization and presentation.

Some parts of the paper are phrased with excessive caution, frequently emphasizing what stronger conclusions cannot be drawn from the existing experimental results. While such qualifiers help ensure the rigor of the arguments, their high density disrupts the flow of the text and makes it difficult for readers to quickly grasp the key findings of each section.

For instance, in Sections 4.1 and 6.3—where the limitations of various benchmarks are discussed—there is a lack of a concise, centralized summary of the core findings.

Moreover, although the paper focuses on evaluation discrepancies across different methods and benchmarks, a significant portion of these cross-study comparisons is presented solely through lengthy descriptive text. While the authors have included some figures, I believe that incorporating more systematic comparison tables, summaries categorized by the quality of evidence, and visualizations of key results would greatly enhance the paper’s readability and help readers more intuitively grasp its primary contributions.

References

[1] Paleka et al. Pitfalls in Evaluating Language Model Forecasters. ICLR 2026.

Questions for authors
  1. Does the analysis of the "Evidence Ledger" reveal any significant conclusions that have not been identified in previous research? What is the incremental value of these findings relative to the existing literature?
  2. Is the sensitivity to weighting schemes observed in ForecastBench also present in other benchmarks? Could the authors conduct extended experiments to verify the generality of this phenomenon?
Reviewer_D2D6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper summarizes current design flaws of LLM forecasting and how to interpret different results.

Critical strengths

Gives a summary of the caveats of LLM forecasting benchmarks.

Critical weaknesses

Little contribution as it reads more like a synthesis of limitations from previous works. Not much insights from the analysis. Everything just reads as being inconclusive, and that give us no idea of how LLM stands today against human forecasters.

Questions for authors

None. The paper seems highly LLM generated with no new insights.

Reviewer_ADboOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper aims to organize the literature on language models and forecasting. First, it searches for papers that use language models for forecasting in a somewhat ad hoc way. Second, it aims to formalize a forecast and its evaluation. Third, it taxonomizes how language models are used for forecasting, and proposes audit questions. Fourth, it categorizes various forecasting benchmarks and audits various reported results. Finally, it suggests very basic data hygiene for language models and forecasting.

Critical strengths

None

Critical weaknesses

The paper makes very little contribution with respect to prior work, including Glasserman and Lin (2023), Sarkar and Vafa (2024), Paleka et al. (2026), and Xu et al. (2026).

There is very little analysis in the paper, and most of its space is dedicated to various taxonomies of forecasting evaluation. The existing analysis has ad hoc sample selection criteria, and the empirics are thin.

Questions for authors

Could you please provide much more detail on how the data was collected, and whether it was done systematically?

What does the paper contribute relative to the literature?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.