acceptodds
Withdrawn from ICLR 2027

DeFuture: Decoupling LLM Forecasting with Hindsight Evidence

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Future event prediction tests whether a large language model (LLM) can turn information about the present into a judgment about the future. Current benchmarks score each system end-to-end, so a failed prediction cannot be traced to specific components, and an agentic forecaster cannot tell whether to improve its search, its processing, or its reasoning. We introduce **DeFuture**, which decouples future event prediction into evidence acquisition, interpretation, and prediction, and anchors this decomposition on golden evidence that a selector picks in hindsight from the same answer-blind evidence pool by reading the resolved answer. Golden evidence makes every phase measurable: it tops a ladder of four evidence tiers, carries known answer-bearing content for scoring each Interpreter, and gives all Predictors a shared input. On 1,254 FutureX-Past questions and 340 MIRAI relation queries, DeFuture shows that (1) decoupling measures each phase at almost no cost in accuracy, (2) evidence weighs more than model capability, since a 4B Interpreter and a 27B Predictor match a frontier model given golden evidence, and (3) even frontier Predictors miss over 25% of questions whose golden evidence contains the answer, with extra reasoning helping only at prediction. We release the golden evidence, evidence pools, model outputs, and code.

Primary area:
datasets and benchmarks
Keywords:
Future Event Prediction, Forecasting, Large Language Models, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_DDkSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents DeFuture, a framework that decouples different components of how a LLM forecast. Notably, they divide this into four phases: evidence acquisition, interpretation, prediction and judging. The paper evaluates each by running controlled experiments, and show how each of them contributes to the final performance. The authors also release the labels, evidence, and outputs.

Critical strengths
  • Interesting aspect to study. Current work mostly focus on optimizing performance, but lack an understanding of which component contributes the most. This study help us understand which aspect needs more attention than others.
  • A lot of effort to ensure statistical robustness like paired bootstrap, holm correction, BCa intervals, etc.
  • The different controls around the experiments allow for informative analysis
  • Authors release outputs, labels, evidence, etc. to support their claims
Critical weaknesses
  • The golden-8 shows hindsight rather than attainable evidence. The reason is because the selector reads the resolved answer before choosing evidence. We can also see this across the results where withholding the answer degrades F1 for DeepSeek and Qwen predictors as well as Table 3 showing how Blind-8 is also a few points below plain retrieval ranking. This means that only knowing the answer makes the selection better than retrieval.
  • The claim about how even the best model gets more than 25% of questions wrong when the evidence contains the answer is a bit overstated. The framework currently only count when the answer's text appears somewhere in the evidence, but don't consider cases where the evidence reveals the answer without explicitly having it in the text.
  • The claim about how evidence matter more than the model seems also to be slightly too overstated with limited experiments with frontier models. The only one shown is GPT-5.6-sol with the answer-picked evidence.

Open to raise score if certain claims are clarified.

Questions for authors

See weaknesses.

Reviewer_pxo9OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces DeFuture, a benchmark for evaluating forecasting performance. The authors decompose forecasting into four components – evidence acquisition, interpretation, prediction, and judging – and evaluate each component. They design the benchmark with different tiers of evidence quality, with Golden evidence selected in hindsight using the true answer as input. The authors additionally evaluate different combinations of Interpreter and Predictor model to separate the effects of interpreting evidence from using evidence to make a forecasting prediction. Using their benchmark, they claim that evidence quality is more important than model capability, that Predictors often fail to reach the correct conclusion even given evidence that includes the true answer, and that increased reasoning effort is more beneficial during prediction than during interpretation.

Critical strengths

There are a few strengths of this paper. First, I find that the decomposition the authors provide is straightforward and well-motivated and is a plausible way to identify the true failure points of forecasting.

In addition, I find that the proposed benchmark is extensive and provides a number of controlled experiments that produce interesting results.

Critical weaknesses

The paper has a few different weaknesses.

First, the writing is not very clear and often appears AI generated. This makes it difficult to read and understand.

An additional point to note is that the authors consistently use the word "decouples" when it seems like they should be saying "decomposes". This is a bit distracting and unclear.

The largest weakness in the paper, however, is a lack of novelty. The decomposition the authors provide seems fairly straightforward and there exists prior work which has evaluated these different components separately before. For example, in Yang et. al (2026) [1], the authors explicitly evaluate "five critical dimensions: source selection, evidence extraction, reasoning synthesis, reasoning-to-prediction alignment, and recognition of prediction uncertainty". In Halawi et. al. (2024) [2], the authors also provide ablations for different components of the prediction pipeline. It is unclear whether this benchmark is needed and if it adds much beyond prior work.

Additionally, there are some methodological issues with the benchmark itself. First, the Golden evidence hindsight selection introduces an confounding signal, namely it may leak information about the target rather than simply being higher quality. To illustrate this, consider a question with many possible future outcomes. Hindsight selection based on the true outcome may include evidence that primarily discusses the specific true outcome, rather than evidence which is simply higher quality in general (e.g. has better reasoning, gets the causal story more correct, etc.). The authors do not separate these two potential effects and do not adequately rule out the possibility of leakage, which impacts the claim that evidence quality is more important than model capability. There is also very little human evaluation of the Golden evidence and it is unclear if it is actually more informative than Silver evidence.

Second, there may be issues with the temporal validity. Table 7 indicates that many documents do not have a publication date and the authors acknowledge that some articles were dated on or after the end date in Appendix A.4. The temporal screening prompt also seems too strong and may exclude articles that contain predictions about the future event we care about.

[1] Yang, Qingchuan, Simon Mahns, Sida Li, Anri Gu, Jibang Wu, and Haifeng Xu. "Llm-as-a-prophet: Understanding predictive intelligence with prophet arena." In International Conference on Learning Representations, vol. 2026, pp. 107020-107072. 2026.

[2] Halawi, Danny, Fred Zhang, Chen Yueh-Han, and Jacob Steinhardt. "Approaching human-level forecasting with language models." Advances in Neural Information Processing Systems 37 (2024): 50426-50468.

Questions for authors

How do you justify treating the Golden evidence condition as a true upper bound? It does not necessarily seem to be the case by construction, since the documents are produced by scoring and selecting documents with hindsight information, which still may not yield the most informative documents.

Did you consider simpler baselines for the structured task performance?

Reviewer_bwCWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a benchmark for the task of "LLM Future event prediction", decoupling the prediction into "Evidence Acquisition, Interpretation, Prediction and Judging" in order to run controlled ablation of each phase contribution on the benchmark performance.

Experiments are run on the FutureX-Past and MIRAI benchmarks showing that "(1) decoupling measures each phase at almost no cost in accuracy, (2) evidence weighs more than model capability, since a 4B Interpreter and a 27B Predictor match a frontier model given golden evidence, and (3) even frontier Predictors miss over 25% of questions whose golden evidence contains the answer, with extra reasoning helping only at prediction" - as per the abstract.

The conclusion "Golden evidence quantifies each phase, and the decoupled phases show what makes evidence golden" is a good summary of what I understood from this paper.

The submission is unclear, poorly structured and poorly motivated, appears to have been mostly AI-generated without proofreading. The experiments poorly support the claims of the paper and do not refer to previous works from the literature. My recommendation is rejection.

Critical strengths

None

Critical weaknesses

The abstract is written as if the reader was already aware of the paper's content: ambiguous terms and concepts in the first few sentences ("judgment", "agentic forecaster", "golden evidence", "answer-blind evidence pool", "ladder of four evidence tiers", "answer-bearing content", "Interpreter, Predictor").

Experimental results are presented before even knowing what the claimed contributions of the paper are, citing metrics that have no definition. The first finding in those results is a tautology, and presents the results in a way that is opposite to the common sense ("adding an Interpreter retains 98.6% of the accuracy of the Predictor" ???).

The Table 2 legend indicates 3 tiers (top, middle, bottom) that do no match the content (two tables are presented).

Results are not grounded nor compared in/with previous literature.

Questions for authors

In one sentence that a human can understand, what is the most valuable finding from the benchmark you are proposing, that you would like to share with the ICLR community?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.