- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents DeFuture, a framework that decouples different components of how a LLM forecast. Notably, they divide this into four phases: evidence acquisition, interpretation, prediction and judging. The paper evaluates each by running controlled experiments, and show how each of them contributes to the final performance. The authors also release the labels, evidence, and outputs.
Critical strengths
- Interesting aspect to study. Current work mostly focus on optimizing performance, but lack an understanding of which component contributes the most. This study help us understand which aspect needs more attention than others.
- A lot of effort to ensure statistical robustness like paired bootstrap, holm correction, BCa intervals, etc.
- The different controls around the experiments allow for informative analysis
- Authors release outputs, labels, evidence, etc. to support their claims
Critical weaknesses
- The golden-8 shows hindsight rather than attainable evidence. The reason is because the selector reads the resolved answer before choosing evidence. We can also see this across the results where withholding the answer degrades F1 for DeepSeek and Qwen predictors as well as Table 3 showing how Blind-8 is also a few points below plain retrieval ranking. This means that only knowing the answer makes the selection better than retrieval.
- The claim about how even the best model gets more than 25% of questions wrong when the evidence contains the answer is a bit overstated. The framework currently only count when the answer's text appears somewhere in the evidence, but don't consider cases where the evidence reveals the answer without explicitly having it in the text.
- The claim about how evidence matter more than the model seems also to be slightly too overstated with limited experiments with frontier models. The only one shown is GPT-5.6-sol with the answer-picked evidence.
Open to raise score if certain claims are clarified.
Questions for authors
See weaknesses.