- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a new method for search-based agent for forecasting named "Bayesian Linguistic Forecaster" where the agent is able to update its belief state consisting of a probability and a rationale. They show how their proposed framework allow models to outperform external agents on ForecastBench.
Critical strengths
- There is a lot of effort into preventing leaks from backtesting such as various auditing, date filtered search, etc.
- Nice analysis with the paired comparisons and variance decomposition. Decoupling question difficulty and method from the outcome allows for better interpretability.
- The research question is interesting, but the setup isn't ideal.
Critical weaknesses
- Paper only studies performance across Gemini models, but uses GPT-5.4 for certain ablations. Not sure why not extend the coverage of models tested and why gemini.
- ForecastBench isn't a very good dataset to use even if others are using it. If you look at the dataset, most questions can be answered without any knowledge of the topic mostly because it is binary and contain logical leakage.
- While the paper state the work was done around 4.20, I think it would have been better to continue the work and test it on new benchmarks like FutureSim who looked at very similar things. https://arxiv.org/abs/2605.15188
- Lack comparison to structured agent memory baselines
- Results seems to be picked to look better. In the appendix, we see how BLF scores considerably lower on AIBQ2 or that the zero-shot baseline ties the best external agent and outperforms their own search agent.
Questions for authors
In Weaknesses section