- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes FABENCH, a backtesting environment for forecasting agents. This system automatically generates questions for past forecast dates. Agents research each question using both an offline archive covering the period prior to the forecast date and live web search. The results of the live web search are filtered by an LLM judge.
Critical strengths
One of the major challenges in existing backtesting with fixed corpus is the gap between backtest results and performance in forward testing or real-world deployment, caused by the limited coverage of the searchable corpus. This paper's attempt to integrate live web search into backtesting is therefore highly valuable.
Critical weaknesses
- Regarding the live web search tool in the backtesting environment, the paper lacks an analysis of the validity of the LLM judge's temporal screening, specifically how accurate it is against ground truth. Simply examining the agreement rate between the low and high reasoning-effort settings is insufficient to assess this point. Have you measured the LLM judge's precision and recall on pages whose publication dates are independently verified (e.g., with web archive snapshots)?
- While the paper addresses the risk of future information leaking through search rankings for the archive, it is unclear what measures, if any, are taken for the live web search (see Paleka et al., "Pitfalls in Evaluating Language Model Forecasters," ICLR 2026). Have you taken any measures against leakage through search engine rankings?
Questions for authors
See critical weaknesses
5 comments
Thank you for your detailed response and new analysis.
- I still have concerns about the first point. A recall of about 95% sounds high, but I would like to know how much impact the remaining 5% had. If important information is concentrated in this 5%, it could have a large effect on the agents' forecasts. Also, real-time news reports (e.g., The New York Times) often update the information on the same page. In such cases, the URL existed before the forecast date, but its content may have been updated after that date. Did you check these cases?
- I also have concerns about the method of the additional experiment for the second point. The experiment tries to measure the effect on forecasts by the direction that each article supports. However, in forecasting questions, the important articles are not always the ones that support one side. Articles that suggest which issues to focus on often play an important role in forecasting. For example, suppose a question asks whether a rocket launch will succeed, and the launch was actually postponed because of the wind direction. If an article that explains the importance of wind direction for rocket launches appears near the top of the search results, this leaks a hint to the agent that it should check the wind direction closely. Wind direction itself is a neutral topic for a rocket launch, so this case is exactly a false negative of your additional experiment. Also, you used 1,035 search queries for 33 questions. How did you compute the confidence intervals? These 1,035 queries are unlikely to be independent of each other, so the confidence intervals need to be computed with care.
Response to the follow-up (1/2)
We thank the reviewer for the careful follow-up and for valuing our integration of live web search into backtesting. The remaining questions relate to the pages that our LLM reviewer misses, to hints in the ranking, and to our confidence intervals.
Question 1: Pages that our LLM reviewer misses
(a) The remaining 5%
"A recall of about 95% sounds high, but I would like to know how much impact the remaining 5% had. If important information is concentrated in this 5%, it could have a large effect on the agents' forecasts."
We believe that the missed pages do not change our conclusions. We traced each missed page to the forecasts that read it. Only five of the 17 relate to the question they were read for. We then remove every affected question, that is, every question for which the agent read a page with information from after t0. Results show that each significant gain from retrieval stays significant.
| Model | Gain from retrieval without affected questions, BI |
|---|---|
| GPT-6 Astra | +3.90 (+1.96, +5.93) |
| GPT-6 Sol | +4.15 (+2.03, +6.27) |
| Claude Opus 5.5 | +2.31 (+0.54, +4.18) |
| DS V4.1 Flash | +4.32 (+2.02, +6.83) |
| Kimi K3 | +5.42 (+3.17, +7.72) |
| GLM 5.3 | +2.75 (+0.78, +4.75) |
| Qwen3.8-27B | +2.03 (−0.51, +4.61) |
| Qwen3.6-35B-A3B | +0.59 (−1.83, +3.18) |
(b) Pages updated after t0
"real-time news reports (e.g., The New York Times) often update the information on the same page. In such cases, the URL existed before the forecast date, but its content may have been updated after that date. Did you check these cases?"
We judge each page by the text that the agent receives, rather than by its URL or date (Sec. 3.2). Our LLM reviewer therefore rejects a page whose text was updated with information from after t0. To check the pages that still passed, two human reviewers read 300 random passed pages. They compared each page with its last archived copy from before t0. They found three pages with information from after t0, none of it related to the question:
- A Nature Medicine report on a virus case, read for a question on new WHO emergencies. Its final version on July 6 added a paragraph on fever surveillance.
- A UN report on the US economy, read for a question on August inflation. It was published on July 6, with data up to May.
- An Instagram post on a UK heatwave, read for a question on the July temperature. Its caption calls June the hottest on record, which was announced on July 1.
If such information related to the question, we would count the question as affected and remove it, as in (a).
Response to Reviewer WCLb
We thank the reviewer.
Yes. We measured both for our LLM reviewer, against human-verified dates and against web archive captures, as the reviewer suggested.
Human-verified dates. We sampled 1,535 pages from our runs. Three human reviewers read the 96 most disputed ones, mostly pages on which GPT-6 Astra, a model from another family, disagreed with the LLM reviewer. They used the original pages and archived copies. On the other pages, the two models agree. Against these labels, the LLM reviewer's recall on pages with real information from after t0 is 99.8% (95% CI 98.0% to 99.9%). Its precision is 100% (95% CI 98.8% to 100%): every page it blocked had some content from after t0.
Web archive captures. CC-News [1], Common Crawl's news archive, records when it captured each page, so it dates pages without any LLM. A page captured before t0 existed before t0. A page first captured after t0, and dated July 1 or later, was published after t0. 2,613 of the pages our LLM reviewer checked fall in one of these two groups.
Recall on pages published after t0 is 95.5%. All 17 misses are dated July 1 or 2, the first two days after t0. From July 3 on, the LLM reviewer blocked every page. Precision is 83.1%: of the 437 pages it blocked, 363 were published after t0. A wrongly blocked page costs the agent some evidence, but it cannot leak.
Yes. Two parts of our design limit this risk, and we measured what remains.
As Paleka et al. [2] note, a search engine queried after the outcome may rank pages that support the outcome higher. Our design limits this in two ways (Section 3.2). The search engine's own titles, snippets and dates never reach the agent, because they are generated today. The agent reads a page only after our LLM reviewer finds it public before t0. The ranking itself could still favor earlier pages that support the outcome. Search results change from one query to the next, even at the same moment [3], so a different list by itself shows nothing. What matters is whether the list favors pages that support the real outcome, and we tested this directly.
The test. In September, our agents searched the live web for 33 FutureX questions [4] before their outcomes were known, and we saved the ranked results of 1,035 queries. After the outcomes were known, we sent the same queries again with the same date limit. For each page in the top five, a model read the page without being told the outcome and labeled which answer it supports. Comparing these labels with the real outcome, we measured how much the top five favor it, with higher-ranked pages counting more ("output search bias" [5], on a scale from −100 to +100).
Result. After the outcomes were known, this bias changed by −1.4 (95% CI −5.5 to +2.4). For 36 control questions whose outcomes were already known at both searches, it changed by +0.2 (−2.7 to +3.2). The ranking does not favor the real outcome.
References
[1] Nagel, S. News dataset available. Common Crawl blog, 2016. https://commoncrawl.org/blog/news-dataset-available
[2] Paleka, D., Goel, S., Geiping, J., and Tramèr, F. Pitfalls in evaluating language model forecasters. In ICLR, 2026.
[3] Hannák, A., et al. Measuring personalization of web search. In WWW, 2013.
[4] Zeng, Z., Liu, J., Chen, S., et al. FutureX: An advanced live benchmark for LLM agents in future prediction. In ICLR, 2026.
[5] Kulshrestha, J., et al. Quantifying search bias: Investigating sources of bias for political searches in social media. In CSCW, 2017.