acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking Forecasting Agents on Yesterday's Tomorrows

Abstract

Digital agents can now gather and organize information, reason over it, and use tools to complete complex tasks. Forecasting puts all of these abilities to the test: it requires integrating incomplete evidence, inferring causal drivers, and committing to calibrated probabilistic judgments. Existing benchmarks are poorly suited to measuring and improving this ability. Live benchmarks provide feedback only after events resolve, so every evaluation cycle waits on the calendar; fixed-corpus evaluations return results immediately but confine research to pre-collected documents. We introduce , a scalable backtesting environment for orecasting gents. An automated pipeline discovers events, generates questions, and resolves outcomes for any past forecast date, so the benchmark can be renewed to stay ahead of model training cutoffs; its first release poses 287 questions across nine domains. Agents research each question freely in an archive environment that combines an offline archive with live web search screened to prevent temporal leakage. Evaluating agents built on eight recent models, we find that retrieval improves the Brier score of seven of them in every retrieval setting and lifts the leading agents to the market consensus on the 54 questions linked to Polymarket, with the best three scoring slightly above it. Research traces reveal recurring failures in how agents weigh and apply evidence, leaving ample room for improvement. We release our code, data, and archive environment as a foundation for community research on forecasting agents.

Primary area:
datasets and benchmarks
Keywords:
Forecasting, Autonomous Agents, Large Language Models, Backtesting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WCLbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes FABENCH, a backtesting environment for forecasting agents. This system automatically generates questions for past forecast dates. Agents research each question using both an offline archive covering the period prior to the forecast date and live web search. The results of the live web search are filtered by an LLM judge.

Critical strengths

One of the major challenges in existing backtesting with fixed corpus is the gap between backtest results and performance in forward testing or real-world deployment, caused by the limited coverage of the searchable corpus. This paper's attempt to integrate live web search into backtesting is therefore highly valuable.

Critical weaknesses
  • Regarding the live web search tool in the backtesting environment, the paper lacks an analysis of the validity of the LLM judge's temporal screening, specifically how accurate it is against ground truth. Simply examining the agreement rate between the low and high reasoning-effort settings is insufficient to assess this point. Have you measured the LLM judge's precision and recall on pages whose publication dates are independently verified (e.g., with web archive snapshots)?
  • While the paper addresses the risk of future information leaking through search rankings for the archive, it is unclear what measures, if any, are taken for the live web search (see Paleka et al., "Pitfalls in Evaluating Language Model Forecasters," ICLR 2026). Have you taken any measures against leakage through search engine rankings?
Questions for authors

See critical weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer WCLb

We thank the reviewer.

Weakness 1. Regarding the live web search tool in the backtesting environment, the paper lacks an analysis of the validity of the LLM judge's temporal screening, specifically how accurate it is against ground truth. Simply examining the agreement rate between the low and high reasoning-effort settings is insufficient to assess this point. Have you measured the LLM judge's precision and recall on pages whose publication dates are independently verified (e.g., with web archive snapshots)?

Yes. We measured both for our LLM reviewer, against human-verified dates and against web archive captures, as the reviewer suggested.

Human-verified dates. We sampled 1,535 pages from our runs. Three human reviewers read the 96 most disputed ones, mostly pages on which GPT-6 Astra, a model from another family, disagreed with the LLM reviewer. They used the original pages and archived copies. On the other pages, the two models agree. Against these labels, the LLM reviewer's recall on pages with real information from after t0 is 99.8% (95% CI 98.0% to 99.9%). Its precision is 100% (95% CI 98.8% to 100%): every page it blocked had some content from after t0.

Web archive captures. CC-News [1], Common Crawl's news archive, records when it captured each page, so it dates pages without any LLM. A page captured before t0 existed before t0. A page first captured after t0, and dated July 1 or later, was published after t0. 2,613 of the pages our LLM reviewer checked fall in one of these two groups.

Pages, by archive capture timePagesLet throughBlocked
Published after t038017363 (95.5%)
— dated July 1–221174
— dated July 3 or later3590359 (100%)
Already existed before t02,2332,159 (96.7%)74

Recall on pages published after t0 is 95.5%. All 17 misses are dated July 1 or 2, the first two days after t0. From July 3 on, the LLM reviewer blocked every page. Precision is 83.1%: of the 437 pages it blocked, 363 were published after t0. A wrongly blocked page costs the agent some evidence, but it cannot leak.

Weakness 2. While the paper addresses the risk of future information leaking through search rankings for the archive, it is unclear what measures, if any, are taken for the live web search (see Paleka et al., "Pitfalls in Evaluating Language Model Forecasters," ICLR 2026). Have you taken any measures against leakage through search engine rankings?

Yes. Two parts of our design limit this risk, and we measured what remains.

As Paleka et al. [2] note, a search engine queried after the outcome may rank pages that support the outcome higher. Our design limits this in two ways (Section 3.2). The search engine's own titles, snippets and dates never reach the agent, because they are generated today. The agent reads a page only after our LLM reviewer finds it public before t0. The ranking itself could still favor earlier pages that support the outcome. Search results change from one query to the next, even at the same moment [3], so a different list by itself shows nothing. What matters is whether the list favors pages that support the real outcome, and we tested this directly.

The test. In September, our agents searched the live web for 33 FutureX questions [4] before their outcomes were known, and we saved the ranked results of 1,035 queries. After the outcomes were known, we sent the same queries again with the same date limit. For each page in the top five, a model read the page without being told the outcome and labeled which answer it supports. Comparing these labels with the real outcome, we measured how much the top five favor it, with higher-ranked pages counting more ("output search bias" [5], on a scale from −100 to +100).

Result. After the outcomes were known, this bias changed by −1.4 (95% CI −5.5 to +2.4). For 36 control questions whose outcomes were already known at both searches, it changed by +0.2 (−2.7 to +3.2). The ranking does not favor the real outcome.

References

[1] Nagel, S. News dataset available. Common Crawl blog, 2016. https://commoncrawl.org/blog/news-dataset-available

[2] Paleka, D., Goel, S., Geiping, J., and Tramèr, F. Pitfalls in evaluating language model forecasters. In ICLR, 2026.

[3] Hannák, A., et al. Measuring personalization of web search. In WWW, 2013.

[4] Zeng, Z., Liu, J., Chen, S., et al. FutureX: An advanced live benchmark for LLM agents in future prediction. In ICLR, 2026.

[5] Kulshrestha, J., et al. Quantifying search bias: Investigating sources of bias for political searches in social media. In CSCW, 2017.

Thank you for your detailed response and new analysis.

  1. I still have concerns about the first point. A recall of about 95% sounds high, but I would like to know how much impact the remaining 5% had. If important information is concentrated in this 5%, it could have a large effect on the agents' forecasts. Also, real-time news reports (e.g., The New York Times) often update the information on the same page. In such cases, the URL existed before the forecast date, but its content may have been updated after that date. Did you check these cases?
  2. I also have concerns about the method of the additional experiment for the second point. The experiment tries to measure the effect on forecasts by the direction that each article supports. However, in forecasting questions, the important articles are not always the ones that support one side. Articles that suggest which issues to focus on often play an important role in forecasting. For example, suppose a question asks whether a rocket launch will succeed, and the launch was actually postponed because of the wind direction. If an article that explains the importance of wind direction for rocket launches appears near the top of the search results, this leaks a hint to the agent that it should check the wind direction closely. Wind direction itself is a neutral topic for a rocket launch, so this case is exactly a false negative of your additional experiment. Also, you used 1,035 search queries for 33 questions. How did you compute the confidence intervals? These 1,035 queries are unlikely to be independent of each other, so the confidence intervals need to be computed with care.
AuthorsOpenReview ↗

Response to the follow-up (1/2)

We thank the reviewer for the careful follow-up and for valuing our integration of live web search into backtesting. The remaining questions relate to the pages that our LLM reviewer misses, to hints in the ranking, and to our confidence intervals.

Question 1: Pages that our LLM reviewer misses

(a) The remaining 5%

"A recall of about 95% sounds high, but I would like to know how much impact the remaining 5% had. If important information is concentrated in this 5%, it could have a large effect on the agents' forecasts."

We believe that the missed pages do not change our conclusions. We traced each missed page to the forecasts that read it. Only five of the 17 relate to the question they were read for. We then remove every affected question, that is, every question for which the agent read a page with information from after t0. Results show that each significant gain from retrieval stays significant.

ModelGain from retrieval without affected questions, BI
GPT-6 Astra+3.90 (+1.96, +5.93)
GPT-6 Sol+4.15 (+2.03, +6.27)
Claude Opus 5.5+2.31 (+0.54, +4.18)
DS V4.1 Flash+4.32 (+2.02, +6.83)
Kimi K3+5.42 (+3.17, +7.72)
GLM 5.3+2.75 (+0.78, +4.75)
Qwen3.8-27B+2.03 (−0.51, +4.61)
Qwen3.6-35B-A3B+0.59 (−1.83, +3.18)

(b) Pages updated after t0

"real-time news reports (e.g., The New York Times) often update the information on the same page. In such cases, the URL existed before the forecast date, but its content may have been updated after that date. Did you check these cases?"

We judge each page by the text that the agent receives, rather than by its URL or date (Sec. 3.2). Our LLM reviewer therefore rejects a page whose text was updated with information from after t0. To check the pages that still passed, two human reviewers read 300 random passed pages. They compared each page with its last archived copy from before t0. They found three pages with information from after t0, none of it related to the question:

  • A Nature Medicine report on a virus case, read for a question on new WHO emergencies. Its final version on July 6 added a paragraph on fever surveillance.
  • A UN report on the US economy, read for a question on August inflation. It was published on July 6, with data up to May.
  • An Instagram post on a UK heatwave, read for a question on the July temperature. Its caption calls June the hottest on record, which was announced on July 1.

If such information related to the question, we would count the question as affected and remove it, as in (a).

Reviewer_exjcOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

FABENCH focuses on backtesting for forecasting agents, introducing a pipeline that generates and resolves questions about historical events and an environment that combines archived news with temporally filtered live web search. The authors show that retrieval generally improves forecasting performance, although the benefits vary across models and domains.

Critical strengths

The paper's setup, which combines archived news with filtered web search, provides a useful environment for backtesting forecasting agents.

Critical weaknesses

Limited methodological novelty

  1. Automated question generation and resolution have already been explored in ForecastBench, FutureX, Paleka et al. [1], and Bosse et al. [2]. While FABENCH focuses on historical events, its question-generation pipeline largely builds on existing techniques, with limited methodological novelty.

  2. The proposed environment mainly integrates existing components, including CC-News as a historical evidence archive and LLM-filtered live web search. While the integration may be useful, the methodological novelty is limited.

[1] Paleka et al., “Consistency Checks for Language Model Forecasters.” ICLR, 2025.

[2] Bosse et al., “Automating Forecasting Question Generation and Resolution for AI Evaluation.” arXiv preprint arXiv:2601.22444, 2026.

Questions for authors

Given that the main contribution of this paper is a benchmark, the following points should be clarified to ensure data quality and reproducibility of the pipeline.

  1. Is there any human validation of leakage filtering, question generation quality, and resolution quality? The current design relies on LLM checking, which may not be sufficient.

  2. Why not use historical questions from prediction markets or questions from existing forecasting benchmarks? What is the advantage of generating new questions through the proposed pipeline?

  3. Could the authors provide more details on event discovery and series selection? Also, how were the 33 Polymarket questions selected by experts (Appendix B, Table 3)?

6 comments
AuthorsOpenReview ↗

Response to Reviewer exjc (1/2)

We thank the reviewer for the comments.

Weakness. Limited methodological novelty

Automated question generation and resolution have already been explored in ForecastBench, FutureX, Paleka et al. [1], and Bosse et al. [2]. While FABENCH focuses on historical events, its question-generation pipeline largely builds on existing techniques, with limited methodological novelty.

The proposed environment mainly integrates existing components, including CC-News as a historical evidence archive and LLM-filtered live web search. While the integration may be useful, the methodological novelty is limited.

[1] Paleka et al., "Consistency Checks for Language Model Forecasters." ICLR, 2025. [2] Bosse et al., "Automating Forecasting Question Generation and Resolution for AI Evaluation." arXiv preprint arXiv:2601.22444, 2026.

We thank the reviewer for pointing out the related work. We agree that automated question generation and individual retrieval components have precedents. Our contribution is an evaluation environment that enables researchers to evaluate forecasting agents promptly and test alternative agent harnesses under open-ended information gathering.

What this environment enables. Researchers need to compare new models and agent designs without waiting weeks or months for each evaluation to resolve. Live benchmarks such as ForecastBench [3], FutureX [4], and Bosse et al. [2] necessarily wait for future outcomes. FABench instead evaluates forecasts at a historical date t0, using questions whose outcomes are already available for scoring. Researchers can therefore obtain feedback immediately after a run and iterate on different harnesses—including search strategies, tool-use policies, and evidence-synthesis procedures—on a common set of questions. The question-generation workflow also allows the benchmark to be renewed using forecast dates after newer models’ training cutoffs.

Why open-ended retrieval matters. Information gathering is a core part of forecasting: an agent must decide what to investigate, which sources to consult, and whether the available evidence is sufficient. A fixed corpus constrains both the breadth of accessible sources and the depth of follow-up investigation, limiting evaluation to the evidence selected by the benchmark’s creators. FABench combines a historical archive with temporally screened live web search, allowing agents to pursue additional sources as their investigation develops. This more closely approximates the information-gathering process in real-world forecasting while retaining immediate scoring. Our web filter assesses temporal eligibility without receiving the forecasting question or its resolved answer, reducing the risk of explicitly outcome-conditioned evidence selection.

What our experiments demonstrate. Our four retrieval settings—no retrieval, archive-only, filtered-web-only, and combined retrieval—allow researchers to examine how access to evidence affects forecasting performance. In the combined setting, retrieval improves the point estimates for all eight models, with 95% paired bootstrap intervals excluding zero for six. After removing all 22 outcome-visible question-construction exceptions, these six gains remain significant. These results show that information access materially affects measured forecasting ability and support evaluating the agent’s research process alongside its underlying model. FABench provides an environment in which researchers can systematically test how alternative harnesses turn that access into better forecasts. We will revise the contribution statement to make this research use explicit and expand Table 1 to include Bosse et al. [2] and Paleka et al. [1].

References

[1] Paleka, D., Pallavi Sudhir, A., Alvarez, A., Bhat, V., Shen, A., Wang, E., and Tramèr, F. Consistency checks for language model forecasters. In ICLR, 2025.

[2] Bosse, N. I., Mühlbacher, P., Wildman, J., Phillips, L., and Schwarz, D. Automating forecasting question generation and resolution for AI evaluation. arXiv:2601.22444, 2026.

[3] Karger, E., Bastani, H., Yueh-Han, C., Jacobs, Z., Halawi, D., Zhang, F., and Tetlock, P. E. ForecastBench: A dynamic benchmark of AI forecasting capabilities. In ICLR, 2025.

[4] Zeng, Z., Liu, J., Chen, S., et al. FutureX: An advanced live benchmark for LLM agents in future prediction. In ICLR, 2026.

AuthorsOpenReview ↗

Response to Reviewer exjc (2/2)

Given that the main contribution of this paper is a benchmark, the following points should be clarified to ensure data quality and reproducibility of the pipeline.

Question 1. Is there any human validation of leakage filtering, question generation quality, and resolution quality? The current design relies on LLM checking, which may not be sufficient.

Yes. Human reviewers validated all three.

Leakage filtering. We sampled 1,535 pages from our runs. Three human reviewers read the 96 most disputed ones, mostly pages on which GPT-6 Astra, a model from another family, disagreed with the LLM reviewer. They used the original pages and archived copies. On the other pages, the two models agree. By their labels, 0.03% (95% CI 0.01% to 0.30%) of the pages that reached the agents contain real information from after t0, not just a date stamp or a live price. The one such page we found is a Reuters report published on July 1. A second check uses no LLM. CC-News, Common Crawl's news archive, records when it captured each page. By these capture times, 380 of the pages our LLM reviewer checked were published after t0. It blocked 363 of them (95.5%), and all 17 it missed are dated July 1 or 2.

Question generation and resolution. Before release, three human reviewers checked every question and its recorded outcome. They checked that the question and its resolution rule are clear, that the event was still open at t0, and that the recorded outcome matches the official source. We will add this check to Appendix B, with the questions it changed or removed. In addition, a model from another family (GPT-6 Astra) resolved all 254 agent-written questions again from the official sources, without seeing our answers. It agreed with us on all 249 whose sources it could open.

Question 2. Why not use historical questions from prediction markets or questions from existing forecasting benchmarks? What is the advantage of generating new questions through the proposed pipeline?

Many important events have no market. Prediction markets open only for events people trade on. Many company results and official statistics are not among them, so a benchmark built only from markets would leave them out.

Questions from existing benchmarks are already old. They were posed before the training cutoffs of current models, so a model may simply remember the answers. With our workflow, we can write new questions for any past date after a new model's cutoff.

Question 3. Could the authors provide more details on event discovery and series selection? Also, how were the 33 Polymarket questions selected by experts (Appendix B, Table 3)?

Event discovery and series selection. We describe the process in Section 2.2 and give its settings, such as how many companies and series it covers, in Appendix B, Table 3. Given t0, the workflow runs in four steps.

  1. Scheduled releases. An LLM agent takes releases due after t0, such as company earnings, central-bank decisions and official statistics, from calendars and lists published before t0.
  2. One-off events. It searches news published before t0 for widely followed events still pending at t0, such as elections and drug approvals.
  3. Writing questions. Each event becomes a question with a clear resolution rule and a named source. For a scheduled release, a fixed rule sets the threshold from values published before t0. The exception is 22 earnings questions, whose thresholds we adjusted after the outcomes to balance Yes and No.
  4. Checking and resolving. A second agent and then three human reviewers check every question (Question 1). After the resolution date, we record the outcome from the named source.

We selected the 33 Polymarket questions in two steps. First, a program searched Polymarket for topics trending at t0, kept markets that were open at t0 and resolved by September 16, and ranked the contracts of each event by trading volume. Second, we reviewed these candidates as they appeared at t0 and kept widely followed events whose outcome can be verified, one contract per event. The review page hid each market's outcome, and we did not use model results. The 33 questions resolved 15 Yes and 18 No. We will add both steps to Appendix B.

Reproducibility. Upon acceptance, we will release every question with its resolution record, and the code to rebuild the news archive, as stated in our Reproducibility statement.

Thank you for your response and the additional analyses, which address most of my questions and I raise my rating from 1 to 2. I am not raising it further because of the limited methodological novelty, as the question-generation and retrieval components largely build on prior work. I am also concerned that the paper overstates the degree of automation in its pipeline. The authors state in the rebuttal that question generation involved substantial human effort, as three human reviewers checked every question, 22 earnings questions were adjusted after seeing the outcomes, and the 33 Polymarket questions were picked by hand.

Reviewer_fjL1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces FABENCH, which turns forecasting at historical dates into a backtesting task with immediate scoring. It combines a question-generation workflow, a pre-cutoff archive, and live web search screened by an LLM reviewer that does not see the question or answer. The initial release contains 287 questions across nine domains, with July 1, 2026 as the forecast date. Experiments compare eight models with no retrieval, archive retrieval, screened web search, and combined retrieval, using Brier Index and ECE to measure forecast error and calibration. Most models benefit from retrieval, and several have point estimates close to or above the market on 54 questions linked to Polymarket. The paper also analyzes retrieval-related failures.

Critical strengths

FABENCH combines renewable questions, historical retrieval, and immediate scoring. Agents can choose their own searches and sources, making the environment useful for repeated evaluation of forecasting methods.

The web reviewer’s design reduces the risk of selecting evidence based on known outcomes. The independent historical archive also provides a control for evaluating the contribution of web search.

The four retrieval settings help distinguish the contributions of different information sources. Archive retrieval alone also improves performance.

Critical weaknesses

Some questions were written with “outcomes visible”, but results excluding them are not reported. Low scores without retrieval do not rule out memorization of some outcomes, so question-selection bias and contamination from model training remain unresolved.

The web-filter audit compares the same model at two reasoning-effort settings. The reported “95.7%” agreement cannot detect errors shared by both settings. Similar archive and web scores also do not establish low leakage, since retrieval noise could offset leakage gains. An independent audit against historical sources is missing.

The market comparison includes 19 of 54 questions that “differ in the settling source or the handling of a postponement” from their matched contracts. These rules can change the appropriate forecast probability. Without separate results on the 35 exactly matched questions, the comparison mixes target differences with forecasting ability.

No confidence intervals or repeated-run results are reported. The market comparison covers only 54 questions, and each model “forecasts each question once per setting”. The best model scores 62.97 BI against the market’s 60.22, but the stability of this difference is unclear.

Questions for authors
  1. Do the retrieval gains and market comparisons still hold after excluding the question-construction exceptions and retaining only models with reliable training cutoffs?

  2. Can you report the proportion of pages containing later information that actually entered the agents’ context, the proportion of affected questions, and the results after excluding those questions, rather than only agreement between two effort settings of the same model?

  3. What are the paired Brier-score differences and confidence intervals between each model and the market on the 35 questions with identical resolution rules? Are the conclusions robust to excluding the other 19 questions and filtering by market liquidity at t0?

6 comments
AuthorsOpenReview ↗

Response to Reviewer fjL1 (1/3)

We thank the reviewer for the precise questions. We ran every analysis the reviewer asked for. All intervals below are 95% paired bootstrap intervals over questions.

Weakness 1. Some questions were written with "outcomes visible", but results excluding them are not reported. Low scores without retrieval do not rule out memorization of some outcomes, so question-selection bias and contamination from model training remain unresolved.

Question 1. Do the retrieval gains and market comparisons still hold after excluding the question-construction exceptions and retaining only models with reliable training cutoffs?

Yes, both still hold.

No model could have seen these outcomes in training. Every outcome in FABench was published after t0 = July 1, 2026. GPT-6 Astra and GPT-6 Sol state a pretraining cutoff of April 2026, and Claude Opus 5.5 of June 2026. The other five models do not state a cutoff, but they were released between April and September 2026, a few months after training usually ends, so their training also ended before t0 (Appendix E).

Why 22 questions were written with the outcome visible. We chose all other questions by how widely followed and how important the events were at t0 (Section 2.2). The 22 exceptions are earnings questions. A threshold at the midpoint of a company's own guidance mostly resolves Yes, because companies usually beat their own guidance. A model that always says Yes would then score well without forecasting anything. To keep these questions informative, we raised 17 thresholds and added 5 questions on capital expenditure, with the outcomes visible. We will add this to Appendix B.

Results without these 22 questions. Gain from retrieval (full minus none), in BI:

ModelAll 287 questionsWithout the 22
GPT-6 Astra+4.09 [+2.20, +6.10]+3.93 [+1.87, +6.13]
GPT-6 Sol+4.09 [+2.12, +6.18]+4.17 [+2.00, +6.38]
Claude Opus 5.5+2.44 [+0.68, +4.27]+2.47 [+0.65, +4.42]
DS V4.1 Flash+4.24 [+2.19, +6.39]+4.39 [+2.21, +6.68]
Kimi K3+5.84 [+3.61, +8.17]+5.69 [+3.33, +8.18]
GLM 5.3+3.03 [+1.05, +5.07]+2.94 [+0.86, +5.14]
Qwen3.8-27B+2.33 [−0.09, +4.78]+1.75 [−0.69, +4.21]
Qwen3.6-35B-A3B+1.47 [−0.79, +3.72]+1.59 [−0.78, +4.02]

The gains change by at most 0.58 BI, and the six gains whose intervals exclude zero still exclude zero. None of the 22 questions is among the 54 market questions, so removing them does not affect the market comparison.

For the three models that state a cutoff, both conclusions hold: their gains stay clearly above zero without the 22 questions, and GPT-6 Sol and GPT-6 Astra score above the market (62.97 and 62.15 BI against 60.22).

AuthorsOpenReview ↗

Response to Reviewer fjL1 (2/3)

Weakness 2. The web-filter audit compares the same model at two reasoning-effort settings. The reported "95.7%" agreement cannot detect errors shared by both settings. Similar archive and web scores also do not establish low leakage, since retrieval noise could offset leakage gains. An independent audit against historical sources is missing.

Question 2. Can you report the proportion of pages containing later information that actually entered the agents' context, the proportion of affected questions, and the results after excluding those questions, rather than only agreement between two effort settings of the same model?

Only live web pages can carry later information. The offline archive holds only news pages captured before t0 and Wikipedia revisions from before t0, so it cannot leak (Section 3.1). Web pages reach the agent only after our LLM reviewer judges them public before t0 (Section 3.2).

We report the three numbers the reviewer asked for.

1. Pages with later information that entered the agents' context: 0.03%. We sampled 1,535 pages from the runs in the paper. Three human reviewers read the 96 most disputed ones, mostly pages on which GPT-6 Astra, a model from another family, disagreed with the LLM reviewer. They used the original pages and archived copies. On the other pages, the two models agree. By these labels, 0.03% (95% CI 0.01% to 0.30%) of the pages that reached the agents contain real information from after t0, not just a date stamp or a live price. The one such page we found is a Reuters report published on July 1.

2. Affected questions: 4 to 20 of the 287 per model (1% to 7%). A question counts as affected for a model if a web page its agent read under full contains information from after t0 about that question. We checked every web page the agents read against its question. DeepSeek V4.1 Flash first marked the pages that might contain such information, and GPT-6 Astra, with web search, checked every marked page. In a random sample of the other pages, it found none. We have the list of pages read for 2,157 of the 2,296 forecasts; the other runs did not keep it.

3. Results without the affected questions.

ModelAffected questionsGain from retrieval, BI: all → without themModel − market, BI: all → without them
GPT-6 Astra8+4.09 → +3.90+1.93 → +1.32
GPT-6 Sol8+4.09 → +4.15+2.74 → +3.17
Claude Opus 5.514+2.44 → +2.31−0.37 → +0.13
DS V4.1 Flash20+4.24 → +3.84−0.73 → −1.12
Kimi K314+5.84 → +5.50+1.67 → +1.25
GLM 5.34+3.03 → +2.75−1.77 → −1.81
Qwen3.8-27B14+2.33 → +2.03−3.25 → −4.07
Qwen3.6-35B-A3B4+1.47 → +1.40−4.94 → −4.94

Every gain stays positive and changes by at most 0.41 BI, and each gap to the market changes by at most 0.82 BI. [Under online-filtered, where agents use only filtered web search, more questions are affected: 6 to 51 per model. Without them, every gain stays positive and changes by at most 0.97 BI. The online-filtered runs of the two Qwen models did not keep the list of pages read.]

Independent audit against historical sources. CC-News [1], Common Crawl's news archive, records when it captured each page, so it dates pages with no LLM involved. A page captured before t0 existed before t0. A page first captured after t0, and dated July 1 or later, was published after t0. 2,613 of the pages our LLM reviewer checked fall in one of these two groups. The LLM reviewer blocked 363 of the 380 pages published after t0 (95.5%). All 17 it missed are dated July 1 or 2, and from July 3 on it blocked every page. It let through 2,159 of the 2,233 pages that existed before t0 (96.7%).

References

[1] Nagel, S. News dataset available. Common Crawl blog, 2016. https://commoncrawl.org/blog/news-dataset-available

AuthorsOpenReview ↗

Response to Reviewer fjL1 (3/3)

Weakness 3. The market comparison includes 19 of 54 questions that "differ in the settling source or the handling of a postponement" from their matched contracts. These rules can change the appropriate forecast probability. Without separate results on the 35 exactly matched questions, the comparison mixes target differences with forecasting ability.

Question 3. What are the paired Brier-score differences and confidence intervals between each model and the market on the 35 questions with identical resolution rules? Are the conclusions robust to excluding the other 19 questions and filtering by market liquidity at t0?

The rule differences do not change the comparison. We wrote these 19 questions with our own workflow and found the matching Polymarket markets afterwards. They ask about the same events, with the same Yes condition, as their markets. Only the source that settles the result, or the handling of a delay, differs. For example, our 14 questions on primary elections settle on the state's official count, while the markets settle on the result the party announces first. So an agent reads and answers each question in the same way under either rule. These differences never applied: no event was delayed, and all 19 markets settled with the result we recorded.

Results on the 35 questions with the same rules as their markets, under full. The second column keeps only the 14 of them whose markets had at least $1,480 traded in the 30 days before t0.

ModelModel − market, BI: 35 questionsModel − market, BI: 14 of them with more tradingModel − market, Brier: 35 questions
Kimi K3+6.31 [−1.61, +14.89]+6.93 [−1.53, +16.62]−0.054 [−0.119, +0.016]
GPT-6 Sol+5.24 [−2.25, +13.49]+9.31 [−0.29, +20.34]−0.046 [−0.112, +0.022]
GPT-6 Astra+4.67 [−2.98, +12.87]+8.02 [−0.91, +17.37]−0.041 [−0.106, +0.027]
Claude Opus 5.5+1.85 [−6.90, +11.12]+1.07 [−10.22, +14.42]−0.017 [−0.092, +0.068]
DS V4.1 Flash+1.50 [−5.62, +9.09]+0.64 [−10.46, +12.06]−0.014 [−0.080, +0.054]
GLM 5.3+1.02 [−4.78, +6.92]+1.99 [−4.94, +9.90]−0.009 [−0.059, +0.046]
Qwen3.8-27B+0.19 [−7.35, +7.97]−3.44 [−16.85, +11.00]−0.002 [−0.069, +0.071]
Qwen3.6-35B-A3B−2.20 [−10.30, +6.19]−4.88 [−13.90, +5.53]+0.021 [−0.053, +0.101]

On both sets, Kimi K3, GPT-6 Sol and GPT-6 Astra score above the market, by more than on all 54 questions. All intervals include zero, so these three models are on par with the market. We give the paired Brier differences in the last column, where negative values favor the model. They rank the models in the same order.

Weakness 4. No confidence intervals or repeated-run results are reported. The market comparison covers only 54 questions, and each model "forecasts each question once per setting". The best model scores 62.97 BI against the market's 60.22, but the stability of this difference is unclear.

GPT-6 Sol stays above the market in repeated runs. We ran GPT-6 Sol three more times on the 54 market questions under full, with the same settings as in the paper. The runs score 60.86, 62.76 and 62.10 BI, against 62.97 in the paper and 60.22 for the market. The run-to-run standard deviation is 1.0 BI. All four runs lie above the market. We will add these runs, and 95% intervals for every comparison, to Appendix E.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

General response

Dear ICLR 2027 PCs, SACs, ACs and Reviewers,

Thank you for your constructive feedback. We address the three main concerns below. All revisions are marked in blue.

  • Leakage (fjL1, WCLb, exjc). New human checks show that leakage is very low and minor. Removing the affected questions barely changes our results (Sec. 4.3, Appendix D).
  • Automatic question generation (exjc, fjL1). Agents write the questions and record the outcomes. People only review them. Our conclusions hold on the questions that no one changed (Sec. 2.2, Appendix B).
  • Novelty (exjc). FABench writes new questions for any past date. Its agents search the live web, and our reviewer checks every page for timing alone, without seeing the question or its answer. This blocks future leakage while keeping useful information from before the cutoff (Sec. 2.3, Table 1).

We give point-by-point responses under each review. Thank you again for your time and consideration.

AuthorsOpenReview ↗

Additional leakage checks

Additional leakage checks — update for all reviewers

We thank the reviewers for their comments on future-information leakage. We have expanded our human audit to cover pages accepted by both models, complementing our earlier checks of cases where the models disagreed.

  1. Pages where the two models disagreed. Our initial audit started with 3,000 randomly sampled pages that passed the filter. A text search flagged 235 pages mentioning a post-cutoff date or a live price. GPT-6 Astra reviewed all 235 flagged pages and a random sample of 1,000 of the remaining 2,765 pages. Human reviewers then examined every case where Astra disagreed with our original LLM reviewer. They identified one page containing post-cutoff information unrelated to its forecasting question, which we classify as minor leakage.
  2. Pages accepted by both models. We added a human spot-check of 300 randomly selected pages accepted by both models. Two human reviewers compared each page with its latest archived version from before the forecast cutoff. They found three pages containing post-cutoff information, all unrelated to the corresponding forecasting questions. These were also classified as minor leakage.

Across both checks, all identified leakage was minor. The human reviewers found no post-cutoff information relevant to the associated forecasting questions. Appendix D provides the audit details.

We also repeated the performance comparisons after excluding affected questions and runs without page lists. All six previously significant retrieval gains remain statistically significant. For each of these six models, the entire paired 95% bootstrap confidence interval indicates a positive improvement from retrieval.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.