acceptodds
Under review as a conference paper at ICLR 2027

What Should We Forecast? Benchmarking Agents on Early Question Discovery

Abstract

Forecasting benchmarks usually assume that the question has already been written. We study the upstream problem: can an agent discover which questions should be asked before the relevant events become obvious? We introduce AnteBench, a benchmark that evaluates whether an agent formulates forecasting questions in advance about the uncertainties that later lead to consequential developments. Using only news published before a cutoff date, the agent produces a list of forecasting questions about the next 30 days; after the 30-day window closes, we construct a reference list of questions from the available information and match the two lists. We use average precision (AP) to evaluate how much of the reference list the agent covers and how highly it ranks those questions in its own list. In a 30-day backtest covering Eastern Europe and Russia in April 2026, we evaluate frontier agents and find that performance differs across agents and improves as reasoning effort increases. We also show that AnteBench serves as a testbed for experimentally studying agents' abilities to gather information and formulate relevant forecasting questions from it.

Primary area:
datasets and benchmarks
Keywords:
AI Forecasting, Question Discovery, LLM Agents, Benchmarking, Foresight Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dzsUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AnteBench, a benchmark for early question discovery: whether an agent can write, before a cutoff, the forecasting questions about uncertainties that later turn out to matter. A forecast-time agent sees only pre-cutoff news (through a cutoff-bounded Perigon wrapper) and submits a ranked list of questions about the next 30 days. After the window closes, a retrospective agent mines what actually happened and builds a gold set of questions that would have represented those issues at the cutoff, and an LLM judge matches the two lists question by question after an embedding prefilter. The score is average precision at a fixed budget. On one backtest, four frontier agents fall in an overlapping band above a GPT-5-mini control. A fixed-news experiment suggests extra reasoning helps mainly when the gathered news is rich. Further analyses cover hinted recovery from precursor signals, prediction-market question lists, second judges and a cross-miner check.

Critical strengths
  1. I think the problem is new and important, and the benchmark design is thoughtful. Forecasting benchmarks usually assume someone already wrote the question, and this paper evaluates the step before that in a way that makes omissions measurable: the gold set is built after the window, matching is at the level of the same underlying uncertainty (polarity doesn't matter, and close neighbors such as preconditions or downstream consequences are rejected), and each gold record carries a boundary note, positive and negative examples and pre-cutoff signals, so individual judgments can be audited. The paper is also careful about what its numbers mean. It says plainly that absolute levels depend on how the pool is built and that the experiment does not resolve the ordering among frontier agents.

  2. The robustness and diagnostic work is thorough. Two second judges (GPT-5.5 and GLM 5.2) confirm 94.1% and 93.3% of the official accepts and agree with 93.0% of verdicts on a boundary audit set, and the prefilter threshold is checked. DeepSeek-family agents are excluded from the main comparison because DeepSeek built the gold set and judges the matches, a choice the cross-miner check supports, and I think that was the right call. The fixed-news experiment is a nice use of the benchmark as a testbed for separating information gathering from question formulation, and the cost and tool-use analysis (GPT-5.5 makes about 650 news-tool calls per run versus about 200 for Claude Fable 5, in the same performance band) is useful context. Code and data are released.

Critical weaknesses
  1. I'm not convinced the gold set is a valid enough reference to support the scores, and since this is a benchmark paper, I think that's the central issue. The gold set is entirely an LLM artifact built with hindsight, and all of its LLM stages use one model, DeepSeek v4 Pro. It keeps only the 428 of 1,403 clusters that the DeepSeek miner submitted at least twice, the mining curves had not saturated when the runs declared completion, and a single run of an independent GPT-5.5 miner recovers 189 of the 329 gold questions; on that subset the DeepSeek agents' relative position changes, which the authors themselves read as consistent with a gold-authorship effect. There's no human check that gold questions were consequential or naturally askable at the cutoff (the only manual checks are a spot-check of category labels and a look at near-threshold judge accepts), and the prefilter was calibrated on positive examples written by the same pipeline. The hinted-recovery experiment makes this concrete: even when given the gold questions' own pre-cutoff signals, GPT-5.5 recovers only 29 of 160 (18.1%), and 16 of the 139 that no main run found. The paper frames this as a probe of whether misses come from sparse evidence or from not turning evidence into questions, but I think the result is also consistent with many gold questions not being askable from pre-cutoff news in the form the judge requires. Without a human-validated sample, I can't tell how much of the low absolute performance (frontier agents cover about 3% of the pool) reflects the agents versus the reference set.

  2. The empirical evidence is thin for a benchmark: one region, one 30-day window, and five runs per agent. The four frontier agents aren't separated, so the only gap between different models that the benchmark resolves is frontier versus GPT-5-mini, and that gap may be confounded, because GPT-5-mini's documented training cutoff is May 2024 versus December 2025 or January 2026 for the frontier models with documented cutoffs. GLM 5.2 was released in June 2026 with no documented cutoff, so parametric knowledge of April 2026 can't be excluded for it, as the authors note. The reasoning-effort finding is for one model, and AP@50 plateaus after medium effort; the conclusion says this, but the abstract's "improves as reasoning effort increases" glosses over it. The fixed-news comparison mixes the amount and the content of news (xhigh-derived sets are about ten times larger, as the authors note), and each article in a fixed set comes with the search queries that retrieved it, so the xhigh-derived sets may carry part of the xhigh agent's question formulation with them. The forward-testing deployment with a public leaderboard is described, but no forward-testing results are reported, and I think those would do the most to address the single-instance concern.

Questions for authors
  1. could you have people rate a random sample of gold questions on whether the issue was consequential and whether the question was naturally askable at the cutoff from pre-cutoff news? Given the 18.1% hinted-recovery rate, I'd like to know how much of the pool is realistically discoverable.

  2. Is there a human-labeled set of candidate-gold pairs for estimating the judge's precision and recall? The second-judge agreement is reassuring, but all three judges are LLMs and the prefilter was calibrated on examples written by the gold pipeline.

  3. How do scores and agent ordering change with a gold pool built by a different model family using the same five-run procedure, or by a union of miners? A single GPT-5.5 run recovers 189 of the 329 gold questions, so I'd like to see how much the pool itself, not just DeepSeek's position, depends on the miner.

  4. The pool keeps only clusters with at least two submissions (428 of 1,403). How many consequential issues are in the clusters you dropped, and how do scores change if they're included?

  5. Do you have results from the live deployment over multiple windows? Stable orderings and absolute levels across windows would address most of my concern about a single instance.

  6. Since GPT-5-mini's documented training cutoff is about a year and a half earlier than those of the frontier models with documented cutoffs, how much of its gap reflects stale background knowledge rather than question-discovery ability? A frontier model with an older cutoff, or a GPT-5-mini run with a background briefing, would help separate these.

5 comments
AuthorsOpenReview ↗

Response to Reviewer dzsU (1/3): questions 1 to 5

Thank you for the detailed and careful review. In this comment, we answer your questions 1 to 5. In the next comment, we answer question 6 and the other points.

Q1. Human rating of gold questions.

As you point out, we did not conduct human validation. We state this as the main limitation in Section 5. On the other hand, as written in the general response and in our reply to Reviewer ZAUr, we measured how robust the results are when we change the models for gold question mining and for the judge, and the results of the additional experiments support this.

About whether a gold question is "naturally askable": we use this phrase not to mean that there was enough information to ask the question in advance, but to mean that the granularity of the question was appropriate. As written in our reply to Reviewer ZAUr, it is not a necessary condition of this benchmark that every gold question could be asked in advance. This is because we cannot judge reliably whether a question could be asked only with hindsight, and a question that looks impossible to ask in advance might have been possible to ask by combining weak signals. To avoid removing such questions, we use all gold questions. So it is quite possible that even a very good agent cannot get a full score on this benchmark. On the other hand, a question that was truly impossible to ask in advance lowers the score of every agent by the same factor. So the comparison between agents does not change, and we think that it does not affect the validity of this benchmark (Section 3.1).

About the hinted-recovery result, we agree with your reading. The 18.1% is the result of one agent setting. It is not an upper bound of what can be asked, and it does not validate the gold pool. We changed Section 4.4 in this way.

Q2&3. Human-labelled pairs for the judge and gold pool

As you point out, we did not conduct human validation. In the current version of the paper, we write this as a limitation. On the other hand, as written in our reply to Reviewer ZAUr, we added ablation results of mining and the judge with a different model. We think that the results show that the main results of the paper are robust to changing the judge model and the mining model (Section 5 and Appendix M).

Q4. Clusters with only one submission.

As you point out, when we built the gold pool, we reported the results using only clusters with two or more elements. We added an additional experiment that checks the effect of this choice of clusters on the results of this benchmark. Specifically, we randomly sampled from the clusters with only one element, and we added them as gold questions. As a result, the main results of the paper did not change, and the ranking of the agent scores did not change much either (Appendix N). So the absolute score values change with this choice of clusters, but in the central part of the benchmark, which is distinguishing the performance between agents, there was almost no change.

Q5. Results from the live deployment.

We do not have results from a live deployment. As written in our reply to Reviewer UEHG, in this paper we ran the experiments with backtesting, in order to do comparative analyses under settings where we change conditions in detail. On the other hand, this benchmark supports forward testing directly with the same protocol. After the acceptance of the paper, we plan to release the forward-testing platform, including a real-time public leaderboard.

About the stability of the ranking between agents, we think that it is further supported by the two instances that we newly added (different regions and a different period). In the three instances, including the first instance, a part of the ranking between agents and the effect of changing the reasoning effort are reproduced. On the other hand, we think that this benchmark may also depend on how recent the knowledge of a model is, so the ranking of the agents may change in comparisons between instances of different periods.

AuthorsOpenReview ↗

Response to Reviewer dzsU (2/3): question 6 and other points

Q6. The older training cutoff of GPT-5-mini.

Thank you for the very useful suggestion. GPT-5-mini and the frontier models have a difference of about one and a half years in their official knowledge cutoffs, so the performance difference between them on this benchmark may include the difference in their knowledge. This does not hurt the validity of this benchmark. This benchmark measures the question generation ability, including the amount and the recency of the knowledge that the agent has about the target region and period. On the other hand, we think that the suggested control experiment is strongly useful for this discussion, so we added it.

Specifically, for the October instances, we compared the performance of GPT-5.4-mini (August 2025) in addition to GPT-5 (knowledge cutoff: Sep 2024) and GPT-5-mini (May 2024). The results were: GPT-5 scores 0.0112 and 0.0074, GPT-5-mini scores 0.0015 and 0.0017, and GPT-5.4-mini scores 0.0092 and 0.0076. This result shows that the performance difference cannot be explained only by the difference in the knowledge cutoffs of the models. This control experiment does not separate the two effects (the model size and the recency of its knowledge) (Appendix L).

GLM 5.2 and knowledge of April 2026.

About the issue that the knowledge cutoff of GLM 5.2 is not officially shown: as written in the general response, we tested it ourselves with questions that check the knowledge of the target period (April 2026) and before. Specifically, we gave descriptions of major events and asked in which month each event happened, without a web search tool. As a result, for events in 2024, the accuracy was 50%, but from June 2025 to March 2026 it was 2/200, and for April 2026 it was 3/317 correct, and these three were also questions about the Hungarian election, whose date was decided in advance. From this, we think that GLM 5.2 does not have the knowledge of April 2026. For comparison, we also asked about events of October 2026, which no model can know, and GLM 5.2 was correct for 0/120.

The frontier agents are not separated.

As also written in point 1 of the general response, in the three instances including the two instances that we added, a statistically significant difference over multiple runs is found for 3 of the 6 possible pairs of the 4 frontier agents.

Reasoning effort.

As you point out, the performance did not increase after medium. We corrected the description in the paper. Thank you for pointing this out. The increase in performance from none to medium was consistently found also in the two added instances, so we described it in this way.

AuthorsOpenReview ↗

Response to Reviewer dzsU (3/3): fixed news, mining, and corrections

The fixed-news comparison.

For the two points that you raised, we ran additional experiments (Appendix C.2).

  • Search queries: as you pointed out, we removed the search queries and ran the experiment again. The results do not change from the trend of the previous results. AP@50 is 0.0008 to 0.0026 with none-derived sets and 0.0060 to 0.0115 with xhigh-derived sets.
  • Amount of news: in addition to the point that you raised, we made one more fix and ran an additional experiment. Specifically, the full text of the xhigh-derived sets included the body text of some articles in addition to the summaries of all articles, while the full text of the none-derived sets included only summaries. We fixed this so that both of them include only summaries, and we also matched the number of tokens of both, and we ran the experiment again. As a result, at high generation effort, the two kinds of set are then within one standard error (0.0026 vs. 0.0040 with 6k to 16k tokens, and 0.0055 vs. 0.0072 with 37k tokens). So we think that these differences came mostly from the amount of articles.
  • Generation effort: with the same news sets, even when the sets include only summaries, high reasoning effort of the writer gave higher performance than no reasoning in all five sets. However, the difference was smaller than when the sets included the full text of articles. We think that this shows that the benchmark can also measure differences in the ability of agents in the part of formulating questions.

The size of the pool and when mining stops.

In the October instances, we stopped the mining of gold questions at 60 passes. To check the effect of this, we also stopped the mining at 60 passes in the April 2026 instance, and computed the benchmark scores again. As a result, we found that almost all gold questions obtained in the later part of mining do not match the forecasting questions that the agents generate. By stopping at 60 passes, the gold question set shrank from 329 questions to 260 questions, but on the other hand, the number of matches with the forecasting questions hardly changed, from 228 to 225. Also, the AP@100 scores of all agents rose almost equally by this, and their relative ranking did not change (Appendix O). We think that this shows that the benchmark is robust to where we stop the mining passes.

Corrections to statements that you cited.

  • Mining curves (Appendix C.6): the text said that the curves had not saturated, but the figure cannot show this. We replaced the text and added a table that counts the new issues from each run. More runs would still add questions, so the pool is not complete.
  • Second judges (Appendix E.2): the disagreement is symmetric only on the audit sample. With the strictest rule, coverage falls by 5% to 19%, depending on the agent.
  • Cross-miner check (Appendix F.2): on the 189 recovered questions, the scores rise mainly because the denominator is smaller. The gain factor is 1.42 to 1.63 for the main agents and 1.10 to 1.25 for the DeepSeek agents.
  • Tool use (Appendix C.4): the figure showed too few returned articles for GPT-5.5. The tool-call counts that you cited are not affected.
Reviewer_ZAUrOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces AnteBench, a benchmark for early forecasting-question discovery. Given a region and a cutoff date, an agent uses pre-cutoff news to produce a ranked list of questions about the next 30 days. A retrospective agent constructs a gold question pool after the window closes, and an LLM judge matches the submitted questions against this pool. The paper evaluates several agents on an Eastern Europe/Russia backtest and investigates how information gathering and reasoning effort affect question discovery.

Critical strengths
  1. The paper addresses an important upstream problem. Identifying which uncertainties deserve attention is a practically relevant capability that complements forecasting on predefined questions.

  2. The paper is clearly written and easy to follow.

Critical weaknesses
  1. The gold pool lacks systematic human validation. Although the paper reports limited manual spot-checks, the gold questions are primarily constructed and validated by LLMs. Independent expert review is needed to establish their importance.

  2. The evaluation covers only a single test scenario. All main results concern one region and one 30-day window, which is insufficient to establish a broadly reliable benchmark.

  3. The low scores in Table 1 are hard to interpret. It is unclear whether the agents perform poorly or the gold set is too demanding or unreliable. The authors also acknowledge possible bias from using DeepSeek to both construct the gold set and judge matches. Without stronger independent validation, these scores tell us little about the agents’ actual question-discovery ability.

Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Response to Reviewer ZAUr

Thank you for your review and for the important points.

A single test scenario.

As you point out, running experiments in only one region and one time window was not enough to show the reliability of the benchmark. So we added two more instances. As written in the general response, we ran additional experiments for October 2026 in two regions ("Middle East and North Africa" and "Latin America and the Caribbean"). The results in the added instances reproduce the trends of the existing instance. We think that this shows that the main results of this benchmark do not depend on the region or the period of the first instance.

DeepSeek builds the gold pool and also judges the matches.

About the concern that we use the same model (DeepSeek v4 Pro) for gold question mining and for the judge: as written in the general response, we ran additional experiments. We used GPT-5.5 instead of DeepSeek for mining and for the judge, and we changed the model for each of them, like a 2x2 matrix.

AgentDeepSeek pool, DeepSeek judgeDeepSeek pool, GPT-5.5 judgeGPT-5.5 pool, DeepSeek judgeGPT-5.5 pool, GPT-5.5 judge
Claude Fable 50.00870.01640.01190.0131
GPT-5.50.01190.01360.00960.0144
Claude Opus 4.80.00570.01030.00560.0092
GLM 5.20.00350.00950.00410.0067
GPT-5-mini0.00150.00270.00100.0018

In the results of this additional experiment, the trend of the ranking of the benchmark does not change much. In particular, the order of Claude Fable 5, Claude Opus 4.8, and GLM 5.2, which are not from the DeepSeek or OpenAI model family, does not change in the four settings of this 2x2 experiment. GPT-5.5 uses a looser standard than DeepSeek (it accepts more pairs), so the scores are higher in the table above. However, we think that this shows that the relative ranking of this benchmark is robust to the models for mining and for the judge. Also, the agreement rate between the two judges, DeepSeek and GPT-5.5, was 96% (Cohen's kappa 0.80). Please see Appendix M for the details.

Human validation of the gold pool.

We showed above that the results are robust when we change the models for mining and for the judge. However, as you point out, the lack of human validation is an important limitation, and we state it clearly in Section 5.

The low scores in Table 1.

Thank you for this point. There is a reason why the scores of this benchmark are low. The gold pool is designed to collect the important questions in the window as completely as possible. This may include questions that could be written only with hindsight. However, we cannot judge reliably whether a question could be asked only with hindsight. A question that looks impossible to ask in advance might have been possible to ask by combining weak signals. To avoid penalizing agents because of such false negatives, we designed the gold pool to be as complete as possible, and this makes the scores low.

Also, a question that was truly impossible to ask in advance lowers the score of every agent by the same factor. So it does not change the comparison, and we think that it does not hurt the validity of this benchmark (Section 3.1).

Also, as an approximate proxy for the set of questions that humans asked in advance, we use the question lists of prediction markets. Of course, these are not all of the questions that were actually asked in advance, but we think that they are useful to some degree as a reference for the scores of the agents (Section 4.5).

I appreciate the additional test scenarios and cross-judge verification, which alleviate some of my concerns. The explanation for the low scores is reasonable, but it remains unclear how much they reflect agent limitations versus questions that are only identifiable in hindsight. Together with the lack of systematic human validation of the gold pool, which I think is very important. These leave my main concerns unresolved. I therefore maintain my original score.

Thank you for your quick reply.

First, let us answer your concern about what share of the gold questions could have been asked.

We think that even if human reviewers judge whether a question needs hindsight, this is a judgment of the human ability to predict the future, and it is not the correct answer to whether hindsight was needed. This is because, even if asking a question seems to need hindsight for humans, a better forecaster might have been able to ask it by combining weak signals.

For example, in December 2019, most people did not recognize the possibility of a pandemic within a few months as a top possibility. On the other hand, BlueDot issued an alert about the outbreak in Wuhan on December 31, and this was about a week earlier than the announcements of the CDC and the WHO (source). In this example the difference is about a week, but a better forecasting agent in the future may point out the possibility at an even earlier time and ask a forecasting question. To human experts this may look like "a question that can be asked only with hindsight", but for that agent it is a question that it could ask well enough.

Our benchmark can give scores even to such future agents that exceed the human ability to forecast, because it judges only from what actually happened after the cutoff, and not from human judgment. Of course, as you point out, we think that the gold question list of this benchmark includes questions that truly can be asked only with hindsight (i.e., questions that even a very good forecaster cannot realistically ask from the given information), because of certain physical limits (the chaotic nature of the real world and the limited amount of input information). However, we believe that this is not a flaw in the design of the benchmark. It is an intentional design: we include problems that may or may not be solvable, in order to measure the ability of forecasting agents that exceed humans.

So, for the upper limit of how many questions an agent can find, which you asked about, we think that the accurate answer is "we do not know". On the other hand, the numbers in the paper give a lower bound. Of the April gold questions, 42 (12.8%) were matched by at least one agent. Also, among a sample of 139 questions that no agent could match, 16 were matched when we told the agent the precursor signals recorded for them and ran it again. We think that these show the lower bound.

Of course, we agree that validation by experts is important. Whether the content written in the gold questions actually happened, whether the verdicts of the LLM judge were appropriate, and so on, can be checked by humans. For these points, we will carry out a human validation for the final version. What we are claiming is that "whether the question could have been asked in advance" cannot be decided reliably by human judgment.

Thank you again for your thoughtful review of our paper.

Reviewer_UEHGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies how capable AI models are of determining what questions are important enough to forecast. The authors create a benchmark for evaluating the early question discovery capability. They provide a backtested experiment and describe that they have deployed a forward-testing experiment but do not discuss the results in the paper.

Critical strengths

Forecasting what should be forecast is a nice question!

Critical weaknesses
  1. The authors do not examine the correlation between question discovery ability and forecast accuracy. This seems natural and important to understand. Ultimately, question discovery is just another kind of (very very high dimensional) forecast. I can appreciate why it makes sense to think of it separately as a "forecast of what to forecast," but I would have been more excited to understand how this ability correlates to classic evaluation benchmarks.
  2. The authors do not include any forward-testing experiment. This feels like the only way to entirely eliminate the risk of polluted data, and also seems very doable.
Questions for authors

Why did you choose to include only back-testing results and not forward-testing?

1 comment
AuthorsOpenReview ↗

Response to Reviewer UEHG

Thank you for your review and for the good questions.

Question discovery and forecast accuracy.

Existing forecasting benchmarks and our new framework, a benchmark of what we should forecast in the first place, are different things at different stages. But as you say, we think that an analysis of how they are related is valuable. Thank you for the suggestion.

In Appendix Q of the paper, we added a table that puts the ranking of the agents that we tested on our benchmark next to their ranking on ForecastBench. We limit it to the agents that appear on both benchmarks. We show the same data below:

ModelForecastBench score (rank)AnteBench AP@100, two October regions
GPT-5.560.2 (7)0.0127 / 0.0122
GPT-559.6 (21)0.0112 / 0.0074
GPT-5-mini58.6 (43)0.0015 / 0.0017
Claude Opus 4.858.6 (43)0.0057 / 0.0069
GPT-5.4-mini55.7 (91)0.0092 / 0.0076

Qualitatively, the ranking trends of the two benchmarks look similar. For example, GPT-5.5 and GPT-5 take the top two places on both. On the other hand, there are also some differences. For example, the scores of GPT-5-mini and GPT-5.4-mini are reversed. The settings of our benchmark, AnteBench, and ForecastBench are very different (e.g., ForecastBench has no search tool), so we do not make a quantitative comparison.

Why only backtesting.

In this paper, we value the advantages of backtesting, such as its high reproducibility and the ability to repeat experiments under different conditions. So we use backtesting as the main experimental method of this paper. For example, for the experiment that changes the reasoning effort and the experiment that generates questions from a fixed news set in Section 4, it is very hard to use a forward test. On the other hand, Section 3.3 of the first submitted version of the paper had wording that could be read as reporting forward-test results, so we corrected it. We still plan to release the public leaderboard and the platform for the forward test after acceptance.

The risk of polluted data in the backtests.

As you point out, data pollution is one of the main problems in backtesting for AI forecasting. However, in our case, we think that this risk is small.

Information about future outcomes can enter in the following two ways:

  • The first way is through the training data of the agents. In the two new instances that we added, the period is after the release of all agents that we tested. Also, for the first instance, we added an experiment that checks the knowledge of GLM 5.2, and we confirmed that this risk is small.
  • The second way is through the search tool. In our backtests, we do not give web search. We give only the Perigon tools. These tools can search news by date, and we give the agents access only to articles published before the cutoff date. Also, after the experiments, we checked the dates of all articles given to the agents in the main runs, and there were no violations (Appendices G and L).

Still, a risk of data pollution may exist, for example through the order of search results. We recognize that a forward test is the most reliable way to remove it. For this reason, we plan to release the forward-testing platform and a real-time public leaderboard after acceptance.

Presentation.

Could you tell us which parts were the hardest to follow? We will rewrite those parts.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2): new instances and experiments

We would like to express our gratitude to the reviewers for carefully reading our paper. We have uploaded a revised PDF. All changes are highlighted in blue. In this comment, we present the main new results and list the changes. Further details are provided in our responses to each reviewer.

Two New Instances.

We added two regions: the Middle East and North Africa, and Latin America and the Caribbean. The cutoff date is October 1, 2026. This period is after the knowledge cutoff dates and the release dates of all models that we used in this paper. So we think that there is no leakage from the internal knowledge of the models.

The table shows AP@100 (average across 5 runs ± 1 standard error). Each instance has its own gold pool. The number of gold questions is listed in parentheses. Please compare scores only within the same column.

AgentEastern Europe/Russia, April (329)Middle East/North Africa, October (279)Latin America/Caribbean, October (336)
Claude Fable 50.0128 ± 0.00150.0087 ± 0.00190.0108 ± 0.0024
GPT-5.50.0111 ± 0.00120.0127 ± 0.00150.0122 ± 0.0019
Claude Opus 4.80.0100 ± 0.00150.0057 ± 0.00100.0069 ± 0.0017
GLM 5.20.0100 ± 0.00130.0035 ± 0.00060.0031 ± 0.0006
GPT-5-mini0.0016 ± 0.00030.0015 ± 0.00080.0017 ± 0.0008
  • The score of GPT-5-mini is lower than that of every frontier agent in all instances.
  • The experiment that changes the reasoning effort step by step was also reproduced in the two new instances. Please see the results in Section 4.2 and Appendix L. We think that it shows the effect of changing the reasoning effort of GPT-5.5 step by step more robustly.
  • About the comparison between frontier agents: over the three instances, three of the six pairs of frontier agents show a statistically significant difference. For example, GPT-5.5 outperformed Claude Opus 4.8 and GLM 5.2, and Claude Fable 5 outperformed GLM 5.2.

Other New Experiments.

  • A reviewer pointed out that we use the same DeepSeek model for gold question mining and for the judge. For this point, we ran additional experiments with GPT-5.5. In these experiments, we swapped GPT-5.5 and the DeepSeek model for gold question mining and for the judge, in a 2×2 matrix. As a result, the agreement rate between the two judge models was 96%, and the main results in the main text of the paper still hold.
  • In the question selection step of gold question mining, small clusters with only one element are not used. There was a concern that this may remove questions that are generated only rarely but are important. For this point, we randomly sampled questions from these removed clusters, added them to the gold question set, and ran the experiment again. As a result, the main results of the paper were reproduced, and the ranking of the agents did not change. We think that this result shows that the ranking of the agents does not depend on this detail of the method design.
  • There was a concern that GPT-5-mini performs poorly not only because of its low question-discovery ability but also because its knowledge cutoff is older than those of the other agents. There was also a concern that this benchmark measures only how recent the knowledge is, and not question-discovery ability. So we added two control agents to the experiment and measured their performance. As a result, we showed that the differences in ability between agents on this benchmark cannot be explained only by how recent the knowledge is.
  • For GLM 5.2, the exact knowledge cutoff is not public. So there was a concern that the model may contain knowledge of April. We tested whether GLM 5.2 has knowledge of April, and we showed that there is no sign of this.
AuthorsOpenReview ↗

General response (2/2): limitations and changes to the PDF

Remaining Limitations.

The biggest limitation is that we did not conduct an audit of the gold questions or the judge by real humans. Instead, as described above, we show that the benchmark results are robust to the way gold questions are selected, and that the results are robust when we change the judge model. On the other hand, we cannot deny that there is a difference between LLMs in general and humans. We consider this validation as future work. We have explicitly stated this as the main limitation (Section 5).

Forward testing

After the acceptance of the paper, we plan to release the platform, including the forward-testing leaderboard.

Changes to the PDF.

  • Table 1, Section 3.3, Section 4.1, and Section 4.2 now cover all three instances.
  • In Section 3.1, we explain that “naturally askable” refers to a condition regarding the granularity of the gold questions. This does not mean that someone could have found these questions in advance.
  • In Section 4.3, we removed the claim that reasoning is primarily useful when news information is abundant.
  • Section 5 states that the lack of human validation is the main limitation.
  • Appendix C.2 and the newly added Appendices L–Q report new experimental results. We have also revised several statements in the appendices that made claims stronger than the data supported. These revisions are listed in our response to reviewer dzsU.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.