- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces AnteBench, a benchmark for early question discovery: whether an agent can write, before a cutoff, the forecasting questions about uncertainties that later turn out to matter. A forecast-time agent sees only pre-cutoff news (through a cutoff-bounded Perigon wrapper) and submits a ranked list of questions about the next 30 days. After the window closes, a retrospective agent mines what actually happened and builds a gold set of questions that would have represented those issues at the cutoff, and an LLM judge matches the two lists question by question after an embedding prefilter. The score is average precision at a fixed budget. On one backtest, four frontier agents fall in an overlapping band above a GPT-5-mini control. A fixed-news experiment suggests extra reasoning helps mainly when the gathered news is rich. Further analyses cover hinted recovery from precursor signals, prediction-market question lists, second judges and a cross-miner check.
Critical strengths
-
I think the problem is new and important, and the benchmark design is thoughtful. Forecasting benchmarks usually assume someone already wrote the question, and this paper evaluates the step before that in a way that makes omissions measurable: the gold set is built after the window, matching is at the level of the same underlying uncertainty (polarity doesn't matter, and close neighbors such as preconditions or downstream consequences are rejected), and each gold record carries a boundary note, positive and negative examples and pre-cutoff signals, so individual judgments can be audited. The paper is also careful about what its numbers mean. It says plainly that absolute levels depend on how the pool is built and that the experiment does not resolve the ordering among frontier agents.
-
The robustness and diagnostic work is thorough. Two second judges (GPT-5.5 and GLM 5.2) confirm 94.1% and 93.3% of the official accepts and agree with 93.0% of verdicts on a boundary audit set, and the prefilter threshold is checked. DeepSeek-family agents are excluded from the main comparison because DeepSeek built the gold set and judges the matches, a choice the cross-miner check supports, and I think that was the right call. The fixed-news experiment is a nice use of the benchmark as a testbed for separating information gathering from question formulation, and the cost and tool-use analysis (GPT-5.5 makes about 650 news-tool calls per run versus about 200 for Claude Fable 5, in the same performance band) is useful context. Code and data are released.
Critical weaknesses
-
I'm not convinced the gold set is a valid enough reference to support the scores, and since this is a benchmark paper, I think that's the central issue. The gold set is entirely an LLM artifact built with hindsight, and all of its LLM stages use one model, DeepSeek v4 Pro. It keeps only the 428 of 1,403 clusters that the DeepSeek miner submitted at least twice, the mining curves had not saturated when the runs declared completion, and a single run of an independent GPT-5.5 miner recovers 189 of the 329 gold questions; on that subset the DeepSeek agents' relative position changes, which the authors themselves read as consistent with a gold-authorship effect. There's no human check that gold questions were consequential or naturally askable at the cutoff (the only manual checks are a spot-check of category labels and a look at near-threshold judge accepts), and the prefilter was calibrated on positive examples written by the same pipeline. The hinted-recovery experiment makes this concrete: even when given the gold questions' own pre-cutoff signals, GPT-5.5 recovers only 29 of 160 (18.1%), and 16 of the 139 that no main run found. The paper frames this as a probe of whether misses come from sparse evidence or from not turning evidence into questions, but I think the result is also consistent with many gold questions not being askable from pre-cutoff news in the form the judge requires. Without a human-validated sample, I can't tell how much of the low absolute performance (frontier agents cover about 3% of the pool) reflects the agents versus the reference set.
-
The empirical evidence is thin for a benchmark: one region, one 30-day window, and five runs per agent. The four frontier agents aren't separated, so the only gap between different models that the benchmark resolves is frontier versus GPT-5-mini, and that gap may be confounded, because GPT-5-mini's documented training cutoff is May 2024 versus December 2025 or January 2026 for the frontier models with documented cutoffs. GLM 5.2 was released in June 2026 with no documented cutoff, so parametric knowledge of April 2026 can't be excluded for it, as the authors note. The reasoning-effort finding is for one model, and AP@50 plateaus after medium effort; the conclusion says this, but the abstract's "improves as reasoning effort increases" glosses over it. The fixed-news comparison mixes the amount and the content of news (xhigh-derived sets are about ten times larger, as the authors note), and each article in a fixed set comes with the search queries that retrieved it, so the xhigh-derived sets may carry part of the xhigh agent's question formulation with them. The forward-testing deployment with a public leaderboard is described, but no forward-testing results are reported, and I think those would do the most to address the single-instance concern.
Questions for authors
-
could you have people rate a random sample of gold questions on whether the issue was consequential and whether the question was naturally askable at the cutoff from pre-cutoff news? Given the 18.1% hinted-recovery rate, I'd like to know how much of the pool is realistically discoverable.
-
Is there a human-labeled set of candidate-gold pairs for estimating the judge's precision and recall? The second-judge agreement is reassuring, but all three judges are LLMs and the prefilter was calibrated on examples written by the gold pipeline.
-
How do scores and agent ordering change with a gold pool built by a different model family using the same five-run procedure, or by a union of miners? A single GPT-5.5 run recovers 189 of the 329 gold questions, so I'd like to see how much the pool itself, not just DeepSeek's position, depends on the miner.
-
The pool keeps only clusters with at least two submissions (428 of 1,403). How many consequential issues are in the clusters you dropped, and how do scores change if they're included?
-
Do you have results from the live deployment over multiple windows? Stable orderings and absolute levels across windows would address most of my concern about a single instance.
-
Since GPT-5-mini's documented training cutoff is about a year and a half earlier than those of the frontier models with documented cutoffs, how much of its gap reflects stale background knowledge rather than question-discovery ability? A frontier model with an older cutoff, or a GPT-5-mini run with a background briefing, would help separate these.
5 comments
Response to Reviewer dzsU (2/3): question 6 and other points
Q6. The older training cutoff of GPT-5-mini.
Thank you for the very useful suggestion. GPT-5-mini and the frontier models have a difference of about one and a half years in their official knowledge cutoffs, so the performance difference between them on this benchmark may include the difference in their knowledge. This does not hurt the validity of this benchmark. This benchmark measures the question generation ability, including the amount and the recency of the knowledge that the agent has about the target region and period. On the other hand, we think that the suggested control experiment is strongly useful for this discussion, so we added it.
Specifically, for the October instances, we compared the performance of GPT-5.4-mini (August 2025) in addition to GPT-5 (knowledge cutoff: Sep 2024) and GPT-5-mini (May 2024). The results were: GPT-5 scores 0.0112 and 0.0074, GPT-5-mini scores 0.0015 and 0.0017, and GPT-5.4-mini scores 0.0092 and 0.0076. This result shows that the performance difference cannot be explained only by the difference in the knowledge cutoffs of the models. This control experiment does not separate the two effects (the model size and the recency of its knowledge) (Appendix L).
GLM 5.2 and knowledge of April 2026.
About the issue that the knowledge cutoff of GLM 5.2 is not officially shown: as written in the general response, we tested it ourselves with questions that check the knowledge of the target period (April 2026) and before. Specifically, we gave descriptions of major events and asked in which month each event happened, without a web search tool. As a result, for events in 2024, the accuracy was 50%, but from June 2025 to March 2026 it was 2/200, and for April 2026 it was 3/317 correct, and these three were also questions about the Hungarian election, whose date was decided in advance. From this, we think that GLM 5.2 does not have the knowledge of April 2026. For comparison, we also asked about events of October 2026, which no model can know, and GLM 5.2 was correct for 0/120.
The frontier agents are not separated.
As also written in point 1 of the general response, in the three instances including the two instances that we added, a statistically significant difference over multiple runs is found for 3 of the 6 possible pairs of the 4 frontier agents.
Reasoning effort.
As you point out, the performance did not increase after medium. We corrected the description in the paper. Thank you for pointing this out. The increase in performance from none to medium was consistently found also in the two added instances, so we described it in this way.
Response to Reviewer dzsU (3/3): fixed news, mining, and corrections
The fixed-news comparison.
For the two points that you raised, we ran additional experiments (Appendix C.2).
- Search queries: as you pointed out, we removed the search queries and ran the experiment again. The results do not change from the trend of the previous results. AP@50 is 0.0008 to 0.0026 with none-derived sets and 0.0060 to 0.0115 with xhigh-derived sets.
- Amount of news: in addition to the point that you raised, we made one more fix and ran an additional experiment. Specifically, the full text of the xhigh-derived sets included the body text of some articles in addition to the summaries of all articles, while the full text of the none-derived sets included only summaries. We fixed this so that both of them include only summaries, and we also matched the number of tokens of both, and we ran the experiment again. As a result, at high generation effort, the two kinds of set are then within one standard error (0.0026 vs. 0.0040 with 6k to 16k tokens, and 0.0055 vs. 0.0072 with 37k tokens). So we think that these differences came mostly from the amount of articles.
- Generation effort: with the same news sets, even when the sets include only summaries, high reasoning effort of the writer gave higher performance than no reasoning in all five sets. However, the difference was smaller than when the sets included the full text of articles. We think that this shows that the benchmark can also measure differences in the ability of agents in the part of formulating questions.
The size of the pool and when mining stops.
In the October instances, we stopped the mining of gold questions at 60 passes. To check the effect of this, we also stopped the mining at 60 passes in the April 2026 instance, and computed the benchmark scores again. As a result, we found that almost all gold questions obtained in the later part of mining do not match the forecasting questions that the agents generate. By stopping at 60 passes, the gold question set shrank from 329 questions to 260 questions, but on the other hand, the number of matches with the forecasting questions hardly changed, from 228 to 225. Also, the AP@100 scores of all agents rose almost equally by this, and their relative ranking did not change (Appendix O). We think that this shows that the benchmark is robust to where we stop the mining passes.
Corrections to statements that you cited.
- Mining curves (Appendix C.6): the text said that the curves had not saturated, but the figure cannot show this. We replaced the text and added a table that counts the new issues from each run. More runs would still add questions, so the pool is not complete.
- Second judges (Appendix E.2): the disagreement is symmetric only on the audit sample. With the strictest rule, coverage falls by 5% to 19%, depending on the agent.
- Cross-miner check (Appendix F.2): on the 189 recovered questions, the scores rise mainly because the denominator is smaller. The gain factor is 1.42 to 1.63 for the main agents and 1.10 to 1.25 for the DeepSeek agents.
- Tool use (Appendix C.4): the figure showed too few returned articles for GPT-5.5. The tool-call counts that you cited are not affected.
Response to Reviewer dzsU (1/3): questions 1 to 5
Thank you for the detailed and careful review. In this comment, we answer your questions 1 to 5. In the next comment, we answer question 6 and the other points.
Q1. Human rating of gold questions.
As you point out, we did not conduct human validation. We state this as the main limitation in Section 5. On the other hand, as written in the general response and in our reply to Reviewer ZAUr, we measured how robust the results are when we change the models for gold question mining and for the judge, and the results of the additional experiments support this.
About whether a gold question is "naturally askable": we use this phrase not to mean that there was enough information to ask the question in advance, but to mean that the granularity of the question was appropriate. As written in our reply to Reviewer ZAUr, it is not a necessary condition of this benchmark that every gold question could be asked in advance. This is because we cannot judge reliably whether a question could be asked only with hindsight, and a question that looks impossible to ask in advance might have been possible to ask by combining weak signals. To avoid removing such questions, we use all gold questions. So it is quite possible that even a very good agent cannot get a full score on this benchmark. On the other hand, a question that was truly impossible to ask in advance lowers the score of every agent by the same factor. So the comparison between agents does not change, and we think that it does not affect the validity of this benchmark (Section 3.1).
About the hinted-recovery result, we agree with your reading. The 18.1% is the result of one agent setting. It is not an upper bound of what can be asked, and it does not validate the gold pool. We changed Section 4.4 in this way.
Q2&3. Human-labelled pairs for the judge and gold pool
As you point out, we did not conduct human validation. In the current version of the paper, we write this as a limitation. On the other hand, as written in our reply to Reviewer ZAUr, we added ablation results of mining and the judge with a different model. We think that the results show that the main results of the paper are robust to changing the judge model and the mining model (Section 5 and Appendix M).
Q4. Clusters with only one submission.
As you point out, when we built the gold pool, we reported the results using only clusters with two or more elements. We added an additional experiment that checks the effect of this choice of clusters on the results of this benchmark. Specifically, we randomly sampled from the clusters with only one element, and we added them as gold questions. As a result, the main results of the paper did not change, and the ranking of the agent scores did not change much either (Appendix N). So the absolute score values change with this choice of clusters, but in the central part of the benchmark, which is distinguishing the performance between agents, there was almost no change.
Q5. Results from the live deployment.
We do not have results from a live deployment. As written in our reply to Reviewer UEHG, in this paper we ran the experiments with backtesting, in order to do comparative analyses under settings where we change conditions in detail. On the other hand, this benchmark supports forward testing directly with the same protocol. After the acceptance of the paper, we plan to release the forward-testing platform, including a real-time public leaderboard.
About the stability of the ranking between agents, we think that it is further supported by the two instances that we newly added (different regions and a different period). In the three instances, including the first instance, a part of the ranking between agents and the effect of changing the reasoning effort are reproduced. On the other hand, we think that this benchmark may also depend on how recent the knowledge of a model is, so the ranking of the agents may change in comparisons between instances of different periods.