- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors propose UncoverBench, a benchmark with 193 tasks built on the recent peer-reviewed papers. Each task gives a model tables and a research objective while leaving one central claim. It also includes false decoy conclusions to check whether the model indeed verifies the patterns. The authors then evaluate 12 frontier models on the dataset and find that the models complete the assigned objective roughly 73.6% of the time, but identify the unstated finding in only 27.9% of cases
Critical strengths
- The proposed study is an interesting alternative for evaluating the models for the scientific task.
- The paper is clearly written, with buzzwords used in some places.
Critical weaknesses
- Limited Novelty: While the idea is interesting, the paper is highly incremental. Similar ideas of extracting information from the paper into tuples are widely used in the literature (see, e.g., arXiv:2608.17501, arXiv:2603.03756). While I understand that the components of the tuple are different, it is still quite incremental for venues such as ICLR.
- Limited Data: The authors provide 193 tasks from recent peer-reviewed venues, which reduces the scope significantly. Additionally, there's no clarification on the choice of papers. An implicit assumption is that the authors select the papers at random, but clarification is required
- Limited Analysis: The authors use two LLM judges and a couple of human reviewers to judge the responses. One of the same judge models is used to generate the data, and both models are evaluated on the same benchmark as well. Additionally, the paper provides no information about the quality of the human reviewers. It also provides no statistical significance. Ideally, the judge model should not be included in the evaluation or data generation, and should be well calibrated and evaluated across multiple seeds, along with reporting of confidence intervals. Human evaluation should also consist of multiple reviewers with expertise in different fields, as there's some noise component coming from every reviewer.
- Wrong choice of models: In addition to frontier models, it is expected to evaluate agentic systems that are meant to solve these tasks
Questions for authors
See Weaknesses.
Additional Question
- Can you elaborate on the choice of models you use for every task? arXiv:2608.17501 shows that a simple Qwen 3.5 9B model can extract the information in the form of tuples. Similarly, arXiv:2603.03756 uses DeepSeek R1-distilled-Qwen-32b.
3 comments
Response to Reviewer (2/2)
Q3: Statistical analysis, judge calibration, and annotator expertise
“It also provides no statistical significance.”
“One of the same judge models is used to generate the data, and both models are evaluated on the same benchmark as well.”
“Human evaluation should also consist of multiple reviewers with expertise in different fields.”
A3: In the original submission, we report task-level statistical uncertainty in Appendix B, Table 5, and blinded human validation of the scoring rule in Section 3.3 and Appendix B, Table 4. We summarize this existing evidence and the additional checks below.
Statistical analysis in the original submission. We report 95% confidence intervals from 4,000 task-level bootstrap resamples for every model under both prompt conditions and for their paired differences. The aggregate reminder effect is +15.6 percentage points, with a 95% interval of [12.7, 18.4], excluding zero. The revised main-text Table 2 also displays the intervals directly.
Blinded judge validation in the original submission. Two independent annotators labelled 200 reports from 50 tasks covering all twelve models and both conditions, without seeing model identity, prompt condition, or judge outputs. Judge–human agreement is and , close to the agreement between the human annotators. Precision is 0.92/0.90 and recall is 0.87/0.87. Agreement remains consistent across model families, with between 0.81 and 0.86.
Additional independent-judge evaluation. We added Qwen3.5-397B, which participates in neither benchmark construction nor the twelve-model evaluation. It agrees with the original dual-judge rule on 92.0% of the 200 human-validated reports and achieves against the human annotators. On the 97-task sample, the discovery rate is 31.0% versus 31.2% under the original scoring rule, with closely matching model rankings (Appendix J, Table 11). The principal findings therefore persist under a judge independent of both construction and the evaluated model set.
Additional repeated-generation evaluation. Across three independent generation seeds on the 97-task subset, run-to-run standard deviations for the four evaluated models range from 1.1 to 1.8 percentage points (Appendix J, Table 11).
Annotator expertise. Across the human validation, task audits, and baseline studies, ten PhD-level researchers contributed expertise in representation learning and computer vision (3), NLP and LLM evaluation (3), optimization and reinforcement learning (2), and AI-for-Science in biology, chemistry, and clinical AI (2). Appendix J now documents this composition.
Q4: Research-agent systems
“In addition to frontier models, it is expected to evaluate agentic systems that are meant to solve these tasks.”
A4: In the original submission, we evaluated active tool-based evidence retrieval. The revision extends this evaluation to code-executing agents and dedicated agentic systems.
In the original retrieval experiment, seven models receive a table catalogue and choose which tables to fetch across 97 tasks (Section 4, Table 3). Discovery remains 33.9%–51.6% even when the model retrieves a table containing evidence for .
We additionally evaluated Claude Code, OpenHands, and four Python-agent backbones on the same 97-task subset (Appendix I, Table 10). The Python agents operate over local CSV workspaces with up to 15 turns of table retrieval and stateful Python execution. Without the reminder, discovery reaches 43.3% for Claude Code, 40.2% for OpenHands, and 35.3% on average for the Python agents, compared with 31.2% for the same four backbones under single-turn reading.
These systems load the relevant evidence in 78.4%–85.6% of tasks, yet still miss most off-objective findings. The recognition gap therefore persists in dedicated agentic systems with active evidence retrieval and code execution.
Following up on our rebuttal
Dear Reviewer 3p8D,
We sincerely appreciate your valuable time devoted to reviewing our manuscript. We would like to gently remind you of the approaching deadline for the discussion phase. We have carefully addressed the concerns you raised in your review, clarifying our problem formulation and paper-selection protocol, highlighting the task-level statistical intervals and human validation, and adding evaluations with an independent open-weight judge, multi-seed runs, and agentic coding systems. Would you kindly take a moment to look at our responses?
We would be very grateful for your thoughts and are happy to discuss any remaining questions.
Response to Reviewer (1/2)
Thank you for the comments. Our responses clarify the scientific contribution of UncoverBench, identify the statistical analysis and judge validation already present in the original submission, and report additional independent-judge and agentic evaluations.
Q1: Novelty and model choice
A1: Our central contribution is introducing off-objective discovery as an evaluation target for autonomous research agents; the data-construction pipeline operationalizes this target. Recognizing important findings beyond an assigned objective is a fundamental capability for autonomous scientific research, yet existing research-agent benchmarks leave this capability unmeasured. Current benchmarks evaluate whether an agent completes a specified research objective, rediscovers a targeted result, or produces insights when explicitly asked to find them. Human scientific discovery also follows another established mode: researchers pursuing one question notice an unexpected result and recognize its significance beyond that question. Studies of scientific serendipity and hypothesis-driven attention examine this process, including how a focused hypothesis can cause researchers to overlook important patterns in their own data (Yaqub, 2018; Yanai & Lercher, 2020). Autonomous research agents likewise need to recognize discoveries that their current objectives do not anticipate.
UncoverBench operationalizes this missing evaluation target by separating the assigned objective A from an important, evidence-supported finding B that the objective does not request. This separation allows us to test whether an agent recognizes findings beyond its instructions even when it successfully completes the assigned task. The resulting simple task structure provides verifiable ground truth and supports controlled comparisons across prompts, models, and agentic workflows.
The cited works address different research questions. SGHA uses structured literature evidence to identify research gaps and formulate research problems. MOOSE-Star studies inspiration retrieval and hypothesis composition to support scientific-discovery training and inference. Neither evaluates whether an agent pursuing objective A independently recognizes an important finding B outside that objective. In UncoverBench, (M,A,B,D) specifies the experimental conditions needed to test this capability. The use of tuples in these works therefore does not establish that UncoverBench’s evaluation target, protocol, or findings are incremental (Sections 2–3, Table 1).
For model selection, we distinguish benchmark construction from capability evaluation. DeepSeek V4 Flash handles table and claim extraction, while GPT-5.6 Sol constructs objectives and decoys under the benchmark’s validity constraints (Section 3.2). The twelve evaluated models are tested on recognizing unstated findings from evidence; successful tuple extraction by a smaller model does not establish its ability to perform this task.
Q2: Dataset size and paper selection
A2: The 193 tasks support the main findings with uncertainty quantified at the task level. The original submission reports a 95% bootstrap interval of [24.0%, 31.9%] around the 27.9% discovery rate (Appendix B, Table 5). Each bootstrap sample resamples source-paper tasks while keeping all reports belonging to a task together, preserving the dependence among reports from the same task.
We sampled 260 empirical papers with quantitative tables, stratified by venue and research area. Programmatic table verification and automated and human validity checks yielded 193 tasks from distinct papers. The final benchmark contains 162 core ML tasks and 31 quantitative AI-for-Science tasks spanning biology, chemistry, physics, climate, and healthcare. The revised Section 3.2 and Appendix H document the sampling procedure, rejection criteria, and selection counts.