acceptodds
Under review as a conference paper at ICLR 2027

Do LLMs See the Gorilla in the Data? A Benchmark for Off-Objective Scientific Discovery

Abstract

Scientific discovery depends on achieving stated objectives and also on noticing what those objectives leave unstated. We call this ability off-objective discovery, the recognition, while pursuing a given research goal, of an important finding that the same materials support but the goal never mentions. Such discoveries are largely attributed to human serendipity, and existing benchmarks evaluate only whether agents complete the objective they were given. We introduce UncoverBench, a benchmark that measures this ability against verifiable answers. Each of its 193 tasks is built from a recent peer-reviewed paper. A model receives the paper’s tables as the working data of an ongoing study, together with a research objective, and is asked what should be done next. One of the paper’s central findings is never mentioned, yet the tables support it. Every task also carries decoy conclusions that the tables contradict, which catch reports that assert patterns without checking them. Across twelve frontier models, reports complete the stated objective about three times in four but state the unstated finding in only 27.9% of cases. The misses are failures of recognition. In a sample labelled by hand, about two thirds of reports interpret the evidence for the finding, yet fewer than one in four state it, and models that fetch the right tables on their own still miss the finding in about half of the tasks or more. A generic reminder to look beyond the objective raises discovery to 43.5% and leaves decoy endorsement unchanged. Off-objective discovery thus remains a distinct capability that current frontier models largely lack, whether they read all materials at once or fetch the evidence with tools. UncoverBench provides a verifiable foundation for measuring and improving this capability, moving research agents from completing the objectives they are given toward noticing the findings those objectives leave unstated.

Primary area:
datasets and benchmarks
Keywords:
off-objective discovery, scientific discovery, large language models, serendipity, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3p8DOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose UncoverBench, a benchmark with 193 tasks built on the recent peer-reviewed papers. Each task gives a model tables and a research objective while leaving one central claim. It also includes false decoy conclusions to check whether the model indeed verifies the patterns. The authors then evaluate 12 frontier models on the dataset and find that the models complete the assigned objective roughly 73.6% of the time, but identify the unstated finding in only 27.9% of cases

Critical strengths
  1. The proposed study is an interesting alternative for evaluating the models for the scientific task.
  2. The paper is clearly written, with buzzwords used in some places.
Critical weaknesses
  1. Limited Novelty: While the idea is interesting, the paper is highly incremental. Similar ideas of extracting information from the paper into tuples are widely used in the literature (see, e.g., arXiv:2608.17501, arXiv:2603.03756). While I understand that the components of the tuple are different, it is still quite incremental for venues such as ICLR.
  2. Limited Data: The authors provide 193 tasks from recent peer-reviewed venues, which reduces the scope significantly. Additionally, there's no clarification on the choice of papers. An implicit assumption is that the authors select the papers at random, but clarification is required
  3. Limited Analysis: The authors use two LLM judges and a couple of human reviewers to judge the responses. One of the same judge models is used to generate the data, and both models are evaluated on the same benchmark as well. Additionally, the paper provides no information about the quality of the human reviewers. It also provides no statistical significance. Ideally, the judge model should not be included in the evaluation or data generation, and should be well calibrated and evaluated across multiple seeds, along with reporting of confidence intervals. Human evaluation should also consist of multiple reviewers with expertise in different fields, as there's some noise component coming from every reviewer.
  4. Wrong choice of models: In addition to frontier models, it is expected to evaluate agentic systems that are meant to solve these tasks
Questions for authors

See Weaknesses.

Additional Question

  1. Can you elaborate on the choice of models you use for every task? arXiv:2608.17501 shows that a simple Qwen 3.5 9B model can extract the information in the form of tuples. Similarly, arXiv:2603.03756 uses DeepSeek R1-distilled-Qwen-32b.
3 comments
AuthorsOpenReview ↗

Response to Reviewer (1/2)

Thank you for the comments. Our responses clarify the scientific contribution of UncoverBench, identify the statistical analysis and judge validation already present in the original submission, and report additional independent-judge and agentic evaluations.

Q1: Novelty and model choice

“Similar ideas of extracting information from the paper into tuples are widely used in the literature … While I understand that the components of the tuple are different, it is still quite incremental for venues such as ICLR.”

“Can you elaborate on the choice of models you use for every task?”

A1: Our central contribution is introducing off-objective discovery as an evaluation target for autonomous research agents; the data-construction pipeline operationalizes this target. Recognizing important findings beyond an assigned objective is a fundamental capability for autonomous scientific research, yet existing research-agent benchmarks leave this capability unmeasured. Current benchmarks evaluate whether an agent completes a specified research objective, rediscovers a targeted result, or produces insights when explicitly asked to find them. Human scientific discovery also follows another established mode: researchers pursuing one question notice an unexpected result and recognize its significance beyond that question. Studies of scientific serendipity and hypothesis-driven attention examine this process, including how a focused hypothesis can cause researchers to overlook important patterns in their own data (Yaqub, 2018; Yanai & Lercher, 2020). Autonomous research agents likewise need to recognize discoveries that their current objectives do not anticipate.

UncoverBench operationalizes this missing evaluation target by separating the assigned objective AA from an important, evidence-supported finding BB that the objective does not request. This separation allows us to test whether an agent recognizes findings beyond its instructions even when it successfully completes the assigned task. The resulting simple task structure provides verifiable ground truth and supports controlled comparisons across prompts, models, and agentic workflows.

The cited works address different research questions. SGHA uses structured literature evidence to identify research gaps and formulate research problems. MOOSE-Star studies inspiration retrieval and hypothesis composition to support scientific-discovery training and inference. Neither evaluates whether an agent pursuing objective AA independently recognizes an important finding BB outside that objective. In UncoverBench, (M,A,B,D)(M,A,B,D) specifies the experimental conditions needed to test this capability. The use of tuples in these works therefore does not establish that UncoverBench’s evaluation target, protocol, or findings are incremental (Sections 2–3, Table 1).

For model selection, we distinguish benchmark construction from capability evaluation. DeepSeek V4 Flash handles table and claim extraction, while GPT-5.6 Sol constructs objectives and decoys under the benchmark’s validity constraints (Section 3.2). The twelve evaluated models are tested on recognizing unstated findings from evidence; successful tuple extraction by a smaller model does not establish its ability to perform this task.

Q2: Dataset size and paper selection

“The authors provide 193 tasks from recent peer-reviewed venues, which reduces the scope significantly. Additionally, there’s no clarification on the choice of papers.”

A2: The 193 tasks support the main findings with uncertainty quantified at the task level. The original submission reports a 95% bootstrap interval of [24.0%, 31.9%] around the 27.9% discovery rate (Appendix B, Table 5). Each bootstrap sample resamples source-paper tasks while keeping all reports belonging to a task together, preserving the dependence among reports from the same task.

We sampled 260 empirical papers with quantitative tables, stratified by venue and research area. Programmatic table verification and automated and human validity checks yielded 193 tasks from distinct papers. The final benchmark contains 162 core ML tasks and 31 quantitative AI-for-Science tasks spanning biology, chemistry, physics, climate, and healthcare. The revised Section 3.2 and Appendix H document the sampling procedure, rejection criteria, and selection counts.

AuthorsOpenReview ↗

Response to Reviewer (2/2)

Q3: Statistical analysis, judge calibration, and annotator expertise

“It also provides no statistical significance.”

“One of the same judge models is used to generate the data, and both models are evaluated on the same benchmark as well.”

“Human evaluation should also consist of multiple reviewers with expertise in different fields.”

A3: In the original submission, we report task-level statistical uncertainty in Appendix B, Table 5, and blinded human validation of the scoring rule in Section 3.3 and Appendix B, Table 4. We summarize this existing evidence and the additional checks below.

Statistical analysis in the original submission. We report 95% confidence intervals from 4,000 task-level bootstrap resamples for every model under both prompt conditions and for their paired differences. The aggregate reminder effect is +15.6 percentage points, with a 95% interval of [12.7, 18.4], excluding zero. The revised main-text Table 2 also displays the intervals directly.

Blinded judge validation in the original submission. Two independent annotators labelled 200 reports from 50 tasks covering all twelve models and both conditions, without seeing model identity, prompt condition, or judge outputs. Judge–human agreement is κ=0.85\kappa=0.85 and 0.830.83, close to the κ=0.87\kappa=0.87 agreement between the human annotators. Precision is 0.92/0.90 and recall is 0.87/0.87. Agreement remains consistent across model families, with κ\kappa between 0.81 and 0.86.

Additional independent-judge evaluation. We added Qwen3.5-397B, which participates in neither benchmark construction nor the twelve-model evaluation. It agrees with the original dual-judge rule on 92.0% of the 200 human-validated reports and achieves κ=0.83/0.81\kappa=0.83/0.81 against the human annotators. On the 97-task sample, the discovery rate is 31.0% versus 31.2% under the original scoring rule, with closely matching model rankings (Appendix J, Table 11). The principal findings therefore persist under a judge independent of both construction and the evaluated model set.

Additional repeated-generation evaluation. Across three independent generation seeds on the 97-task subset, run-to-run standard deviations for the four evaluated models range from 1.1 to 1.8 percentage points (Appendix J, Table 11).

Annotator expertise. Across the human validation, task audits, and baseline studies, ten PhD-level researchers contributed expertise in representation learning and computer vision (3), NLP and LLM evaluation (3), optimization and reinforcement learning (2), and AI-for-Science in biology, chemistry, and clinical AI (2). Appendix J now documents this composition.

Q4: Research-agent systems

“In addition to frontier models, it is expected to evaluate agentic systems that are meant to solve these tasks.”

A4: In the original submission, we evaluated active tool-based evidence retrieval. The revision extends this evaluation to code-executing agents and dedicated agentic systems.

In the original retrieval experiment, seven models receive a table catalogue and choose which tables to fetch across 97 tasks (Section 4, Table 3). Discovery remains 33.9%–51.6% even when the model retrieves a table containing evidence for BB.

We additionally evaluated Claude Code, OpenHands, and four Python-agent backbones on the same 97-task subset (Appendix I, Table 10). The Python agents operate over local CSV workspaces with up to 15 turns of table retrieval and stateful Python execution. Without the reminder, discovery reaches 43.3% for Claude Code, 40.2% for OpenHands, and 35.3% on average for the Python agents, compared with 31.2% for the same four backbones under single-turn reading.

These systems load the relevant evidence in 78.4%–85.6% of tasks, yet still miss most off-objective findings. The recognition gap therefore persists in dedicated agentic systems with active evidence retrieval and code execution.

AuthorsOpenReview ↗

Following up on our rebuttal

Dear Reviewer 3p8D,

We sincerely appreciate your valuable time devoted to reviewing our manuscript. We would like to gently remind you of the approaching deadline for the discussion phase. We have carefully addressed the concerns you raised in your review, clarifying our problem formulation and paper-selection protocol, highlighting the task-level statistical intervals and human validation, and adding evaluations with an independent open-weight judge, multi-seed runs, and agentic coding systems. Would you kindly take a moment to look at our responses?

We would be very grateful for your thoughts and are happy to discuss any remaining questions.

Reviewer_ijqrOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies off-objective discovery and introduces UncoverBench. The benchmark is evaluated on 12 frontier models across 4,632 reports. The authors characterize the gap as a failure of recognition and position UncoverBench as measuring a capability that objective-completion benchmarks do not capture.

Critical strengths

This is an interesting research question. The paper introduces off-objective discovery, evaluating whether LLMs can recognize important findings beyond their explicitly assigned research objectives. This addresses a meaningful gap in existing scientific discovery benchmarks.

Critical weaknesses

Mismatch between the benchmark and real scientific research agents. Although UncoverBench is motivated by autonomous scientific discovery, its main evaluation relies entirely on single-turn API calls. Real research agents conduct iterative exploration, retrieve literature, use tools, and validate hypotheses. The current setup primarily measures whether an LLM can recognize an unstated finding from provided tables. More importantly, the benchmark is constructed from publicly available papers. Although the authors remove paper identifiers and conduct a masking experiment to assess memorization, these controls do not address external information retrieval. Once agents are allowed to search the literature, they may identify the original papers and directly retrieve the hidden findings. The paper does not explain how to support realistic research workflows without turning the discovery task into an answer-retrieval task.

Potentially artificial task construction. Both the stated objective (A) and hidden finding (B) are selected through an LLM-based process explicitly designed to keep them separate. This may create artificial omissions, instead of reflecting how unexpected discoveries arise in real scientific research.

Questions for authors

Why is a single-turn, table-based evaluation sufficient to support conclusions about autonomous scientific discovery agents? How would the findings generalize to agents capable of iterative analysis, tool use, and experimental validation?

Have the authors independently evaluated whether B is scientifically important, directly supported by M, and genuinely outside A? Can the authors report quantitative human assessments of these properties?

How would UncoverBench prevent agents with internet access from identifying the source papers and retrieving the hidden findings? Have the authors tested whether the research briefs are sufficient to locate their original papers?

8 comments
AuthorsOpenReview ↗

Response to Reviewer (1/2)

We thank the reviewer for the constructive review.

Q1: Iterative research agents

“Why is a single-turn, table-based evaluation sufficient to support conclusions about autonomous scientific discovery agents? How would the findings generalize to agents capable of iterative analysis, tool use, and experimental validation?”

A1: In the original submission, we also evaluated models that actively retrieve evidence using tools (Section 4, Table 3). Across 97 tasks, seven models receive a table catalogue and call a tool to retrieve tables. They retrieve a table supporting BB in 61.9%–73.2% of tasks, yet discovery remains 33.9%–51.6% when that evidence has been fetched.

To test whether iterative computation helps agents make that connection, we evaluated multi-turn code-executing agents and agentic coding systems on the same 97 tasks (Section 5, Appendix I, Table 10). Four model backbones operate in a workspace with the tables stored as CSV files and up to 15 interactive turns of list_tables, read_table, and run_python (pandas, numpy, scipy), alongside Claude Code and OpenHands. Mean discovery rises from 31.2% (all tables provided) to 35.3% with the 15-turn Python agent, while Claude Code reaches 43.3% and OpenHands reaches 40.2%.

The recognition gap persists despite active analysis: the Python agents execute 5.8–6.4 tool calls per task and load the tables supporting BB in 78.4%–82.5% of tasks, performing iterative computational analysis of the study data for the assigned objective AA while still leaving the majority of unstated findings BB unreported (Section 5, Appendix I).

Q2: Task construction and independent validity assessment

“Both the stated objective (A) and hidden finding (B) are selected through an LLM-based process explicitly designed to keep them separate. This may create artificial omissions, instead of reflecting how unexpected discoveries arise in real scientific research.”

“Have the authors independently evaluated whether B is scientifically important, directly supported by M, and genuinely outside A? Can the authors report quantitative human assessments of these properties?”

A2: Separating AA from BB makes off-objective discovery measurable: a finding explicitly requested by the task would instead measure instruction following. Section 3.1 formalizes that MM supports BB while AA can be completed without stating BB. Section 3.2 grounds BB in a verbatim central claim of a peer-reviewed paper, verifies its numerical cells programmatically, uses a brief writer that never sees BB, and subjects all candidates to two-stage review (55/248 = 22.2% rejected prior to benchmark freeze).

The independent assessment checks whether this separation remains scientifically meaningful in the resulting tasks (Appendix H, Table 9). Two external PhD researchers, uninvolved in benchmark construction, independently audited 50 randomly sampled tasks:

Criterion (50-Task External Validity Audit)Evaluator 1Evaluator 2Both evaluators passRaw agreement / κ\kappa
BB is directly supported by the tables MM98.0% (49/50)98.0% (49/50)98.0%100.0% / κ=1.00\kappa=1.00
BB lies genuinely outside the stated objective AA96.0% (48/50)94.0% (47/50)94.0%98.0% / κ=0.79\kappa=0.79
BB is scientifically important (central claim)96.0% (48/50)98.0% (49/50)96.0%98.0% / κ=0.66\kappa=0.66
Brief (M,A)(M, A) reads as a realistic ongoing study94.0% (47/50)92.0% (46/50)92.0%98.0% / κ=0.85\kappa=0.85

The paired human baseline provides a behavioral check as well: domain researchers working from the same briefs achieve 87.5% objective completion and 60.4% discovery, versus 28.8% discovery across the twelve models on the same 24 tasks (p=0.0012p = 0.0012; Appendix H, Table 9). Thus, the task supports both pursuing AA and recognizing BB under the original instructions.

AuthorsOpenReview ↗

Response to Reviewer (2/2)

Q3: Internet retrieval of source papers

“How would UncoverBench prevent agents with internet access from identifying the source papers and retrieving the hidden findings? Have the authors tested whether the research briefs are sufficient to locate their original papers?”

A3: We use real, peer-reviewed papers because they provide genuine scientific findings whose significance has been recognized through peer review. An answer invented for the benchmark cannot supply this independent scientific grounding. Each target finding BB is a central claim supported by the paper's numerical tables. During evaluation, these tables are available to the agent as working data, while the source paper's statement of BB is held out as the answer key. Controlling access to that answer key allows us to evaluate whether the agent recognizes the finding in the supplied evidence.

Agents can actively investigate the data within this setting. Our added evaluations allow multi-turn Python analysis and run Claude Code and OpenHands over local CSV workspaces, with external network access disabled. Discovery reaches 43.3% for Claude Code and 40.2% for OpenHands on the 97-task sample. These experiments extend the evaluation to agents that choose and execute their own analyses, as detailed in our response to Q1 (Section 5, Appendix I, Table 10).

We also tested source identifiability directly. In a separate web-search audit of 30 randomly sampled tasks, an agent located the source paper in 23/30 cases (76.7%) using exact table numbers and study identifiers, and in 13/30 cases (43.3%) when verbatim numbers were excluded from queries. Thus, removing paper identifiers or exact numbers is insufficient to prevent source lookup. These findings motivate the explicit answer-access control in our evaluation protocol. The revised Section 5 and Appendix I report both the agent environment and the identification audit.

AuthorsOpenReview ↗

Follow-up Response to Reviewer ijqr (1/4): Modular Benchmark Scope

Thank you for confirming that the independent human audit and multi-turn agent evaluations address several of your original concerns. Your remaining concern is whether evaluating off-objective recognition in a controlled research setting meaningfully captures a capability relevant to autonomous scientific discovery. It does, both because established scientific-agent benchmarks routinely evaluate constituent research capabilities and because our empirical results directly establish the scientific significance and distinctiveness of the recognition gap.

Q1: Evaluating a targeted capability vs. reproducing the full research lifecycle

“However, scientific research agents typically engage in a broader process involving literature review, hypothesis generation, experimental design, and validation. The current benchmark does not capture these activities.”

A1: Evaluating targeted capabilities within scientific workflows is an established principle across the scientific-agent literature. ScienceAgentBench explicitly calls for rigorous assessment of individual tasks within scientific workflows before making broad claims about end-to-end scientific automation, evaluating 102 programming tasks drawn from published studies [1], and AstaBench decomposes scientific research assistance into separate stage-level tasks [2]. Widely used benchmarks likewise isolate specific stages (Section 2, Table 1), including data-driven discovery over supplied datasets (DiscoveryBench [3]), hypothesis generation (HypoBench [4]), and paper replication (PaperBench [5]). Even full-cycle rediscovery benchmarks such as FIRE-Bench [6] direct the agent toward a target research question, while combining errors across planning, execution, and result interpretation into a multi-stage outcome.

Isolating the recognition step is a deliberate methodological choice that enables controlled measurement of off-objective discovery. Holding the working study evidence MM fixed ensures that every evaluated agent has access to the same evidence supporting the peer-reviewed finding BB while pursuing AA. If agents instead designed and executed open-ended experiments, their resulting evidence could differ and might not support BB, making it difficult to distinguish failures of evidence generation from failures of recognition. Within an autonomous research workflow, recognizing an unexpected finding is a necessary step toward investigating that finding further. An agent cannot deliberately formulate follow-up hypotheses, design targeted validation experiments, or communicate the scientific significance of a finding that it has failed to notice. Evaluating this recognition step independently therefore identifies a bottleneck that successful completion of other research stages may conceal.

Reviewer_ta6eOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The author(s) proposes UncoverBench, a benchmark for what the authors call off-objective discovery. This means noticing an important finding that the materials support but that the stated research objective never mentions. There are 193 tasks, each built from a recent peer-reviewed paper (175 from ICML 2026). The model gets all of the paper's tables, presented as the working data of an ongoing study, plus an objective that would be a natural next step. It is asked what should be done next. One of the paper's central claims is left out of the brief, but the tables support it. Each task also has three decoys, which are plausible misreadings that the tables contradict. The construction is mostly done by LLMs. Twelve frontier models are evaluated. They complete the objective in 73.6% of reports but state the hidden objective in only 27.9%, and no model gets above half. Adding a generic reminder to look beyond the objective raises discovery to 43.5%, while decoy endorsement stays at 4.1%.

Critical strengths

These are the overall strengths of the work:

A. The overall task and research is novel and the approach is clear and well written.

B. Completing the objective and making the discovery are scored separately, the ground truth is fixed by the source paper, and the decoys guard against a model getting credit for listing many patterns without checking them. This is a well constructged methodology.

C. The diagnostic analysis is well presented through the work. The Stated/Interpreted/Cited/Untouched breakdown indicates that most misses are not cases of ignoring the evidence. The results clearly show the gap and problem of models to perform this task.

Critical weaknesses

Given the strengths, I still think there are weaknesses that needs to be addressed:

A. I am not convinced that missing ‘B’ is always a failure of recognition, as opposed to a mismatch with the authors' framing. Several things make me worry that the 27.9% partly measures agreement with how the source authors framed their result:

a. The prompt asks "what should be done next". A good next-steps report has no particular reason to restate a result the study has apparently already produced.

b. The scoring is strict. In Figure 1, MiMo singles out the near-perfect reconstruction for further study but is counted as a miss because it does not set that result against decoding. A report that mentions both halves of ‘B’ in different places also does not count.

c. Most tables support several findings. Only ‘B’ is credited, so a report that raises a different, equally valid and important finding scores zero.

The paper does not measure how often reports state other valid off-objective findings.

B. There is also no human baseline therefore we don't know how often expert researchers given the same brief and prompt would state ‘B’. Without that reference point it is hard to tell whether 27.9% is low. The gorilla analogy in the introduction suggests humans would miss it fairly often too.

C. The construction pipeline is LLM-heavy, and some of the models involved are also being evaluated. And the human review step is described in one sentence. We are not told how many reviewers there were, what the rejection or edit rate was, or what criteria were applied.

D. The benchmark is also almost entirely ML papers (175 of 193 from ICML 2026), so the framing as a benchmark for "scientific discovery" is broader than what is tested.

E. All evaluations are single-turn and single-sample, with no code execution, even though the conclusion argues for applicability to research agents. This cannot be translated until proven otherwise.

Questions for authors

Please see the weaknesses mentioned.

3 comments
AuthorsOpenReview ↗

Response to Reviewer (1/2)

Thank you for your positive assessment and thoughtful questions. We provide detailed responses below.

Q1: Prompt framing, scoring, and other valid discoveries

“I am not convinced that missing ‘B’ is always a failure of recognition, as opposed to a mismatch with the authors' framing.”

“The prompt asks "what should be done next". A good next-steps report has no particular reason to restate a result the study has apparently already produced.”

“A report that mentions both halves of ‘B’ in different places also does not count.”

“The paper does not measure how often reports state other valid off-objective findings.”

A1: The next-step request requires interpreting the working evidence; it does not disclose the finding we score. The finding BB is absent from the brief and appears only as a relationship in the tables. Section 3.3 credits any statement of that relationship, including a passing observation, a hedged statement, or a justification for a next step. In Figure 1, noting the single high reconstruction score on its own is distinct from recognizing the contrast between near-perfect reconstruction and slightly worse behavioral decoding—and it is that cross-metric relationship that constitutes the finding BB and informs what to investigate next.

Your questions identify three alternative explanations for missing that relationship. We tested each directly (Section 5, Appendix G, Table 8):

  • Prompt framing: On the 97-task stratified subset across four models, explicitly instructing models to summarize all key findings already established by the tables before proposing next steps raises discovery from 31.2% to 34.8%, compared with 54.9% under the generic off-objective reminder on the same tasks.
  • Scoring strictness: Crediting unlinked mentions of both halves of BB in the 200 human-labelled reports moves discovery from 23.5% to 26.5% without the reminder and from 39.0% to 43.0% with it.
  • Other valid discoveries: Two blinded researchers examined 120 missed-BB reports for any other substantive off-objective finding supported by the tables. They identified such findings in 14 reports (11.2% stratum-weighted among misses, κ=0.82\kappa = 0.82), giving an estimated combined discovery ceiling of 36.0% (versus 27.9% for BB alone).

These controls address different possible explanations, but each leaves the majority of reports without an off-objective discovery. The recognition gap therefore persists when the prompt explicitly requests established findings, when scoring accepts disconnected mentions, and when other valid discoveries receive credit.

Q2: Human researcher baseline

“There is also no human baseline therefore we don't know how often expert researchers given the same brief and prompt would state ‘B’. Without that reference point it is hard to tell whether 27.9% is low. The gorilla analogy in the introduction suggests humans would miss it fairly often too.”

A2: The original comparison shows that models often complete the assigned objective while overlooking an important finding in the same evidence. A human reference tests how often researchers recognize that finding under the same instructions. Thank you for this suggestion. We evaluated six blinded PhD-level ML researchers on 24 stratified tasks (four tasks per finding type, two independent researchers per task = 48 human reports). They received the exact same no-reminder briefs, tables, and closing request as the models, with a 35-minute limit per task and no internet access (Section 5, Appendix H, Table 9):

Group (Paired 24-Task Subset, No Reminder)Objective completionOff-objective discovery
Human researchers (48 reports)87.5%60.4%
Claude Opus 591.7%45.8%
GPT-5.6 Sol95.8%41.7%
All twelve models (288 reports)74.3%28.8%

Blinded human researchers discover BB more than twice as often as the twelve-model average on the exact same tasks (60.4%, 95% CI [45.8%, 72.9%], versus 28.8%; paired permutation p=0.0012p = 0.0012), providing a direct human reference under identical materials and instructions (Section 5, Appendix H).

AuthorsOpenReview ↗

Response to Reviewer (2/2)

Q3: Construction overlap and human review

“We are not told how many reviewers there were, what the rejection or edit rate was, or what criteria were applied.”

A3: The scientific finding is grounded in the source paper, and the brief writer never sees it. Section 3.2 requires BB to be a verbatim central claim of the peer-reviewed source paper supported by the printed tables, while the model that writes the public brief receives only the continuation objective AA and the tables without access to BB or the answer key.

All 248 candidate tasks underwent a two-stage review against five explicit criteria: (1) BB is verbatim-grounded in the source paper and verified in the table cells of MM; (2) AA can be completed without stating BB; (3) the public brief contains zero cues for BB; (4) the tables strictly contradict all three decoys DD; and (5) all metric and setting definitions are preserved. Of the 248 candidate tasks entering Step 5 review, 33 (13.3% of 248) were rejected at the automated checklist screen, and 22 of the remaining 215 candidates (8.9% of 248; 10.2% of 215) were rejected during manual inspection by four senior ML doctoral researchers (55/248 = 22.2% rejected in total), yielding the final 248−33−22=193248 - 33 - 22 = 193 tasks; among the 193 retained tasks, 18 (9.3%) received minor human wording edits to metric or setting definitions before freeze.

To check validity independently of this construction process, two external PhD researchers (uninvolved in task construction) audited 50 randomly sampled final tasks (Appendix H, Table 9): both evaluators confirmed 98.0% direct support for BB in MM (100.0% raw agreement, κ=1.00\kappa = 1.00), 94.0% separation from AA (98.0% agreement, κ=0.79\kappa = 0.79), 96.0% scientific importance (98.0% agreement, κ=0.66\kappa = 0.66), and 92.0% natural ongoing-study brief quality (98.0% agreement, κ=0.85\kappa = 0.85).

Q4: Scientific domain coverage

“The benchmark is also almost entirely ML papers (175 of 193 from ICML 2026), so the framing as a benchmark for "scientific discovery" is broader than what is tested.”

A4: Off-objective discovery is the ability to recognize an important empirical finding beyond the assigned objective. Recent peer-reviewed papers with dense quantitative tables allow this capability to be evaluated against verifiable findings whose importance was established by the original authors and peer reviewers.

The task subjects also extend beyond the publication venue’s name. Across the 193 tasks, 162 cover core machine learning and 31 cover quantitative AI-for-Science studies spanning computational biology, molecular chemistry, physics, climate, and clinical AI (Section 1, Section 3.4). Across the AI-for-Science tasks, discovery is 26.8% without the reminder and 42.2% with it, closely matching 28.1% and 43.8% on the core ML tasks. We have made this domain composition explicit in Section 1 and Section 3.4.

Q5: Multi-turn agents, code execution, and repeated samples

“All evaluations are single-turn and single-sample, with no code execution, even though the conclusion argues for applicability to research agents. This cannot be translated until proven otherwise.”

A5: The original submission includes active tool-based evidence retrieval (Section 4, Table 3). Seven models receive a table catalogue and call a tool to retrieve tables across 97 tasks: discovery is 21.6%–35.1% overall and 33.9%–51.6% even on tasks where the model fetches the table containing evidence for BB.

This experiment establishes that accessing the relevant evidence does not ensure discovery. The requested code-execution and repeated-run checks test whether iterative analysis changes that result. We added multi-turn Python agents, dedicated coding harnesses, and multi-seed runs on the same 97 tasks (Section 5, Appendices I–J, Tables 10–11).

The Python agents operate in a workspace storing all tables as CSV files with up to 15 turns of list_tables, read_table, and a stateful run_python interpreter (pandas, numpy, scipy). We also evaluated Claude Code and OpenHands directly on the same workspace directories:

Model / Backbone (97 Tasks, No Reminder)All tables providedTool retrieval15-turn Python agent
Claude Opus 539.2%35.1%41.2%
GPT-5.6 Sol37.1%24.7%39.2%
Gemini 3.8 Flash26.8%26.8%30.9%
DeepSeek V4.1 Flash21.6%27.8%29.9%
4-Model Mean Discovery31.2%28.6%35.3%

Claude Code (Claude Opus 5) reaches 43.3% discovery and OpenHands (GPT-5.6 Sol) reaches 40.2%. The Python agents make 5.8–6.4 tool calls per task and load the evidence tables supporting BB in 78.4%–82.5% of tasks, yet still miss BB in 58.8%–70.1% of tasks without the reminder. Across three independent generation seeds on the 97-task subset (Appendix J, Table 11), run-to-run standard deviations range from 1.1 to 1.8 percentage points.

AuthorsOpenReview ↗

Gentle reminder regarding the discussion phase

Dear Reviewer ta6e,

We sincerely appreciate your valuable time devoted to reviewing our manuscript and your positive initial assessment. With the discussion deadline approaching today, we would like to gently follow up on our rebuttal. In response to your feedback, we have added the prompt-framing and scoring controls, the paired PhD researcher baseline, the external task-validity audit, and multi-turn coding-agent evaluations. Would you kindly take a moment to look at our responses?

We are very enthusiastic about engaging in further discussions with you and addressing any remaining questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their feedback and for recognizing the importance of evaluating whether research models notice findings beyond their assigned objectives. Autonomous research agents need to recognize important findings that emerge while pursuing a scientific task, including findings beyond the assigned objective. Our central contribution is to make this capability measurable through UncoverBench and reveal where current llms fail to exercise it. Across twelve frontier models, objective completion reaches 73.6%, while off-objective discovery reaches 27.9% despite models frequently reading and interpreting the relevant evidence. In the revised manuscript (changes highlighted in blue), we complement the original tool-retrieval, judge-calibration, and task-level bootstrap analyses with prompt-framing and scoring controls (Appendix G, Table 8), a paired PhD researcher baseline and independent task-validity audit (Appendix H, Table 9), multi-turn Python and coding-agent evaluations (Appendix I, Table 10), and a disjoint open-weight judge and multi-seed study (Appendix J, Table 11).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.