acceptodds
Under review as a conference paper at ICLR 2027

Can Scientific Agents Make Research State Transitions as Evidence Evolves?

Abstract

Scientific discovery unfolds through evolving research states: researchers reason from the available evidence context, identify what remains unresolved, and update conclusions as new evidence arrives. However, scientific agents are often evaluated on isolated tasks or with the full research context available at once, leaving unclear whether they can construct, advance, carry and even revise a state along a real research trajectory. To make these capabilities measurable, we collect recent biomedical studies as source-grounded discovery graphs that record what was known at each point, what evidence supported each claim, and which analyses followed. This structure enables us to derive evidence boundaries and research checkpoints naturally from each study’s own trajectory, rather than impose them manually. We instantiate three settings: Initial Discovery from the data available at the outset; Continued Investigation from an source-grounded checkpoint, with only subsequent evidence revealed; and Full-Trajectory, where agents continue from the state they constructed themselves. We evaluate seven frontier agentic systems with graph-grounded rubrics that separately assess scientific setup, analysis design, and empirical endpoint, while allowing equivalent alternatives to the authors’ exact implementations. Agents often identify relevant analyses but rarely carry them through to supported conclusions. Supplying a grounded research state substantially reduces stable cross-system differences, indicating that state construction is a major source of capability variation. Strikingly, on matched downstream tasks, performance drops from 61.1% with the source-grounded checkpoint to 35.2% when agents continue from their own state. The loss is concentrated in analyses missed at earlier stages, which agents rarely revisit and repair. Our work shifts evaluation from whether an agent can rediscover scientific findings to whether it can advance a scientifically valid research state as evidence evolves.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Scientific Discovery, Agentic AI, System Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_usg5OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper reframes scientific-agent evaluation based on continual research states. Discovery graphs are constructed based on reconstructed biomedical studies, and the graphs define staged evidence boundaries and the rubric criteria used to score agent reports.

Critical strengths

None

Critical weaknesses
  • Contamination is never tested, and the stated mitigation does not really mitigate it: The used LLMs are recent models that might already be trained on the source studies. Conditioning the criteria on the current evidence boundary only blocks credit for memorized conclusions, leaving the setup and analysis-design criteria to be exposed. The consistent ordering S > A > E could be consistent with memorization, and no post-cutoff split or publication-year correlation is reported to separate the two.
  • Scoring reads only the report text: The judge evaluates only reports, and nothing else is evaluated, including cited code, tables, etc. This weakens the setup and analysis-design criteria, as the analysis and design might be very relevant to tables and code. Moreover, as a correctly described plan and a correctly executed analysis might look identical in text, the scoring must also inspect the execution itself.
Questions for authors
  • Is there any evidence that memorization of the source studies does not contribute to the setup and analysis-design scores?
  • Does the judge inspect the cited code, tables, or figures to verify that a described analysis was actually executed?
4 comments
AuthorsOpenReview ↗

Response to usg5

We appreciate the reviewer’s critical points, and we respond point by point below.

W1 and Q1. Memorization and the Scientific setup and Analysis design criteria

We ran a no-data control. Each LLM backbone, without a harness, datasets or tools, received the same instructions as in the Initial Discovery, Continued Investigation and Full Trajectory tasks (research background, data description and research question) for the first 32 studies, and its reports were scored with the same S/A/E rubrics (Table 9 in the updated PDF).

Table R4. Rubric scores (%) of reports written from model knowledge alone, without data or tools (mean ± s.d. across papers).

LLM backboneSAEAll
Claude-Opus-517.51 ± 16.4034.23 ± 14.7539.59 ± 24.5529.79 ± 11.70
Claude-Sonnet-512.36 ± 14.1526.16 ± 15.7931.64 ± 27.3922.37 ± 13.38
GPT-5.47.76 ± 12.8922.68 ± 14.1131.43 ± 25.9720.04 ± 11.85
GPT-5.6-Sol11.74 ± 12.9922.81 ± 11.9834.17 ± 26.5622.14 ± 10.85
DeepSeek-V4-Flash14.40 ± 15.3327.93 ± 19.3724.92 ± 27.3921.70 ± 14.50
DeepSeek-V4-Pro12.38 ± 13.0524.38 ± 15.6126.04 ± 26.3420.08 ± 12.28
Kimi-K2.611.33 ± 13.0125.64 ± 15.2530.06 ± 26.1321.32 ± 13.62
All backbones12.50 ± 14.1226.26 ± 15.6431.12 ± 26.4022.49 ± 12.86

Memory alone earns little Scientific setup or Analysis design credit. Averaged over backbones, the no-data reports score 12.5% on Scientific setup, against 60.6% (Initial Discovery) and 68.5% (Continued Investigation) for the full systems working with data. Analysis design scores 26.3%, against 51.7% and 65.1%. The residual Analysis design credit is expected: these criteria include standard design steps (e.g., quality control, or inference at the participant rather than the cell level) that a model can name from general knowledge of omics analysis without knowing the specific paper, and our collection excludes method-development papers.

The S > A > E ordering is expected from the task structure, not from memorization. The three categories form a chain: a valid analysis design requires the correct setup, and a supported empirical endpoint requires both. Pass rates therefore fall along the chain, and the node-level analysis in Figure 3 shows this directly. Memory does not produce this ordering: with data, every system scores Scientific setup highest, but without data, Scientific setup is the lowest category for all seven backbones.

Retrieval during the runs is also blocked: direct Internet access is disabled, author-generated results are removed from the data packages, and the reference databases are frozen at releases that predate the 2023–2025 source papers (Appendices B and I.1).

W2 and Q2. Code and artifacts inspection

In the original protocol, the judge scores the report text, but several features tie the reported analyses to what was executed. First, the shared reporting contract (Figure 9) requires every analysis to end with an “Evidence:” sentence that names the saved tables, figures or outputs and identifies the relevant value, column or row, and the task prompt instructs agents to describe only analyses they actually performed. Second, a report written without running anything earns little credit: in our no-data control, Scientific setup and Analysis design scores fall to 12.5% and 26.3%, against 60.6% and 51.7% for systems working with data in Initial Discovery. Third, to check execution directly, we added a code-facing evaluation on ten Full Trajectory studies (Appendix F). A separate extractor reads each agent’s inputs, code and saved artifacts and records every analysis it implements, with its inputs, outputs, file paths and code locations; the same judge then scores the Scientific setup and Analysis design criteria against these records. Code-facing scores are higher than report-facing ones in every setting, by 7.3 points in Initial Discovery, 8.7 in Full Trajectory later stages (own state) and 9.1 in Continued Investigation (supplied prior state), so reports omit some implemented work rather than describe work that was not done. Code inspection also works in the other direction, exposing operations a report implies but the code does not contain. Most importantly, the gap between continuing from the supplied and from the agent’s own prior state is the same under both evidence sources (22.0 vs. 22.3 points, Table 14 in the updated PDF), so the own-state cost we report does not arise from how reports are scored.

Table R5. Mean of Scientific setup and Analysis design scores under code-facing and report-facing evaluation (ten Full Trajectory studies).

SettingCode-facing (structured records)Report-facing (original)
Initial Discovery59.5652.24
Later stages (own state)46.9438.23
Late stages (supplied state)69.2760.19

Dear Reviewer usg5,

We have provided point-by-point responses to your comments in our rebuttal, along with additional clarifications where appropriate. We hope you will have an opportunity to review our responses during the rebuttal period. If you find that our responses have adequately addressed your concerns, we would appreciate your consideration of updating your evaluation accordingly.

Thanks,
Paper 24518 Authors

Thank you for the additional experiments.

Both of my concerns are addressed. I am raising my rating from 1 to 2. My remaining concern is the scope of evidence. In detail, the framework is only validated on biomedical task, and the expert validation of the LLM judge’s scoring is limited to a small number of studies.

Thanks for your acknowledgement and raising the score. We further clarify two points:

  1. We chose biomedical studies because they offer natural and verifiable stage boundaries: follow-up evidence typically requires a new cohort, assay, or platform. Our 40 studies span 4 research areas, 3 organism types, and multiple data modalities, while the framework itself is domain-general. Our findings also have broader implications for open-ended AI research (Kirgis et al., 2026).

  2. Binary, single-requirement criteria minimize subjectivity. Expert evaluation of 20 reports across all settings and 4 systems shows 96% agreement, with the judge being stricter in 80% of disagreements. Two judge backbones agree on >93% of decisions (κ ≈ 0.87), and our main findings rely on same-judge paired comparisons, further mitigating systematic bias. Human experts also reviewed the graphs and rubrics for 25% of the studies.

Hope this discussion has helped clarify and strengthen our contributions.

Reviewer_HqTgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a benchmark for evaluating scientific agents' ability to construct, advance, and revise research states. The authors build discovery graphs from biomedical studies and introduce three settings: Initial Discovery, Continued Investigation, and Full Trajectory. Experiments across seven agentic systems show that agents often identify relevant analyses but struggle to complete them, with substantial performance degradation when continuing from their own research states.

Critical strengths
  1. The paper addresses an important problem in evaluating scientific agents' ability to conduct long-horizon research and revise previous findings.
  2. The discovery graph-based evaluation framework is well structured and provides a useful way to assess different stages of scientific research.
  3. The experiments cover multiple agentic systems and include informative analyses of error propagation, node completion, and evaluator agreement.
Critical weaknesses
  1. The performance drop in Full Trajectory does not necessarily indicate a lack of self-correction ability. Continued Investigation provides a calibrated research state, while Full Trajectory relies on potentially incomplete agent-generated states. Thus, the drop from 61.1% to 35.2% may primarily reflect differences in prior-state quality. An experiment explicitly prompting agents to revisit previous analyses would help distinguish these factors.
  2. The discovery graph-based evaluation may be overly dependent on the original published findings. Different but scientifically valid analyses may lead to different conclusions. How does the benchmark handle results that differ from the source paper but remain scientifically justified?
  3. The reliability of LLM-as-Judge remains a concern. Although the authors report high agreement across judges and with human experts, agreement does not necessarily imply correctness, especially for complex scientific reasoning and empirical findings. Further validation of the judge's ability to identify scientific errors and potential scoring biases would strengthen the evaluation.
Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to HqTg (Part 1)

We thank the reviewer for recognizing the importance of evaluating long-horizon research and revision, the structure of our evaluation framework, and the breadth of our analyses. We respond point by point below.

W1. Prior-state quality versus self-correction

We agree that prior-state quality accounts for much of the drop, and this is how the original manuscript explains it. In the updated manuscript, we expanded the Full Trajectory experiments to 18 studies and compared later-stage performance from the source-grounded checkpoint with that from the agent’s own earlier state (Section 4.4 and Table 3 in the updated PDF). Later-stage analyses with an earlier predecessor that received no credit fall from 54.2% with the supplied prior state to 15.8% from the agent’s own state (−38.4 points, 95% CI [−53.9, −20.5]), about twice the loss of analyses whose predecessors all received some credit (−17.5 points; Table 18 in the updated PDF). Neither conversation length nor unreported work explains the cost: resuming in a fresh session leaves performance unchanged (36.0 vs. 36.0), and re-scoring from agents’ code and artifacts leaves the gap unchanged (22.0 vs. 22.3 points).

Our analysis of revisiting addresses a separate question: whether agents update their earlier analyses and conclusions when new data become available.

Following the reviewer’s suggestion, we tested this directly with an explicit revision instruction on 12 follow-up tasks from 8 Full Trajectory studies.

In the original Full Trajectory setting, agents were instructed to “use your new findings to update and revise your prior report.” To test whether a more explicit instruction changes the behavior, we reran Full Trajectory on the 12 tasks from 8 studies with a revision hint added to the prompt: “As new evidence becomes available, consider whether any earlier analyses or conclusions need to be revisited. You may inspect and, when warranted, revise prior work using the available data and artifacts before completing the current investigation.” All other inputs, including the prior-stage data, code and artifacts, were identical. We call the original instruction the default prompt.

Table R2. Later-stage Full Trajectory scores under the default prompt and the revision hint (12 tasks from 8 studies) (Table 19 in Appendix H).

SystemDefault promptRevision hintChange
Claude-Opus-548.08%45.70%−2.38 pp
Claude-Sonnet-536.23%35.42%−0.81 pp
GPT-5.430.88%33.13%+2.25 pp
GPT-5.6-Sol33.50%39.61%+6.11 pp
DeepSeek-V4-Flash19.82%25.10%+5.28 pp
DeepSeek-V4-Pro24.38%25.30%+0.92 pp
Mean32.15%34.04%+1.89 pp

The revision hint changes later-stage scores only marginally (mean +1.89 points): scores rise for GPT-5.6-Sol and DeepSeek-V4-Flash and fall slightly for both Claude backbones.

AuthorsOpenReview ↗

Response to HqTg (Part 2)

To see whether the hint changes what agents do, we classified revisiting behavior into three categories: Review, inspecting prior work or rerunning its unmodified analysis code; Revise, adding a new analysis on earlier data or results; and Repair, modifying or correcting a prior analysis. A run counts as Revisited when any of these behaviors is observed.

Table R3. Number of follow-up tasks (out of 12 per system, 72 runs in total) showing each revisiting behavior, reported as default prompt / revision hint. (Table 20 in Appendix H)

SystemRevisitedReviewReviseRepair
Claude-Opus-512 / 1212 / 123 / 20 / 0
Claude-Sonnet-54 / 73 / 71 / 10 / 0
GPT-5.412 / 1212 / 121 / 11 / 0
GPT-5.6-Sol11 / 1011 / 101 / 10 / 0
DeepSeek-V4-Flash11 / 711 / 61 / 10 / 0
DeepSeek-V4-Pro11 / 811 / 80 / 00 / 0
Total61 / 5660 / 557 / 61 / 0

Under both prompts, almost every system reviewed its earlier work in most of its runs (Claude-Sonnet-5 is the exception), but revisions were rare (7 / 6 runs in total) and repairs almost absent: one repair under the default prompt (GPT-5.4) and none under the revision hint. The hint did not increase the number of runs with any revisit (61 vs. 56 of 72, Figure 5A and Table 20 in the updated PDF).

We also examined what the hint changed in the reports. We compared the two runs (default vs. hint) of each pair criterion by criterion: an earlier-stage criterion is considered carried forward if it passes in one run but not in the paired run; it is considered linked if a subsequent analysis built on it also passes a criterion only in the same run (Appendix H). Under the hint, agents carry forward more earlier-stage criteria (152 vs. 123), and more of them are linked (32.2% vs. 18.7%; Figure 5B), most clearly in the Analysis design category (40.0% vs. 19.6%; Figure 5C; Table 21). The hint thus helps agents carry forward not only earlier results but also earlier design choices, such as which cells or comparison units an analysis uses, that later analyses build on. The pattern is not driven by a few runs: 19 hint runs contain at least one linked criterion, against 10 under Default (Appendix H).

Together with Tables R2 and R3, this suggests that these modest gains came from Review, not Repair: agents reread their earlier work and carried relevant first-stage results into the later-stage report, but did not correct the earlier analyses. The explicit instruction thus helps agents bring forward what they already had, not fix what they missed, so the later-stage score barely moves (Table R2). The Full Trajectory loss therefore reflects a limited ability to repair one’s own state rather than an artifact of the instructions. The revision-hint experiment is in Section 4.5 and Appendix H of the updated PDF.

This mirrors failure modes recently reported for agents attempting open-ended AI research, which settled early on one approach, backtracked ineffectively when it was not working, and did not review their own work critically enough to change course (Kirgis et al., 2026 and Hutson, 2026).

AuthorsOpenReview ↗

Response to HqTg (Part 3)

W2. Dependence on the original publications

We agree that the same data can support more than one valid line of investigation, and a publication records one of them. Our design limits dependence on the source paper in three ways.

  • Criteria follow the evidence boundary, not the paper’s final conclusions. Each stage is scored only on what the evidence released at that stage can support.
  • Methods are not fixed. Analysis design accepts equivalent algorithms and implementations. Only Scientific setup is held fixed (the population, comparison and unit of inference), because changing these changes the scientific question rather than the route to answering it.
  • Endpoints are scored semantically. Empirical endpoint criteria assess a finding’s object, direction, relationship and evidential scope; exact numerical agreement is required only when a value defines the study design.

The published path also has a practical advantage as a reference: it has passed peer review and, in many studies, is supported by orthogonal or experimental validation at later stages. A genuinely different but valid conclusion can still earn Scientific setup and Analysis design credit, though not Empirical endpoint credit.

W3. Correctness and bias of the LLM judge

In our design, what counts as correct is fixed when the rubrics are built, not by the judge: criteria are derived from peer-reviewed publications through discovery graphs that pass automated verification and independent expert review. The judge’s task is narrower: to decide whether a report meets each binary criterion, such as a finding’s object, direction and stated scope. We validated this step against a domain expert who scored the same reports against the same criteria: the two agree on 96% of criterion-level decisions, and in 80% of the disagreements the judge is stricter, failing a criterion the expert passes. There is also no sign of self-preference: re-scoring the reports with DeepSeek-V4-Flash gives nearly identical rankings to GPT-5.4 as judge (ρ = 0.976), and the DeepSeek-V4-Flash judge scores the DeepSeek-V4-Flash system 3.1 points lower than the GPT-5.4 judge does (Table 11 in the updated PDF).

Dear Reviewer HqTg,

We have provided point-by-point responses to your comments in our rebuttal, along with additional clarifications where appropriate. We hope you will have an opportunity to review our responses during the rebuttal period. If you find that our responses have adequately addressed your concerns, we would appreciate your consideration of updating your evaluation accordingly.

Thanks,
Paper 24518 Authors

Reviewer_8oZsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that scientific agents should be tested on how they build, extend, and revise a research state as evidence arrives in stages, not only on isolated tasks or with all evidence available at once. To do this, the authors convert 32 published biomedical studies into "discovery graphs" whose nodes are analyses and claims and whose edges record evidential dependencies. From these graphs they derive stage boundaries and binary rubrics for scientific setup, analysis design, and empirical endpoint. They define three settings: Initial Discovery, Continued Investigation from a source-grounded checkpoint, and Full Trajectory, in which an agent continues from its own earlier report. Across seven agent systems, agents touch most relevant analyses but complete few. A supplied checkpoint reduces differences between systems, and on matched later-stage tasks performance falls sharply with the agent's own state, because agents almost never return to fix first-stage analyses they missed.

Critical strengths
  1. New staged evaluation framework. Releasing evidence in stages taken from each study's own dependency structure, and freezing each report before the next release, lets the authors separate three things existing benchmarks combine: building a state, extending a given state, and carrying one's own state forward. The comparison of Continued Investigation with Full Trajectory on the same later-stage tasks and the same rubrics is a clean way to measure the cost of a self-built state.

  2. Reasonable diagnostic analyses beyond a single score. The discovery graph lets the authors trace where the Full Trajectory loss comes from: mostly from later analyses whose first-stage predecessors the agent never addressed. A comparison of intermediate and final reports gives direct evidence that agents rarely repair earlier failures.

Critical weaknesses
  1. The headline Full Trajectory result rests on a small and partly confounded sample. Only 10 studies support Full Trajectory, only 5 were completed by every system and the 95% CI is wide, and each system ran once per task. The prompts do not explicitly hint agents to re-examine earlier conclusions, "agents rarely revisit their state" may partly reflect the instructions.

  2. The evaluation's validity rests on thin evidence. Independent expert review of the graphs covers only 5 of 32 studies, and judge–human agreement comes from one expert scoring 3 studies for 2 systems.

Questions for authors
  1. What will happen if we hint agents that prior states may not be fully correct in Full Trajectory?
  2. Have you tested partial credit instead of binary pass/fail for rubric criteria? If not, how do you expect the results to change? Could the current binary rubric miss credit for nodes that are mostly right, for example a correct direction with an overstated scope?
4 comments
AuthorsOpenReview ↗

Response to 8oZs (Part 1)

We thank the reviewer for recognizing the staged evaluation framework and the diagnostic analyses it enables. We respond point by point below.

W1. Size of the Full Trajectory sample and single runs

We expanded the Full Trajectory analyses from 10 to 18 studies, 16 of which were completed by all systems (up from 5). Two factors limit the number of eligible studies. First, a study qualifies for Full Trajectory only if (1) each later stage builds on results from the earlier stage, and (2) no later stage depends on something only the authors had and never released, such as their own cell-type labels or processed tables. Second, all systems must be compared on the same inputs. We therefore excluded studies with datasets that require image understanding (e.g., spatial transcriptomics with histology images), because the evaluated backbones do not handle image inputs at a comparable level. Some runs also ended early because system-side content filters terminated execution on biological content (most often for Claude-Opus-5 and Kimi-K2.6).

This sample size is comparable to those of recent benchmarks for biomedical agents; for example, BioAgent Bench (Fa et al., ICML 2026) comprises 10 curated bioinformatics tasks, while HeurekaBench (Panigrahi et al., ICLR 2026) was built on 13 published biomedical studies. What matters for this setting is that each study supports a genuine multi-stage trajectory, which our selection criteria ensure.

Although the confidence interval is wide, the direction of the effect is consistent. On the 16 studies completed by all systems, the mean later-stage score falls from 57.3% with the source-grounded checkpoint to 40.2% from the agent’s own state (−17.1 points, 95% CI [−27.3, −7.8]; Table R1). Every system declines, and the interval excludes zero for six of the seven; the exception, Claude-Opus-5, has both the smallest loss and the strongest Initial Discovery performance. Across all 18 studies, the drop is −16.3 points (95% CI [−25.7, −7.7]). The largest loss is in analysis design (A: −21.8 vs. S: −14.4 and E: −15.0 points; Sec. 4.4).

Table R1. Later-stage scores (%) from the supplied and own prior states, with 95% study-cluster bootstrap CIs (also see Table 3 and Table 15 in the updated PDF).

SystemSuppliedOwnΔ95% CI for Δ
Claude-Opus-561.850.6−11.2[−24.7, 1.4]
Claude-Sonnet-563.447.2−16.1[−29.6, −3.9]
GPT-5.6-Sol58.343.3−15.0[−27.9, −3.3]
GPT-5.455.237.8−17.5[−25.5, −9.5]
DeepSeek-V4-Flash54.632.6−22.1[−32.3, −12.3]
DeepSeek-V4-Pro50.038.4−11.6[−22.0, −1.4]
Kimi-K2.657.631.6−26.0[−39.1, −14.0]
Pooled, 16 common studies57.340.2−17.1[−27.3, −7.8]
Pooled, all 18 studies57.641.3−16.3[−25.7, −7.7]
AuthorsOpenReview ↗

Response to 8oZs (Part 2)

Q1. Instructions to revisit earlier stages

In the original Full Trajectory setting, agents were instructed to “use your new findings to update and revise your prior report.” To test whether a more explicit instruction changes the behavior, we reran Full Trajectory on 8 studies (12 tasks in total since some studies have more than one later-stage) with a revision hint prompt: “As new evidence becomes available, consider whether any earlier analyses or conclusions need to be revisited. You may inspect and, when warranted, revise prior work using the available data and artifacts before completing the current investigation.” All other inputs, including the prior-stage data, code and artifacts, were identical. We call the original instruction the default prompt.

Table R2. Later-stage Full Trajectory scores under the default prompt and the revision hint (12 tasks from 8 studies) (Table 19 in Appendix H).

SystemDefault promptRevision hintChange
Claude-Opus-548.08%45.70%−2.38 pp
Claude-Sonnet-536.23%35.42%−0.81 pp
GPT-5.430.88%33.13%+2.25 pp
GPT-5.6-Sol33.50%39.61%+6.11 pp
DeepSeek-V4-Flash19.82%25.10%+5.28 pp
DeepSeek-V4-Pro24.38%25.30%+0.92 pp
Mean32.15%34.04%+1.89 pp

The revision hint changes later-stage scores only marginally (mean +1.89 points): scores rise for GPT-5.6-Sol and DeepSeek-V4-Flash and fall slightly for both Claude backbones.

To see whether the hint changes what agents do, we classified revisiting behavior into three categories: Review, inspecting prior work or rerunning its unmodified analysis code; Revise, adding a new analysis on earlier data or results; and Repair, modifying or correcting a prior analysis. A run counts as Revisited when any of these behaviors is observed.

Table R3. Number of follow-up tasks (out of 12 per system, 72 runs in total) showing each revisiting behavior, reported as default prompt / revision hint. (Table 20 in Appendix H)

SystemRevisitedReviewReviseRepair
Claude-Opus-512 / 1212 / 123 / 20 / 0
Claude-Sonnet-54 / 73 / 71 / 10 / 0
GPT-5.412 / 1212 / 121 / 11 / 0
GPT-5.6-Sol11 / 1011 / 101 / 10 / 0
DeepSeek-V4-Flash11 / 711 / 61 / 10 / 0
DeepSeek-V4-Pro11 / 811 / 80 / 00 / 0
Total61 / 5660 / 557 / 61 / 0

Under both prompts, almost every system reviewed its earlier work in most of its runs (Claude-Sonnet-5 is the exception), but revisions were rare (7 / 6 runs in total) and repairs almost absent: one repair under the default prompt (GPT-5.4) and none under the revision hint. The hint did not increase the number of runs with any revisit (61 vs. 56 of 72, Figure 5A and Table 20 in the updated PDF).

We also examined what the hint changed in the reports. We compared the two runs (default vs. hint) of each pair criterion by criterion: an earlier-stage criterion is considered carried forward if it passes in one run but not in the paired run; it is considered linked if a subsequent analysis built on it also passes a criterion only in the same run (Appendix H). Under the hint, agents carry forward more earlier-stage criteria (152 vs. 123), and more of them are linked (32.2% vs. 18.7%; Figure 5B), most clearly in the Analysis design category (40.0% vs. 19.6%; Figure 5C; Table 21). The hint thus helps agents carry forward not only earlier results but also earlier design choices, such as which cells or comparison units an analysis uses, that later analyses build on. The pattern is not driven by a few runs: 19 hint runs contain at least one linked criterion, against 10 under Default (Appendix H).

Together with Tables R2 and R3, this suggests that these modest gains came from Review, not Repair: agents reread their earlier work and carried relevant first-stage results into the later-stage report, but did not correct the earlier analyses. The explicit instruction thus helps agents bring forward what they already had, not fix what they missed, so the later-stage score barely moves (Table R2). The Full Trajectory loss therefore reflects a limited ability to repair one’s own state rather than an artifact of the instructions. The revision-hint experiment is in Section 4.5 and Appendix H of the updated PDF.

This mirrors failure modes recently reported for agents attempting open-ended AI research, which settled early on one approach, backtracked ineffectively when it was not working, and did not review their own work critically enough to change course (Kirgis et al., 2026 and Hutson, 2026).

AuthorsOpenReview ↗

Response to 8oZs (Part 3)

W2. Validity of the discovery graphs and the LLM judge

Discovery graphs. Independent expert review is an additional external audit, not the only check. Before use, every one of the 40 graphs passes automated verification of schema, identifiers, source anchors, output lineage and dependency closure (verifier accuracy in Table 8 in the updated PDF) and a review against its source materials. We expanded the independent expert review from 5 to 10 of 40 studies; agreement with the constructed nodes and edges remains about 98%.

Judge–human agreement. We would like to clarify how the expert-scored reports were selected. To compare the judge with the expert across all settings, we restricted the selection to studies that (1) support all three settings (Initial Discovery, Continued Investigation and Full Trajectory) and (2) were completed by both systems. In the original manuscript, only 5 studies met both conditions, and the expert scored 3 of them, with at least 4 reports per study, at the level of individual rubric criteria. The expert therefore covered most of the eligible pool rather than a small arbitrary sample. During the rebuttal, the expert additionally scored 3 Initial Discovery and 3 Continued Investigation reports from another two systems (DeepSeek-V4-Flash and Claude-Sonnet-5). Across all four systems, judge–human agreement is 96%.

The judge is also robust to its own backbone: re-scoring with DeepSeek-V4-Flash instead of GPT-5.4 agrees on over 93% of decisions (κ ≈ 0.87) and gives nearly identical system rankings (Spearman’s ρ = 0.976). Each criterion is decided by majority vote over three judge runs, with 95.6% within-judge agreement.

Q2. Binary criteria and partial credit

Binary criteria and partial credit operate at different levels. When converting a graph node into rubrics, we split it into several binary criteria, each checking a single, clearly bounded requirement and each assigned to Scientific setup (S), Analysis design (A) or Empirical endpoint (E). Partial credit comes from aggregation: a node’s score, like the task score, is the equal-weight mean of its S, A and E pass rates. For example, a node with two S criteria, two A criteria and one E criterion that passes both S criteria, one A criterion and the E criterion scores (1 + 0.5 + 1) / 3 = 0.83, not 0 or 1. We kept each criterion binary because binary checks are easier to grade consistently, whether by a human or an LLM judge.

This design can mitigate the reviewer’s concern on “correct direction but overstated scope”. E criteria separately specify a finding’s object, direction, relationship and evidential scope (Section 3.3). A finding with the correct direction but an overstated scope passes the direction criterion, fails the scope criterion, and receives partial credit. The coverage and completion metrics in Table 2 report this “mostly right” case explicitly: in Initial Discovery, 80.6% of analysis nodes are covered (at least one criterion passes) but only 14.3% are complete.

Dear Reviewer 8oZs,

We have provided point-by-point responses to your comments in our rebuttal, along with additional clarifications where appropriate. We hope you will have an opportunity to review our responses during the rebuttal period. If you find that our responses have adequately addressed your concerns, we would appreciate your consideration of updating your evaluation accordingly.

Thanks,
Paper 24518 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.