- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper reframes scientific-agent evaluation based on continual research states. Discovery graphs are constructed based on reconstructed biomedical studies, and the graphs define staged evidence boundaries and the rubric criteria used to score agent reports.
Critical strengths
None
Critical weaknesses
- Contamination is never tested, and the stated mitigation does not really mitigate it: The used LLMs are recent models that might already be trained on the source studies. Conditioning the criteria on the current evidence boundary only blocks credit for memorized conclusions, leaving the setup and analysis-design criteria to be exposed. The consistent ordering S > A > E could be consistent with memorization, and no post-cutoff split or publication-year correlation is reported to separate the two.
- Scoring reads only the report text: The judge evaluates only reports, and nothing else is evaluated, including cited code, tables, etc. This weakens the setup and analysis-design criteria, as the analysis and design might be very relevant to tables and code. Moreover, as a correctly described plan and a correctly executed analysis might look identical in text, the scoring must also inspect the execution itself.
Questions for authors
- Is there any evidence that memorization of the source studies does not contribute to the setup and analysis-design scores?
- Does the judge inspect the cited code, tables, or figures to verify that a described analysis was actually executed?
4 comments
Dear Reviewer usg5,
We have provided point-by-point responses to your comments in our rebuttal, along with additional clarifications where appropriate. We hope you will have an opportunity to review our responses during the rebuttal period. If you find that our responses have adequately addressed your concerns, we would appreciate your consideration of updating your evaluation accordingly.
Thanks,
Paper 24518 Authors
Thank you for the additional experiments.
Both of my concerns are addressed. I am raising my rating from 1 to 2. My remaining concern is the scope of evidence. In detail, the framework is only validated on biomedical task, and the expert validation of the LLM judge’s scoring is limited to a small number of studies.
Thanks for your acknowledgement and raising the score. We further clarify two points:
-
We chose biomedical studies because they offer natural and verifiable stage boundaries: follow-up evidence typically requires a new cohort, assay, or platform. Our 40 studies span 4 research areas, 3 organism types, and multiple data modalities, while the framework itself is domain-general. Our findings also have broader implications for open-ended AI research (Kirgis et al., 2026).
-
Binary, single-requirement criteria minimize subjectivity. Expert evaluation of 20 reports across all settings and 4 systems shows 96% agreement, with the judge being stricter in 80% of disagreements. Two judge backbones agree on >93% of decisions (κ ≈ 0.87), and our main findings rely on same-judge paired comparisons, further mitigating systematic bias. Human experts also reviewed the graphs and rubrics for 25% of the studies.
Hope this discussion has helped clarify and strengthen our contributions.
Response to usg5
We appreciate the reviewer’s critical points, and we respond point by point below.
W1 and Q1. Memorization and the Scientific setup and Analysis design criteria
We ran a no-data control. Each LLM backbone, without a harness, datasets or tools, received the same instructions as in the Initial Discovery, Continued Investigation and Full Trajectory tasks (research background, data description and research question) for the first 32 studies, and its reports were scored with the same S/A/E rubrics (Table 9 in the updated PDF).
Table R4. Rubric scores (%) of reports written from model knowledge alone, without data or tools (mean ± s.d. across papers).
Memory alone earns little Scientific setup or Analysis design credit. Averaged over backbones, the no-data reports score 12.5% on Scientific setup, against 60.6% (Initial Discovery) and 68.5% (Continued Investigation) for the full systems working with data. Analysis design scores 26.3%, against 51.7% and 65.1%. The residual Analysis design credit is expected: these criteria include standard design steps (e.g., quality control, or inference at the participant rather than the cell level) that a model can name from general knowledge of omics analysis without knowing the specific paper, and our collection excludes method-development papers.
The S > A > E ordering is expected from the task structure, not from memorization. The three categories form a chain: a valid analysis design requires the correct setup, and a supported empirical endpoint requires both. Pass rates therefore fall along the chain, and the node-level analysis in Figure 3 shows this directly. Memory does not produce this ordering: with data, every system scores Scientific setup highest, but without data, Scientific setup is the lowest category for all seven backbones.
Retrieval during the runs is also blocked: direct Internet access is disabled, author-generated results are removed from the data packages, and the reference databases are frozen at releases that predate the 2023–2025 source papers (Appendices B and I.1).
W2 and Q2. Code and artifacts inspection
In the original protocol, the judge scores the report text, but several features tie the reported analyses to what was executed. First, the shared reporting contract (Figure 9) requires every analysis to end with an “Evidence:” sentence that names the saved tables, figures or outputs and identifies the relevant value, column or row, and the task prompt instructs agents to describe only analyses they actually performed. Second, a report written without running anything earns little credit: in our no-data control, Scientific setup and Analysis design scores fall to 12.5% and 26.3%, against 60.6% and 51.7% for systems working with data in Initial Discovery. Third, to check execution directly, we added a code-facing evaluation on ten Full Trajectory studies (Appendix F). A separate extractor reads each agent’s inputs, code and saved artifacts and records every analysis it implements, with its inputs, outputs, file paths and code locations; the same judge then scores the Scientific setup and Analysis design criteria against these records. Code-facing scores are higher than report-facing ones in every setting, by 7.3 points in Initial Discovery, 8.7 in Full Trajectory later stages (own state) and 9.1 in Continued Investigation (supplied prior state), so reports omit some implemented work rather than describe work that was not done. Code inspection also works in the other direction, exposing operations a report implies but the code does not contain. Most importantly, the gap between continuing from the supplied and from the agent’s own prior state is the same under both evidence sources (22.0 vs. 22.3 points, Table 14 in the updated PDF), so the own-state cost we report does not arise from how reports are scored.
Table R5. Mean of Scientific setup and Analysis design scores under code-facing and report-facing evaluation (ten Full Trajectory studies).