- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies how scientific agents inspect computational outputs, separating check selection, execution, and interpretation. Using a controlled benchmark of simulated neoantigen-selection outputs, the authors find that models can execute specified checks but often fail to propose relevant ones. Tool-triggered retrieval and context distillation improve detection of withheld defects, while increasing false alarms on clean inputs. An independent deterministic verifier then recomputes claimed findings from the underlying files and substantially reduces these false alarms.
Critical strengths
The distinction between check selection, execution, and interpretation provides a useful way to diagnose agent failures. Reporting performance on clean inputs exposes the false alarms introduced by retrieval and distillation. The replication across two serving configurations is also useful.
Critical weaknesses
-
The motivation is insufficiently focused. The paper begins with scalable feedback for scientific self-improvement, but studies consistency checks on completed files. It does not clearly explain why this task requires the proposed architecture rather than conventional pipeline validation.
-
The narrative lacks a clear central contribution. The abstract and main text move between benchmark construction, retrieval, distillation, verification, and serving sensitivity. Their relationship is difficult to follow, leaving the paper closer to a collection of experiments than a coherent methodological argument.
-
The benchmark makes the central problem somewhat artificial. All 17 checks are predefined, and a short deterministic program solves the task when given the complete rules. Withholding selected rules demonstrates a knowledge-access gap, but provides limited evidence of open-ended scientific check selection.
-
The main failure mode is substantially driven by an underspecified rule. Most false alarms concern the purity check, whose statistic and tolerance are omitted from the teacher prompt. Simply supplying the full definition largely resolves the problem, weakening the case that the observed failures require the broader framework.
-
The validation is too narrow for the broader claims. The main experiments concern one simulated workflow, with only 42 clean test bundles. Coverage is self-reported, and the external experiments do not establish transfer of the complete framework to other scientific workflows.
-
The presentation needs substantial revision. The figures are sparse and visually underdeveloped, offering little help in understanding the architecture or experimental logic. Repeated explanations and numerous implementation details further obscure the main message.
Questions for authors
see Weaknesses
2 comments
Thank you for the response. I have read it carefully, but my concerns remain. The figures, the core idea, and the overall presentation are still far below the standard for acceptance, and the authors have not provided further empirical work to address the specific weaknesses I raised. The response explains and reframes rather than demonstrates. I am not persuaded, so I am keeping my rating and confidence unchanged.
Thank you for the review. You are right that the motivation was not focused and the central contribution was hard to find. That is our failure of presentation. The general response restates the paper in plain terms. Here we answer your points in order, in the same plain terms; the exact numbers and intervals are in the revised PDF (blue text).
W1. Why is conventional pipeline validation not enough?
Conventional validation is a fixed set of checks that a person writes in advance. It works well for problems someone has already thought of, and our verifier is exactly this kind of validation. The paper asks a different question: when an agent reviews a pipeline output, who decides which checks this output needs? For a new pipeline, or a new kind of error, nobody has written that list yet.
Our results show that both parts are needed:
So the model supplies the choice, and validation supplies the reliability. §1 now makes this argument ("Why validation alone is not enough").
W2, W6. The central contribution is unclear and the paper is hard to follow.
We agree, and thank you for saying it plainly. The revision is organised around one message: the hard part for an agent is choosing what to check, not running the check; when it checks more broadly it makes errors in interpreting the results; verifying its claims fixes those errors but cannot do the choosing for it. Concretely:
We will keep improving the writing.
W3. The benchmark makes check selection artificial.
Partly yes, on purpose. To measure whether an agent chooses the right checks, we need to know the right answer for every possible check, and only a controlled benchmark gives that. The revision states that we do not claim open-ended scientific check selection.
Your concern also has a stronger form: perhaps the model knew the checks and simply was not asked. We tested this with new prompts that name no hidden check and only say that the list is incomplete. You were partly right: such a prompt recovers most of the gap, so we now call it a gap in eliciting checks rather than missing knowledge. Training still adds about 10 points on top of the best prompt, and the difference is significant (§3, Table 1).
W4. The false alarms come almost entirely from one underspecified rule (purity).
Agreed, and this turned out to be central. The new prompts make the same mistake on the same rule, with no training involved. So it is a natural trap for any agent that checks broadly, not a side effect of our method.
One paragraph defining the rule removes these false alarms (§6). But we could write that paragraph only after we had found the rule by studying test failures. The verifier was fixed before that analysis, and it removed the same false alarms without anyone pointing to the rule. A real pipeline has many such rules, and nobody knows in advance which ones will cause trouble. That is the case for verification.
W5. The validation is narrow, and coverage is self-reported.
Thank you again for the careful review.