- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether the loop that most scientific agents now follow (retrieve, update some intermediate record, decide whether to stop) actually helps, and if so, which part helps. The authors split the loop into two knobs: what the recursion builds (the evidence state) and what controls it (the stopping judge). They need a task where the loop matters, so they define "consensus claim verification": given a single-sentence claim, decide its stance with respect to the open literature, not a single paired passage. To evaluate this, they build ProClaim-eval, 718 claims drawn from five curated databases. Most REFUTE labels come from curator decisions such as removed edges, rejected entries, refuted planets and disproved isomeric states.
Critical strengths
The benchmark is the real contribution, and I think it will outlast the method. Getting claim-level labels whose REFUTE side comes from experts is hard, and the authors found a sensible way to do it: track what curators removed or rejected between releases, and use archives that explicitly record refutations. It spans three fields, and the Figure 3 experiment makes a fair case that reading one reference abstract is not enough. The serialized release to keep it out of pretraining crawls is a nice touch. Even if every method claim in the paper had to be softened, I would still want this dataset to exist, and that carries a lot of weight in my assessment.
Critical weaknesses
- The comparisons don't isolate the evidence state, which is the variable the paper is about. The central claim is that keeping state outside the context window beats accumulating evidence in context. But Appendix D.1 says the full execution log is sent to the model at every step, and C.4 confirms ProClaim re-sends its accumulated history. Table 7 shows ProClaim's main agent consuming roughly 610K to 1.16M input tokens per claim, against 82K to 261K for ReAct. So ProClaim's context grows more than ReAct's, not less. The real difference is what sits in the context: printed state summaries and stance-labelled facts in one case, raw paper text in the other. That is a perfectly interesting result, but it is the "what it stores" half of Finding 2, not the "outside the context" half. As far as I can tell, the paper has no experiment that tests the second half on its own.
- The labels don't quite mean "consensus," and the ablation evidence is statistically fragile. I like the dataset, but I'm not convinced its labels measure consensus in the sense the title suggests. On top of that, the verdict definition in B.1 counts "a thorough search finds nothing" as REFUTE, while SUPPORT often requires two to five papers. That rewards a "can't find it, so it's false" policy, and it makes the task partly a test of retrieval recall. It also weakens the Figure 3 argument. For a REFUTE claim, the reference abstract is the very paper the database later decided was wrong, so of course reading it alone produces poor agreement.
Questions for authors
- Is the final verdict produced from the state file or from the accumulated log? Could you run a variant where the agent sees only the current state snapshot and not the history? This would directly test the "outside the context" claim.
- Which tools and access did the ReAct controls and the harnesses have? Specifically, did they get full-text APIs, S2 recommendations, gap search, the three-paper minimum, and identical label definitions? And which backbone did Codex use? §4 says every agentic system runs on Sonnet 4.6, which is unusual for Codex.
- Are the 40-claim results from the same run as Table 2, or from a rerun?
5 comments
Clarification on the consensus claim dataset
Comment eFDS-C2 (Clarification on the consensus claim dataset):
I like the dataset, but I'm not convinced its labels measure consensus in the sense the title suggests.
We appreciate the reviewer's thoughtful comments.
We want to clarify how each database represents domain consensus. For example, in the SIGNOR subset, our re-curation guideline explicitly instructed curators to examine multiple publications, and the final verdict was decided by majority voting among curators. Curators in the ConnectomeDB 2025 release examined more than one paper and documented a consensual justification per entry. The ExTRI corpus was read by at least two domain experts, with disagreements resolved to consensus. In NUBASE, evaluators weighed published measurements and flagged states that were later shown not to hold. The NASA Exoplanet Archive maintains curated dispositions of entries, which are periodically examined by domain experts from the peer-reviewed literature. We have explained how the ProClaim-eval subsets relate to domain consensus in Appendix B. 7.
Therefore, we consider the databases the best proxy for consensus these fields maintain, which others already rely on for knowledge-intensive work, for example, as prior knowledge for downstream bioinformatics analysis [1,2]. This is highlighted in Section 1.
On top of that, the verdict definition in B.1 counts "a thorough search finds nothing" as REFUTE, while SUPPORT often requires two to five papers. That rewards a "can't find it, so it's false" policy, and it makes the task partly a test of retrieval recall.
We agree that the "can't find it, so it's false" policy lets poor retrieval get correct REFUTE answers. However, this policy would be penalised by the SUPPORT class. A label states whether the claim is supported as consensus, and we design it to carry the burden of proof (Appendix. B.1). In Figure 3, the naive top-5 retrieval does not guarantee finding the right papers and is a proxy for this policy. It reaches 92% on gold REFUTE but only 19% on Gold SUPPORT. Therefore, a naive policy would not perform well on our dataset.
It also weakens the Figure 3 argument. For a REFUTE claim, the reference abstract is the very paper the database later decided was wrong, so of course reading it alone produces poor agreement.
We further clarify this in our explanation of Figure 3 in Section 2. Having the reference abstract gives high agreement (0.82) in the REFUTE class, which is expected because the abstract is wrong. The low agreement (0.43) actually comes from the SUPPORT class, where the abstract is supposed to be correct. Therefore, a single-reference-guided verification does not predict consensus, which is what Figure 3 aims to show.
Clarification on the shared tooling
Comment eFDS-C3 (Clarification on the shared tooling):
Which tools and access did the ReAct controls and the harnesses have? Specifically, did they get full-text APIs, S2 recommendations, gap search, the three-paper minimum, and identical label definitions? And which backbone did Codex use? §4 says every agentic system runs on Sonnet 4.6, which is unusual for Codex.
We thank the reviewer for these questions. We clarified tools and access per system in Appendix C.4.
Shared components. Every system searched PubMed and Semantic Scholar (S2 only on NUBASE / NASA), with web search off (to avoid database leakage), and received the same label definitions. Static retrieval only used Semantic Scholar. The three-paper minimum rule also applied to the LLM judges in Table 4.
ProClaim-specific components.
- Full text. Giving it to the agents is not viable in Table 2. For example, ReAct re-sends its whole conversation each step (its total input grows quadratically) with full-text-sized documents would raise input from 127K to 0.8–1.2M tokens per claim, and 22% of claims would exceed a 200K window. In ProClaim, a local extractor reads full text and only facts enter the main context.
- Gap-targeted retrieval and S2 recommendations are part of ProClaim's implementation, not the subject of study, and so the baselines did not get them. Our separate analysis suggests they do not affect ProClaim’s performance: runs with gap-targeted retrieval do not perform better than runs without; S2 recommendations only supply < 1% of final-state facts. Both have negligible performance effects.
Codex backbone runs on Sonnet 4.6. In Table 2, the intention is to let Claude Code, Codex and ProClaim share one backbone, with only the harness varying. Codex uses only OpenAI's API, so a small local bridge forwards its requests to Sonnet 4.6 (Vertex AI, via LiteLLM) and returns OpenAI-format replies.
Clarification and results on the evidence state
Comment eFDS-C1 (Clarification and results on the evidence state):
We thank the reviewer for raising these comments and questions. Since weakness 1 and question 1 are related, we explain them together.
Verdicts are produced from the log. ProClaim writes the verdict from its execution log. As a safeguard, the verdict is derived by the same LLM from state only when the agent fails to give a verdict from the log (e.g., when it is hard stopped by allocated budget and has to be reprompted).
Storing evidence "outside the context" gives higher agreement. We agree with the reviewer that “outside the context” half of our claim deserves to be tested. In Section 4.2, we ran the suggested experiment variant on the same 200 claims (same 40-claim samples from each dataset subset), where each call sees the current evidence state JSON and only the previous step, with no earlier history. The previous step is needed to avoid tool calling loops. We also removed full text from the JSON state to avoid overflowing the context.
Pooled over the 200 claims, ProClaim with the state in-context (0.69) has lower agreement than ProClaim (0.77, p = 0.049), with paired exact McNemar test. This shows that the evidence state should be kept outside the context.
Yes, Table 3 rows come from the Table 2 runs, restricted to the 40 claims randomly sampled from each domain subset. In Table 4, the trained-judge row comes from the Table 2 runs, and the other stopping rules are separate runs on the same 40 claims, because ablating the judge requires new ProClaim runs.
We clarified this point in the “setting” paragraph of Section 4.2.