- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1(−1)
Read review
Paper summary
This paper introduces EpiGraph, a meta-graph execution policy for agentic scientific reasoning, designed to address the lack of coherent management of task requirements, execution prerequisites, intermediate actions, and supporting evidence in multi-step scientific tasks. The method first represents a task as a contract specifying goals, execution boundaries, and evidence requirements, and then constructs an execution graph consisting of prerequisites, actions, expected observations, and persisted evidence receipts. Experiments span multiple task domains and demonstrate performance improvements. Overall, the paper’s main contribution is to treat execution validity and evidence completeness as distinct objects of prompt engineering and to propose a reusable approach to organizing execution across tasks and host systems, maintaining inspectable, traceable, and revisable links among the premises, computational results, and final conclusions of scientific analyses.
Critical strengths
-
EpiGraph provides a practically useful way to organize workflows by explicitly distinguishing execution prerequisites, expected outputs, and acceptance evidence. Through Guards, Predicts, and Verifies, it connects “what conditions permit an action to proceed”, “what results the action is expected to produce”, and “what evidence is sufficient to accept those results”. This allows support relationships and outstanding obligations to be explicitly managed throughout execution.
-
The same execution policy is evaluated across multiple tasks and backbone models, with results showing broadly positive evidence of improvement over the baselines.
Critical weaknesses
It remains necessary to establish whether explicitly typed dependencies offer additional benefits over other well-designed execution policies. The authors provide a Checklist baseline aligned with the task requirements, along with edge and operator ablations. These are valuable, but they do not fully resolve the attribution issue. The Checklist is an author-constructed baseline inspired by plan-and-execute and reason–act approaches, and may not be sufficiently strong or representative of these approaches in realistic practice.
Questions for authors
Clarification request:
Could the authors clarify why Claude / CSSwitch was not included in the BiomniBench-DA comparison in Figures 4 and 11? Its results are available in the same BiomniBench-AI4S repository from which the paper imports results for EvoScientist, the two OpenScience systems, and Wisp Science.
Suggestions:
In Figure 4, Codex + EpiGraph uses GPT 5.5, whereas EvoScientist and several other specialized scientific-agent baselines use DeepSeek V4 Pro, as specified in Appendix E.2. While the within-Codex comparison provides evidence of EpiGraph’s incremental benefit, the cross-system comparison leaves it unclear whether its advantage over these strong baselines persists when the backbone is held constant. Could the authors add at least one same-backbone comparison on BiomniBench-DA, for example by evaluating EvoScientist with GPT 5.5 against Codex + EpiGraph, or by evaluating a DeepSeek V4 Pro-compatible host both with and without EpiGraph alongside EvoScientist using the same model?
Thank you for the additional experiments. They raise several further questions:
1. Discrepancy between the original and repeated-run results
Appendix I states that the three runs include the original main-text run. For Sonnet 5 + EpiGraph on GPQA-Diamond, Table 1 reports 191/198 correct (96.46%), while Table 13 implies 192 questions correct at least once and 159 correct at least twice.
Thus, 33 questions were correct exactly once, and the total correct answers across three runs cannot exceed
33 + 3 × 159 = 510. Consequently:These numbers are not mathematically contradictory, but under the stated protocol they imply substantial run-to-run variability in this EpiGraph configuration. Please explain the gap from the original 96.46% and confirm whether the original run was included under identical settings.
2. Report individual and average Pass@1
As an extension question from question 1, please report each run’s Pass@1, mean Pass@1, and run-to-run standard deviation for both baseline and +EpiGraph, together with per-task correctness records identifying the original run. Pass@3, Cons@3, and task-bootstrap standard errors do not directly establish the reproducibility of the original single-run accuracy.
3. Align with the published DeepSeek configuration
Why not evaluate Claude Code / CSSwitch with and without EpiGraph using DeepSeek V4 Pro as in the released BiomniBench-AI4S comparison? The new Sonnet 5 experiment does not address the omitted published configuration. I acknowledge the new GPT 5.5 same-backbone comparison; this question concerns alignment with the original public baseline.
4. Can the authur provide minimal reproducible code and records?
Could the authors provide an anonymous reproduction package containing the exact prompts, baseline and ablation implementations, host/model configurations, execution and evaluation scripts, and raw per-task outputs? Appendix H is useful, but the policy description alone does not allow independent reconstruction of the reported experiments.
5. Clarify the scope of the claim
Is the intended claim that EpiGraph provides reliable plug-in improvements across both general-purpose and specialized agents? Which paired experiments establish this, particularly for specialized agents?
The revised manuscript still emphasizes a ranking reversal, but the new same-backbone comparison gives Codex + EpiGraph 68.29 versus EvoScientist 70.62. Meanwhile, adding EpiGraph to EvoScientist yields +1.73, with a 95% CI of [−0.71, +4.14], leaving that incremental benefit uncertain.
Please distinguish outperforming specialized systems from potential complementarity with them, and align the manuscript’s claims with the evidence.
Thank you again for your clarification and experiments.