acceptodds
Under review as a conference paper at ICLR 2027

EpiGraph: Meta-Graph Engineering for Agentic Scientific Reasoning

Abstract

Agentic scientific reasoning is attracting growing attention because it enables agent systems to support scientific decisions and discoveries. Despite rapid progress with increasingly capable large language models, task requirements, intermediate actions, expected observations, and supporting evidence are essential for building capable agent systems in this domain, yet they are often not maintained as a coherent execution structure. This makes execution validity a general and central challenge across scientific tasks, even when the underlying domain knowledge and tools differ. We name this engineering perspective Meta-Graph Engineering and introduce \ourmethod, a method that represents scientific execution as a structured graph maintained throughout the task. First, graph operators construct, expand, prune, admit, close, and revise the execution graph; second, typed edges determine whether actions are licensed, what they are expected to produce, and whether their saved results satisfy the corresponding obligations; third, this shared execution structure provides a reusable way to organize agentic scientific reasoning across tasks and systems. We evaluate \ourmethod across scientific, mathematical, and biomedical benchmarks with multiple agent configurations and find consistent outcome improvements. Notably, it surpasses biomedical agent systems designed by human experts on BiomniBench without supplying biological prior knowledge or task-specific biological interpretations during execution. These results suggest that \ourmethod provides a general execution layer for making agentic scientific reasoning more reliable, inspectable, and reusable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Systems, Large Language Models, Scientific Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_pbkVOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
1(−1)
Read review
Paper summary

This paper introduces EpiGraph, a meta-graph execution policy for agentic scientific reasoning, designed to address the lack of coherent management of task requirements, execution prerequisites, intermediate actions, and supporting evidence in multi-step scientific tasks. The method first represents a task as a contract specifying goals, execution boundaries, and evidence requirements, and then constructs an execution graph consisting of prerequisites, actions, expected observations, and persisted evidence receipts. Experiments span multiple task domains and demonstrate performance improvements. Overall, the paper’s main contribution is to treat execution validity and evidence completeness as distinct objects of prompt engineering and to propose a reusable approach to organizing execution across tasks and host systems, maintaining inspectable, traceable, and revisable links among the premises, computational results, and final conclusions of scientific analyses.

Critical strengths
  1. EpiGraph provides a practically useful way to organize workflows by explicitly distinguishing execution prerequisites, expected outputs, and acceptance evidence. Through Guards, Predicts, and Verifies, it connects “what conditions permit an action to proceed”, “what results the action is expected to produce”, and “what evidence is sufficient to accept those results”. This allows support relationships and outstanding obligations to be explicitly managed throughout execution.

  2. The same execution policy is evaluated across multiple tasks and backbone models, with results showing broadly positive evidence of improvement over the baselines.

Critical weaknesses

It remains necessary to establish whether explicitly typed dependencies offer additional benefits over other well-designed execution policies. The authors provide a Checklist baseline aligned with the task requirements, along with edge and operator ablations. These are valuable, but they do not fully resolve the attribution issue. The Checklist is an author-constructed baseline inspired by plan-and-execute and reason–act approaches, and may not be sufficiently strong or representative of these approaches in realistic practice.

Questions for authors

Clarification request:

Could the authors clarify why Claude / CSSwitch was not included in the BiomniBench-DA comparison in Figures 4 and 11? Its results are available in the same BiomniBench-AI4S repository from which the paper imports results for EvoScientist, the two OpenScience systems, and Wisp Science.

Suggestions:

In Figure 4, Codex + EpiGraph uses GPT 5.5, whereas EvoScientist and several other specialized scientific-agent baselines use DeepSeek V4 Pro, as specified in Appendix E.2. While the within-Codex comparison provides evidence of EpiGraph’s incremental benefit, the cross-system comparison leaves it unclear whether its advantage over these strong baselines persists when the backbone is held constant. Could the authors add at least one same-backbone comparison on BiomniBench-DA, for example by evaluating EvoScientist with GPT 5.5 against Codex + EpiGraph, or by evaluating a DeepSeek V4 Pro-compatible host both with and without EpiGraph alongside EvoScientist using the same model?

Reviewer_tGDAOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The article presents EpiGraph, an execution policy provided in the form of instructions to an existing agent. It presents scientific workflows as a graph in which actions are related to their prerequisites, expected results, and the evidence needed to accept those results. The method formalizes how these dependencies are created and updated, including which later steps need to be reviewed if earlier assumptions change. The agent stores this information in its transcript or task state. The assessment covers scientific questions, mathematical problems, coding tasks, and biomedical data analysis using multiple agent models and environments. The presented results are usually better than those of the same agents without EpiGraph. As for biomedical tasks, Gemini 3.1 Pro evaluates recorded work and final answers according to criteria compiled by experts. These results are also compared with published scores for specialized scientific agents. Further experiments on science and mathematics benchmarks compare the full EpiGraph policy with versions that omit individual dependency types or execution operators. In a comparison on the same benchmarks, the graph is replaced by a checklist containing similar requirements for tasks. Biomedical case studies illustrate the resulting workflows, and one case is replayed after changing the premise, removing evidence, or introducing a conflicting label to test how subsequent execution changes. Each benchmark task is performed once for each condition.

Critical strengths

The evaluation is broad and covers several models and agent environments across scientific reasoning, mathematics, coding, and biomedical data analysis. It includes comparisons with the same agents without EpiGraph, component ablations, a checklist baseline, and targeted interventions. Biomedical case studies provide specific examples of how the policy relates to analysis decisions and supporting evidence. This helps to assess how useful this approach is for various tasks and agent settings.

Critical weaknesses

Each task is run once per condition, and run-to-run variability is not reported. Since LLM agents can produce different outcomes on the same task, repeated runs are needed to assess whether the smaller gains are consistent.

Questions for authors
  1. How were the comparison systems chosen? The cited evaluation reports a higher Gemini score for OmicOS, which is discussed in the related work but not included in the comparison.
  2. Could you report run-to-run variability and confidence intervals for the performance differences in the main comparisons and ablations?
Reviewer_4AETOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The agent maintains a typed execution graph whose nodes are prerequisites, actions, expected observations and receipts, connected by Guards/Predicts/Verifies edges and maintained by six operators. An action is legal only when its support set is satisfied, an expectation is declared and a receipt obligation is specified. The method is a 4-page natural-language policy document injected into off-the-shelf harnesses. Results: 6 models x 6 QA/math benchmarks with 35/36 pairs improving, plus CMT-Bench and SWE-bench; a process-level comparison on 50 BiomniBench-DA tasks judged by Gemini 3.1 Pro where Codex+EpiGraph (67.48) beats Biomni (48.92) and EvoScientist (65.26); matched ablations on two models; and a three-variant replay intervention on one trace.

Critical strengths
  1. Breadth and consistency of outcome-level evidence: nine datasets, three harnesses, eight models, positive direction in 35/36 Table 1 pairs, all CMT-Bench overall scores and both SWE-bench pairs. Without variance estimates this does not pin magnitudes, but it makes a zero aggregate effect unlikely, and harness-agnostic transfer across Codex, Claude Code and Pi is practically useful.
  2. Process-level evaluation with inspectable artifacts. Using BiomniBench-DA's expert rubric rather than final-answer accuracy, then opening six traces to show the saved tables, assertions and receipts, is the right evidence for a thesis about execution validity. The RQ4 replay (premise mutation 100 to 66, receipt deletion 100 to 85) is a concrete test of the claimed dependency semantics.
Critical weaknesses
  1. The causal claim, that graph structure rather than a long careful-execution prompt drives the gains, is not established, and the control evidence is anomalous. The baseline is the raw harness with no policy prompt. The only control for "any careful prompt" is the matched-nograph checklist on two models and two datasets, and it behaves implausibly: GPT 5.6 Terra scores 85.35% on GPQA with the checklist against an 86.36% no-prompt baseline, and 68.00% vs 74.00% on BeyondAIME; Claude Sonnet 5 gets 55% vs a 63% baseline. The "w/o Admit" ablation is 7.6 points below having no policy at all. A checklist of the same obligations hurting relative to no prompt fits single-run variance better than the paper's reading. No other ablation numbers are tabulated. There is also no comparison against Plan-and-Solve, ReAct-style planning or Reflexion, which the paper itself cites as the alternatives, and no matched token budget. Harness choice alone moves CMT-Bench by 22 points for one model, larger than any EpiGraph delta in that table.
  2. No variance estimates, and most effects are within single-run noise. Each task is run once per condition. AIME and HMMT have n=30-33, so one task is 3.3 points; many deltas are one or two tasks. BiomniBench-DA has n=50 with an LLM judge, and the margin over EvoScientist is 2.22 points with no judge-variance or bootstrap. Wall-clock deltas are reported to two decimals though single-run API latency is dominated by server load.
  3. The headline biomedical claim is confounded and likely inflated by the judge. Plain Codex without EpiGraph already scores 59.16 against Biomni's 48.92, so beating Biomni is a property of the harness. The other comparison systems run on DeepSeek V4 Pro with scores taken from the released evaluation, not re-run. Gains concentrate in rubric dimensions that reward documented process (source reliability 82.69 to 93.27, data handling 59.34 to 70.88) while biological interpretation is flat and scientific reasoning decreases. The policy explicitly instructs the agent to write contracts, receipts and evidence records, and a judge reading the trajectory will reward that regardless of whether the science improved. No human evaluation or judge-blinding. Appendix E.4 reports 11 task-level losses to Biomni, one around 40 points, undiscussed in the main text. Finally, none of Eqs. 12-22 is implemented by a checker, so the claim that legality is checked by executable predicates rather than model judgment is not supported.

Minor: no code, baseline prompts or traces released; closed models throughout; whether web search was disabled for GPQA is not stated; a 96.46% GPQA score obtained faster than a 76.77% baseline needs explanation.

Questions for authors
  1. Please run 3-5 repetitions for GPQA-Diamond and BeyondAIME on two models and BiomniBench-DA on Codex, in baseline, checklist and +EpiGraph conditions, with paired bootstrap. Is the checklist-below-baseline pattern stable?
  2. What exactly is the baseline prompt, and were web search and other tools identical across conditions?
  3. Can you re-run one cross-system comparison with a shared base model, and report judge scores when Gemini sees only the final artifacts rather than the trajectory text?
  4. Which predicates in Eqs. 12-22 are evaluated by code at any point? If none, please reword the "executable predicates" claims.
  5. How does the policy produce +13 to +21 points on multiple-choice GPQA at equal or lower wall-clock time? A short qualitative trace analysis would help.
Reviewer_8BvAOpenReview ↗
Rating
2(+1)
Confidence
2(+1)
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I do not believe that a 64-page submission containing 9 tables and 33 figures (excluding some unlabeled tables and subfigures) can be reviewed thoroughly within a short review period. Nevertheless, I tried to understand the problem the paper wants to solve and the paper's main claims. While reading the submission, unfortunately, I found it extremely difficult to understand the paper even at a high level. The paper does not seem to clearly explain the very specific problem the authors aim to solve or explain how the proposed EpiGraph addresses that problem, relying instead on generic statements (e.g., statements in the abstract or introduction).

While I should acknowledge that some of the difficulty may stem from my own lack of expertise in this field, considering the overall quality of the writing and presentation, I do not believe the paper is sufficiently clear or well-prepared to be ready for peer review in its current form. For example:

  • In line 1803, the figure reference is missing and appears as “Figure ??”.
  • Appendix B.4 contains no content. I suspect that some tables may have been intended to appear there.
  • Figure 1 appears to be one of the paper's most important figures. However, it contains excessive meaningless text and inconsistent spacing, making it difficult to read and, in my view, not ready for publication.
  • Figures 22 and 26 are noticeably blurry and difficult to inspect.
  • Table 1 compares EpiGraph with a baseline, yet the baseline methodology is not described. For example, it is unclear how the baseline was implemented (e.g., the prompts used and the scaffold).
    • The results are based on a single run, which raises a question on whether the improvement is truly significant.
    • Related to Table 1, a more objective metric you would consider is the cost in USD. Because runtime is often affected by many factors (e.g., API or GPU health). In figure 9, EpiGraph used a shorter time with more tokens. Is it because it uses more system prompts?
  • The formal definitions in Appendix B are excessively complex and difficult to assess within a limited review period.

Given the level of complexity and the amount of content, I do not believe I can adequately evaluate the technical soundness of the proposed approach within the time available. I also do not think it would be appropriate to assess the paper solely on the basis of the broader goals it claims to be achieved without being able to properly examine its technical contributions.

With all sincerity, I fully appreciate the substantial time, financial resources, and effort the authors have invested in their experiments and ideas (I understand how difficult everything is). I hope other reviewers better recognize the value of this paper. However, I do not believe the manuscript, in its current form, is sufficiently clear or complete to allow for a proper and rigorous review.

Critical strengths

None

Critical weaknesses

None

Questions for authors

I do not have any questions.

Comments on the paper

1 comment

Thank you for the additional experiments. They raise several further questions:


1. Discrepancy between the original and repeated-run results

Appendix I states that the three runs include the original main-text run. For Sonnet 5 + EpiGraph on GPQA-Diamond, Table 1 reports 191/198 correct (96.46%), while Table 13 implies 192 questions correct at least once and 159 correct at least twice.

Thus, 33 questions were correct exactly once, and the total correct answers across three runs cannot exceed 33 + 3 × 159 = 510. Consequently:

  • Mean Pass@1 across all three runs ≤ 85.86%.
  • Mean Pass@1 of the two additional runs ≤ 80.56%, assuming the original 191-correct run is included.

These numbers are not mathematically contradictory, but under the stated protocol they imply substantial run-to-run variability in this EpiGraph configuration. Please explain the gap from the original 96.46% and confirm whether the original run was included under identical settings.


2. Report individual and average Pass@1

As an extension question from question 1, please report each run’s Pass@1, mean Pass@1, and run-to-run standard deviation for both baseline and +EpiGraph, together with per-task correctness records identifying the original run. Pass@3, Cons@3, and task-bootstrap standard errors do not directly establish the reproducibility of the original single-run accuracy.


3. Align with the published DeepSeek configuration

Why not evaluate Claude Code / CSSwitch with and without EpiGraph using DeepSeek V4 Pro as in the released BiomniBench-AI4S comparison? The new Sonnet 5 experiment does not address the omitted published configuration. I acknowledge the new GPT 5.5 same-backbone comparison; this question concerns alignment with the original public baseline.


4. Can the authur provide minimal reproducible code and records?

Could the authors provide an anonymous reproduction package containing the exact prompts, baseline and ablation implementations, host/model configurations, execution and evaluation scripts, and raw per-task outputs? Appendix H is useful, but the policy description alone does not allow independent reconstruction of the reported experiments.


5. Clarify the scope of the claim

Is the intended claim that EpiGraph provides reliable plug-in improvements across both general-purpose and specialized agents? Which paired experiments establish this, particularly for specialized agents?

The revised manuscript still emphasizes a ranking reversal, but the new same-backbone comparison gives Codex + EpiGraph 68.29 versus EvoScientist 70.62. Meanwhile, adding EpiGraph to EvoScientist yields +1.73, with a 95% CI of [−0.71, +4.14], leaving that incremental benefit uncertain.

Please distinguish outperforming specialized systems from potential complementarity with them, and align the manuscript’s claims with the evidence.


Thank you again for your clarification and experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.