acceptodds
Under review as a conference paper at ICLR 2027

ResearchFlow-Eval: Tracing the Execution Gap in End-to-End Multi-Claim Autoresearch

Abstract

Autoresearch systems increasingly undertake entire research projects. In practice, scientific research requires systems to pursue multiple interdependent claims through a continuous, multi-stage workflow and revise experiments as evidence accumulates. Existing evaluations of autoresearch systems, however, typically score isolated tasks or final artifacts, leaving these process-level capabilities largely unmeasured. We therefore introduce ResearchFlow-Eval, a fine-grained framework for assessing continuous scientific execution and iteration at both the stage and claim levels. Each evaluation is grounded in a published anchor paper, from which we derive stage-specific rubrics that decompose each research stage into dimensions and separately scored rubric items. Using these rubrics, we score each research stage and examine experimental execution in detail, focusing on multi-claim completion and revisions across attempts. We apply the evaluation framework to general-purpose agent harnesses, skill-augmented harnesses, and systems with research-specific orchestration. Across all evaluated systems, experimental execution consistently lags behind design and planning. Within execution, systems struggle to produce high-quality results and meet claim requirements, and repeated attempts rarely yield measurable improvements. Moreover, neither skill augmentation nor research-specific orchestration reliably yields higher scores. ResearchFlow-Eval thus extends autoresearch evaluation beyond isolated, single-objective runs to continuous workflows in which systems pursue multiple claims and refine their work across iterations.

Primary area:
datasets and benchmarks
Keywords:
Multi-claim research workflows, Autonomous scientific research, Fine-grained evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UUNpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ResearchFlow-Eval, a framework for evaluating research agents across seven stages, from literature review to paper writing. It tracks whether code, experiments, and results actually support individual research claims. On a neural-decoding task, the authors find that agents perform better at planning than execution, and that many claims they report as successful lack sufficient supporting evidence. Providing reference materials from earlier stages substantially improves performance.

Critical strengths
  1. The framework looks beyond a polished final paper and checks whether the underlying research was actually done. Separating implementation, experiment execution, and evidence for each claim makes it easier to see where an agent falls short.

  2. The comparison between end-to-end and stagewise runs provides useful diagnostics. Tracking individual claims also reveals problems that an overall score would miss, such as agents implementing a method but failing to establish its empirical benefits.

Critical weaknesses
  1. The evaluation mixes achieving the expected results with doing sound research. A valid negative result cannot receive full credit, even if the experiment and analysis are rigorous. The two evaluation modes also provide different levels of detail about the target claims. This makes it hard to tell how much of the stagewise improvement comes from better upstream work versus clearer targets. Similarly, reporting success on a narrower claim does not necessarily mean an agent overstates its evidence.

  2. The validation is limited to one research task, without repeated runs or quantitative human–judge agreement. Some systems also receive different workflow instructions. The observed failures are informative, but these factors make it difficult to draw broader conclusions about orchestration, iteration, or error propagation. A small human audit and repeated scoring or runs would help establish how stable the findings are.

Questions for authors
  1. Could you clarify which fixed claims and success criteria were visible to the end-to-end agents? For the gap between reported and confirmed success, it would help to distinguish unsupported claims from cases where an agent correctly reported a narrower result that did not meet the benchmark’s full requirements.

  2. Could you separate research rigor from achieving the expected positive results? A complete, well-run experiment may produce a valid negative result, but the current scoring limits its result-quality and analysis scores. Could you reanalyze the existing outputs with these aspects reported separately and check whether the main execution-gap findings still hold?

  3. Could you provide more detail on the human validation? How many items and claim verdicts were checked, how were they selected, and how closely did the human judgments agree with the automated judge? Repeating a small set of key judge calls would also help show whether the conclusions are stable.

  4. What drives the improvement in stagewise runs? These runs receive reference upstream materials, including more explicit target claims, so the improvement may reflect several factors. A small comparison with the same visible claims across both modes would help separate these effects. Please also explain why agents stopped well before the time limit and whether repeated runs show similar behavior.

  5. Could you evaluate another project or research domain? Additional projects would help establish whether the framework works across different research settings and whether the observed execution gap is a broader pattern. This would reduce my concern about generalizability.

4 comments
AuthorsOpenReview ↗

Response to Reviewer UUNp

We thank the reviewer for a constructive review whose questions each point to a concrete analysis. Points shared with the other reviewers are answered in the General Response (G1–G4) and summarized in the first section below. The sections after it answer the remaining points.

Points answered in the General Response

Critical Weaknesses, ¶1: "The evaluation mixes achieving the expected results ... A valid negative result cannot receive full credit ... reporting success on a narrower claim does ... overstates its evidence." Critical Weaknesses, ¶2: "The validation is limited to one research task … how stable the findings are." Questions ¶1: "... clarify which fixed claims and success criteria were visible ...", "... distinguish unsupported claims from cases where an agent correctly reported a narrower result ..." Questions ¶2: "... separate research rigor from achieving the expected positive results …", "... reanalyze the existing outputs ... check whether the main execution-gap findings still hold?" Questions ¶4: "… A small comparison with the same visible claims ...", "... explain why agents stopped ... whether repeated runs show similar behavior." Questions ¶5: "... evaluate another project or research domain? …"

  • Repeated runs and scoring (G1). Every configuration now has two runs, each scored by two judges. The revised Table 2 and Figure 3 report the mean of the four evaluations and Appendix H reports each one. Execution lags design for all nine systems on the two-judge mean (18.7 points, p = 0.004), stagewise runs score above end-to-end runs in all twelve run × judge comparisons, and the two judges rank the systems at Spearman 0.98.
  • What the end-to-end agents could see (G3 (a), Table G5). Not the ten required claims or their criteria. They received the brief, the stage instructions and the resource descriptions, and had to formulate and freeze claims of their own.
  • Sound research versus the expected result, and narrower results versus unsupported claims (G3 (c), Table G4, G4). Result quality and claim completion measure attainment of the paper-derived claims by design, so a valid negative result cannot reach the top of their scales. The analysis cap is different, since the quality of an analysis should not depend on the direction of the result. We regard it as a flaw in the rubric and will revise it so that the S6 analysis stage assesses how completely and how rigorously a system analyzed its own claims, with the outcome requirement removed, while the execution stage keeps the requirement that results match the required claims. The success a system reports refers to the claims it froze itself, which in several runs were materially narrower than the required ones (G4). A reported success can therefore be a correct report of a narrower result rather than an unsupported claim.
  • The same visible claims in both modes (G3, Table G6). We ran this as a new mode, Continuous from S3, in which Codex / GPT-5.6 Sol receives the reference related work and the required claims and still carries out S3 to S7 itself. What it does and does not separate is discussed below.
  • Why agents stopped early, and whether it repeats (G4, Table G7). It repeats. No run reached the 72-hour limit, and each system ends the same way in both runs. Ten runs stop when the system judges its own, usually narrower, claims met. The bare harness also stops early on the second anchor paper.
  • Another research domain (G2). We built one, FITS (ICLR 2024), on long-horizon time-series forecasting and anomaly detection. Eight end-to-end systems and three stagewise harnesses ran it twice, and the first run is scored (new Appendix I, Table 20). Stagewise above end-to-end replicates for all three harnesses, execution below design and planning replicates on average (41.8 against 66.9), and end-to-end claim attainment stays low (25 of 88).

What drives the stagewise improvement

Critical Weaknesses, ¶1: "The two evaluation modes also provide different ... This makes it hard to tell how much of the stagewise improvement ..." Questions ¶4: "What drives the improvement in stagewise runs ... improvement may reflect several factors. …"

The improvement is positive in all twelve run × judge comparisons, by +21.3, +16.7 and +12.2 overview points on the two-judge mean (Table G3 in General Response). The run with the same visible claims (Table G6 in General Response) separates the two factors only in part. Making the required claims visible raises claims met well above the end-to-end run. Adding the reference method and plan on top raises the process sub-score (module development and code execution) by a further 20.0 and 18.9 points under the two judges but does not raise claims met. Target clarity therefore accounts for much of the claim attainment, and reference upstream work for much of the execution score.

Different workflow instructions

Critical Weaknesses, ¶2: "… Some systems also receive different workflow instructions. The observed failures are informative, but these factors make it difficult to draw broader conclusions about orchestration, iteration, or error propagation. …"

Two of the four pilot-driven revisions, the stage order in the task file and the polling of background jobs, are written for the general-purpose harnesses. AutoResearchClaw and DeepScientist instead receive the stages through their native entry points and lifecycle, which we keep as part of the system under test.

Thank you for the additional experiments and clarifications. The second project, repeated runs, and second judge reduce my concerns about coverage and stability. The visible-target and own-plan analyses are particularly helpful, though they also show that the original execution gap depends heavily on target specification.

My remaining concerns are that execution scoring still mixes research quality with attainment of expected results, and quantitative human–judge agreement remains unclear. I appreciate the planned rubric revision, but these concerns keep me at weak reject. I have no further requests for experiments.

Thank you for the follow-up. We agree that the current execution score combines research rigour with attainment of expected results. As discussed in G3 of our General Response (table G4), we propose assessing analysis (stage s6) rigour separately from the outcome so that a well-supported negative result can receive full analysis credit. And in G3 of our General Response (table G6), We also ran a condition with the reference claims visible (Continuous from S3) to examine the role of target specification. We hope you’ll consider revisiting your score in light of the additional evidence and the narrower interpretation of the execution score.

Reviewer_e6HnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ResearchFlow-Eval, an evaluation framework designed to trace the "execution gap" (the disparity between planning/writing research and carrying out experiments that substantiate scientific claims) in end-to-end autonomous research systems. Grounded in a published anchor paper—specifically SPINT (Le et al., 2025) on cross-session intracortical motor decoding using three subjects from the public FALCON dataset—the framework decomposes a complete research workflow into seven stages (literature review, research claims, method design, experiment planning, experimental execution, results analysis, and paper writing), 10 required interdependent claims (4 method claims and 6 empirical claims with frozen, hidden success criteria), and 99 rubric items scored on a 0–4 scale by an LLM judge (claude-opus-5) supported by eight integrity gates.

The authors evaluate nine configurations across three categories: general-purpose agent harnesses (Codex with GPT-5.5, GPT-5.6 Sol, and GPT-6 Astra; Claude Code with Opus 4.8; Kimi Code with K3), skill-augmented harnesses (ARIS and Orchestra on Codex/GPT-5.5), and research-orchestration systems (AutoResearchClaw and DeepScientist on GPT-5.5). By comparing 72-hour end-to-end runs against stagewise runs (where each stage receives paper-derived upstream inputs) and tracking per-attempt claim trajectories, the paper reports three main findings: (1) across all nine systems, experimental execution (mean 43.0, result quality 26.7) lags behind design and planning (mean 61.6), with only 14 of 90 total required claims met; (2) systems frequently over-report claim completion (reporting up to 8 claims met when at most 3–4 satisfy frozen criteria), and feeding paper-derived upstream designs in stagewise runs raises met claims from 1–3 to 7–8; and (3) under a shared GPT-5.5 backbone, skill augmentation improves paper writing (+15–20 points) while lowering execution result quality (30.0 to 20.0), and complex research orchestration degrades execution further.

Critical strengths
  1. Diagnostic Decoupling via Paired End-to-End vs. Stagewise Runs and Self-Report Auditing: Pairing end-to-end runs with stagewise runs (which replace the agent's upstream outputs with paper-derived reference inputs at each stage) provides a clean diagnostic for separating downstream coding failures from upstream specification errors. Showing that general-purpose harnesses jump from 1–3 met claims in end-to-end mode to 7–8 met claims in stagewise mode—combined with the trajectory analysis exposing how agents self-report up to 8 completed claims by quietly narrowing experimental scopes or swapping in weaker baselines—offers a valuable diagnostic insight into how current autoresearch pipelines fail.

  2. Controlled Comparison of General Harnesses, Skill Augmentation, and Research Orchestration: Holding the backbone model fixed (GPT-5.5) across bare Codex, skill-augmented harnesses (ARIS, Orchestra), and research-orchestration systems (AutoResearchClaw, DeepScientist) reveals an important cross-stage trade-off: adding research skills and complex multi-agent orchestration substantially boosts manuscript writing scores (+15 to +20 points) while degrading experimental result quality (from 30.0 down to 20.0 or 0.0). This highlights that surface-level workflow scaffolding often optimizes paper presentation at the expense of rigorous experimentation.

Critical weaknesses
  1. Limited Conceptual Novelty Over Prior Benchmarks (Notably Omitting EXP-Bench) and Constrained Task Formulation: While the paper positions itself as uniquely bridging stage-level planning and experimental execution, the core paradigm—extracting fine-grained rubrics from published papers and their official repositories to evaluate hypothesis/method formulation, experimental design, implementation, execution, and conclusion validity—has already been extensively explored. Most notably, the paper completely omits EXP-Bench (Kon et al., 2025), which evaluated AI agents across 461 tasks from 51 NeurIPS/ICLR papers on experimental design, implementation, execution, and conclusions, and already established the exact headline phenomenon termed the "execution gap" here (finding 20–35% scores on design and implementation aspects vs. only 0.5% on complete executable experiments). Furthermore, to make ground-truth grading tractable on a paper-level task, the initial research brief (S0, Appendix B) explicitly feeds the anchor paper's core methodological priors (an unordered, permutation-invariant transformer set decoder using cross-attention and unlabeled calibration identity inference). Once the high-level direction and dataset split are prescribed, the task largely reduces to guided method/experiment specification and code replication—capabilities already evaluated across PaperBench, EXP-Bench, MLR-Bench, and ResearchGym—while simultaneously penalizing agents that explore valid alternative architectures outside the anchor paper's exact trajectory.

  2. Single-Paper Instantiation (N=1N=1) Cannot Support Broad Generalizations About Autoresearch Systems: The entire empirical study is instantiated on a single task from a single anchor paper (SPINT, on intracortical motor decoding over the FALCON dataset), with one end-to-end run per system. Drawing sweeping conclusions about the general failure of skill augmentation, research orchestration, and iterative refinement across all of autonomous scientific research from N=1N=1 domain-specific neuroscience task is statistically fragile. For instance, AutoResearchClaw crashed on a missing import before making any scientific attempt (0 attempts, score 18.6), and Codex / GPT-5.5 terminated its 72-hour end-to-end run after just 0.30 hours (18 minutes). On a single task without multi-task coverage or repeated runs, it is impossible to disentangle systemic architectural flaws from task-specific brittle failures or prompt/environment idiosyncrasies.

Questions for authors
  1. Clarification on Conceptual Differentiation and Relation to EXP-Bench: How does ResearchFlow-Eval fundamentally advance our understanding of the "execution gap" beyond what was already demonstrated at scale in EXP-Bench (Kon et al., 2025; 51 papers, 461 tasks), as well as PaperBench (Starace et al., 2025), MLR-Bench (Chen et al., 2025a), and ARAC (Cui et al., 2026)? Please include EXP-Bench in Table 1 and Section 2.2 and clarify the specific methodological novelty of your rubric and evaluation pipeline relative to it.

  2. Generalizability Beyond a Single Anchor Paper (N=1N=1) and Single-Run Variance: Can you evaluate at least 2–3 additional anchor papers from standard ML/AI domains (e.g., vision, NLP, or general representation learning, rather than solely specialized intracortical neural spike decoding in NWB format) and report multi-run variance? In particular, why did Codex / GPT-5.5 exit after only 0.30 hours out of its 72-hour budget despite the pilot prompt revisions (Table 7), and does the negative effect of ARIS, Orchestra, and DeepScientist persist across other tasks?

  3. Handling Valid Alternative Designs Under Anchor-Bound Rubrics: Because the research brief prescribes SPINT's core intuition and the 99 rubric items strictly enforce SPINT's 8 method components and 10 claims (e.g., dynamic channel dropout and specific calibration pooling), how does the evaluator avoid unfairly penalizing a system that formulates a scientifically sound alternative cross-session alignment method that achieves strong R2R^2 on the frozen FALCON splits without following SPINT's exact architectural components?

3 comments
AuthorsOpenReview ↗

Response to Reviewer e6Hn

We thank the reviewer for a direct and well-informed assessment. Points shared with the other reviewers are answered in the General Response (G1–G4), and the revised PDF marks every change in red. Below we answer the remaining points.

"Limited Conceptual Novelty Over Prior Benchmarks (Notably Omitting EXP-Bench)"

"Most notably, the paper completely omits EXP-Bench (Kon et al., 2025), which … already established the exact headline phenomenon termed the 'execution gap' here ..." "Clarification on Conceptual Differentiation and Relation to EXP-Bench ... methodological novelty of your rubric and evaluation pipeline relative to it."

We acknowledge that EXP-Bench is closely related to our work, and the revised PDF now adds it to Table 1, to Sec. 2.2 and to the introduction. Both benchmarks follow research execution from experimental design through implementation and execution to a conclusion, and both can expose an execution gap, the distance between what a system designs and what it carries out. EXP-Bench, however, measures whether an agent can carry one specified experiment through to a correct and executable result. Our emphasis is on a research process in which several logically connected claims are executed and revised in sequence. Design, execution and analysis must all account for the whole workflow, because a research project is not one independent experiment but a chain of experiments built around multiple claims. MLR-Bench and ARAC also cover the whole research workflow, MLR-Bench in one run and ARAC one stage at a time with reference inputs from the preceding stage, but neither scores the continuous execution of multiple claims, and their experiments are closer to a single independent task. PaperBench decomposes a paper into many claim-level rubric items, but each item is graded on its own, and the system is given the paper, so no design of its own is carried through to later stages.

Table E1. The four benchmarks the reviewer names, as they now appear in Table 1 of the revised PDF. ✓, △ and × mean that a benchmark evaluates an item explicitly, partially, or not at all.

BenchmarkDesign planExp. executeAnalyze writeFull E2EMulti-claimPaper rubricIteration diagnostics
PaperBench×✓△△✓✓✓
EXP-Bench✓✓△△×✓×
MLR-Bench✓✓✓✓××△
ARAC✓△✓××✓×
ResearchFlow-Eval (ours)✓✓✓✓✓✓✓

"Constrained Task Formulation" and "Handling Valid Alternative Designs Under Anchor-Bound Rubrics"

"Furthermore, … the initial research brief (S0, Appendix B) explicitly feeds the anchor paper's core methodological priors … the task largely reduces to guided method/experiment specification … while simultaneously penalizing agents that ..." "Handling Valid Alternative Designs Under Anchor-Bound Rubrics ... how does the evaluator avoid unfairly penalizing a system that formulates a scientifically sound alternative cross-session alignment method … without following SPINT's exact architectural components?"

The priors are described correctly, and they are deliberate. Constraining the research question, the method search space and the scientific and engineering boundaries keeps what different systems produce inside a common, auditable scope, so their artifacts can be checked against one paper-derived reference and against each other. That is a trade of openness for assessability, and it means our conclusions are about research carried out under given priors. G3 in General Response states the full design argument.

What we change. Following G3 in General Response, we respond in two ways.

  1. Analysis rigour assessed separately from the outcome. In the rubric, the S6 analysis stage will carry the assessment of how completely and how rigorously a system analysed its own claims, with the outcome requirement removed (the last row of Table G4 in General Response), while the execution stage keeps the requirement that results match the required claims.
  2. A run with the reference claims visible. To see how much of the end-to-end shortfall arises when a system must set its own targets, we introduce a new execution mode, Continuous from S3. A system in this mode receives the reference S1 (related work) and the reference S2 (the required claims with their criteria), and then carries out S3 to S7 itself, as in an end-to-end run. Table G5 in General Response states what each mode supplies.

"Single-Paper Instantiation … Cannot Support Broad Generalizations"

"On a single task without multi-task coverage ... impossible to disentangle systemic architectural flaws ..." "Generalizability Beyond a Single Anchor Paper … Can you evaluate at least 2–3 additional anchor papers ... In particular, why did Codex / GPT-5.5 exit after only 0.30 hours ... and does the negative effect of ARIS, Orchestra, and DeepScientist persist across other tasks?"

G1, G2 and G4 in General Response answer these points in full, for multi-run variance, additional anchor papers and the early termination of end-to-end runs respectively. We answer each briefly here.

(a) Multi-run variance. Every configuration now has two runs, each scored by two judges (G1 in General Response). Table 2 and Figure 3 report the mean of the four evaluations and Appendix H reports each one. Three findings hold. Design exceeds execution on average under every evaluation and for all nine systems on the two-judge mean (gap 18.7, p = 0.004). Stagewise runs beat end-to-end runs for all three harnesses in every run under both judges. And the two judges rank the systems similarly (Spearman 0.98).

(b) Additional anchor papers. We built and ran one, FITS (ICLR 2024), on long-horizon time-series forecasting and anomaly detection, a standard ML setting. Eight end-to-end systems and three stagewise harnesses ran it twice, and the first run of all eleven workflows is now scored, with the per-system table in the new Appendix I (G2 in General Response). Within the discussion period one additional anchor paper is what we could build and run end to end. The contribution we claim at present is the evaluation protocol rather than broad task coverage, and we are continuing to add anchor papers.

What the second task shows so far, on a single run scored by one judge, is the following.

  • Stagewise above end-to-end replicates for all three harnesses (by 9.0, 33.5 and 33.4 overview points).
  • Execution below design and planning replicates on average (41.8 against 66.9).
  • End-to-end claim attainment stays low. End-to-end systems met 25 of 88 required claims on FITS.

(c) The 0.30-hour run. It was a normal exit after 18.2 minutes, with exit code 0, no crash, no timeout and no operator intervention. Its final message states that "all frozen claims reached their stated standards", meaning the claims it had written itself, and adds that the positive aggregate result is driven mostly by one dataset. Its own criteria were materially narrower than the anchor's. All four pilot-driven revisions were present in its instructions. They ask the agent to continue until its claims are met or 90% of the allowance is used, but nothing external checks completion, so once it judged its own criteria met it stopped. Its second run repeated this in 0.73 hours. The pattern recurs across systems and on the second task (G4 in General Response).

(d) Does the negative effect persist across tasks? On the first task the group-level result holds. On the mean of two runs and two judges, both skill-augmented harnesses and both orchestration systems stay at or below the bare harness on overview and on execution (Table T1 in our response to Reviewer trEt). AutoResearchClaw failed again in its second run at the same pipeline stage for a different reason, a misidentified dataset instead of a missing import, and it also failed to deliver one of its FITS runs. The revised Sec. 5.3 reports these as observed behaviour rather than as evidence about architecture.

Thank you again for your review. We wanted to briefly follow up on two points. Our response explains how tracking multiple linked claims within one research run extends the experiment-level evaluation in EXP-Bench. G1 and G2 of our General Response summarize the repeated runs and the results on the second task. If you have a chance to review these updates, we’d welcome your thoughts and would appreciate your reconsidering your score.

Reviewer_trEtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ResearchFlow-Eval, a framework for evaluating autoresearch systems on a continuous seven-stage workflow derived from a published anchor paper. The stages run from literature review, claims, method and planning through execution, analysis and writing. An LLM judge (claude-opus-5) scores each stage's artifact against hidden, paper-derived rubrics: 99 items, each rated 0-4 against item-specific anchors. Execution is also assessed per required claim. There are 10 claims with frozen success criteria, each judged met or unmet, and diagnostics compare system-reported against judge-confirmed attainment over time, as well as metric trajectories across attempts. End-to-end runs are paired with stagewise runs, in which each stage receives the paper-derived upstream outputs. The framework is instantiated on a single task (SPINT; FALCON M1/M2/H1). Nine configurations are run once each end-to-end: five general-purpose harness/model pairs, plus two skill-augmented harnesses and two research-orchestration systems on GPT-5.5. Three configurations are also run stagewise. Reported findings: (1) execution scores below design and planning in all end-to-end runs (mean 43.0 vs. 61.6); (2) 14 of 90 required claims are met end-to-end, only one of them empirical, while systems report up to eight claims as attained; (3) stagewise inputs raise met claims to 7-8; (4) on GPT-5.5, skill augmentation raises writing scores but lowers execution scores, and orchestration lowers execution further.

Critical strengths
  1. The evaluation protocol is careful and auditable. Evaluator assets never enter the workspace, and inputs are allowlisted and hashed (p.14); claim events go into a hash-chained log with externally added timestamps (App. D, p.21); every item stays in its denominator (p.20); integrity gates are defined (Table 12, p.23), and the one gate that fired is disclosed (p.22); pilot-driven instruction revisions and system versions/commits are reported (Table 7, p.17; Table 14, p.25). The numbers are internally consistent: I recomputed the Table 2 overview scores via Eq. (1)-(2), the part means (61.6/43.0/46.5), the 14/90 met count from Table 16 and the Spearman values in Sec. 5.4, and all of them match.

  2. The claim-level view separates what systems report from what they actually establish, and it exposes concrete failure modes that a final-artifact score would hide. GPT-5.6 Sol and GPT-6 Astra each report eight attained claims, but only three and four meet their criteria (Fig. 4a/b, p.8; Table 16, p.26). The Sol case study shows self-chosen baselines replacing the frozen ones, a missing 20% retention condition, and absolute latency reported instead of the required ratio (pp.26-27). Skill-augmented harnesses raise writing scores while result quality drops (Table 2, p.6).

Critical weaknesses
  1. The evidence base is too narrow for the general conclusions, and the judge is not validated quantitatively.

All results come from one anchor task (Sec. 4.1, p.6), one run per configuration (Table 13, p.24), and one judge sample per call, with temperature and seed not controllable (p.22). There are no repeated runs, variance estimates, confidence intervals or significance tests, and the limitations paragraph (p.9) does not mention this. Judge validity rests on one sentence: a single domain researcher "spot-checked ... and found them largely consistent" (p.6), with no sample size or agreement statistic. The judge comes from the same model family as one of the evaluated systems, and both run through Claude Code (p.22; Table 14); the paper does not discuss this, and there is no cross-family check. The step that maps each system's own claim records onto the ten required claims (p.22) determines Fig. 4 and is not validated either.

Within this task, the direction of the design-execution difference is consistent: all nine end-to-end runs show it, with gaps of 8.6 to 44.1 points (my computation from Table 2). The finer comparisons in Sec. 5.3 (contribution 3) are much less secure. Result quality differs by 4 integer rating points over 10 items (12/40 vs 8/40), and claim completion by 2-4 points. That comparison is also confounded: the three groups ran on different hosts (p.24); the orchestration systems did not receive two of the four instruction revisions (p.14); AutoResearchClaw's score comes from a single crash on a missing import (p.9); and the Codex/GPT-5.5 reference run stopped after 0.30 h and had two writing items zeroed by a gate (Table 13; p.22). The causal reading that the comparison "reflects how research is organized rather than which model is used" (p.6) is therefore not supported.

  1. In end-to-end mode, the measured "execution gap" mixes execution ability with alignment to a hidden, anchor-specific specification.

In S2/S4, systems write their own claims, criteria and baselines (Table 3, p.14; Table 7, p.17), but result quality, claim completion and met verdicts are judged against the anchor's frozen criteria, which are "all hidden from the system under evaluation" (p.1; p.22; Table 9, p.19). No score is computed against a system's own frozen plan, and the unmet claims are not broken down into not planned, planned but not run, and run but missing the criterion. When the stagewise runs supply claims.md with the criteria and per-claim execution contracts (Table 4, p.15), met claims rise to 7-8, execution minus design becomes −4.7/+5.3/−3.1 instead of −8.6/−16.4/−10.0 end-to-end (my computation from Table 15), and result quality is no longer the weakest dimension. The authors partly acknowledge this (p.2, "part of this deficit"; p.7; p.9; p.27 notes that reported successes can refer to narrower experiments), but the abstract and conclusion still present the gap as an execution deficit, and the reported-vs-confirmed gap is partly a product of the same design. There are genuine execution failures as well: a crash (p.9), negative R² on M2/H1 (p.27), and code execution remaining the weakest dimension even in stagewise runs (Table 15).

Questions for authors

Questions whose answers could change my assessment:

  1. If the end-to-end runs are re-adjudicated against each system's own frozen S2/S4 claims and criteria, do "execution lags design" and "self-reports are unreliable" still hold? How do the 76 unmet end-to-end claims split into not planned, planned but not executed, and executed but missing the criterion? What does the "success boundary" in the S0 brief (Table 4) reveal?
  2. Is there any variance estimate, e.g., repeated runs of at least Codex/GPT-5.5, ARIS and Orchestra, or repeated judge calls on the same artifacts? How do you explain the 8-15 point stagewise gain on S1, whose inputs are identical in both modes?
  3. Can you quantify judge-human agreement (number of items checked, exact agreement or kappa, number of raters)? Was the mapping of system records to required claims validated? Do the rankings and met/unmet verdicts hold with a non-Anthropic judge?
  4. Why did each end-to-end run terminate? The claim log records budget stops (p.21). Does forcing further iteration, or giving criterion-level pass/fail feedback, change the met counts or trajectories?
  5. Were the pilot runs (Table 7) carried out on the same SPINT task and harnesses, and were any rubric items or criteria changed after the pilots?
4 comments
AuthorsOpenReview ↗

Response to Reviewer trEt

We thank the reviewer for a careful and specific review. Several points are shared with the other reviewers and are answered in the General Response (G1–G4). The revised PDF marks all changes in red. Below we answer the remaining points.

Repeated runs, variance, and a non-Anthropic judge

Critical Weaknesses, ¶1: "All results come from one anchor task ... There are no repeated runs...", "The judge comes from the same model family ... no cross-family check." Questions ¶2: "Is there any variance estimate, ... How do you explain the 8-15 point stagewise gain on S1, ..." Questions ¶3: "… Do the rankings and met/unmet verdicts hold with a non-Anthropic judge?"

Every configuration now has two research runs, each scored by the original judge and by gpt-6-astra, a non-Anthropic model. The tables and the check against each conclusion are in G1 in General Response. In the revised PDF, Table 2 and Figure 3 report the mean of the four evaluations, and Appendix H gives every run under every judge. The limitations paragraph (Sec. 6) now states that the evidence comes from two tasks, each run twice.

Rankings and met or unmet verdicts under a non-Anthropic judge. The Spearman correlation between the two judges' run means is 0.98, so the ranking is stable. Few claims are met under Astra either, 5/90 and 9/90 end-to-end, with no empirical claim met in either run. On the submitted runs Astra agrees with the submitted verdict on 79 of 90 claims, 75 of which are joint "unmet", and it is stricter on ten of the remaining eleven.

The stagewise S1 gain. It only partly replicates. On the two-judge means, stagewise minus end-to-end S1 is +13.0, +3.1 and −3.6 for Codex, Claude Code and Kimi Code. Since the inputs to this stage are identical in both modes, what differs is how much of the run goes into the stage. In a stagewise run the literature review is a deliverable on its own, whereas in an end-to-end run the system writes it while scoping the whole project and tends to keep it short, which may account for the difference.

The fine-grained comparisons of Sec. 5.3

Critical Weaknesses, ¶2: "The finer comparisons in Sec. 5.3 are much less secure. ... the comparison 'reflects how research is organized rather than which model is used' is therefore not supported."

Rather than argue this point we re-checked it on the two runs and the two judges. The conclusion of Sec. 5.3 holds, so we keep it.

1. On the mean of two runs and two judges the comparison stands. All five configurations share GPT-5.5 as their backbone model, so Table T1 isolates what the added structure contributes.

Table T1. The five GPT-5.5 configurations on the mean of two runs and two judges. The bare harness, the two skill-augmented harnesses and the two research-orchestration systems. The bare harness is the highest of the five on overview, on experimental execution and on result quality, and it has the smallest design-minus-execution gap, so neither skill augmentation nor research orchestration closes the gap on this task.

SystemOverviewExecution (S5)Result qualityWriting (S7)Design − execution
Codex / GPT-5.5 (bare harness)45.040.525.637.515.2
ARIS / GPT-5.543.036.419.443.817.7
Orchestra / GPT-5.539.230.813.140.621.1
AutoResearchClaw / GPT-5.521.418.40.00.022.2
DeepScientist / GPT-5.537.321.40.046.937.7

2. The confounds the reviewer lists do not produce this ordering.

Table T2. The confounds the reviewer raises, and how each one should be read.

Point raisedHow it should be read
Different hostsEach group runs in its own environment so that no system can read another's research state or interfere with it. The hosts are built from the same container image and run the same task, so the research state every system starts from is the same
Two instruction revisions missingOne states the stage order in the task file, and the other asks the system to keep polling background experiments. AutoResearchClaw and DeepScientist already receive the stages through their native entry points and lifecycle, so these two revisions bring the general-purpose harnesses to the same footing rather than giving them an advantage
AutoResearchClaw crashReproducible small failures of the system itself. A missing import at pipeline stage 12 in the first run, and 71 data files misidentified in the second
Codex / GPT-5.5 at 0.30 hA normal exit for this system. It declared the claims it had frozen itself satisfied and stopped, which we report as system behaviour (G4 in General Response)
Writing gate on CodexThe two writing items the gate zeroed are the gate working as intended, since one figure had no traceable provenance and one table was inconsistent with the text, which is what it checks

The anchor-derived specification, and scoring against each system's own plan

Critical Weaknesses, ¶3: "the measured 'execution gap' mixes execution ability ...", "No score is computed against a system's own frozen plan, and the unmet claims are not broken down into not planned, ..." Questions ¶1: "If the end-to-end runs are re-adjudicated ... still hold? How do the 76 unmet ... split into not planned, ... What does the 'success boundary' ..."

We agree that the submission over-read the end-to-end score as an execution deficit. We now describe it as attainment of an anchor-derived specification under the given brief, reported separately from fulfilment of a system's own plan. G3 in General Response gives the design argument. The direct answers to this question follow.

(a) Re-adjudicated against each system's own frozen claims, "execution lags design" does not hold for seven of the nine systems, and both sides of that comparison then sit near the ceiling. We scored S2–S7 of all nine first runs against the 68 claims, criteria and comparators the systems themselves froze in S2 and S4, with no anchor access. Seven systems' design-minus-execution difference collapses into [−0.8, 1.3], on own-chain design and execution scores that are all above 98. Part of the reason is that several systems froze claims that are easy to satisfy, which raises both sides at once, and that is why we also score every run against one fixed set of required claims. It remains true that the gap we reported is a gap against a fixed external specification rather than a general property of these systems' own execution. The new execution mode introduced in G3 in General Response, Continuous from S3, addresses this by making the required claims visible to the system while leaving the research work to it.

Table T3. Each system scored against its own frozen claim chain. Nine submitted end-to-end runs, stages S2 to S7, own claims and own criteria.

SystemDesign − execution
Codex / GPT-5.5−0.8
Codex / GPT-5.6 Sol0.0
Codex / GPT-6 Astra0.0
Claude Code / Opus 4.81.3
Kimi Code / K31.3
ARIS / GPT-5.51.3
Orchestra / GPT-5.50.1
AutoResearchClaw / GPT-5.535.8
DeepScientist / GPT-5.528.2

(b) Split of the 76 unmet claims. From the submitted first-run verdicts. The categories were fixed after seeing the data, so the split is retrospective. Only three are complete executions that miss the criterion. All nine systems omit training-day scaling (E3), and five omit dynamic channel dropout (MC4). The twelve planned-but-not-executed cases are concentrated in AutoResearchClaw (8) and DeepScientist (4). The shortfall is therefore not only a matter of how demanding the required criteria are.

Table T4. Where end-to-end attainment is lost. The 76 unmet claims of the nine submitted runs.

CategoryCount
Not planned (no corresponding claim or experiment in the system's S2/S4)15
Planned but not executed12
Executed but incomplete (missing comparator, scope, protocol or evidence)37
Executed completely, criterion missed3
Unresolved9

(c) The success boundary. It reads, "A successful solution should form a complete chain from gap, testable claims, method, frozen experiment plan and real run evidence to calibrated conclusions." It lists no claims, comparators or thresholds.

Why each run terminated

Questions For Authors, ¶4: "Why did each end-to-end run terminate? ... Does forcing further iteration, ... change the met counts or trajectories?"

See G4 in General Response for the termination table and the per-system reasons. No end-to-end run reached the limit, and the longest used 38% of it. The "budget stop" in the claim log is the instructed stop line at 90% of the allowance, and no selected run reached it, so it does not describe why these runs ended. Runs end because the system declares its own criteria met (ten runs), closes while acknowledging unmet claims (six), or crashes (two), and each system ends the same way in both runs. We did not arrange a forced-iteration condition, and the reason lies in the design of the mode. Forcing further iteration needs a completion signal, and the only signal available to a system that has not seen the required claims is its own frozen claim set, which these systems declare satisfied long before the required criteria are met.

The pilot runs, and what changed after them

Questions ¶5: "Were the pilot runs ... on the same SPINT task and harnesses, and were any rubric items or criteria changed ..."

The pilots used the same task and the Codex, Claude Code and Kimi Code harnesses under earlier protocol versions. Their purpose is described in G4 in General Response. Systems were ending their turn while experiments were still running, so the instruction revisions were written to keep them executing. No required claim or criterion changed after the pilots.

Thank you for the additional experiments. The repeated runs and second task address several of my concerns, and the own-plan analysis helps explain what the execution gap measures.

My remaining concern is the contribution beyond existing benchmarks. Tracking claims across stages is useful, but I do not see enough new findings from this approach to support acceptance. I will keep my score at weak reject.

Thank you for clarifying your concern about the contribution. Many existing benchmarks, including EXP-Bench, evaluate individual tasks or experiments. ResearchFlow-Eval follows multiple linked claims through successive rounds of planning, execution, and revision in a single research run. In our evaluation, agents often left experiments incomplete and met few of the required claims. These difficulties persisted with skill-augmented harnesses and purpose-built research agents. We hope this distinction and the additional evidence give you reason to reconsider your score.

Comments on the paper

5 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all three reviewers. Several concerns were shared, so we answer them once here and refer to these sections from the individual responses. We have uploaded a revised PDF in which every change is set in red.

G1. Two runs per configuration, two independent judges

Reviewer trEt: "All results come from one anchor task … one run per configuration ...", "… Do the rankings and met/unmet verdicts hold with a non-Anthropic judge?" Reviewer e6Hn ("Generalizability Beyond a Single Anchor Paper … and Single-Run Variance"). Reviewer UUNp: "The validation is limited to one research task… repeated scoring or runs ..."

Every configuration now has two research runs. Each run is scored by two judges, the original claude-opus-5 and gpt-6-astra with xhigh reasoning, which is a non-Anthropic model. Both judges use the same rubrics, the same evidence and the same frozen criteria. With n = 2 we report raw values and means. The only statistical test treats the nine end-to-end systems as units (exact two-sided Wilcoxon signed-rank) and is descriptive within this task.

In the revised PDF. Table 2 and Figure 3 now report the mean of the four evaluations of each configuration (two runs × two judges). Appendix H gives every run under every judge. Table 17 summarizes, Table 18 covers stagewise versus end-to-end, and Table 19 covers all end-to-end runs.

Table G1. Original judge (claude-opus-5), two runs, nine end-to-end systems. Here and below, r1 and r2 are the first and the second run of a configuration.

SystemOverview r1 / r2Overview meanDesign − exec. r1 / r2Design − exec. mean
Codex / GPT-5.545.1 / 48.947.08.6 / 10.79.6
Codex / GPT-5.6 Sol59.5 / 82.070.821.7 / 14.017.9
Codex / GPT-6 Astra64.9 / 92.378.615.6 / −13.21.2
Claude Code / Opus 4.861.3 / 74.968.116.4 / 8.312.4
Kimi Code / K364.2 / 73.168.710.0 / 14.112.1
ARIS / GPT-5.544.5 / 45.044.815.4 / 16.415.9
Orchestra / GPT-5.545.2 / 30.337.814.1 / 29.721.9
AutoResearchClaw / GPT-5.518.6 / 20.019.321.9 / 16.819.3
DeepScientist / GPT-5.539.9 / 37.438.644.1 / 33.839.0
Nine systems49.3 / 56.052.618.7 / 14.5 (9/9, 8/9 positive)16.6 (9/9 positive)

Table G2. Mean of the two runs, by judge, and the two-judge mean. Nine end-to-end systems. The two-judge mean is the value in the revised Table 2.

SystemOverview (Claude / Astra / both)Design − exec. (Claude / Astra / both)
Codex / GPT-5.547.0 / 43.0 / 45.09.6 / 20.8 / 15.2
Codex / GPT-5.6 Sol70.8 / 56.1 / 63.417.9 / 20.3 / 19.1
Codex / GPT-6 Astra78.6 / 62.5 / 70.61.2 / 23.1 / 12.1
Claude Code / Opus 4.868.1 / 46.3 / 57.212.4 / 20.9 / 16.6
Kimi Code / K368.7 / 50.3 / 59.512.1 / 0.6 / 6.3
ARIS / GPT-5.544.8 / 41.2 / 43.015.9 / 19.5 / 17.7
Orchestra / GPT-5.537.8 / 40.6 / 39.221.9 / 20.3 / 21.1
AutoResearchClaw / GPT-5.519.3 / 23.4 / 21.419.3 / 25.1 / 22.2
DeepScientist / GPT-5.538.6 / 36.0 / 37.339.0 / 36.5 / 37.7
Nine systems52.6 / 44.4 / 48.516.6 / 20.8 / 18.7 (9/9, 9/9, 9/9 positive)

Table G3. Stagewise versus end-to-end for the three harnesses run in both modes (overview).

HarnessModeClaude r1 / r2 / meanAstra r1 / r2 / meanTwo-judge mean
Codex / GPT-5.5Stagewise78.5 / 62.7 / 70.664.5 / 59.3 / 61.966.3
End-to-end45.1 / 48.9 / 47.043.1 / 42.8 / 43.045.0
Difference+33.4 / +13.8 / +23.6+21.4 / +16.5 / +19.0+21.3
Claude Code / Opus 4.8Stagewise81.0 / 79.9 / 80.463.7 / 70.8 / 67.373.9
End-to-end61.3 / 74.9 / 68.150.9 / 41.6 / 46.357.2
Difference+19.7 / +5.0 / +12.3+12.8 / +29.2 / +21.0+16.7
Kimi Code / K3Stagewise74.8 / 81.0 / 77.962.5 / 68.3 / 65.471.6
End-to-end64.2 / 73.1 / 68.746.8 / 53.7 / 50.359.5
Difference+10.6 / +7.9 / +9.2+15.7 / +14.6 / +15.1+12.2

What holds against the paper's conclusions.

  1. Execution lags design (holds). It holds on average under every evaluation, and for 9/9 systems on the Claude mean (gap 16.6), the Astra mean (20.8) and the two-judge mean (18.7, p = 0.004).
  2. Few required claims met end-to-end (holds). 14/90 required claims are met in the submitted first run and 16/90 in the second run under the original judge, and 5/90 and 9/90 under the second judge gpt-6-astra.
  3. Stagewise above end-to-end (holds). It holds for all three harnesses in all twelve run × judge comparisons, by 5.0 to 33.4 overview points (Table G3). The size varies across runs and judges.
  4. Rankings (stable). Spearman correlation is 0.98 between the two judges' run means and 0.83 between the two runs under the original judge.

G2. A second anchor paper and its first results

Reviewer e6Hn ("Single-Paper Instantiation … Cannot Support Broad Generalizations About Autoresearch Systems"), and in the questions: "Can you evaluate at least 2–3 additional anchor papers ... report multi-run variance?" Reviewer UUNp (Questions For Authors, ¶5): "Could you evaluate another project or research domain? Additional projects would help establish whether the framework works across different research settings …" Reviewer trEt (Critical Weaknesses, ¶1): "The evidence base is too narrow for the general conclusions …"

Within the discussion period one additional anchor paper is what we could build and run end to end. We built a second task from FITS: Modeling Time Series with 10k Parameters (ICLR 2024), a compact frequency-domain model for long-horizon forecasting and anomaly detection. Eight end-to-end systems and three stagewise harnesses ran it twice each. The new task has eleven required claims and ninety-three rubric items.

Results. The first run of all eleven FITS workflows has been scored by claude-opus-5. The per-system table, with every stage and the four execution dimensions, is Table 20 in the new Appendix I of the revised PDF.

  • Stagewise above end-to-end replicates. All three harnesses run in both modes score higher stagewise, by 9.0, 33.5 and 33.4 overview points, and their required claims met go from 11, 0 and 2 to 9, 5 and 10 of eleven.
  • Execution below design and planning replicates on average. The mean over the eight end-to-end systems is 66.9 for design and planning against 41.8 for experimental execution, a gap of 25.1 points, and the gap is positive for six of the eight systems.
  • End-to-end claim attainment stays low. The end-to-end systems met 25 of 88 required claims on FITS.
  • Early stopping also recurs. The bare harness (Codex with GPT-5.6 Sol) ended after 2.13 and 0.60 hours, both times at exit 0 after declaring its own claims met. AutoResearchClaw again failed to deliver one of its FITS runs. G4 gives the termination audit.

G3. Required claims versus the agent's own claims

Reviewer trEt: "... the measured 'execution gap' mixes execution ability with alignment ...", "If the end-to-end runs are re-adjudicated ... 'self-reports are unreliable' still hold? …" Reviewer e6Hn ("Handling Valid Alternative Designs Under Anchor-Bound Rubrics") Reviewer UUNp: "... mixes achieving the expected results ... A valid negative result cannot receive full credit ...", "... which fixed claims and success criteria were visible ...", "... separate research rigor from achieving the expected positive results"

This is the concern all three reviewers raise. It has three distinct parts, namely what the agent was told, whether it is fair to score a faithfully executed but different plan against our claims, and whether process rigour can be separated from hitting those claims. We answer them in that order and then report a new execution mode.

(a) What the end-to-end condition supplies, and what it withholds. The complete set of ten required claims and their claim-specific frozen criteria was not supplied as a reference input. Agents received the research brief, the stage instructions and the resource descriptions, and the S2 instructions required them to formulate their own claims. The brief's "success boundary" paragraph describes the expected chain of evidence, which does not enumerate the ten claims or their numeric thresholds.

(b) Why we score against paper-derived claims at all. The priors in the brief are a deliberate evaluation design, not an oversight. By constraining the research question, the method search space and the scientific and engineering boundaries, we try to keep what different systems produce inside a common, traceable and auditable range, so that stage-level and claim-level evidence checks can be applied to all of them. Without a paper-derived ground truth and rubric, heterogeneous solutions cannot be scored comparably, and the judge has nothing to audit a self-report against. That is the trade-off we chose. The measured quantity is continuous research performance under given priors, relative to a fixed paper-derived target, not a general ranking of scientific merit.

(c) Rigour versus the expected outcome. It follows from (b) that our scores measure each run against the required claims, and therefore that they give little weight to how completely and how soundly a system carried out claims of its own. We accept that this is a weakness of the present design. Table G4 shows where it sits in the scoring.

Table G4. Can a complete, valid negative result receive full credit?

Score componentFull credit without a positive result?
Execution, module developmentYes
Execution, code executionYes
Execution, result qualityNo. A complete and clear disconfirmation scores 1 of 4
Execution, claim completionNo. A complete execution that misses the criterion scores 3 of 4
AnalysisNo. Capped at 3 of 4 when the claim is unmet
Analysis (after the planned rubric revision)Yes. A complete and clear analysis can reach 4 of 4 when the claim is unmet

We therefore propose two changes to separate the rigour of an analysis from attainment of the required claims.

  1. Analysis rigour assessed separately from the outcome. In the rubric, the S6 analysis stage will carry the assessment of how completely and how rigorously a system analysed its own claims, with the outcome requirement removed (the last row of Table G4), while the execution stage keeps the requirement that results match the required claims.
  2. A run with the reference claims visible. To see how much of the end-to-end shortfall arises when a system must set its own targets, we introduce a new execution mode, Continuous from S3. A system in this mode receives the reference S1 (related work) and the reference S2 (the required claims with their criteria), and then carries out S3 to S7 itself, as in an end-to-end run. Table G5 states what each mode supplies.

Table G5. What each mode supplies to the system under evaluation.

InformationEnd-to-endContinuous from S3 (new)Stagewise
Brief with methodological priorsyesyesyes
Ten required claims and their criterianoyesfrom S3
Reference related work (S1)noyesfrom S2
Reference method and plan (S3, S4)nonofrom S4 / S5
Evaluator reference and grading rubricnonono

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.