- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces ResearchFlow-Eval, a framework for evaluating research agents across seven stages, from literature review to paper writing. It tracks whether code, experiments, and results actually support individual research claims. On a neural-decoding task, the authors find that agents perform better at planning than execution, and that many claims they report as successful lack sufficient supporting evidence. Providing reference materials from earlier stages substantially improves performance.
Critical strengths
-
The framework looks beyond a polished final paper and checks whether the underlying research was actually done. Separating implementation, experiment execution, and evidence for each claim makes it easier to see where an agent falls short.
-
The comparison between end-to-end and stagewise runs provides useful diagnostics. Tracking individual claims also reveals problems that an overall score would miss, such as agents implementing a method but failing to establish its empirical benefits.
Critical weaknesses
-
The evaluation mixes achieving the expected results with doing sound research. A valid negative result cannot receive full credit, even if the experiment and analysis are rigorous. The two evaluation modes also provide different levels of detail about the target claims. This makes it hard to tell how much of the stagewise improvement comes from better upstream work versus clearer targets. Similarly, reporting success on a narrower claim does not necessarily mean an agent overstates its evidence.
-
The validation is limited to one research task, without repeated runs or quantitative human–judge agreement. Some systems also receive different workflow instructions. The observed failures are informative, but these factors make it difficult to draw broader conclusions about orchestration, iteration, or error propagation. A small human audit and repeated scoring or runs would help establish how stable the findings are.
Questions for authors
-
Could you clarify which fixed claims and success criteria were visible to the end-to-end agents? For the gap between reported and confirmed success, it would help to distinguish unsupported claims from cases where an agent correctly reported a narrower result that did not meet the benchmark’s full requirements.
-
Could you separate research rigor from achieving the expected positive results? A complete, well-run experiment may produce a valid negative result, but the current scoring limits its result-quality and analysis scores. Could you reanalyze the existing outputs with these aspects reported separately and check whether the main execution-gap findings still hold?
-
Could you provide more detail on the human validation? How many items and claim verdicts were checked, how were they selected, and how closely did the human judgments agree with the automated judge? Repeating a small set of key judge calls would also help show whether the conclusions are stable.
-
What drives the improvement in stagewise runs? These runs receive reference upstream materials, including more explicit target claims, so the improvement may reflect several factors. A small comparison with the same visible claims across both modes would help separate these effects. Please also explain why agents stopped well before the time limit and whether repeated runs show similar behavior.
-
Could you evaluate another project or research domain? Additional projects would help establish whether the framework works across different research settings and whether the observed execution gap is a broader pattern. This would reduce my concern about generalizability.
4 comments
What drives the stagewise improvement
Critical Weaknesses, ¶1: "The two evaluation modes also provide different ... This makes it hard to tell how much of the stagewise improvement ..." Questions ¶4: "What drives the improvement in stagewise runs ... improvement may reflect several factors. …"
The improvement is positive in all twelve run × judge comparisons, by +21.3, +16.7 and +12.2 overview points on the two-judge mean (Table G3 in General Response). The run with the same visible claims (Table G6 in General Response) separates the two factors only in part. Making the required claims visible raises claims met well above the end-to-end run. Adding the reference method and plan on top raises the process sub-score (module development and code execution) by a further 20.0 and 18.9 points under the two judges but does not raise claims met. Target clarity therefore accounts for much of the claim attainment, and reference upstream work for much of the execution score.
Different workflow instructions
Critical Weaknesses, ¶2: "… Some systems also receive different workflow instructions. The observed failures are informative, but these factors make it difficult to draw broader conclusions about orchestration, iteration, or error propagation. …"
Two of the four pilot-driven revisions, the stage order in the task file and the polling of background jobs, are written for the general-purpose harnesses. AutoResearchClaw and DeepScientist instead receive the stages through their native entry points and lifecycle, which we keep as part of the system under test.
Thank you for the additional experiments and clarifications. The second project, repeated runs, and second judge reduce my concerns about coverage and stability. The visible-target and own-plan analyses are particularly helpful, though they also show that the original execution gap depends heavily on target specification.
My remaining concerns are that execution scoring still mixes research quality with attainment of expected results, and quantitative human–judge agreement remains unclear. I appreciate the planned rubric revision, but these concerns keep me at weak reject. I have no further requests for experiments.
Thank you for the follow-up. We agree that the current execution score combines research rigour with attainment of expected results. As discussed in G3 of our General Response (table G4), we propose assessing analysis (stage s6) rigour separately from the outcome so that a well-supported negative result can receive full analysis credit. And in G3 of our General Response (table G6), We also ran a condition with the reference claims visible (Continuous from S3) to examine the role of target specification. We hope you’ll consider revisiting your score in light of the additional evidence and the narrower interpretation of the execution score.
Response to Reviewer UUNp
We thank the reviewer for a constructive review whose questions each point to a concrete analysis. Points shared with the other reviewers are answered in the General Response (G1–G4) and summarized in the first section below. The sections after it answer the remaining points.
Points answered in the General Response