- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper introduces a benchmark to evaluate whether scientific agents can reconstruct the evidence-supported stages of research. The proposed benchmark represents 53 papers spanning 15 scientific domains and 131 research cases. Averaged across all the domains, the authors observe that the agents can recover conclusions more reliably than the evidence that supports them.
Critical strengths
- The paper is clearly written and easy to understand.
- The benchmark consists of 15 different scientific domains, which ensures that the dataset is diverse enough.
- There's an interesting finding in Table 2, which shows that the Claude Code harness performs better than Codex in motivation and method
Critical weaknesses
- The paper, in its current form, is not well placed within the literature. While the authors have a dedicated related work section in the Appendix, they don't compare it with closely related works such as arXiv:2608.05235, arXiv:2602.02905v2, arXiv:2601.03260, etc which concerns the limited technical novelty.
- The comparison seems to not be fair. There are many more harnesses, e.g., The AI Scientist (arXiv:2504.08066), Virtual Laboratories (arXiv:2507.06271v1), etc that the authors would evaluate the benchmark on
Questions for authors
- Wondered if the authors also thought of treating this as a dependent process, meaning there could be a clear motivation for a problem, but it could be simply infeasible. Did the authors also consider looking into such problems? If so, what was the observation?
Also note that I am open to reconsidering my score. On a convincing answer to both weaknesses, I will consider adjusting my score to Weak Accept
3 comments
Response on Related Work and Novelty
Thank you for highlighting these closely related studies. We agree that the manuscript should better acknowledge their contributions and clarify SCIONBench's position.
SCIONBench evaluates evidence-preserving reconstruction of scientific research. Its five-stage, paper-grounded dependency graphs define reference targets for methods, experimental designs, evidence, and conclusions. Four independently initialized disclosure conditions enable paired comparisons of the same downstream targets evaluable in both conditions, examining how reconstruction changes with the supplied upstream reference information.
1. From Trajectories to Evidence (arXiv:2608.05235)
This work examines which findings are supported by multi-round experiments in industrial recommendation. It verifies and repairs research artifacts, checks execution validity and attribution, and retains supported claims with provenance and applicability boundaries for selective reuse in subsequent tasks.
Both works emphasize evidential support, but their evaluation objects differ. Its records qualify and preserve findings from executed trajectories; SCIONBench uses published papers across scientific domains to define reference targets for reconstructing the content and connections supporting conclusions. Our automatic node scoring evaluates the scientific content, evidential relationships, and conditions presented in the final research report, using reference nodes, scoring criteria, and relevant evidence.
2. FIRE-Bench (arXiv:2602.02905v2)
FIRE-Bench evaluates whether agents can independently design and execute experiments to rediscover published empirical findings from a research question. It provides experimental scope and evaluation requirements while withholding the authors' specific implementations and conclusions. Its primary metrics compare atomic empirical claims through precision, recall, and F1, complemented by execution-trace analysis of planning, implementation, execution, and conclusion errors.
Both works use published research as a reference and examine multi-stage scientific tasks. SCIONBench provides complementary evaluation capabilities in two respects.
Evaluation targets and the role of the reference structure. FIRE-Bench selects a central experiment and its parent question from a research-problem tree to instantiate a rediscovery task, with the published finding as the scoring reference. SCIONBench organizes methods, experimental designs, evidence, and conclusions into reference nodes linked by scientific dependencies. These stage-specific nodes directly define reconstruction and scoring targets. Consequently, the protocol separately assesses the recovery of conclusions and the scientific content supporting them; agreement with the final conclusion does not substitute for evaluating intermediate content.
Comparisons across information conditions. FIRE-Bench's trajectory analysis traces missed or incorrect findings back to research errors. SCIONBench additionally asks how reconstruction of the same downstream target changes when reference methods, designs, or observations are supplied. For example, the same experimental-design target is evaluated from motivation alone and from motivation plus the reference method, using independent runs with the same paper-local resources. This characterizes both reconstruction from the research motivation and downstream reconstruction supported by supplied reference content.
The evaluated FIRE-Bench tasks focus on empirical ML research, whereas SCIONBench covers 53 papers across 15 scientific domains. For modes requiring experimentation, resource-dependent applicability decisions are fixed before execution and propagate to dependent conclusions, preserving evaluable branches of studies that cannot be executed in full. This supports cross-domain reconstruction evaluation under resource constraints.
3. SciNet (arXiv:2601.03260)
SciNet uses citation networks and citation contexts to evaluate relation-aware scientific literature retrieval. Its tasks cover retrieving papers by scholarly attributes such as novelty, identifying support, contradiction, or extension relationships between papers, and retrieving citation paths reflecting scientific evolution. It therefore evaluates scientific relationships beyond topical relevance.
Both works use relational structures, but their units and tasks differ. SciNet organizes papers and inter-paper relationships. SCIONBench represents claims and evidence units within a paper, with dependencies specifying how upstream content supports downstream construction, execution, or justification. Reconstruction targets are thus evaluated with their scientific conditions and evidential basis.
We will expand Related Work and revise the Introduction to acknowledge these contributions and clarify the overlaps, differences, and their evaluation significance.
Response on Additional Research Harnesses
Thank you for recommending these systems. Following your suggestion, we have added experiments with The AI Scientist-v2 (arXiv:2504.08066) and a research workflow built using VAILabs, the framework introduced in Virtual Laboratories (arXiv:2507.06271v1), alongside Claude Code and Codex, expanding the comparison to four harness configurations. Both added configurations use DeepSeek-V4-Pro and are compared with the original two harnesses using the same model. The added evaluations use the same frozen local resources, disclosure conditions, applicability masks, and node-scoring protocol as the original experiments.
1. Integration of the two research workflows
For The AI Scientist-v2, we retain the authors' experiment manager, experiment tree, candidate selection, code interpreter, and execution-feedback loop, supplying the scientific task and permitted resources for the current mode. The model proposes experimental plans and generates code; the interpreter executes the code; and the experiment manager determines subsequent research steps from the execution results and feedback. We replace manuscript writing with a research report generated from saved code, numerical results, tables, and execution outputs. Its internal stages govern the experimentation workflow and are distinct from SCIONBench's four independently initialized disclosure modes. Because our DeepSeek-V4-Pro configuration supports text input only, we disable The AI Scientist-v2's original visual-feedback loop. To keep the model and input modality consistent across configurations, we do not introduce an additional vision model.
For VAILabs, the framework provides modular components and plugin interfaces for organizing scientific research, while task-specific execution steps, module coordination, and loop control require a workflow definition. To enable these components to perform SCIONBench tasks, we reuse its workflow execution core and plugin interfaces to construct a supervisor-executor configuration. The executor selects research methods, conducts analyses, and inspects results; the supervisor reviews submitted intermediate findings and reports, requesting additional checks or revisions when needed. Both roles use DeepSeek-V4-Pro and maintain separate conversation histories. We also add long-context management. The supervisor is an internal research-workflow role and cannot access hidden reference answers or judge scores.
2. Results across four harnesses
All configurations in the table use DeepSeek-V4-Pro.
| Harness | Motivation | Method | Experiment | Conclusion | Overall |
|---|---|---|---|---|---|
| Claude Code | 3.97 | 5.51 | 6.59 | 9.01 | 6.27 |
| Codex | 3.66 | 5.44 | 6.86 | 9.03 | 6.25 |
| The AI Scientist-v2 | 1.72 | 3.28 | 4.37 | 8.88 | 4.56 |
| VAILabs (our workflow) | 3.53 | 4.88 | 6.43 | 8.98 | 5.96 |
Both added configurations, as adapted for SCIONBench, exhibit the pattern reported in Sec. 3.2.1: mean scores increase across the Motivation, Method, Experiment, and Conclusion modes. Their Conclusion scores are 8.88 and 8.98, compared with Motivation scores of 1.72 and 3.53, yielding gaps of 7.16 and 5.45 points, respectively. Under this evaluation protocol, forming conclusions from supplied upstream content and experimental observations yields higher reconstruction scores than reconstructing the scientific content supporting those conclusions from motivation alone.
This result is consistent with the capability gap emphasized in the paper: when reference methods, experimental designs, and observations are available, agents can synthesize conclusions relatively well; however, high Conclusion scores do not imply the ability to autonomously recover the investigation that produced the evidence. Despite differences in overall performance across workflows, the asymmetry between recovering conclusions and reconstructing the scientific content supporting them persists, indicating that this observation also holds for the two added research workflows.
Regarding execution outcomes, none of the 212 tasks run with our VAILabs-based workflow reached a terminal timeout; The AI Scientist-v2 had 26 tasks that timed out and were scored as zero.
These additional evaluations broaden the range of research workflows covered by SCIONBench and further illustrate the distinction emphasized in the paper: recovering conclusions from supplied evidence and reconstructing the research that scientifically supports those conclusions require separate evaluation. We will incorporate the integration configurations, execution-accounting details, and additional results into the revised manuscript.
Response to Reviewer soPi: Research Dependencies and Feasibility
Thank you for raising this important question about dependencies and feasibility in scientific research. We agree that accounting for the feasibility of downstream steps is essential when evaluating scientific agents.
This consideration is reflected in the original design of SCIONBench through its research graphs and applicability masks (Sections 2.1–2.3 and Appendix B.4). These mechanisms account for investigations that are well motivated but cannot be completed with the available resources.
Specifically, before evaluation, each experiment is assessed against the frozen inputs and computational budget. Experiments whose required evidence cannot be recovered because essential inputs or physical measurements are unavailable, or the necessary computation exceeds the budget, are marked NOT_EVALUABLE and excluded from scoring. In the Motivation, Method, and Experiment modes, this exclusion also applies to conclusions that depend on the unavailable evidence, following the explicit dependency edges. Other evaluable branches remain scored. In Conclusion mode, experimental observations are provided as context, so the assessment concerns reasoning from those supplied observations.
These applicability masks are fixed before evaluation: an agent’s failure or timeout on an otherwise evaluable task still receives a zero score. The protocol therefore distinguishes resource-related infeasibility and its downstream consequences from failures in agent execution. Our treatment of feasibility is scoped to the information and resources specified by the benchmark, and we will make its connection to research-stage dependencies clearer in the revised manuscript.
Additional experiments are ongoing, and we will follow up on your remaining comments separately.