acceptodds
Under review as a conference paper at ICLR 2027

SCIONBench: Evaluating Evidence-Preserving Research Trajectories from Motivation to Conclusion in Scientific Discovery

Abstract

A correct scientific conclusion can conceal an incomplete or unsupported research trajectory. Existing evaluations emphasize task outcomes, leaving the continuity of evidence across research stages insufficiently characterized. In order to address the problem, we introduce SCIONBench, a benchmark for evidence-preserving reconstruction of scientific research trajectories. SCIONBench represents 53 papers spanning 15 scientific domains and 131 research cases as source-grounded graphs linking motivation, method, experimental design, experiment, and conclusion. Four controlled disclosure modes progressively expose upstream information and assess reconstruction of held-out target nodes. Frozen evidence packages and explicit applicability decisions distinguish unavailable resources from execution failures, enabling stage-specific assessment under controlled information access. Node-level scoring assesses scientific correctness, experimental adequacy, evidence grounding, and conclusion scope. Across six model-agent configurations, agents recover conclusions more reliably when reference evidence is supplied than they reconstruct the investigation from motivation alone. On matched conclusion targets across 53 papers, evidence disclosure yields domain-macro gains of 2.68-2.88 points on a 0-10 scale for Claude Code configurations. However, providing a correct experimental design does not improve every independently executed continuation. These findings distinguish evidence-conditioned synthesis from reconstructing the investigation that supports a conclusion. SCIONBench provides a diagnostic framework for evaluating how scientific agents preserve the evidential dependencies that connect research stages.

Primary area:
datasets and benchmarks
Keywords:
Scientific Agents, Large Language Models (LLMs), Research Trajectories Reconstruction, AI for Science

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_soPiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a benchmark to evaluate whether scientific agents can reconstruct the evidence-supported stages of research. The proposed benchmark represents 53 papers spanning 15 scientific domains and 131 research cases. Averaged across all the domains, the authors observe that the agents can recover conclusions more reliably than the evidence that supports them.

Critical strengths
  1. The paper is clearly written and easy to understand.
  2. The benchmark consists of 15 different scientific domains, which ensures that the dataset is diverse enough.
  3. There's an interesting finding in Table 2, which shows that the Claude Code harness performs better than Codex in motivation and method
Critical weaknesses
  1. The paper, in its current form, is not well placed within the literature. While the authors have a dedicated related work section in the Appendix, they don't compare it with closely related works such as arXiv:2608.05235, arXiv:2602.02905v2, arXiv:2601.03260, etc which concerns the limited technical novelty.
  2. The comparison seems to not be fair. There are many more harnesses, e.g., The AI Scientist (arXiv:2504.08066), Virtual Laboratories (arXiv:2507.06271v1), etc that the authors would evaluate the benchmark on
Questions for authors
  1. Wondered if the authors also thought of treating this as a dependent process, meaning there could be a clear motivation for a problem, but it could be simply infeasible. Did the authors also consider looking into such problems? If so, what was the observation?

Also note that I am open to reconsidering my score. On a convincing answer to both weaknesses, I will consider adjusting my score to Weak Accept

3 comments
AuthorsOpenReview ↗

Response to Reviewer soPi: Research Dependencies and Feasibility

Thank you for raising this important question about dependencies and feasibility in scientific research. We agree that accounting for the feasibility of downstream steps is essential when evaluating scientific agents.

This consideration is reflected in the original design of SCIONBench through its research graphs and applicability masks (Sections 2.1–2.3 and Appendix B.4). These mechanisms account for investigations that are well motivated but cannot be completed with the available resources.

Specifically, before evaluation, each experiment is assessed against the frozen inputs and computational budget. Experiments whose required evidence cannot be recovered because essential inputs or physical measurements are unavailable, or the necessary computation exceeds the budget, are marked NOT_EVALUABLE and excluded from scoring. In the Motivation, Method, and Experiment modes, this exclusion also applies to conclusions that depend on the unavailable evidence, following the explicit dependency edges. Other evaluable branches remain scored. In Conclusion mode, experimental observations are provided as context, so the assessment concerns reasoning from those supplied observations.

These applicability masks are fixed before evaluation: an agent’s failure or timeout on an otherwise evaluable task still receives a zero score. The protocol therefore distinguishes resource-related infeasibility and its downstream consequences from failures in agent execution. Our treatment of feasibility is scoped to the information and resources specified by the benchmark, and we will make its connection to research-stage dependencies clearer in the revised manuscript.

Additional experiments are ongoing, and we will follow up on your remaining comments separately.

AuthorsOpenReview ↗

Response on Related Work and Novelty

Thank you for highlighting these closely related studies. We agree that the manuscript should better acknowledge their contributions and clarify SCIONBench's position.

SCIONBench evaluates evidence-preserving reconstruction of scientific research. Its five-stage, paper-grounded dependency graphs define reference targets for methods, experimental designs, evidence, and conclusions. Four independently initialized disclosure conditions enable paired comparisons of the same downstream targets evaluable in both conditions, examining how reconstruction changes with the supplied upstream reference information.

1. From Trajectories to Evidence (arXiv:2608.05235)

This work examines which findings are supported by multi-round experiments in industrial recommendation. It verifies and repairs research artifacts, checks execution validity and attribution, and retains supported claims with provenance and applicability boundaries for selective reuse in subsequent tasks.

Both works emphasize evidential support, but their evaluation objects differ. Its records qualify and preserve findings from executed trajectories; SCIONBench uses published papers across scientific domains to define reference targets for reconstructing the content and connections supporting conclusions. Our automatic node scoring evaluates the scientific content, evidential relationships, and conditions presented in the final research report, using reference nodes, scoring criteria, and relevant evidence.

2. FIRE-Bench (arXiv:2602.02905v2)

FIRE-Bench evaluates whether agents can independently design and execute experiments to rediscover published empirical findings from a research question. It provides experimental scope and evaluation requirements while withholding the authors' specific implementations and conclusions. Its primary metrics compare atomic empirical claims through precision, recall, and F1, complemented by execution-trace analysis of planning, implementation, execution, and conclusion errors.

Both works use published research as a reference and examine multi-stage scientific tasks. SCIONBench provides complementary evaluation capabilities in two respects.

Evaluation targets and the role of the reference structure. FIRE-Bench selects a central experiment and its parent question from a research-problem tree to instantiate a rediscovery task, with the published finding as the scoring reference. SCIONBench organizes methods, experimental designs, evidence, and conclusions into reference nodes linked by scientific dependencies. These stage-specific nodes directly define reconstruction and scoring targets. Consequently, the protocol separately assesses the recovery of conclusions and the scientific content supporting them; agreement with the final conclusion does not substitute for evaluating intermediate content.

Comparisons across information conditions. FIRE-Bench's trajectory analysis traces missed or incorrect findings back to research errors. SCIONBench additionally asks how reconstruction of the same downstream target changes when reference methods, designs, or observations are supplied. For example, the same experimental-design target is evaluated from motivation alone and from motivation plus the reference method, using independent runs with the same paper-local resources. This characterizes both reconstruction from the research motivation and downstream reconstruction supported by supplied reference content.

The evaluated FIRE-Bench tasks focus on empirical ML research, whereas SCIONBench covers 53 papers across 15 scientific domains. For modes requiring experimentation, resource-dependent applicability decisions are fixed before execution and propagate to dependent conclusions, preserving evaluable branches of studies that cannot be executed in full. This supports cross-domain reconstruction evaluation under resource constraints.

3. SciNet (arXiv:2601.03260)

SciNet uses citation networks and citation contexts to evaluate relation-aware scientific literature retrieval. Its tasks cover retrieving papers by scholarly attributes such as novelty, identifying support, contradiction, or extension relationships between papers, and retrieving citation paths reflecting scientific evolution. It therefore evaluates scientific relationships beyond topical relevance.

Both works use relational structures, but their units and tasks differ. SciNet organizes papers and inter-paper relationships. SCIONBench represents claims and evidence units within a paper, with dependencies specifying how upstream content supports downstream construction, execution, or justification. Reconstruction targets are thus evaluated with their scientific conditions and evidential basis.

We will expand Related Work and revise the Introduction to acknowledge these contributions and clarify the overlaps, differences, and their evaluation significance.

AuthorsOpenReview ↗

Response on Additional Research Harnesses

Thank you for recommending these systems. Following your suggestion, we have added experiments with The AI Scientist-v2 (arXiv:2504.08066) and a research workflow built using VAILabs, the framework introduced in Virtual Laboratories (arXiv:2507.06271v1), alongside Claude Code and Codex, expanding the comparison to four harness configurations. Both added configurations use DeepSeek-V4-Pro and are compared with the original two harnesses using the same model. The added evaluations use the same frozen local resources, disclosure conditions, applicability masks, and node-scoring protocol as the original experiments.

1. Integration of the two research workflows

For The AI Scientist-v2, we retain the authors' experiment manager, experiment tree, candidate selection, code interpreter, and execution-feedback loop, supplying the scientific task and permitted resources for the current mode. The model proposes experimental plans and generates code; the interpreter executes the code; and the experiment manager determines subsequent research steps from the execution results and feedback. We replace manuscript writing with a research report generated from saved code, numerical results, tables, and execution outputs. Its internal stages govern the experimentation workflow and are distinct from SCIONBench's four independently initialized disclosure modes. Because our DeepSeek-V4-Pro configuration supports text input only, we disable The AI Scientist-v2's original visual-feedback loop. To keep the model and input modality consistent across configurations, we do not introduce an additional vision model.

For VAILabs, the framework provides modular components and plugin interfaces for organizing scientific research, while task-specific execution steps, module coordination, and loop control require a workflow definition. To enable these components to perform SCIONBench tasks, we reuse its workflow execution core and plugin interfaces to construct a supervisor-executor configuration. The executor selects research methods, conducts analyses, and inspects results; the supervisor reviews submitted intermediate findings and reports, requesting additional checks or revisions when needed. Both roles use DeepSeek-V4-Pro and maintain separate conversation histories. We also add long-context management. The supervisor is an internal research-workflow role and cannot access hidden reference answers or judge scores.

2. Results across four harnesses

All configurations in the table use DeepSeek-V4-Pro.

HarnessMotivationMethodExperimentConclusionOverall
Claude Code3.975.516.599.016.27
Codex3.665.446.869.036.25
The AI Scientist-v21.723.284.378.884.56
VAILabs (our workflow)3.534.886.438.985.96

Both added configurations, as adapted for SCIONBench, exhibit the pattern reported in Sec. 3.2.1: mean scores increase across the Motivation, Method, Experiment, and Conclusion modes. Their Conclusion scores are 8.88 and 8.98, compared with Motivation scores of 1.72 and 3.53, yielding gaps of 7.16 and 5.45 points, respectively. Under this evaluation protocol, forming conclusions from supplied upstream content and experimental observations yields higher reconstruction scores than reconstructing the scientific content supporting those conclusions from motivation alone.

This result is consistent with the capability gap emphasized in the paper: when reference methods, experimental designs, and observations are available, agents can synthesize conclusions relatively well; however, high Conclusion scores do not imply the ability to autonomously recover the investigation that produced the evidence. Despite differences in overall performance across workflows, the asymmetry between recovering conclusions and reconstructing the scientific content supporting them persists, indicating that this observation also holds for the two added research workflows.

Regarding execution outcomes, none of the 212 tasks run with our VAILabs-based workflow reached a terminal timeout; The AI Scientist-v2 had 26 tasks that timed out and were scored as zero.

These additional evaluations broaden the range of research workflows covered by SCIONBench and further illustrate the distinction emphasized in the paper: recovering conclusions from supplied evidence and reconstructing the research that scientifically supports those conclusions require separate evaluation. We will incorporate the integration configurations, execution-accounting details, and additional results into the revised manuscript.

Reviewer_7S9LOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents 53 papers from 15 scientific domains as graphs connecting motivation, method, experimental design, experiments and conclusions. Four independent disclosure methods expose different amounts of original information and ask the agents to reconstruct the remaining content. The benchmark uses frozen resources and applicability masks, then evaluates the final reports in a node level. The paper shows that agents perform better when the upstream method, design, and observations are provided.

Critical strengths
  1. The benchmark protocol handles unavailable resources explicitly, it excludes experiments that cannot be performed within the resources available, and further propagates these exclusions to conclusions. This is a useful design for evaluating scientific tasks whose physical or computational requirements cannot all be reproduced in the same environment.

  2. Matched comparisons provide evidence that upstream information helps. The analysis compares the same downstream targets within each paper. This control supports the finding that supplying the reference method, design, or observations improves reconstruction. The separate blinded audit also gives the evaluation empirical support: two domain-matched experts score sampled outputs from 20 papers, with a third expert resolving major disagreements.

Critical weaknesses
  1. The results do not locate a bottleneck in the early stage scientific reasoning. For example, Motivation mode requires an agent to propose a method, design and execute experiments, and derive conclusions. Conclusion mode supplies the preceding work and observations and explicitly avoids new experiments. These tasks differ in workload and scientific information available. So the conclusion might not be fair.

  2. It is unclear whether the scoring verifies execution and evidence source. The judge does not receive intermediate tool traces, and it is unclear how the results reported by agents are checked against the actual artifacts.

  3. The reported scores do not explain what drives the measured failures. Table 2 reports only aggregate scores. Those averages combine semantic quality with zeros for execution failure. The paper-level disclosure plots mark timeout pairs, which helps, but do not provide the completion and failure breakdown needed to explain the overall gaps.

Questions for authors
  1. what execution evidence is actually available to the judge?

  2. Do the stage gaps remain after separating completion from quality?

3 comments
AuthorsOpenReview ↗

Response on Workload and Information Availability Across Modes

We thank you for highlighting the differences in workload and available scientific information across evaluation modes. We agree that these factors may jointly affect agent performance, and therefore, differences in aggregate scores across modes cannot be directly attributed to bottlenecks in early-stage scientific reasoning.

However, we would like to clarify that SCIONBench does not aim to construct tasks with identical workloads or information availability across its four disclosure modes. Instead, we intentionally adopt progressive disclosure of upstream scientific information to evaluate agents' ability to reconstruct research trajectories under different information conditions. We believe that variations in workload and available information are inherent to different stages of scientific research. Our goal is to examine whether scientific agents can maintain consistent research performance despite these variations.

To investigate how these differences affect evaluation scores, we conducted matched comparisons on the same papers and target nodes in Section 3.2.2 (Figures 5–6). The results show that, when downstream evaluation targets are held fixed, disclosing reference methods, experimental designs, and experimental observations consistently improves agents' reconstruction performance on average. These findings indicate that downstream reconstruction performance is closely associated with the availability of upstream scientific information, rather than merely reflecting differences in aggregate scores across modes. Nevertheless, we acknowledge that these comparisons cannot fully disentangle the effects of increased information availability from reduced workload.

We further analyzed the average task execution time across evaluation modes to assess the reasonableness of the workload.

Average Task Execution Time (hours/task)

ConfigurationMotivationMethodExperimentConclusion
GLM-5.2 + Codex3.894.124.600.07
DeepSeek-V4-Pro + Claude Code3.173.794.460.19
Kimi K3 + Claude Code5.768.158.335.08
GPT-5.6 Sol + Codex5.656.909.034.48
Claude Sonnet 5 + Claude Code3.533.603.510.74
DeepSeek-V4-Pro + Codex9.9512.8013.914.01

SCIONBench imposes a uniform 18-hour execution limit per task. Across the six model–agent configurations, the average execution times for Motivation, Method, Experiment, and Conclusion modes are 5.33, 6.56, 7.31, and 2.43 hours, respectively. All configurations have average execution times below the 18-hour limit in every mode.

Notably, Motivation is not the most time-consuming mode. All six configurations exhibit longer average execution times in Experiment mode than in Motivation mode. This suggests that actual execution workload does not simply increase with the number of research stages to be reconstructed. Although workloads differ across modes, these results indicate that the 18-hour budget is generally sufficient and does not appear to be a systematic constraint. Nevertheless, some tasks may still exceed the limit due to limitations in agents' long-horizon planning, experimental execution, or tool management capabilities.

Therefore, we will further clarify the scope of our conclusions. Our findings reveal the substantial challenges current scientific agents face in reconstructing end-to-end research trajectories from limited initial information, as well as their dependence on upstream scientific information. However, they do not establish that early-stage scientific reasoning is the sole or an independently isolated bottleneck. We will revise the corresponding statements in the manuscript to reflect this distinction more precisely.

AuthorsOpenReview ↗

Response on Execution Traceability and Evidence Consistency

We sincerely thank you for raising this concern regarding the authenticity of experimental execution and evidence traceability. We fully agree that evaluating only the scientific quality of final research reports is insufficient to verify whether the reported experimental results are supported by actual execution evidence. We therefore clarify SCIONBench's evaluation design and provide an additional report–execution trace consistency analysis.

Regarding the evidence accessible to the Judge and its evaluation responsibilities, SCIONBench intentionally separates scientific quality assessment of final reports from auditing the underlying execution traces. In the current evaluation protocol, the Judge receives the agent's final research report, reference answers from the original paper, node-specific scoring criteria, and relevant reference evidence, but does not have access to intermediate tool-call traces or execution logs. This design aims to approximate scientific peer review, where reviewers primarily assess the correctness, experimental adequacy, evidence grounding, and scope of scientific conclusions based on the submitted report and its supporting arguments and evidence. Accordingly, the primary scores reflect the scientific quality of the agent's final report rather than independently verifying the authenticity of every experimental execution. We agree that whether reported experimental results are supported by actual execution is another important capability that warrants separate evaluation.

To further examine execution authenticity, we conducted an independent report–execution trace consistency audit across all 1,272 evaluation tasks. We used GPT-6 Sol to cross-check agents' final reports against their actual execution traces, focusing on whether reported numerical results could be traced to execution outputs and whether they were consistent with the recorded results. The audit identified 474 tasks (37.3%) with aligned reports and traces, 786 tasks (61.8%) with partially supported results that could not be fully reconciled with execution traces, and 12 tasks (0.9%) without valid reports. We also randomly sampled 90 of the partially supported tasks for manual verification, confirming the presence of the inconsistencies identified by the automated audit.

Model–Agent ConfigurationTotal TasksTrace-AlignedPartially Trace-SupportedNo Valid Report
GLM-5.2 + Codex212551525
DeepSeek-V4-Pro + Codex212831290
GPT-5.6 Sol + Codex212179303
DeepSeek-V4-Pro + Claude Code212691403
Kimi K3 + Claude Code212541571
Claude Sonnet 5 + Claude Code212341780
Total1,27247478612

Further analysis of the partially trace-supported tasks reveals that the main issues involve numerical misreporting and incorrect evidence attribution. Among the 786 tasks, 649 contained numerical results that did not match the actual execution outputs, 18 reported numerical results without corresponding execution evidence, and 119 exhibited both types of inconsistencies. Representative cases include reporting an extreme value from a single experiment as an average, confusing metrics obtained under different experimental conditions, and citing outdated results that had been superseded by subsequent experiments. These findings indicate that, even when agents successfully execute experiments, they may incorrectly use or interpret existing results when synthesizing and writing their research reports. We consider such numerical hallucinations and evidence misattribution to be important aspects of scientific agent reliability that warrant more fine-grained evaluation in future work.

We also observed substantial differences in report–trace consistency across model–agent configurations. Notably, GPT-5.6 Sol + Codex achieved trace alignment in 179 of 212 tasks (84.4%), substantially exceeding the other five configurations (34–83 aligned tasks) and exhibiting the lowest proportion of partially trace-supported reports. This suggests that different model–agent configurations vary considerably in their ability to faithfully preserve and report actual experimental results.

We thank the reviewer again for highlighting this important issue. We believe that the scientific quality of final research reports and the execution traceability of their reported results are related but distinct evaluation dimensions. While the primary evaluation focuses on the former, our additional report–trace consistency audit provides complementary evidence on the latter and reveals specific failure modes. In the revised manuscript, we will clarify the Judge's evidence access and further discuss numerical misreporting and evidence inconsistencies in scientific reports to improve the transparency and interpretability of SCIONBench's evaluation results.

AuthorsOpenReview ↗

Response on Execution Failures and Stage-Wise Performance Gaps

We thank you for highlighting the potential confounding between scientific content quality and execution failures in the evaluation scores. We agree that, in our original evaluation, some tasks may have received zero scores due to execution timeouts or unexpected termination. Consequently, aggregate scores that include these zeros may not adequately distinguish agents' scientific reconstruction capabilities from their execution reliability. To address this concern, we further analyzed the impact of zero-score tasks and conducted additional comparisons across disclosure modes.

1. The stage-wise performance gaps observed in the original paper remain substantial after excluding zero-score tasks. Across all six model–agent configurations, the scores consistently increase from Motivation through Method and Experiment to Conclusion. Specifically, the average scores range from 2.56 to 3.97 in Motivation mode and from 8.65 to 9.03 in Conclusion mode, leaving gaps of 5.04–6.09 points. Moreover, among the 1,272 paper-level evaluation tasks, 1,256 (98.74%) received non-zero scores. Notably, all 212 tasks in each of the Claude Sonnet 5 + Claude Code and DeepSeek-V4-Pro + Codex configurations received non-zero scores, yet both configurations still exhibit substantial stage-wise performance differences. These results indicate that the observed performance gaps cannot be explained solely by zero-score tasks and persist in the reconstruction quality of non-zero-score tasks.

Model–Agent ConfigurationMotivationMethodExperimentConclusionOverall
GLM-5.2 + Codex2.56 (51)4.13 (50)5.85 (50)8.65 (52)5.32 (203)
DeepSeek-V4-Pro + Claude Code3.97 (53)5.51 (53)6.99 (50)9.01 (53)6.36 (209)
Kimi K3 + Claude Code3.76 (53)5.01 (53)6.58 (52)8.88 (53)6.05 (211)
GPT-5.6 Sol + Codex2.96 (52)4.04 (52)5.68 (52)8.87 (53)5.41 (209)
Claude Sonnet 5 + Claude Code3.26 (53)4.71 (53)6.26 (53)8.78 (53)5.75 (212)
DeepSeek-V4-Pro + Codex3.66 (53)5.44 (53)6.86 (53)9.03 (53)6.25 (212)

Numbers in parentheses indicate the number of tasks with non-zero scores.

2. An additional matched comparisons on the same downstream target nodes after excluding zero-score tasks. Across all six model–agent configurations, all comparison categories show positive average gains, ranging from 0.89 to 3.00 points. These findings further indicate that, even among non-zero-score tasks, disclosing upstream reference methods, experimental designs, and experimental observations improves agents' ability to reconstruct the same downstream scientific content.

ConfigurationDesign: M−MotivationExperiment: M−MotivationConclusion: M−MotivationExperiment: E−MConclusion: E−MConclusion: C−E
GLM-5.2 + Codex+2.08 (50)+1.97 (50)+1.82 (41)+1.89 (50)+1.73 (41)+2.56 (41)
DeepSeek-V4-Pro + Claude Code+1.98 (53)+1.77 (53)+1.57 (43)+1.58 (50)+1.18 (40)+1.82 (40)
Kimi K3 + Claude Code+1.73 (53)+1.51 (53)+1.39 (43)+1.66 (52)+1.30 (42)+2.08 (42)
GPT-5.6 Sol + Codex+1.53 (52)+1.23 (52)+1.06 (42)+1.83 (52)+1.57 (42)+3.00 (42)
Claude Sonnet 5 + Claude Code+1.75 (53)+1.62 (53)+1.50 (43)+1.55 (53)+1.24 (43)+2.38 (43)
DeepSeek-V4-Pro + Codex+2.12 (53)+2.09 (53)+2.16 (43)+1.70 (53)+0.89 (43)+2.19 (43)

Values represent average score gains on matched downstream target nodes. Numbers in parentheses indicate the number of valid matched paper pairs. M, E, and C denote the Method, Experiment, and Conclusion disclosure modes, respectively.

3. The major types of task-level execution failures. We identified two primary failure modes:

(1) Execution timeout, where an agent spends an extended period running experiments or tests and fails to complete the task and submit a report within the specified time limit.

(2) Agent-induced premature termination, where an agent inadvertently executes a termination command that affects its own process, causing the task to end prematurely.

These failures primarily reflect limitations in research task planning and execution, tool use, and process management, and should indeed be distinguished from scientific content reconstruction quality.

We thank you again for this valuable suggestion. Our additional analyses demonstrate that, although execution failures may affect aggregate scores, the stage-wise performance gaps and the positive gains from upstream information disclosure persist after excluding zero-score tasks. We will incorporate these analyses into the revised manuscript and more explicitly distinguish task execution status from scientific content scores to provide a clearer interpretation of our experimental findings.

Reviewer_jj9uOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper builds a benchmark from roughly 50 published papers in 15 fields by splitting the paper into five stages: motivation, method, experiment design, experiment and conclusion.

They do quite an interesting experiment: for each step, the agent is shown the paper up to that step and asked to fill in the rest. so the the agent sees only the motivation, then up to the method, then up to the design, and then everything including the results.

they use LLM judge (k3 model) to score the agent's answer against that of the real paper, from 0 to 10. they test six agents incl. claude code and codex with different models or versions.

Their experiment is a good way to show that ai for science is bottlenecked by the early stages of research (028-029).

Critical strengths

i like the simplicity of the idea.

The agent gets more and more of a paper and is asked to do the rest from that point. seems simple but very interesting premise to work out of cause it keeps the question fixed and only changes how much the agent is shown.

Critical weaknesses
  1. I would like the authors to add a score in the paper that doesn't depend on how close the agent's answer to that of the paper.

A really great check would actually to find if the AI agent's method surpasses that of the paper, or that it given a motivation it deviates from the human paper and comes up with a completely new method, design and experiments etc. Given a motivation, no two human researchers will come up with the same method/design/experiments. they will have different journeys and the journey is judged based on how it pushes the field forward not how close they are in semantic space. Think of this as coming up with metrics that can capture the "open-ended" process of research. you have the human "open-ended" search in the "final form" as the paper itself.

Human knowledge is not what the agent should be asked to recover, it should be surpass human knowledge.

  1. It would be good for the authors to figure out if the models have already seen the paper since they are published in well known journals. An ablation with papers after the cutoff date would be good to show as well.
Questions for authors
  1. The main critical weakness i see is weakness 1. I would like the authors to talk about this in the rebuttal phase and see if they can actually come up with open ended assessments that calculate the divergence, or the judges if the new trajectory was something new entirely or opened up a new direction that the paper didn't. Think about these artificial trees as "trees of life" the divergence and open ended areas those trees take is >>> if they match with human trajectories.

If you can't do this, please write this aspect in the limitations section. tbh i would want the authors to actually display this information as a tree of life kind of structure. thanks you.

My score isn't hinging on the simplicity of the method, it's that the authors could do a lot more analysis with this simple premise and the dataset that they have collected.

  1. Ablation that i ask in weakness.
2 comments
AuthorsOpenReview ↗

Response on Open-Ended Scientific Evaluation

We sincerely thank you for this important and insightful suggestion. We fully agree that scientific research is inherently an open-ended exploratory process. Even when starting from the same research motivation, different researchers may pursue distinct research trajectories and develop methods and experiments with independent scientific value. Therefore, a comprehensive evaluation of scientific agents should not be limited to their ability to reconstruct existing research, but should also assess their capacity to explore new directions, develop novel methods, and potentially contribute to scientific discovery. We also appreciate the reviewer’s analogy to the “tree of life,” which offers a valuable perspective for characterizing the branching structure and open-ended exploration of scientific research trajectories.

We would like to further clarify the current scope of SCIONBench. Our primary objective is to evaluate whether agents can reconstruct the research process from motivation to conclusion under controlled information disclosure, while preserving scientific reasoning continuity and evidential dependencies across research stages. Accordingly, our evaluation does not rely solely on textual or semantic similarity between agent outputs and the original papers. Instead, it assesses reconstruction quality across multiple dimensions, including scientific correctness, experimental adequacy, evidence grounding, and conclusion scope. Nevertheless, we acknowledge that, because our evaluation is still grounded in the research nodes and scientific evidence of the original papers, the current framework cannot adequately assess alternative research trajectories that diverge from the original studies but may possess independent scientific value.

Regarding the proposed open-ended evaluation, we fully recognize its importance. However, we have not yet identified a sufficiently reliable and well-validated metric that can effectively determine whether an agent-generated research trajectory demonstrates genuine scientific novelty, explores directions not investigated in the original paper, or makes substantive scientific contributions. Importantly, divergence between research trajectories does not necessarily imply scientific innovation, nor does the emergence of a new research branch necessarily indicate greater scientific value. Therefore, relying solely on semantic differences, structural divergence, or the number of branches may fail to accurately capture the actual value of open-ended scientific exploration and could even lead to misleading assessments.

We recognize this as an important direction for future research in scientific agent evaluation. Given the current methodological and validation limitations, we are not yet able to propose a sufficiently rigorous and trustworthy open-ended evaluation metric within the scope of this work. Following the reviewer’s suggestion, we will explicitly acknowledge this limitation in the Limitations section and discuss potential future approaches that combine structural analysis of research trajectories, scientific novelty assessment, and domain-expert evaluation to better assess agents’ open-ended scientific exploration capabilities.

We sincerely thank the reviewer again for this constructive suggestion. It has helped us more clearly define the scope of our current work and provided valuable guidance for extending the evaluation capabilities of SCIONBench in future research.

AuthorsOpenReview ↗

Response on Pretraining Data Contamination and Ablation Study

We thank you for raising the important concern of potential data contamination from model pretraining. We agree that, since SCIONBench is constructed from published scientific papers, some of these papers may have appeared in the models' pretraining data. This could allow agents to reconstruct research content using memorized knowledge rather than relying solely on scientific reasoning during evaluation.

To investigate this potential issue, we conducted an additional ablation study using papers published after the models' knowledge cutoff dates. After examining the relevant dates of all six models, we adopted August 13, 2026, as a conservative, unified cutoff and collected 5 newly published scientific papers after this date. We evaluated them using the same dataset construction and evaluation protocol as SCIONBench, testing three model–agent configurations: DeepSeek-V4-Pro + Claude Code, DeepSeek-V4-Pro + Codex, and GPT-5.6 Sol + Codex.

The results show that the stage-wise performance gaps observed in SCIONBench persist on these newly collected papers. Across all three configurations, the average scores increase consistently from Motivation to Conclusion, following the same pattern as in the original experiments.

DeepSeek-V4-Pro + Claude Code

ModeOriginal PapersNew PapersDifference (New − Original)
Motivation3.974.38+0.41
Method5.515.90+0.39
Experiment6.596.35−0.24
Conclusion9.018.49−0.52

DeepSeek-V4-Pro + Codex

ModeOriginal PapersNew PapersDifference (New − Original)
Motivation3.662.73−0.93
Method5.444.21−1.23
Experiment6.865.92−0.94
Conclusion9.039.15+0.12

GPT-5.6 Sol + Codex

ModeOriginal PapersNew PapersDifference (New − Original)
Motivation2.903.63+0.73
Method3.965.58+1.62
Experiment5.586.57+0.99
Conclusion8.879.19+0.32

These results indicate that, across all three model–agent configurations, agents still struggle to reconstruct complete research trajectories under limited upstream information, while their reconstruction performance improves as more upstream scientific information becomes available, particularly when drawing conclusions from experimental evidence. This is consistent with the main findings of the original experiments.

We acknowledge that filtering papers by publication date cannot completely rule out the possibility that models encountered their content through other sources. Moreover, this additional study includes only five papers and cannot replace a larger-scale independent evaluation. Nevertheless, the results provide further empirical evidence that the stage-wise performance gaps observed in SCIONBench are not limited to the original paper collection but also persist on more recently published scientific studies.

In the revised manuscript, we will include the paper selection criteria, model–agent configurations, task completion statistics, and detailed results of this ablation study. We will also explicitly discuss the potential risks of pretraining data contamination and the limitations of the current evaluation. We thank the reviewer again for this valuable suggestion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the Area Chair and all reviewers for their thoughtful feedback and constructive suggestions. To address the concerns raised, we conducted additional analyses and experiments.

1. Open-ended scientific exploration and evaluation scope. We agree on the ultimate goal. Scientific agents should be evaluated not only on reconstructing existing research. They should also be evaluated on discovering novel research directions. That said, SCIONBench focuses on one specific setting. It evaluates evidence-preserving reconstruction of scientific research trajectories under controlled information disclosure. Our evaluation considers scientific correctness, experimental adequacy, evidence grounding, and conclusion scope. It does not rely merely on semantic similarity. We acknowledge, however, that it cannot adequately assess alternative research trajectories with independent scientific value. We will explicitly discuss this limitation in the revised manuscript. We will also discuss potential directions for open-ended evaluation.

2. Pretraining data contamination. We investigated potential contamination from previously published papers. We conducted an additional evaluation using 5 papers published after August 13, 2026. This evaluation covered three model–agent configurations. All three configurations reproduced the original stage-wise performance pattern. Scores consistently increased from Motivation to Conclusion. These results provide additional evidence. The observed performance gaps persist on recently published research.

3. Workload differences across evaluation modes. We acknowledge that the 4 disclosure modes differ in workload and in the available scientific information. These differences are intentional. SCIONBench aims to assess research reconstruction under progressively disclosed upstream evidence. Our matched comparisons use identical downstream target nodes (Section 3.2.2). They demonstrate consistent performance improvements with additional upstream information. We further analyzed task execution times across all 6 configurations. Average durations were 5.33, 6.56, 7.31, and 2.43 hours for Motivation, Method, Experiment, and Conclusion modes, respectively. All were below the 18-hour task limit.

4. Execution failures versus scientific reconstruction quality. We addressed the concern that execution failures may distort aggregate scores. We reanalyzed performance after excluding zero-score tasks. Among 1,272 evaluation tasks, 1,256 (98.74%) received non-zero scores. The stage-wise performance gaps remained substantial across all 6 configurations. Motivation–Conclusion differences ranged from 5.04 to 6.09 points. Additional matched comparisons showed positive average gains across all 6 configurations and 6 comparison categories. These gains ranged from 0.89 to 3.00 points. We further identified 2 major task-level failure modes. They were execution timeouts and agent-induced premature termination. These findings indicate that zero-score tasks alone cannot explain the observed performance differences.

5. Execution evidence and report–trace consistency. We clarified the role of SCIONBench's primary Judge. It evaluates final research reports against reference answers and supporting scientific evidence. It does not access intermediate execution traces. To complement this evaluation, we conducted an independent report–execution trace consistency audit. The audit covered all 1,272 tasks and used GPT-6 Sol Max. We also manually verified 90 sampled cases. The audit identified 474 trace-aligned tasks, 786 partially trace-supported tasks, and 12 tasks without valid reports. Further analysis revealed the main sources of inconsistency. They were primarily numerical misreporting and incorrect evidence attribution. Notably, GPT-5.6 Sol + Codex achieved the highest report–trace alignment (179/212 tasks, 84.4%). These findings reveal important differences in agents' ability to faithfully preserve and report experimental evidence. They complement the original scientific reconstruction scores.

Overall, our additional experiments support the robustness of SCIONBench's central findings. Scientific agents perform substantially better when upstream research information is available. Reconstructing complete research trajectories from limited initial information remains challenging. At the same time, we recognize a limitation. These results do not establish early-stage scientific reasoning as an independently isolated bottleneck. We will revise the manuscript to clarify this distinction. We will incorporate the additional analyses. We will also explicitly acknowledge the limitations of the current framework.

We thank the reviewers for their feedback. It has helped us strengthen the empirical evidence. It has improved the transparency of our evaluation protocol. It has also helped us more precisely characterize the contributions and limitations of SCIONBench.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.