- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ScienceIDE, an infrastructure for converting scientific software repositories into executable environments for training and evaluating AI agents. The authors argue that scientific experience is a bottleneck for developing discovery-oriented intelligence, because existing scientific codebases contain complex dependencies, implicit conventions, and domain-specific notions of correctness that are difficult to directly use as agent learning environments.
ScienceIDE addresses this problem by packaging scientific modules with executable runtimes, expert-defined scientific checks, and task factories that generate validated tasks. The resulting environments provide trajectories, verifier-based rewards, and evaluation protocols that can be used for supervised fine-tuning, reinforcement learning, and agent benchmarking.
The authors construct a registry containing 64 environments from 27 scientific codebases and 2,812 generated tasks. They evaluate frontier agents on ScienceIDE-Hard (85 tasks from 18 environments), showing that current agents solve only a subset of scientific software tasks. They further demonstrate that trajectories collected from ScienceIDE can improve public benchmark performance through supervised fine-tuning and can provide verifier-based rewards for reinforcement learning on scientific repair tasks.
Critical strengths
-
The paper addresses an important and timely problem: how to create scalable, executable experience for scientific AI agents. The proposed abstraction of combining scientific repositories, expert-defined verification contracts, task generation, and learning interfaces is conceptually valuable and extends the idea of executable environments beyond conventional coding tasks.
-
The empirical evaluation is relatively comprehensive. The authors evaluate many frontier models on a challenging scientific software benchmark and demonstrate both supervised fine-tuning and reinforcement learning use cases, showing that the proposed infrastructure can support multiple learning paradigms.
Critical weaknesses
- The paper's central motivation emphasizes discovery-oriented intelligence, but the demonstrated tasks are predominantly software engineering tasks (repair and implementation). Most experiments evaluate whether agents can modify existing scientific code rather than whether they can formulate hypotheses, design experiments, or generate new scientific knowledge. Therefore, the connection between ScienceIDE and autonomous scientific discovery remains largely aspirational.
2.The proposed framework relies heavily on predefined scientific checks and acceptance criteria. While this enables reliable evaluation for repair and implementation tasks, it may fundamentally limit the ability of ScienceIDE to support open-ended scientific discovery, where multiple hypotheses, unexpected outcomes, or novel mechanisms may not have predefined observables or reference solutions. Therefore, it remains unclear whether the proposed verification paradigm can extend beyond reference-based software engineering toward genuinely exploratory scientific reasoning.
Questions for authors
-
The paper emphasizes a bidirectional loop between Science and AI, where scientific environments improve AI capabilities and improved agents in turn contribute back to science. However, the current experiments mainly validate the first direction (using scientific experience for SFT/RL), while the second direction appears largely conceptual. Can the authors provide empirical evidence that agents trained with ScienceIDE can improve scientific workflows, generate useful scientific hypotheses, or create new validated scientific experiences?
-
The SFT and RL studies evaluate transferability in substantially different ways. SFT is evaluated on public benchmarks outside the training environments, whereas RL improvements are only demonstrated on held-out tasks within the same scientific environments and do not test transfer across codebases. Since a central motivation of ScienceIDE is to provide reusable scientific experience, can the authors clarify whether RL learns environment-specific repair strategies or transferable scientific agent capabilities? What evidence supports generalization beyond the training repositories?
Since I have not received the rebuttal, I will retain my original score for the time being.