acceptodds
Under review as a conference paper at ICLR 2027

ScienceIDE: Turning World’s Scientific Codebase into Agent Learnable Environments

Abstract

Scientific code repositories encode decades of human knowledge in executable models, methods, and tools. Yet fragmented toolchains, implicit domain conventions, and specialized correctness criteria make this knowledge difficult to convert into reliable learning experience—a challenge we call the scientific experience bottleneck. We introduce ScienceIDE, infrastructure for turning the world's scientific code into programmable environments for scientific agents. Guided by expert-defined scientific cases and acceptance criteria, agents transform repositories into executable environments that support task generation, execution, and scientific verification. These environments provide a shared foundation for supervised fine-tuning, reinforcement learning, and evaluation. Using verified interaction trajectories, we train PhAI-IDE-72B, PhAI-IDE-9B, and PhAI-IDE-4B. The model family shows gains in held-out scientific-code repair and across selected general-purpose benchmarks in code, reasoning, and knowledge, providing evidence of positive transfer from scientific experience to broader capabilities. ScienceIDE lays the foundation for an integrated workspace for agent learning and scientific practice, making humanity's scientific software a shared substrate for developing scientific intelligence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI for Discovery

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tJTwOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ScienceIDE, an infrastructure for converting scientific software repositories into executable environments for training and evaluating AI agents. The authors argue that scientific experience is a bottleneck for developing discovery-oriented intelligence, because existing scientific codebases contain complex dependencies, implicit conventions, and domain-specific notions of correctness that are difficult to directly use as agent learning environments.

ScienceIDE addresses this problem by packaging scientific modules with executable runtimes, expert-defined scientific checks, and task factories that generate validated tasks. The resulting environments provide trajectories, verifier-based rewards, and evaluation protocols that can be used for supervised fine-tuning, reinforcement learning, and agent benchmarking.

The authors construct a registry containing 64 environments from 27 scientific codebases and 2,812 generated tasks. They evaluate frontier agents on ScienceIDE-Hard (85 tasks from 18 environments), showing that current agents solve only a subset of scientific software tasks. They further demonstrate that trajectories collected from ScienceIDE can improve public benchmark performance through supervised fine-tuning and can provide verifier-based rewards for reinforcement learning on scientific repair tasks.

Critical strengths
  1. The paper addresses an important and timely problem: how to create scalable, executable experience for scientific AI agents. The proposed abstraction of combining scientific repositories, expert-defined verification contracts, task generation, and learning interfaces is conceptually valuable and extends the idea of executable environments beyond conventional coding tasks.

  2. The empirical evaluation is relatively comprehensive. The authors evaluate many frontier models on a challenging scientific software benchmark and demonstrate both supervised fine-tuning and reinforcement learning use cases, showing that the proposed infrastructure can support multiple learning paradigms.

Critical weaknesses
  1. The paper's central motivation emphasizes discovery-oriented intelligence, but the demonstrated tasks are predominantly software engineering tasks (repair and implementation). Most experiments evaluate whether agents can modify existing scientific code rather than whether they can formulate hypotheses, design experiments, or generate new scientific knowledge. Therefore, the connection between ScienceIDE and autonomous scientific discovery remains largely aspirational.

2.The proposed framework relies heavily on predefined scientific checks and acceptance criteria. While this enables reliable evaluation for repair and implementation tasks, it may fundamentally limit the ability of ScienceIDE to support open-ended scientific discovery, where multiple hypotheses, unexpected outcomes, or novel mechanisms may not have predefined observables or reference solutions. Therefore, it remains unclear whether the proposed verification paradigm can extend beyond reference-based software engineering toward genuinely exploratory scientific reasoning.

Questions for authors
  1. The paper emphasizes a bidirectional loop between Science and AI, where scientific environments improve AI capabilities and improved agents in turn contribute back to science. However, the current experiments mainly validate the first direction (using scientific experience for SFT/RL), while the second direction appears largely conceptual. Can the authors provide empirical evidence that agents trained with ScienceIDE can improve scientific workflows, generate useful scientific hypotheses, or create new validated scientific experiences?

  2. The SFT and RL studies evaluate transferability in substantially different ways. SFT is evaluated on public benchmarks outside the training environments, whereas RL improvements are only demonstrated on held-out tasks within the same scientific environments and do not test transfer across codebases. Since a central motivation of ScienceIDE is to provide reusable scientific experience, can the authors clarify whether RL learns environment-specific repair strategies or transferable scientific agent capabilities? What evidence supports generalization beyond the training repositories?

1 comment

Since I have not received the rebuttal, I will retain my original score for the time being.

Reviewer_19eBOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces ScienceIDE, an infrastructure that translates scientific expertise into executable environments, providing verifiable tasks and feedback for training and evaluating AI agents.

Critical strengths
  1. The paper is well motivated by the scientific experience bottleneck and the need to make scientific expertise reusable for agent development.

  2. The dataset covers a broad range of scientific domains, making it a potentially useful resource for training and evaluating scientific agents.

Critical weaknesses
  1. The writing lacks clarity and precision, with several terms and symbols introduced without adequate definitions. For example, line 207 presents the criterion ∣c−r∣≤a+ρ∣r∣|c-r| \leq a+\rho|r| without explicitly defining cc, rr, aa, or ρ\rho. Such omissions make the technical description unnecessarily difficult to follow.

  2. The methodological novelty appears limited. The proposed pipeline seems to rely primarily on prompting agents to organize existing scientific resources, such as GitHub repositories, into executable environments and training data. The paper does not clearly articulate what methodological innovations distinguish this approach from a straightforward application of existing agent capabilities.

  3. The SFT evaluation compares fine-tuned models against their initial checkpoints, without a baseline trained on general coding or agent trajectories under matched data and compute budgets. Consequently, it remains unclear whether the reported gains reflect the value of scientific expertise and verification, or simply additional training on high-quality teacher-generated demonstrations.

Questions for authors

NA

Reviewer_EtUiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ScienceIDE, an infrastructure that turns scientific codebases into executable environments for evaluating and training coding agents. Experts define scientific checks first, then task factories generate repair and implementation tasks, which are kept only if they pass execution-based validation.

Critical strengths
  • The idea of fixing scientific checks before generating tasks is reasonable and addresses a real problem as unit tests often do not capture scientific correctness. Even though similar ideas has been used in some other new works, this is a meaningful effort for the validity of the data generation.

  • An unsaturated and large scale new benchmark for scientific discovery and training code agents would is a meaninful contribution -The failure analysis shows that strong models mostly fail not because their code doesn’t run, but because they miss scientific conventions of the codebase or edit the wrong place. This is a useful insight about what makes scientific coding hard.

Critical weaknesses

1- No code, environments or benchmark are released. There is no link or clear release plan for codebase. Given this paper is mostly proposing a sound engineering pipeline, without accessing and reviewing the code and the resulting benchmark, it is impossible for the reviewer to validate the claims and evaluate the quality and significance of the work.

2-The contribution is mainly engineering, so it depends on the artifact. The method (inject bugs or remove code, then validate by execution) follows SWE-smith and R2E-Gym. The new part is the expert-calibrated checks and the environments themselves. Without access to them, the main contribution cannot be verified or used by the community.

Questions for authors

How many of the 2812 tasks are fully validated under the current pipeline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.