- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SciEnBuilder, a framework that automatically constructs a reliable scientific research environment from pairs of papers and repositories by linking scientific claims, verifiable experimental evidence, and protected evaluation mechanisms. Using this framework, this paper constructed 30 types of scientific research environments for Coding Agents from a large collection of top-tier academic papers. Furthermore, through manual evaluation, this paper confirmed that the constructed environments remain faithful to the scientific objectives of the original papers and that the evaluation mechanisms are reliable. Additionally, this paper demonstrated that utilizing these environments enables improved research efficiency, the elimination of observed shortcut behaviors, and the achievement of more advanced long-term optimization.
Critical strengths
-
S1. Automation of Environment Preparation. In previous studies, environment preparation has mainly been done manually or has not been considered in detail. Therefore, proposing an automated method for this process is a meaningful contribution.
-
S2. Quantitative Demonstration of Efficiency Improvement. The paper quantitatively shows how much efficiency can be improved simply by preparing the environment. This result highlights the importance of environment preparation for future research.
Critical weaknesses
-
W1. The novelty and significance of the contribution are limited: The main contribution of this study is the automatic construction of a scientific research environment from existing papers and repositories; it does not propose new methods for improving the capabilities of the agent itself. Therefore, the novelty relative to existing research and the significance for the development of scientific research agents appear somewhat limited.
-
W2. The scale and diversity of the constructed scientific research environments are limited: Although this study constructs 30 types of scientific research environments, this number is relatively small, and the target research fields are also limited. And out of these environments, only 10 environments have actually been run as experiments.
-
W3. Lack of comparison with other simple baselines: To prevent shortcut behavior, one possible approach is to prohibit improper operations through instruction files such as AGENTS.md or system prompts. Therefore, it would be desirable to compare these simple prompt-based methods as a baseline and verify whether the environment configuration using the proposed method is truly necessary.
-
W4. I am not sure how effective the details of the proposed method actually are. Specifically, I am not sure whether a DAG is necessary, nor am I sure whether the paper and the code need to be examined separately (L305, L307). It might be helpful to have experiments that demonstrate the effectiveness of these approaches.
Questions for authors
For more details, please refer to “Critical Weaknesses.” In particular, I am concerned about the significance of the paper's contribution.
Thank you for the substantial additional work. The new experiments and analyses would benefit from clearer integration into the manuscript, with a more focused presentation of the main claims, supporting evidence, and remaining limitations. My concern about the reproducibility of the long-horizon results remains, and I therefore maintain my current rating.