- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Rules to Tools (R2T), a framework that provides LLM-based scientific coding agents with pre-built executable checks derived from scientific requirements. Instead of requiring agents to implement verification procedures themselves, R2T allows them to invoke prepared checking tools and use the resulting diagnostic feedback to guide iterative code repair. The authors evaluate R2T across four scientific computing benchmarks. Controlled experiments compare agents equipped with written scientific rules against those additionally provided with executable checks. The results show task-dependent improvements in repair success rates and occasional reductions in output tokens, although the benefits are inconsistent across benchmarks and accompanied by increased computational costs in several settings.
Critical strengths
The paper attempts to address a meaningful challenge in scientific coding agents: ensuring that generated programs satisfy scientific constraints beyond basic execution correctness.
Critical weaknesses
-
Limited technical novelty. The central idea of providing pre-built executable checks to guide agent-based code repair closely resembles existing approaches based on executable specifications, test-driven development, and tool-based feedback. The paper does not introduce a substantially new agent architecture, verification algorithm, or automated mechanism for converting scientific requirements into executable tools.
-
Limited and statistically inconclusive empirical improvements. The performance gains are modest and concentrated on a small number of tasks. In the main held-out SciCode experiments, success rates improve from 13/16 to 15/16 and from 13/14 to 14/14, while the shared-definition cohort shows no improvement (13/24 for both groups). The reported statistical analyses provide insufficient evidence of consistent improvements.
-
Poor writing quality. The paper suffers from significant writing and presentation issues that substantially undermine its clarity and readability. Rather than developing a coherent scientific narrative around a well-defined research question and a clear set of contributions, the manuscript largely consists of lengthy descriptions of experimental settings and repetitive enumerations of numerical results. The unclear presentation of contributions is not merely a writing issue; it also reflects the paper's failure to establish a sufficiently distinct and compelling research contribution.
Questions for authors
Could the authors clarify what the key methodological contribution of R2T is beyond existing executable specifications and tool-based feedback approaches? Additionally, given the limited and inconsistent performance gains, what evidence supports the claim that the improvements are specifically attributable to checker feedback rather than stochastic variation or changes in agent behavior?