acceptodds
Under review as a conference paper at ICLR 2027

Rules to Tools: Executable Checks for LLM Agents in Scientific Computing

Abstract

Scientific coding agents receive equations, boundary conditions, and output requirements in writing, then must assess the programs they revise. Rules to Tools (R2T) supplies prepared executable checks of public scientific requirements. Matched SciCode repair groups share written checks, starting programs, model, and budgets; the tool group receives a callable implementation. Across two task-ID cohorts, complete repair is 26/30 with text and 29/30 with the prepared checks. Three task IDs favor tools, one favors text, and eleven tie. The eight-ID cohort scores 13/16 versus 15/16, with a task-cluster bootstrap 95% interval of [−12.5, 43.75] percentage points for the difference. The larger shared-definition SciCode cohort ties at 13/24 per group. Five development-exposed tasks with alternate starting programs score 3/10 versus 7/10. The tool group favors tasks 17, 77, and 11; initial checks flag task 17 and report no violation for tasks 77 and 11. Task 37 favors text and has no initial reported violation. A fresh source-through-Python arm also reaches 15/16, matching the dedicated command’s aggregate. In a matched PDE comparison, detailed text scores 23/24 and checks score 24/24, with 31.2% lower reported model output for checks. Agent-side output savings vary by cohort, while public CPU use rises in both task-ID cohorts. These results measure task-dependent repair outcomes and agent-side costs with prepared checks.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Scientific Computing, AI for Science, LLM Agents, Executable Specifications, Tool Use, Execution Feedback, Code Generation and Repair

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7rNPOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Rules to Tools (R2T), a framework that provides LLM-based scientific coding agents with pre-built executable checks derived from scientific requirements. Instead of requiring agents to implement verification procedures themselves, R2T allows them to invoke prepared checking tools and use the resulting diagnostic feedback to guide iterative code repair. The authors evaluate R2T across four scientific computing benchmarks. Controlled experiments compare agents equipped with written scientific rules against those additionally provided with executable checks. The results show task-dependent improvements in repair success rates and occasional reductions in output tokens, although the benefits are inconsistent across benchmarks and accompanied by increased computational costs in several settings.

Critical strengths

The paper attempts to address a meaningful challenge in scientific coding agents: ensuring that generated programs satisfy scientific constraints beyond basic execution correctness.

Critical weaknesses
  • Limited technical novelty. The central idea of providing pre-built executable checks to guide agent-based code repair closely resembles existing approaches based on executable specifications, test-driven development, and tool-based feedback. The paper does not introduce a substantially new agent architecture, verification algorithm, or automated mechanism for converting scientific requirements into executable tools.

  • Limited and statistically inconclusive empirical improvements. The performance gains are modest and concentrated on a small number of tasks. In the main held-out SciCode experiments, success rates improve from 13/16 to 15/16 and from 13/14 to 14/14, while the shared-definition cohort shows no improvement (13/24 for both groups). The reported statistical analyses provide insufficient evidence of consistent improvements.

  • Poor writing quality. The paper suffers from significant writing and presentation issues that substantially undermine its clarity and readability. Rather than developing a coherent scientific narrative around a well-defined research question and a clear set of contributions, the manuscript largely consists of lengthy descriptions of experimental settings and repetitive enumerations of numerical results. The unclear presentation of contributions is not merely a writing issue; it also reflects the paper's failure to establish a sufficiently distinct and compelling research contribution.

Questions for authors

Could the authors clarify what the key methodological contribution of R2T is beyond existing executable specifications and tool-based feedback approaches? Additionally, given the limited and inconsistent performance gains, what evidence supports the claim that the improvements are specifically attributable to checker feedback rather than stochastic variation or changes in agent behavior?

Reviewer_CVL1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

I understand R2T as giving a coding agent some prepared checking functions while it repairs a scientific program. In the main SciCode comparisons, both groups get the same failed program, written rules, probe inputs, tolerances, Python access, and budgets. The text group can implement the checks itself, while the tool group can call an implementation the authors have already prepared. The authors grade the final programs with separate tests. Across two task-ID cohorts they report 26/30 complete repairs with text and 29/30 with tools. The larger shared-definition cohort ties at 13/24. They also examine other ways of giving the checks and report additional scientific programming experiments and costs.

Critical strengths
  1. I appreciate that the authors keep the written checking rules the same and let both groups use Python, since otherwise I would not know whether the tool group just gets a better explanation of the problem. I also find their reporting quite honest. They include the ties, increased CPU use, development exposure, and the selection problem in the flow experiment, and they describe the closest related work fairly. I think using separate final tests is important here, as passing a few public checks should not count as finishing the task.
Critical weaknesses

I can follow the paper, although I keep going back to check which cohort a number belongs to, as the authors use several support forms and grading protocols.

I think the code problems below are fixable, while I still need to know whether they influenced the actual repairs. The supplement does not give the final programs and complete repair traces to check this part.

  1. I feel like the contribution is rather thin. Both groups already know what to check, with which inputs and tolerances, while one group also gets the checking program. If the text group wants the same measurements, it still needs to write and debug this program using its repair allowance, so I do not think the same online budget means the same amount of work. Someone has already done a part of the work for the tool group. I would expect this to save effort, although it does not have to improve every repair. This reminds me of a programming assignment where the teacher gives one student runnable public tests and gives the other student descriptions of those tests. I can see why it helps. I fail to see what I learn from calling this a scientific-computing experiment unless the results explain something more.

    I find the related work fairly described, although the overlap is large. SecTDD already studies repair from the same starting programs with separate hidden tests. CodeSpec compares textual and executable specifications, while Paper2Agent compares scientific tools with Markdown skills. I understand that R2T fixes the numerical relations and probes more carefully, but I do not get much extra understanding of verification from this difference.

    I get especially stuck at the tasks with improvement. Task 77 goes from 0/2 to 2/2 and contributes the whole net gain in the first cohort, while all five initial checks pass. Task 11 gives the extension its only gain, from 1/2 to 2/2, and both initial checks pass too. As I discuss below, its partial-trace check even passes a function that returns the input without tracing anything. A passing check could rule out something the agent was unsure about, and other checks might help after an edit. I just do not know what happened here. Across the two cohorts only three tasks favor tools, one favors text, and eleven tie, with sign-test p=0.625, while the larger shared-definition cohort ties at 13/24. Giving checker source through Python also gets 15/16, the same aggregate as the dedicated command. With this sample size I would not call the two forms equivalent, but I do not see an advantage for the dedicated interface either.

    The PDE output-token saving is useful, while the two SciCode cohorts together use 87.3% more CPU. The authors record these separately, which is good. However, the prepared checking code is not free of investment, as somebody still has to write and review it, and the paper has no authoring-time measurement. I would like to know when preparing the code pays back, instead of only knowing that it changes where the work happens.

  2. I am also bothered by whether the prepared measurements are worth-trusting. The reported initial violations do reproduce across the 15 supplied starting programs and 39 diagnostics, but the supplied code can still give a large discrepancy to a correct numerical result.

    For task 34, depletion_charge_neutrality_and_width_scaling in scicode_extension/34/checker.py adds abs(Na*xp - Nd*xn) and abs(xn + xp - expected_total) and uses 1e-18 as the threshold. I find the sum kind of suspicious already, as one term has concentration times length and the other has length. I can just regroup the arithmetic in the same depletion-width formula, changing the widths by about 1.36e-16 relatively, and the checker residual jumps from zero to 7.63e-6. It then says the equivalent implementation violates the rule. I do not think this is a reasonable numerical tolerance.

    I have another problem with public_checker_sources/pde_delivery_checker.py, which concerns the separate scalar-PDE support, not the main SciCode counts or the steady-flow checker. It changes t_end to request solutions at T, T-h, and T-2h, but leaves dt unchanged, while a fixed-step solver can use ceil(t_end/dt) steps and return the same actual time layer for all three requests. With T=0.1, dt=0.02, and the checker's h=0.005, the requests at 0.1, 0.095, and 0.09 all return the field at 0.1. For the heat equation u_t - Laplacian(u) = 1, with zero initial data and boundary value u=t, the exact solution is u=t. The primary output is exact, while the checker estimates zero time derivative and reports a normalized residual near 1. This example runs the supplied diagnostic with exact scalar time stepping; I have not run a complete DOLFINx experiment. The time layers simply do not match the times in the difference formula. The supplement does not tell me which individual PDE runs used this source, so I also need the authors to locate this problem in their reported experiments.

    Task 11 bothers me for its connection with the only extension gain. Its partial-trace metric only compares the total traces. I can trace the wrong subsystem, or return the original matrix without doing any trace, and still get zero residual. Therefore I do not see how it detects wrong subsystem ordering as its justification says. I would not normally ask one invariant to check everything, but here I want to know what checking information actually helped in this favorable task. I guess the separate Kraus-completeness check or later measurements could help, but the final success count does not tell me this.

    The reported accepted-program and targeted-fault test counts do not answer these examples. I have not recomputed the independent final scores, since the final programs and grading records are missing from the supplement. I am concerned that the authors offer these prepared measurements to guide the agent while the measurements themselves can mislead it, and I still do not know how this affects the repairs.

I recommend rejection. I think the authors report their experiments honestly, but I learn too little beyond the expected help from giving an agent finished checking code, and I also find errors in that code. I would need more explanation of what the checks actually contribute and more support that their measurements are worth-trusting.

Questions for authors
  1. What do the saved trajectories show for tasks 77 and 11, whose initial checks all pass? I would like to know which later measurements helped the repairs, including whether the Kraus check helped on task 11. How much effort did the text group actually spend writing and debugging its checks? This could explain whether the benefit is mostly from having this work done in advance.

  2. How do the authors handle the task-34 false alarm and the PDE time-layer problem? Which PDE arms use pde_delivery_checker.py: the detailed-text comparison in Table 3 and Appendix H, with 23/24 versus 24/24 and the 31.2% output-token saving, the scalar pilot in Appendix G, or both? I would like the relevant final programs and diagnostic/repair traces to see whether these errors occurred and which conclusions hold after correction.

Reviewer_aGDxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a prepared executable implementation of public scientific checks helps an LLM repair programs when the comparison group already receives the same written relations, probe inputs, tolerances, and ordinary Python access. It packages task-specific checks as callable tools and evaluates complete repair, native partial progress, and resource use. Two SciCode cohorts contain 15 task IDs and two continuations per condition: text repairs 26/30 programs and tools repair 29/30. The original twelve-task matched cohort ties at 13/24. Additional comparisons examine an initial diagnostic report, checker source executed through Python, development-exposed alternate programs, PDE repair, and historical scientific-computing tasks. The paper's main contribution is a controlled empirical study of prepared executable support. Its results support task-dependent effects and a generation-versus-execution cost tradeoff rather than a general repair improvement.

Critical strengths
  1. The central control isolates a meaningful practical question. Both SciCode conditions receive the same public requirements, probes, tolerances, starting program, model, and repair budget. The text group retains Python access. The source-through-Python and initial-report arms further distinguish access to a prepared implementation from a dedicated command and one-time feedback. This design is substantially more informative about the value of executable support than comparing a detailed checker with vague textual advice (Sections 3–4; Table 4; Appendix D).

  2. The reporting preserves evidence that limits the positive interpretation. The paper retains the original 13/24 tie, the first cohort's wide task-bootstrap interval, the selected nature of the flow panel, and the absence of initial violations on key positive tasks. It reports native partial progress and both output generation and public CPU cost. Across the two new SciCode cohorts, CPU rises 87.3%, even though reported output falls by at least 4.9%. These details make the tradeoffs and limits inspectable (Section 5; Tables 3–7).

Critical weaknesses
  1. The positive repair effects remain uncertain and concentrated. The combined SciCode advantage comes from three task IDs favoring tools, one favoring text, and eleven ties; the exact task-level sign test is p = 0.625. In the first cohort, excluding task 77 removes the average gain, and native-step completion is identical between text and tools. The larger original matched cohort ties. The matched PDE result is near ceiling, while the flow comparison selects text-probe failures and changes the capture gate. The latter is appropriately disclosed but cannot strengthen the same controlled treatment estimate. This limits how strongly the results can justify deploying checks for expected repair gains. It does not invalidate the descriptive contribution, and I credit the manuscript for preserving these qualifications (Section 5; Appendices C–G).

  2. Checker preparation and the repair endpoint limit practical interpretation. The first cohort's checks were authored after failed programs and earlier context records existed, without a blinded authoring log; task-ID separation therefore does not establish independence from the evaluated failure. The seven-ID extension improves qualification, but only one of its seven actual failed starts triggers a selected check. Authoring and review time are unrecorded, so trajectory costs do not measure total preparation effort. Separately, the new SciCode cohorts count a substep as passing if either the 2024 or 2025 environment accepts it. The manuscript does not report whether every successful complete module passes all steps within one environment. Reporting that breakdown would clarify the endpoint and its comparison with the original 2024-only cohort (Appendices C–D; Section 6).

Questions for authors
  1. Please report complete-task success in the 2024 environment and identify repairs that rely on 2025 step-level retries. Because the 2025 environment only retries previously failed steps, full 2025 results may be unavailable; could the same final programs then be graded on all steps in that environment? How many of the reported 26/30 versus 29/30 repairs pass every step within at least one single environment, and does this change the task-level comparison? This asks for clarification or regrading of existing final programs, not new agent generations (Section 5; Appendices C–D).

  2. What archived failures, diagnoses, and historical context were visible when the first cohort's checkers were written, and which checks were changed after inspecting them? Please distinguish task-ID separation from independence of checker design, including any relevance of the shared molecular-dynamics family of tasks 77 and 80. The seven-ID protocol is a useful stronger control. If broader generalization is intended, is there prospective evidence with checkers frozen before the new failed programs are seen, or should the conclusions remain limited to the evaluated cohorts (Appendices C–D; Section 6)?

  3. Can the logs connect checker calls, returned violations, subsequent edits, and hidden-test outcomes on the improved tasks? Tasks 77 and 11 initially pass the public checks, while task 17 is initially flagged; only one of seven extension starts triggers a selected check. What evidence distinguishes diagnosis of an existing defect, prevention of a later regression, and reduced checker-implementation effort? Please also explain what interface-specific conclusion is justified when source-plus-Python and dedicated tools both achieve 15/16 (Tables 4 and 7; Appendices C–D).

  4. How should the claimed repair benefit be interpreted given three positive task IDs, one negative ID, eleven ties, the sign-test p-value of 0.625, and the original 13/24 versus 13/24 result? Please identify which conclusions are descriptive and which predict gains on new tasks, retaining task-level uncertainty and the sensitivity to task 77. If a reliable average advantage is intended, what independent task-level evidence supports it? I do not require every task or panel to show a positive effect (Table 3; Section 5).

  5. Are the cost conclusions explicitly limited to agent execution, given the approximately 87.3% increase in public CPU and the reduction in generated output? If an end-to-end efficiency benefit is intended, can checker authoring, human qualification, maintenance, and reuse assumptions be included in cost per correct repair? Otherwise, please retain the narrower resource-tradeoff conclusion. The selected flow panel should also remain distinct from matched cohorts because its selection and capture criteria differ (Sections 5–6; Appendices E–I).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.