acceptodds
Under review as a conference paper at ICLR 2027

NeutronGym: Physics-Graded Neutron Instrument Design for LLM Agents

Abstract

Designing a scientific instrument tests whether language-model agents can do physics rather than recall it, provided the grading cannot be argued with. We introduce NeutronGym, to our knowledge the first executable environment for neutron instrument design: agents build instruments through validating tools, McStas ray-traces what they build, and a level-resolved ladder grades syntax, runtime, structure and science with no LLM judge. Procedural families supply unlimited instances of a fixed layout whose design parameters the agent must set, with held-out parameter regimes; a curated slice, McStasBench, adds 16 tasks from published instruments behind memorization probes and a sandbox. Seven models reproduce at most 7 of the 16, none retrieves a reference, and none meets an improvement target. The environment also trains. Reinforcement learning on its reward takes Qwen3-8B from 11% to 77% of held-out instances of a family whose targets come from a hidden design (69% at a second seed), past an untrained Qwen3-32B, and the recipe holds, at one seed each, on three further gated families. The analysis says what that gain is. Without the ladder’s partial credit it collapses by 60 points. From reward alone the trained model reaches what a classical optimizer reaches, at the agent’s simulation budget, only when handed the closed-form physics (77% against 81%, a gap that does not separate at this size), while frontier models still solve 98–99%. Getting a trustworthy result meant failing four task designs that no-model baselines could solve, and we release the probes that found them.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
LLM agents, executable environments, benchmarks, reinforcement learning, scientific instrument design, neutron scattering, reward hacking, contamination

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9GXqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces NeutronGym, a simulator-grounded environment for evaluating and training LLM agents on neutron-instrument tasks. Agents interact through validated tools, designs are executed in McStas, and a reward ladder scores syntax, runtime, structure, and physics. The paper also introduces McStasBench and studies reinforcement learning on several procedural task families.

Critical strengths

The benchmark design is careful. In particular, I appreciate the use of simulator-based grading rather than an LLM judge and the explicit attempts to discover shortcut solutions before accepting a task.

The reward-ladder experiment is also informative: removing dense partial credit causes a very large drop in performance, providing a plausible explanation for why the environment is useful for RL training.

Critical weaknesses

My main concern is scope. The scored tasks do not require topology design: layouts are fixed, and the procedural tasks mainly require selecting a small number of parameters. The genuinely open-design setting is not scored. This makes the evidence considerably narrower than the paper's “instrument design” framing suggests.

The RL evidence is also still fairly exploratory. Most configurations use a single seed, the benchmark has only 16 scored tasks with one episode per task/model, and the manuscript itself notes instability across seeds and limited task coverage.

Questions for authors

Can the authors provide additional independent training seeds for the auxiliary task families? I would also like the authors to clarify which main claims remain if the results based on single-seed configurations are treated as exploratory only.

1 comment
AuthorsOpenReview ↗

Reply to Reviewer 9GXq

Thank you for recognizing the simulator grading, shortcut checks and reward-ladder ablation. We agree that the paper needed to distinguish those contributions more clearly from the narrower design and training results.

A. Scope: fixed layouts (weakness 1)

We agree that our framing should have made the fixed-layout scope clearer. We have now stated that limit throughout the revision. The abstract says that "the agent sets the parameters of a fixed instrument layout", and the introduction explains that "choosing a layout (the open-design tier) is not scored". Section 6 states that "No scored task asks for a topology choice" and limits the conclusion to "the level of a fixed layout's parameters". We release the open-design tier unscored because we do not yet have an expert rubric for grading a free layout.

Within this limited scope, we believe the parameters pose a useful design problem: their effects interact through physics that the agent must reason about or measure. Our shortcut probes offer some support for this interpretation. The best fixed answer or lookup passes at most 12.7% in any admitted family, and no rule that simply copies prompt quantities succeeds; on TOF chopper, no readout rule passes at all. Training raises the untrained 8B's 2–26% pass rates to 44–77% across the four held-out families. We interpret these results as evidence for parameter-level design; topology design remains open. Extending the families to component types or counts would require applying the same admission checks before interpreting any new training gains.

A. Size of the benchmark (weakness 2, first part)

We also agree that 16 tasks with one episode per task and model cannot establish a ranking. As Section 4.2 notes, the Wilson interval for 7/16 spans 23–67%, and Table 1's caption says that no per-model difference is significant. We therefore use McStasBench to establish whether reproduction is possible and to describe failures: some instruments were rebuilt, no episode retrieved a reference, and no agent met an improvement target. The procedural evaluations address learning on larger sets of instances, but do not remove the curated benchmark's coverage limit.

B. Exploratory RL evidence and which claims remain (weakness 2, question)

We could not add seeds because our training cluster is down. Topic B of the general comment explains how we used the archived runs instead: Appendix O (Table 12) reports 15 run–family comparisons from specialist and joint training, of which 12 beat the untrained model. Since the joint runs pool families, these comparisons provide supporting evidence across runs rather than additional seeds of the specialist recipe.

To answer your question about what remains, Appendix R (Table 14) assesses each claim separately. The environment, admission test and four rejected task designs do not depend on single-seed training results. We believe the McStasBench observations remain supported as existence results, and the frontier models remain ahead on the reported evaluation.

Among the training findings, guide match has two specialist seeds, at 76.7% and 69.0% against the untrained 32B's 11.3%. Gains over the untrained 8B also appear in 3/4 TOF-chopper runs and 2/2 bender runs, although those counts include joint training. The physics-seeded search matches or beats the trained model at equal simulation budget, and all three attempts to fine-tune on the model's own successes made it worse. Together, these results offer support beyond one training run, although they do not settle the reproducibility of each specialist recipe.

The remaining findings retain the exploratory label: SANS match improves in only 2/4 runs; removing partial credit costs 60 points in a one-run-per-condition ablation; and the four-family policy, transfer, tolerance and range tests each rely on one seed. Although the ablation's effect is about eight times the 7.7-point gap between the two guide-match seeds, we do not treat that comparison as replication.

We hope the revision makes it easier to assess the environment and admission protocol, the RL gains supported across runs, and the findings that still need replication. Thank you for prompting us to make these distinctions clearer. We would be grateful if you would consider the revised scope and evidence in reassessing your rating.

Reviewer_iuS8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces NeutronGym, an executable environment for evaluating and training LLM agents in neutron-instrument design using McStas simulations. Rather than demonstrating a general-purpose AI scientist capable of designing complete neutron instruments, the paper develops a relatively reliable training and evaluation environment and shows that reinforcement learning can produce effective strategies for fixed-layout tasks with low-dimensional design parameters. A central contribution is its systematic use of no-model baselines and diagnostic probes to identify spurious training gains caused by task shortcuts and reward exploitation. The empirical evaluation covers both prompted agents and reinforcement-learned policies. The trained model performs comparably to a budget-matched optimizer initialized with a closed-form physics solution, while remaining substantially behind frontier models.

Critical strengths
  • The paper treats task validity as something that must be tested rather than assumed. Its fixed-policy, readout-rule, cross-instance, contamination, and concentration probes uncover several tasks that would otherwise have produced impressive but misleading results. The admission protocol and documented failure cases could be valuable for future simulator-based agent benchmarks.

  • The evaluation is objective, reproducible, and unusually transparent. McStas directly simulates each proposed instrument, while the reward ladder shows where and why an attempt fails. The authors also report negative results, sensitivity to training seeds, instability under new grading seeds, and a previously discovered measurement error. This transparency gives me greater confidence that the main improvement on the guide-matching task is real under the reported evaluation conditions.

Critical weaknesses
  • The training results are not sufficiently replicated. Three of the four specialist results come from a single training run. On the main guide-matching task, the second seed drops from 76.7% to 69.0%. The instability is even clearer in multi-family training, where changing only the seed reduces performance on the TOF task from 55.0% to 2.3%. The reported confidence intervals cover variation across test instances, but not variation across training runs. This makes it difficult to know whether the reported gains across four families are reliable or whether some depend on favorable seeds.

  • The experiments do not fully establish that the model has learned transferable physics or that it is cost-effective. The procedural tasks use fixed layouts with only two or three adjustable parameters, so the model may have learned family-specific mappings or search strategies rather than general physical principles. The comparison with classical optimization is also matched only by the number of simulator calls. It does not include the cost of LLM inference, tool interaction, or reinforcement learning. As a result, the paper shows that the trained model can perform well under a fixed simulation-call budget, but it does not yet show that the model is more useful or economical than classical optimization under an equal total-compute budget.

Questions for authors
  • Could the authors report results from additional training seeds for the three specialist families that currently have only one run? Given the large difference between the two multi-family runs, the variation across training seeds could materially affect the claim that the method works across four families.

  • What evidence shows that the trained model has learned a transferable physical relationship rather than a family-specific mapping or search strategy? For example, does its behavior follow the expected scaling laws or monotonic relationships when wavelength, geometry, or target conditions are changed?

  • How does the trained agent compare with classical optimization under an equal total-compute budget? Please include simulator calls, LLM inference, tool interaction, and amortized training cost. It would also be helpful to report how many downstream design instances are needed before the cost of training a specialist model is recovered.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer iuS8

Thank you for the careful review. Your questions helped us distinguish the evidence for learning from the stronger claims about transfer and practical cost that it does not yet establish.

B. Replication across training seeds (weakness 1, question 1)

As discussed in topic B of the general comment, our GPU cluster is down, so we could not add training seeds. We instead brought together the archived specialist and joint runs in Appendix O (Table 12). Of 15 run–family comparisons, 12 beat the untrained 8B. The joint runs use a different training mixture, so they do not replace additional specialist seeds, but they offer some evidence that gains on guide match, TOF chopper and bender appear in more than one run. SANS match remains mixed and is labelled exploratory.

C. Transferable physics or a family-specific mapping (weakness 2, question 2)

We agree that successful designs alone do not establish that the model learned a transferable physical law. To examine what the existing evaluations can tell us, Appendix P (Table 13) now groups instances by how far their geometry lies beyond the training range. For example, TOF chopper uses a spacing of 9.5–12 m in evaluation, compared with 5–9 m in training. Splitting that interval into thirds lets us ask whether performance deteriorates as the extrapolation grows.

We find no significant downward trend on guide match, SANS match or bender (trend p ≥ 0.15). On TOF chopper, however, the trained pass rate falls from 57% to 33% (p < 0.001), while on guide match's further out-of-range split it falls from 56% to 38% (p = 0.07). The gain nevertheless persists beyond the training range: the untrained model passes only 2%, 0% and 4% of the three TOF-chopper thirds. We take this as evidence of limited extrapolation, with performance weakening at the largest shifts.

The design records help narrow the interpretation. Of the passing designs, 225/230 on guide match and 138/140 on TOF chopper are distinct, whereas a fixed answer or lookup passes only 4.0% and 2.7% of those families. No TOF-chopper readout rule succeeds. Yet neither the trained nor untrained 8B passes either family on its first turn, showing that the policy relies on simulator feedback. Because the records retain final designs, we cannot tell whether that refinement uses a learned physical relation or an effective family-specific search strategy. We therefore interpret the gain as feedback-driven search; Section 5.3 describes what transfers as "the habit RL teaches (propose a valid geometry, then refine), not design skill".

Your proposed controlled sweep would help distinguish these explanations by checking whether the first design changes as the time-of-flight relation predicts. It requires new inference, which we cannot run without usable GPU compute, so it remains unperformed. The revised discussion keeps this limit explicit rather than interpreting the extrapolation analysis as proof of learned physics.

D. Comparison with classical optimization at equal total compute (weakness 3, question 3)

We agree that matching simulator calls leaves out much of the agent's cost. Appendix Q now uses existing timing and usage records to include it. A family requires about 3 hours of GRPO plus 0.7–1.6 hours of state collection, or about 4 A100-hours overall. Once trained, an episode takes about 4–7 s, including tools and simulations. By comparison, physics-seeded search needs about 0.3 CPU-s per design and no training; the frontier models cost about $0.05 per design.

At an assumed $1–2 per A100-hour, this puts training at $4–8 per family and inference at $0.001–0.004 per design. Under these assumptions, training would recover its monetary cost against the frontier models after roughly 85–180 designs, but with a lower pass rate, 77% against 98–99%. Against physics-seeded search, it does not recover that cost in this comparison: search is cheaper per design and at least as successful. We have added this qualification to the limitations.

The RL comparison is useful for a narrower reason. The closed-form start has to be derived by hand, whereas the trained model comes within 4–12 points of the seeded search at its two seeds from reward alone; search from a random start reaches only 14%. We see this as evidence for studying learning in the environment, with no cost advantage over the physics-seeded optimizer. We have not run budget-matched classical search on families without such an inversion and make no comparative claim there.

We hope these additions help address your concerns and make the remaining limits easier to assess. Thank you again for the suggestions; we would be grateful if you would consider the revision in reassessing your score.

I thank the authors for their detailed response, which addresses most of my concerns. I will maintain my score and support acceptance if necessary.

Reviewer_Gi7GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

 

The authors introduce NeutronGym, an RL environment where language agents are tasked with constructing instrumentation for neutron scattering experiments. The instrument designs are assessed by ray tracing experiments in McStas. The core findings of the paper appear to be that language agents can indeed design neutron scattering instrumentation and that post-training improves the performance of small models on the environment. The main limitation of the paper would seem to be that, by the authors' own admission, the environment does not admit relative ranking between frontier models. As such, the utility of the benchmark would seem to hinge on the fact that post-training a small 8B model improves performance. It remains unclear what the intended purpose of the benchmark is. Nonetheless, the main issue with the paper is that it is clearly LLM-written. This, in and of itself, is not necessarily problematic but the manner in which it was conducted makes the paper incredibly difficult to follow. As things stand, both the contribution of the paper, as well as the content, are highly unclear and as such I recommend rejection. If the authors can provide convincing arguments for the utility of the benchmark as well as substantially improve the clarity of the paper during the rebuttal phase I will be inclined to raise my score.

 

Critical strengths

 

The main strength of the paper is that, if properly designed, the RL environment could provide a path towards autonomous generation of neutron instrumentation which could in turn give rise to autonomous laboratories in which language agents both design and operate instrumentation. A further strength of the paper is that the codebase is made fully available and appears to be thoroughly documented by Claude Code (barring function-level docstrings which I describe below).

 

Critical weaknesses

 

I demarcate between major points and minor points below.

 

MAJOR POINTS

 

  1. The key weakness in my view is the lack of a clear purpose for the benchmark? Is the benchmark designed to galvanize advances in language agent methodology? If so, by the authors' own admission, it seems that 16 tasks is too narrow for this purpose? Is the intention for the benchmark to serve as an RL training ground for small models? If so, this should be stated and motivation given for why small models would be practically useful in real neutron instrumentation design loops.

  2. A gaping weakness in my view is that the LLM writing renders the manuscript almost completely opaque. I found the manuscript grating to read, with concepts being introduced in a very roundabout way. One illustrative example is Section 3.1 which is intended to introduce the McStas software. The authors could have stated very simply here that McStas is an open-source software package used to simulate neutron scattering experiments. Instead, what is written appears to dance about a basic explanation of what the software is. While the first sentence, "Everything in NeutronGym rests on one fact about the simulator...", may be a suitable opening credit to build audience suspense for the much-anticipated cinematic adaption of NeutronGym, I would rather the authors just tell me the fact about the simulator upfront. It would life much easier for the reader if the paper was self-contained.

 

MINOR POINTS

 

  1. There are some missing conference citations in the references e.g., [1] was accepted to ICML 2026, [2] was accepted to COLM 2026, [3] was accepted to ACL 2026, [4] was published in MLST, [5] was accepted to ICLR 2024, and [6] was published in the Journal of Neutron Research.

  2. While the documentation in the codebase is good, it could be improved slightly by requesting that Claude Code make use of function-level docstrings.

  3. The LLM-based writing style is somewhat roundabout e.g., the first sentence, "Designing a scientific instrument asks more of an agent than writing code that runs". Later on in the passage, "agreement with a reference answer" is assumed to be semantically similar to "writing code that runs". The opening passage centers on comparing the tasks agents are typically used for with the challenge of designing a scientific instrument. It would be a lot simpler for the reader if no contrast was used and the challenges for LLM agents to design scientific instruments were laid out plainly.

  4. A further example of the LLM writing style is line 053, "with no LLM judge". It would be more understandable for the reader to state what you did rather than what you didn't do? What is being used for grading in place of an LLM judge?

  5. Line 060, "The paper follows what happened when we used it" is yet another example of an LLM-generated sentence. It is unclear here what the referrent, "it" refers to?

  6. Line 066, "past an untrained 32B that is no stronger than the 8B there" is another weird and LLM-like sentence.

  7. The text of Appendix F does not read as a complete sentence.

  8. The text of Appendix G does not reference any figure or table. I believe it should reference Table 6.

  9. The source paper for AdamW should be cited given that it is used [7].

  10. Table 12 is very difficult to read. It is difficult to understand what is being commented on in the Note column. How are descriptions such as "independent repeat" and "$9.69 for both" meant to be understood?

 

REFERENCES

 

[1] Yujiong Shen, yajie yang, Zhiheng Xi, Binze Hu, Huayu Sha, Qiyuan Peng, Jiazheng Zhang, Junlin Shang, Jixuan Huang, Yutao Fan, Jingqi Tong, Ming Zhang, Shihan Dou, Zhenfei Yin, Xingjun Ma, LEI BAI, Tao Gui, Qi Zhang, Xuanjing Huang, & Yu-Gang Jiang (2026). SciAgentGym: Benchmarking Multi-Step Scientific Tool-Use in LLM Agents. In Forty-third International Conference on Machine Learning.

[2] Xinge Liu, Terry Jingchen Zhang, Bernhard Scholkopf, Zhijing Jin, & Kristen Menou (2026). Stargazer: A Scalable Model-fitting Benchmark Environment for AI Agents under Astrophysical Constraints. In Third Conference on Language Modeling.

[3] Li, X., Chen, J., Fang, X., Ding, S., Duan, H., Liu, Q. and Chen, K., 2026, July. OPT-BENCH: Evaluating the Iterative Self-Optimization of LLM Agents in Large-Scale Search Spaces. In Findings of the Association for Computational Linguistics: ACL 2026 (pp. 28437-28452).

[4] Qasim, S.R., Owen, P. and Serra, N., 2025. Physics instrument design with reinforcement learning. Machine Learning: Science and Technology, 6(3), p.035033.

[5] Peng, B., Quesnelle, J., Fan, H. and Shippole, E., 2024, May. YaRN: Efficient context window extension of large language models. In International Conference on Learning Representations (Vol. 2024, pp. 31932-31951).

[6] Robledo, J.I., Lieutenant, K. and Willendrup, P., 2024. Small angle neutron scattering in McStas: Optimization for high-throughput virtual experiments. Journal of Neutron Research, 26(4), pp.173-185.

[7] Ilya Loshchilov, & Frank Hutter (2019). Decoupled Weight Decay Regularization. In International Conference on Learning Representations.

 

Questions for authors

 

  1. For the T2 task (guide_divergence) if the goal is simply to identify the optimal settings of 5 continuous parameters why can't a surrogate model-based optimization approach such as Bayesian optimization be used instead of a language agent?

 

2 comments
AuthorsOpenReview ↗

Reply to Reviewer Gi7G

Thank you for the detailed examples. They helped us revise both the framing and the prose.

A. Purpose and small models (major point 1)

We should have distinguished the two purposes more clearly. NeutronGym's procedural families provide unlimited graded instances for training and evaluation, with 300 held-out instances per family and model. McStasBench then asks whether an agent can rebuild a published instrument from its specification and where it fails. We see its 16 tasks as a diagnostic study and agree that they cannot support a model ranking. The abstract and introduction now explain this distinction and make clear that all scored tasks fix the layout; open design is released unscored.

We chose an 8B model to test whether an agent could learn from the environment's reward. Its improvement from 11% to 77% on guide match provides evidence that it can. We do not present it as the best practical design tool: frontier models reach 98–99%, and physics-seeded search matches or beats it. Each 8B training run fits one 40 GB A100 and takes about 3.5 hours; a 32B model's bf16 weights alone exceed 60 GB. Small open models can also run on site at facilities that restrict external services, which may make this environment useful for training and testing them.

We hope the training study also offers lessons for building such environments. Imitating the model's own successes made it worse, while removing partial credit cost 60 points in the one-run-per-condition ablation, which remains exploratory. Alongside the four task designs rejected by our no-model probes, these findings suggest why the reward and tasks need testing before training gains can be interpreted.

E. Writing and presentation (major point 2; minor points 3–8, 10)

We rewrote the abstract and Sections 1–5 with your examples in mind. Section 3.1 now begins with a direct definition: "McStas is an open-source Monte Carlo package for simulating neutron scattering instruments." The introduction likewise starts with the task itself: turning a specification into a layout whose simulated readings meet several targets.

We also revised the descriptions of grading and results to make them more explicit. In place of "with no LLM judge" at line 053, we write: "The environment simulates what they build, and the simulated monitor readings are graded on four levels: L1 syntax, L2 runtime, L3 structure and L4 physics." The unclear "it" sentence at line 060 is replaced by "Our main results are as follows." The comparison at line 066 now reads "the untrained Qwen3-32B scores 11.3%, the same as the untrained 8B."

The appendix changes follow the same approach. Appendix F now uses complete sentences, and Appendix G refers to Table 6. In the submission's Table 12 (now Table 15), we grouped rows by policy type and expanded the notes: "Independent repeat" becomes "Same recipe, second training seed". The caption explains that "59 vs. 46" counts instances passed only by one policy or its comparator, and that $9.69 is the combined API cost of the two frontier runs. We also added the best fixed design's count, 6/150.

D. Why not Bayesian optimization? (question)

We agree that Bayesian optimization is suitable here. We have also clarified in Appendix C that guide divergence has three free parameters and is a development task; the scored T2 task, SANS collimation, has two. Both prompts are included there.

Classical search supplies T2's targets, each closing 80% of the gap from the baseline to a fresh-seed-verified best design. In both tasks, the Nelder–Mead candidate was discarded: it left the bounds on guide divergence and violated the beam-width constraint on SANS. A 30-sample random search supplied the accepted point, yet no agent reached either target. Our intention with T2 is to test whether an agent can reach a target that simple search shows is attainable. We did not run a new BO experiment. Appendix C now explains this role, consistent with Section 5.3's finding that physics-seeded search matches or beats the trained model at equal simulation budget. Choosing components or layouts remains outside the scored tasks.

F. References and code (minor points 1, 2, 9)

We updated all six references to their published venues: SciAgentGym (ICML 2026), Stargazer (COLM 2026), OPT-BENCH (Findings of ACL 2026), Qasim et al. (MLST 2025), YaRN (ICLR 2024) and Robledo et al. (Journal of Neutron Research, 2024). Appendix K also cites Loshchilov and Hutter (2019) for AdamW.

For the code, all 452 functions and 10 classes across the 60 package and harness files now have docstrings. This patch preserves executable behavior; the full suite has 484 passes and one fixture skip. The anonymous repository is updated and verified.

We appreciate your willingness to reconsider the paper if its purpose and clarity improve. We hope these revisions help address both concerns and would be grateful for your reassessment.

Reviewer_Gi7GOpenReview ↗

Many Thanks to the Authors for their Response

 

While I appreciate the efforts the authors have made to improve the clarity of the manuscript, improve the code, and answer questions raised during the initial review, unfortunately there is still much to be desired. For example the first contribution in the introduction reads,

 

"Tasks worth training on. An admission test for procedurally generated scientific tasks: three no-model probes (the best fixed answer, the best rule that reads the prompt, and every instance’s solution applied to every other), decided on a one-sided confidence bound, plus a concentration diagnostic for class-level shortcuts the bound cannot see. Nine documented reward failures motivated it; four of our own designs failed it."

 

For me, this contribution is unintelligible and have the hallmarks of being LLM-written cf. the mirrored clauses joined by a semicolon comprising the final sentence. From the authors' rebuttal it seems as though the main contributions are:

 

  1. Unlimited graded instances for training and evaluation for building instruments.
  2. That small models can learn from the environment's reward.

 

However, the contributions in the paper's introduction seem to have no bearing to the contributions listed in the rebuttal and the remainder of the paper is similarly confusing.

As such, I will be maintaining my score. For the benefit of clarity my recommendation would be that the authors write the manuscript in their own words first, before invoking an LLM to polish the text.

 

Comments on the paper

2 comments
AuthorsOpenReview ↗

general response and summary of revisions

Summary of the reviews and of the revision

We thank the reviewers for their careful reading and specific suggestions. Their comments helped us see where the paper needed a clearer account of its purpose and where the evidence needed qualification. We have uploaded a revised PDF that aims to address both. We discuss the shared concerns below, followed by the remaining questions in each reviewer's thread, using these topic letters throughout.

TopicRaised by
A. Purpose and scope of NeutronGym and McStasBenchGi7G, 9GXq
B. Whether the training results hold across runsiuS8, 9GXq
C. What the trained policy learnediuS8
D. Classical optimization and costiuS8, Gi7G
E. Clarity of the writingGi7G
F. References and codeGi7G

A. Purpose and scope

We should have explained the distinction between NeutronGym's two parts more clearly. The procedural families provide an environment for training and evaluating agents: they generate unlimited graded instances, with 300 held-out instances per family and model for paired comparisons. McStasBench complements this with 16 tasks drawn from published instruments. Its purpose is to examine whether an agent can rebuild an instrument from its specification and where it fails. We agree that so few tasks cannot support a model ranking.

This distinction also helps define the scope of the results. All scored tasks fix the instrument's layout, so the procedural results concern choosing its parameters. Choosing the layout itself belongs to the open-design tier, T3, which we release unscored because we do not yet have an expert rubric for grading it. The revised abstract and introduction now explain these purposes and limits together.

B. Whether the training results hold across runs

We agree that Figure 2's intervals describe variation across test instances, not across training runs. Three of the four specialists were trained at one seed, and the joint-run collapse gives a concrete reason to be cautious. We could not add seeds during the rebuttal: our training cluster has been out of service since late October because of a system failure, and we have no other usable GPU compute. All additional evidence therefore comes from analysing existing runs and logs; no new training or model evaluations were performed.

To make the available evidence easier to assess, Appendix O (Table 12) brings together the specialist and joint runs, paired against the untrained Qwen3-8B on the same held-out instances. The joint runs share the base model and reward but pool several families at one specialist's budget, so they are not repetitions of the specialist recipe. They nevertheless let us ask whether improvement on a family appears in more than one training run. Each contributes a comparison for every family it covers; across these 15 run–family comparisons, 12 beat the untrained 8B (exact McNemar, p < 0.05):

FamilyRuns that beat the untrained 8B
Guide match5 of 5
TOF chopper3 of 4
Bender2 of 2
SANS match2 of 4

The exceptions are joint3rep on TOF chopper and SANS match, and joint3 on SANS match. These comparisons offer some reassurance that the gains on guide match, TOF chopper and bender appear across independent runs, while SANS match remains mixed. For the reported specialists, the gains of 45 to 65 points also exceed the 7.7-point difference between the two guide-match seeds, although this comparison cannot replace seed replication.

Additional seeds would help establish the variability of each recipe, especially on SANS match. We believe the available evidence supports the paper's conclusions within the limits now stated, but it does not resolve the replication concern. Appendix R (Table 14) therefore separates claims supported by multiple runs from single-run findings, which remain exploratory.

Changes in the revised PDF

The abstract and Sections 1–5 have been rewritten to explain the purpose, scope and methods more directly, while keeping the main text within nine pages. Section 5.2 now points to the run comparisons in Appendix O (Table 12), and Section 5.3 discusses the analysis beyond the training range in Appendix P (Table 13). Section 6 brings these qualifications together with the cost comparison in Appendix Q and the claim-by-claim assessment in Appendix R (Table 14).

We also revised Appendix C to explain the role of classical search in T2. For Gi7G's presentation comments, we revised Appendices F and G, restructured the submission's Table 12 (now Table 15), updated six references to their published venues and added the AdamW citation in Appendix K. The accompanying code update adds docstrings to every function and class in the released package and harness; the anonymous repository has been updated and verified.

Hi reviewers,

Thank you for your reviewing service. The authors have provided rebuttals and clarifications. Please confirm that you have read them, state which of your concerns are or are not addressed, and whether that changes your recommendations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.