I demarcate between major points and minor points below.
MAJOR POINTS
-
The key weakness in my view is the lack of a clear purpose for the benchmark? Is the benchmark designed to galvanize advances in language agent methodology? If so, by the authors' own admission, it seems that 16 tasks is too narrow for this purpose? Is the intention for the benchmark to serve as an RL training ground for small models? If so, this should be stated and motivation given for why small models would be practically useful in real neutron instrumentation design loops.
-
A gaping weakness in my view is that the LLM writing renders the manuscript almost completely opaque. I found the manuscript grating to read, with concepts being introduced in a very roundabout way. One illustrative example is Section 3.1 which is intended to introduce the McStas software. The authors could have stated very simply here that McStas is an open-source software package used to simulate neutron scattering experiments. Instead, what is written appears to dance about a basic explanation of what the software is. While the first sentence, "Everything in NeutronGym rests on one fact about the simulator...", may be a suitable opening credit to build audience suspense for the much-anticipated cinematic adaption of NeutronGym, I would rather the authors just tell me the fact about the simulator upfront. It would life much easier for the reader if the paper was self-contained.
MINOR POINTS
-
There are some missing conference citations in the references e.g., [1] was accepted to ICML 2026, [2] was accepted to COLM 2026, [3] was accepted to ACL 2026, [4] was published in MLST, [5] was accepted to ICLR 2024, and [6] was published in the Journal of Neutron Research.
-
While the documentation in the codebase is good, it could be improved slightly by requesting that Claude Code make use of function-level docstrings.
-
The LLM-based writing style is somewhat roundabout e.g., the first sentence, "Designing a scientific instrument asks more of an agent than writing code that runs". Later on in the passage, "agreement with a reference answer" is assumed to be semantically similar to "writing code that runs". The opening passage centers on comparing the tasks agents are typically used for with the challenge of designing a scientific instrument. It would be a lot simpler for the reader if no contrast was used and the challenges for LLM agents to design scientific instruments were laid out plainly.
-
A further example of the LLM writing style is line 053, "with no LLM judge". It would be more understandable for the reader to state what you did rather than what you didn't do? What is being used for grading in place of an LLM judge?
-
Line 060, "The paper follows what happened when we used it" is yet another example of an LLM-generated sentence. It is unclear here what the referrent, "it" refers to?
-
Line 066, "past an untrained 32B that is no stronger than the 8B there" is another weird and LLM-like sentence.
-
The text of Appendix F does not read as a complete sentence.
-
The text of Appendix G does not reference any figure or table. I believe it should reference Table 6.
-
The source paper for AdamW should be cited given that it is used [7].
-
Table 12 is very difficult to read. It is difficult to understand what is being commented on in the Note column. How are descriptions such as "independent repeat" and "$9.69 for both" meant to be understood?
REFERENCES
[1] Yujiong Shen, yajie yang, Zhiheng Xi, Binze Hu, Huayu Sha, Qiyuan Peng, Jiazheng Zhang, Junlin Shang, Jixuan Huang, Yutao Fan, Jingqi Tong, Ming Zhang, Shihan Dou, Zhenfei Yin, Xingjun Ma, LEI BAI, Tao Gui, Qi Zhang, Xuanjing Huang, & Yu-Gang Jiang (2026). SciAgentGym: Benchmarking Multi-Step Scientific Tool-Use in LLM Agents. In Forty-third International Conference on Machine Learning.
[2] Xinge Liu, Terry Jingchen Zhang, Bernhard Scholkopf, Zhijing Jin, & Kristen Menou (2026). Stargazer: A Scalable Model-fitting Benchmark Environment for AI Agents under Astrophysical Constraints. In Third Conference on Language Modeling.
[3] Li, X., Chen, J., Fang, X., Ding, S., Duan, H., Liu, Q. and Chen, K., 2026, July. OPT-BENCH: Evaluating the Iterative Self-Optimization of LLM Agents in Large-Scale Search Spaces. In Findings of the Association for Computational Linguistics: ACL 2026 (pp. 28437-28452).
[4] Qasim, S.R., Owen, P. and Serra, N., 2025. Physics instrument design with reinforcement learning. Machine Learning: Science and Technology, 6(3), p.035033.
[5] Peng, B., Quesnelle, J., Fan, H. and Shippole, E., 2024, May. YaRN: Efficient context window extension of large language models. In International Conference on Learning Representations (Vol. 2024, pp. 31932-31951).
[6] Robledo, J.I., Lieutenant, K. and Willendrup, P., 2024. Small angle neutron scattering in McStas: Optimization for high-throughput virtual experiments. Journal of Neutron Research, 26(4), pp.173-185.
[7] Ilya Loshchilov, & Frank Hutter (2019). Decoupled Weight Decay Regularization. In International Conference on Learning Representations.
Reply to Reviewer 9GXq
Thank you for recognizing the simulator grading, shortcut checks and reward-ladder ablation. We agree that the paper needed to distinguish those contributions more clearly from the narrower design and training results.
A. Scope: fixed layouts (weakness 1)
We agree that our framing should have made the fixed-layout scope clearer. We have now stated that limit throughout the revision. The abstract says that "the agent sets the parameters of a fixed instrument layout", and the introduction explains that "choosing a layout (the open-design tier) is not scored". Section 6 states that "No scored task asks for a topology choice" and limits the conclusion to "the level of a fixed layout's parameters". We release the open-design tier unscored because we do not yet have an expert rubric for grading a free layout.
Within this limited scope, we believe the parameters pose a useful design problem: their effects interact through physics that the agent must reason about or measure. Our shortcut probes offer some support for this interpretation. The best fixed answer or lookup passes at most 12.7% in any admitted family, and no rule that simply copies prompt quantities succeeds; on TOF chopper, no readout rule passes at all. Training raises the untrained 8B's 2–26% pass rates to 44–77% across the four held-out families. We interpret these results as evidence for parameter-level design; topology design remains open. Extending the families to component types or counts would require applying the same admission checks before interpreting any new training gains.
A. Size of the benchmark (weakness 2, first part)
We also agree that 16 tasks with one episode per task and model cannot establish a ranking. As Section 4.2 notes, the Wilson interval for 7/16 spans 23–67%, and Table 1's caption says that no per-model difference is significant. We therefore use McStasBench to establish whether reproduction is possible and to describe failures: some instruments were rebuilt, no episode retrieved a reference, and no agent met an improvement target. The procedural evaluations address learning on larger sets of instances, but do not remove the curated benchmark's coverage limit.
B. Exploratory RL evidence and which claims remain (weakness 2, question)
We could not add seeds because our training cluster is down. Topic B of the general comment explains how we used the archived runs instead: Appendix O (Table 12) reports 15 run–family comparisons from specialist and joint training, of which 12 beat the untrained model. Since the joint runs pool families, these comparisons provide supporting evidence across runs rather than additional seeds of the specialist recipe.
To answer your question about what remains, Appendix R (Table 14) assesses each claim separately. The environment, admission test and four rejected task designs do not depend on single-seed training results. We believe the McStasBench observations remain supported as existence results, and the frontier models remain ahead on the reported evaluation.
Among the training findings, guide match has two specialist seeds, at 76.7% and 69.0% against the untrained 32B's 11.3%. Gains over the untrained 8B also appear in 3/4 TOF-chopper runs and 2/2 bender runs, although those counts include joint training. The physics-seeded search matches or beats the trained model at equal simulation budget, and all three attempts to fine-tune on the model's own successes made it worse. Together, these results offer support beyond one training run, although they do not settle the reproducibility of each specialist recipe.
The remaining findings retain the exploratory label: SANS match improves in only 2/4 runs; removing partial credit costs 60 points in a one-run-per-condition ablation; and the four-family policy, transfer, tolerance and range tests each rely on one seed. Although the ablation's effect is about eight times the 7.7-point gap between the two guide-match seeds, we do not treat that comparison as replication.
We hope the revision makes it easier to assess the environment and admission protocol, the RL gains supported across runs, and the findings that still need replication. Thank you for prompting us to make these distinctions clearer. We would be grateful if you would consider the revised scope and evidence in reassessing your rating.