- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents PreWeave, a Qwen3.5-9B agent for inorganic precursor recommendation with four tools: retrieval of similar recipes and a conditional precursor completion model (both from He et al. 2023), plus valence and element-coverage checks. The agent is trained by SFT on teacher trajectories that match a reference answer, followed by online RL (GSPO) with an exact-match reward. On the He et al. benchmark (2,654 test targets), Pass@1 goes from 53.81 (untrained agent) to 71.89 (SFT) to 73.02 (SFT+RL); answer-only SFT gives 68.88, and MSP-LLM on the same backbone gives Hit@1 73.89. Analyses show that tools add 3.6 points, that gains saturate at four calls, and that training mainly makes answers consistent across runs while coverage stays flat.
Critical strengths
-
The experiments have the right controls: an untrained agent with tools, answer-only SFT without tools, the same 9B backbone under a task-specific method (MSP-LLM), and frontier LLMs with the identical tools. The appendix is complete (sampled MSP-LLM, a fixed-tool-order control, per-target gain and loss counts, RL dynamics, dataset accounting and costs).
-
The pipeline is specified well enough to reproduce, on a public benchmark, with tools, trajectories and code to be released. The analysis is candid even where it undercuts the story: tools add little, the chemical checks do not help, and training does not expand coverage.
Critical weaknesses
-
The headline 53.81 to 73.02 is against an untrained agent. Against answer-only SFT the gain is 4.1 points at Pass@1 and 1.9 at Pass@8, and that control was decoded with settings tuned for Pass@4, so the matched gap is probably 2-3 points. MSP-LLM on the same backbone, with no tools, no teacher and no RL, has Hit@1 73.89 vs Pass@1 73.02 and Hit@8 85.34 vs 78.33; under sampling (Table 7) PreWeave leads by 0.64 points, i.e. 17 targets. RL adds 1.13 points on a single run. Pass@1 is the first run only (App. B.1), and no standard error or paired test appears anywhere although eight runs exist. With one standard error is about 0.9 points, so most of the differences discussed in Sec. 4-5 (the RL gain, the tool ablations of 0.1-0.2 points, the budget effects) is inside noise.
-
Tools add 3.58 points at inference, about half from a single retrieval call; the SFT agent with tools removed (68.31) is below answer-only SFT (68.88), so trajectory supervision by itself gives nothing; removing either chemical check slightly raises Pass@1. The consistency result (8-of-8 hits from 14% to 66%) comes with an unchanged zero-hit set (578, 577, 575 targets), so training sharpens the output distribution rather than solving new targets; a low-temperature or majority-vote evaluation of the base agent, which costs nothing extra, would show how much of the headline gain is sharpening. Finally, the recipe (distilled SFT, then group-relative RL with an outcome reward) is the standard one from Search-R1, ToolRL and ReTool, none of which is cited, and there is no RL-only run. The real contribution is the environment, the data and the characterization, and the framing should say so.
Questions for authors
- Please report mean SD over the eight runs for every row in Table 2 Panel B and Table 3, and a paired test for SFT vs SFT+RL.
- What does the base agent reach with greedy or low-temperature decoding, or with majority vote over its eight runs?
- How often does the final answer equal the top retrieved recipe or the top MPC suggestion?
- MSP-LLM (Qwen3.5-9B) is your own re-training on this split; please give its training details, and say why SynReason is cited but not compared.