- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
In one sentence, this paper defines inorganic synthesis planning simply as precursor prediction, redefines the metrics based on lab execution, and proposes a refinement loop driven by LLM-as-judge feedback. From the reviewer’s perspective, the content of the paper, including the title, is considerably overclaimed, the methodology is not technically novel, and it does not present a new methodology that integrates domain knowledge well. In addition, comparisons with essential baselines are missing.
Critical strengths
- The motivation is good. The authors consider executability and attempt to evaluate beyond literature recipe matching.
Critical weaknesses
-
This work performs precursor recommendation. Synthesis planning broadly consists of precursors, synthesis operations, and conditions, yet the authors ignore the latter and speak as if precursor prediction were the entirety of synthesis planning. Without operations and conditions, the authors claim “executable” and “Can a route be run?” Precursors are certainly important in synthesis because they are the starting materials, but without considering operations and conditions, precursors alone cannot constitute an executable route. Given that many prior works are extending synthesis planning by addressing each of these components individually or in an integrated manner, it is difficult to agree with the authors’ claims.
-
Paradoxically, the authors state in their comparison with LLMs that LLMs perform poorly at precursor prediction, yet they use that same LLM as the judge, which introduces circular reasoning. It is difficult to claim that a closed loop for materials synthesis has been built through this kind of modeling alone. Materials synthesis differs from general domains where LLM judges can be readily applied, so treating the evaluator simply as an LLM has limitations and requires care.
-
(1) The authors did not verify how well the judge’s decisions agree with experts or with deterministic classification rules, and they even use an outdated model. In addition, validation of agreement with experts is also needed.
-
(2) Furthermore, I cannot agree that the LLM judge constitutes a chemist’s feedback. Real experimentalists obtain feedback from synthesis outcomes (e.g., XRD) and then run experiments again. The type of feedback introduced in the paper is unlikely to produce practical value.
-
There is no methodological comparison at all with the specialized precursor prediction models introduced as existing methods. If precursor prediction is treated as the main scope, as it currently is, comparison with these models is essential. Such baseline comparisons are entirely absent. [1,2,3,4,5]
-
The chemical validity, precursor obtainability, etc. described in SynBench are overly simple, and prior work has not been sufficiently explored. Many recent works already cover nearly all elements of the target material. In addition, prior works also (1) perform matching via CAS, and (2) since environments differ across laboratories and users can sufficiently adapt precursor availability to their own settings, this is not a major issue.
-
The proposed FirstFail@K also does not seem to be an appropriate metric. What matters to a chemist is not the position of the first failure but the position of the first success.
[1] Kim, Seongmin, Yousung Jung, and Joshua Schrier. "Large language models for inorganic synthesis predictions." Journal of the American Chemical Society 146.29 (2024): 19654-19659.
[2] Kim, Seongmin, et al. "Predicting synthesis recipes of inorganic crystal materials using elementwise template formulation." Chemical Science 15.3 (2024): 1039-1045.
[3] Noh, Heewoong, et al. "Retrieval-retro: retrieval-based inorganic retrosynthesis with expert knowledge." Advances in Neural Information Processing Systems 37 (2024): 25375-25400.
[4] Prein, Thorben, et al. "Retro-rank-in: a ranking-based approach for inorganic materials synthesis planning." arXiv preprint arXiv:2502.04289 (2025).
[5] Noh, Heewoong, et al. "MSP-LLM: A Unified Large Language Model Framework for Complete Material Synthesis Planning." arXiv preprint arXiv:2602.07543 (2026).
Questions for authors
None.