acceptodds
Withdrawn from ICLR 2027

Can an Inorganic Synthesis Route Be Run? Benchmarking Laboratory Executability and Improving It from Practice Feedback

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Synthesis planning for inorganic materials is evaluated by whether a generated route matches a recorded literature recipe. That criterion rewards reproducing knowledge that has already been published, and it leaves three things undetermined: whether a route can actually be run, whether the candidates are ordered by how reliable they are, and whether a model can revise its output once a route fails in the laboratory. We introduce SynBench, an executability benchmark that scores a route by whether its chemistry balances and whether every precursor can be commercially obtained, verified one by one against a real reagent platform. On 185 target materials, we evaluate sixteen LLM configurations and find that executability collapses as more candidates are requested: the executable share of the returned routes falls by 37% to 66% between the first candidate and the tenth, and the first route that cannot be run appears among the first five. We then present ExecuSyn, which uses SynBench as a training environment and, by having an LLM judge simulate a chemist’s feedback, closes the loop between the planner and the chemist so that it keeps refining its decisions during an experiment. Against the strongest evaluated model, ExecuSyn raises the executable share of the top ten predictions by 44% and moves the first unexecutable route 86% later, while the rubric score rises throughout training.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Inorganic synthesis planning, large language models, continual learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_E2TzOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

In one sentence, this paper defines inorganic synthesis planning simply as precursor prediction, redefines the metrics based on lab execution, and proposes a refinement loop driven by LLM-as-judge feedback. From the reviewer’s perspective, the content of the paper, including the title, is considerably overclaimed, the methodology is not technically novel, and it does not present a new methodology that integrates domain knowledge well. In addition, comparisons with essential baselines are missing.

Critical strengths
  • The motivation is good. The authors consider executability and attempt to evaluate beyond literature recipe matching.
Critical weaknesses
  • This work performs precursor recommendation. Synthesis planning broadly consists of precursors, synthesis operations, and conditions, yet the authors ignore the latter and speak as if precursor prediction were the entirety of synthesis planning. Without operations and conditions, the authors claim “executable” and “Can a route be run?” Precursors are certainly important in synthesis because they are the starting materials, but without considering operations and conditions, precursors alone cannot constitute an executable route. Given that many prior works are extending synthesis planning by addressing each of these components individually or in an integrated manner, it is difficult to agree with the authors’ claims.

  • Paradoxically, the authors state in their comparison with LLMs that LLMs perform poorly at precursor prediction, yet they use that same LLM as the judge, which introduces circular reasoning. It is difficult to claim that a closed loop for materials synthesis has been built through this kind of modeling alone. Materials synthesis differs from general domains where LLM judges can be readily applied, so treating the evaluator simply as an LLM has limitations and requires care.

  • (1) The authors did not verify how well the judge’s decisions agree with experts or with deterministic classification rules, and they even use an outdated model. In addition, validation of agreement with experts is also needed.

  • (2) Furthermore, I cannot agree that the LLM judge constitutes a chemist’s feedback. Real experimentalists obtain feedback from synthesis outcomes (e.g., XRD) and then run experiments again. The type of feedback introduced in the paper is unlikely to produce practical value.

  • There is no methodological comparison at all with the specialized precursor prediction models introduced as existing methods. If precursor prediction is treated as the main scope, as it currently is, comparison with these models is essential. Such baseline comparisons are entirely absent. [1,2,3,4,5]

  • The chemical validity, precursor obtainability, etc. described in SynBench are overly simple, and prior work has not been sufficiently explored. Many recent works already cover nearly all elements of the target material. In addition, prior works also (1) perform matching via CAS, and (2) since environments differ across laboratories and users can sufficiently adapt precursor availability to their own settings, this is not a major issue.

  • The proposed FirstFail@K also does not seem to be an appropriate metric. What matters to a chemist is not the position of the first failure but the position of the first success.

[1] Kim, Seongmin, Yousung Jung, and Joshua Schrier. "Large language models for inorganic synthesis predictions." Journal of the American Chemical Society 146.29 (2024): 19654-19659.

[2] Kim, Seongmin, et al. "Predicting synthesis recipes of inorganic crystal materials using elementwise template formulation." Chemical Science 15.3 (2024): 1039-1045.

[3] Noh, Heewoong, et al. "Retrieval-retro: retrieval-based inorganic retrosynthesis with expert knowledge." Advances in Neural Information Processing Systems 37 (2024): 25375-25400.

[4] Prein, Thorben, et al. "Retro-rank-in: a ranking-based approach for inorganic materials synthesis planning." arXiv preprint arXiv:2502.04289 (2025).

[5] Noh, Heewoong, et al. "MSP-LLM: A Unified Large Language Model Framework for Complete Material Synthesis Planning." arXiv preprint arXiv:2602.07543 (2026).

Questions for authors

None.

Reviewer_K85MOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces SynBench, a benchmark for precursor recommendation in solid-state inorganic synthesis that combines a formula-based chemical-validity check with precursor obtainability. The authors constructs synthesis routes and metrics named Exec@K and FirstFail@K to evaluate LLM performance.

Critical strengths
  • The proposed refinement process with simulated chemist feedbacks improves the performance a lot.
  • The presentation is clear, the paper is easy to follow.
Critical weaknesses

See the questions.

Questions for authors
  • Why are C, N, H, and O excluded from the chemical-validity check? The criteria may reject some routes that are executable in practice. If these elements are exempted, what justifies treating P and the halogens differently? The design of this criteria may be problematic which requires further discussion. A rule without counterexamples may be more appropriate for evaluating synthesis routes.

  • For a benchmark with llms, the selection of evaluated models is relatively outdated.

  • The paper lacks of validation such as real synthesis experiments or evaluation on out-of-distribution synthesis routes.

  • The source corpus contains 1,987 distinct precursor formulas, but only 251 are labeled obtainable from supplier. Please provide more information about the numeric difference. The standard of starting materials may be "common precursors" not materials from supplier, since the author want to achieve "a route can actually be run" and "how reliable they are" (words from abstract), materials from inorganic materials literature are already "real to run, reliable to synthesize‌", which is different from organic retro-synthesis (where lots of starting molecules require complex and multiple synthesis routes).

Reviewer_bwq9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a framework to evaluat (SynBench) and finetune (ExecuSyn) inorganic synthesis planners to take into account 1) the chemical validity of the proposed routes, and 2) the practical availability of their precursors. They show that fine-tuning LLMs within this framework leads to considerably more executable routes per target.

Critical strengths
  • Good presentation: The paper is well written with clear figures and a well-organized appendix.
  • Clear motivation: The authors motivate their work with an evaluation of existing methods identifying a clear gap in executability as the number of routes suggested increases. Their new method is then designed to target specifically this gap.
Critical weaknesses

My two main issues with the current version of the paper are the difficulty establishing the method's performance gains under the current evaluation setup, and the overselling of the practice-feedback contribution. Below are suggestions to address both and thus increase my score:

  • [W1] The current empirical evaluation is weak, and possibly inflates ExecuSyn's performance:

    • Any precursor not in the fixed precursor database (375 precursors, of which 66.9% are labelled obtainable) or in a cache of previous online lookups (size not reported) is considered unobtainable (App. C.4): this penalizes baselines which may return obtainable precursors not in the precursor list.
    • (a) The gains in Exec@10 are expected, given that the model was fine-tuned on this specific metric. It does not mean the model learned better planning or chemistry, as opposed to simply memorizing (the rather small dataset) of obtainable material: The authors report topk accuracy in Fig. 4 (f) to show that matching the literature is unaffected, but they report no baselines on the same metric to compare to (e.g. the same models before fine-tuning, the methods reported in Table 1, other inorganic methods cited in the literature, etc). A LitHit@10 of 0.65–0.75 (Table G.1) cannot be interpreted as 'no cost to literature knowledge'. I suggest the authors include LitHit@10 for baselines to quantify their claim. The authors should also use additional metrics that approximate executability, and which were not used to finetune the model, like a thermodynamic check [1, 2], precursor diversity to check for potential mode collapse, and blind expert evaluation if accessible.
    • (b) The claim that the precursor database can be swapped for another catalogue (App. B.2) is not tested: the authors could use a different list of purchasable materials (e.g. the Thermo Fisher Scientific catalogue [3], or a laboratory's own stock list), unseen during finetuning, to test this claim.
    • (c) Unfair comparison to baselines: the baselines never see the list of obtainable materials. It would be interesting to see the effect of providing this list in the prompt, or of using it to filter the precursors returned by the baselines, so that they have access to the same level of information as ExecuSyn.
    • (d) Missing baselines: dedicated inorganic methods are cited but never evaluated (e.g. Retrieval-Retro [4], Retro-Rank-In [5], MSP-LLM [6]). Another interesting simple baseline would be to enumerate all the obtainable precursors from the training routes that supply a single element (e.g. Li → Li2CO3, LiOH, LiNO3; Fe → Fe2O3, Fe3O4, FeC2O4·2H2O; P → NH4H2PO4, (NH4)2HPO4.), and for each target list all possible combinations and rank them based on their occurrence in the training set, and choose the top 10 as viable routes. Since the validity check ignores C, N, H and O (Algorithm C.1), all such routes are chemically valid by construction. This would show whether Exec@10 can be increased without learning any chemistry at all, by simply composing routes from the obtainable list of precursors.
    • (e) The practice-feedback evaluation is circular: the same LLM judge (Claude-Sonnet-5) and rubric provide both the phase 2 training reward and the held-out practice-rubric score (Fig. 5c, Table 3), and the judge's verdicts are never compared to those of chemists.
  • [W2] The practice-feedback simulated chemist (Figure 2 b) is oversold:

    • (a) Overclaiming on closing the loop (point (iii) in the introduction): the LLM standing in for a chemist only checks whether each precursor has literature precedents for a given target; this is a far cry from providing experimental information to refine the precursor choice. I think the claim on closing the loop should be walked back accordingly.
    • (b) The feedback is only used during training: at test time, the model receives no feedback and does not revise its routes for a given target (App. F). The feedback is thus akin to fine-tuning with a second reward, while it is described as a planner refining its own experiment.
    • (c) The GRPO objective in phase 2 is not well defined: phase 2 uses a single rollout per target (Table F.2, App. F.3), while GRPO estimates the advantage of a rollout relative to the other rollouts of the same target (Sec. 4.1). With a group of one, this advantage is zero, so it is unclear what the GRPO term contributes, and what the 'GRPO-only' ablation in Table 3 measures.
  • (nitpicks) Small changes to the paper's presentation:

    • Some of the fonts used in Figure 2 are too small and make the figure difficult to read.
    • The 'lab-executable' criterion is essentially chemical validity and precursor availability. Executability is a big claim based on these metrics. I suggest renaming the criterion to something like 'chemical or execution plausibility'.
    • The evaluation metrics used in the protocol resemble commonly used metrics in organic synthesis: Solve Rate is the fraction of targets for which at least one route connecting the target to purchasable starting materials can be found. Syntheseus [7] further proposes to filter out any chemically invalid intermediate steps (evaluated in organic chemistry through parsing with RDKit [8]). Furthermore, Exec@K is close to the top-k validity metrics used in single-step retrosynthesis [9]. This resemblance should be acknowledge in the paper, otherwise it seems like the evaluation protocol overclaims novelty.
Questions for authors

Questions

  • Which baseline is used to compute the advantage in the phase 2 GRPO loss, given a single rollout per target?
  • How large is the cache of online lookups, and how was it populated?

References

[1] Muratahan Aykol, Joseph H. Montoya, and Jens Hummelshøj. Rational solid-state synthesis routes for inorganic materials. Journal of the American Chemical Society, 143(24):9244–9259, 2021.

[2] Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, and Kristin A. Persson. Commentary: The Materials Project: A materials genome approach to accelerating materials innovation. APL Materials, 1(1):011002, 2013.

[3] Thermo Fisher Scientific. Chemicals catalogue. https://www.thermofisher.com.

[4] Heewoong Noh, Namkyeong Lee, Gyoung S. Na, and Chanyoung Park. Retrieval-Retro: Retrieval-based inorganic retrosynthesis with expert knowledge. In Advances in Neural Information Processing Systems, volume 37, 2024.

[5] Thorben Prein, Elton Pan, Sami Haddouti, Marco Lorenz, Janik Jehkul, Tymoteusz Wilk, Cansu Moran, Menelaos Panagiotis Fotiadis, Artur P. Toshev, Elsa Olivetti, and Jennifer L. M. Rupp. Retro-Rank-In: A ranking-based approach for inorganic materials synthesis planning. arXiv preprint arXiv:2502.04289, 2025.

[6] Heewoong Noh, Gyoung S. Na, Namkyeong Lee, and Chanyoung Park. MSP-LLM: A unified large language model framework for complete material synthesis planning. arXiv preprint arXiv:2602.07543, 2026.

[7] Krzysztof Maziarz, Austin Tripp, Guoqing Liu, Megan Stanley, Shufang Xie, Piotr Gaiński, Philipp Seidl, and Marwin Segler. Re-evaluating retrosynthesis algorithms with Syntheseus. arXiv preprint arXiv:2310.19796, 2023.

[8] RDKit: Open-source cheminformatics. https://www.rdkit.org.

[9] Yue Wan, Chang-Yu Hsieh, Benben Liao, and Shengyu Zhang. Retroformer: Pushing the limits of end-to-end retrosynthesis transformer. In Proceedings of the 39th International Conference on Machine Learning, PMLR 162, 2022.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.