Can an Inorganic Synthesis Route Be Run? Benchmarking Laboratory Executability and Improving It from Practice Feedback

Synthesis planning for inorganic materials is evaluated by whether a generated route matches a recorded literature recipe. That criterion rewards reproducing knowledge that has already been published, and it leaves three things undetermined: whether a route can actually be run, whether the candidates are ordered by how reliable they are, and whether a model can revise its output once a route fails in the laboratory. We introduce SynBench, an executability benchmark that scores a route by whether its chemistry balances and whether every precursor can be commercially obtained, verified one by one against a real reagent platform. On 185 target materials, we evaluate sixteen LLM configurations and find that executability collapses as more candidates are requested: the executable share of the returned routes falls by 37% to 66% between the first candidate and the tenth, and the first route that cannot be run appears among the first five. We then present ExecuSyn, which uses SynBench as a training environment and, by having an LLM judge simulate a chemist’s feedback, closes the loop between the planner and the chemist so that it keeps refining its decisions during an experiment. Against the strongest evaluated model, ExecuSyn raises the executable share of the top ten predictions by 44% and moves the first unexecutable route 86% later, while the rubric score rises throughout training.