acceptodds
Under review as a conference paper at ICLR 2027

FREA: A Multi-Source Expert Benchmark for Reaction Feasibility Verification

Abstract

Reaction feasibility verification is essential for synthesis planning, yet progress is constrained by limited evaluation resources and scarce negative data. We introduce FREA, an expert-annotated benchmark of reactions spanning retrosynthesis proposals, zero-yield experimental records, perturbations by large language models (LLMs), and candidates from five generation methods. This multi-source design enables a broad assessment of verifier strengths and failure modes. Each reaction is manually annotated by expert chemists under carefully specified criteria. We evaluate frontier LLMs, forward models, and learned classifiers on FREA. Frontier LLMs show strong performance, with the best attaining the highest mean balanced accuracy across sources and closely matching expert rankings of retrosynthesis models. Forward models achieve the best performance on retrosynthesis proposals, with likelihood scoring providing an efficient option for large-scale screening. However, no model performs best across all four sources, and each exhibits distinct failure patterns. We also release a corpus of over 14 million recorded reactions and generated negative candidates. Exploratory training experiments suggest some transfer from generated negatives to other reaction sources, and generating negative data that reliably improve generalization remains an open question. Together, the benchmark and corpus provide a shared platform for diagnosing verifiers and studying how to improve them.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
chemistry, AI for Science, retrosynthesis, synthesis planning, chemical reaction feasibility, benchmark, dataset, molecule

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1usyOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose FREA - a novel expert-curated test and corresponding benchmark for assessing retrosynthesis model ability to distinguish feasible and non-feasible reactions.

The main novelty of the paper is coming from chemical expert curation of this benchmark. The FREA test set was constructed from 4 different sources (both tested on practice and synthetic ones) and further was assessed manually by two experts who also discussed their disagreements and came to a joint decision. The only comment and corresponding question about data construction - it’s clear from Table 1, that synthetic (LP and GR) data sources disbalanced and provide significantly more infeasiable designs, while sources constructed from ORDerly (RP and ZR) provide more feasible designs than infeasible. Can models exploit this disbalance to cheat on the proposed benchmark? Let say, a model can distinguish distribution of ORDerly and synthetic sources, for example if they differ in the size of products/reactants or preference of subfragments in the source, and use it as a strong prior for feasibility/infeasibility decisions.

The experiments part is sufficient enough for this type of work, still leaving several open questions and inconsistencies. The authors compare set open-weight and 2 proprietary LLMs. Obviously, more proprietary LLMs and open-source foundation models for science in benchmark would give more significance to results, but I don't consider this as a drawback in my assessment. The main inconsistency with LLM comparison is the lack of analysis of potential data leaks. For LLMs it’s necessary to provide at least analysis of how many test object was published before each model release/publishing data, if the model has documented training procedure with explicitly stated training dataset it’s necessary to make an analysis on how many benchmark objects or similar ones (for instance, with similar scaffold) was presented during training since this could lead to test set leakage and hence make some models results less reliable.

Also, while the experiment section results analysis is pretty wide and sufficient for benchmark paper, the authors mainly focus on analysis of the results in most cases on low level (comparison of metrics values, confidence intervals) instead of analysing what exactly model lacks, on which types of test objects they make a mistake and what models lack to improve their performance.

Overall, the quality and the significance of the paper is high and raises the important question on feasibility detection in the molecular synthesis domain. The paper is easy-to-read, the text is clear. The only problem with the text is stated above - the frequent mentioning of exact model metrics values instead of high-level analysis, which breaks the paper understanding flow while not providing any useful insights.

Critical strengths

As stated in Summary, the main strength of the paper is the raising of the important question “Do molecular synthesis models determine if reaction is feasible or not”. To my knowledge, none of the previous works made systematic analysis of this question, so the paper provides a significant novelty. Also high-quality expert-curated data also make the paper significant for the area.

Critical weaknesses

None of the stated weaknesses in Summary is critical. Still authors could increase the importance and significance of the paper if they can cover stated inconsistencies from Summary.

Questions for authors

It is not clear for me after reading the text for how many products at least one feasible reaction exists. For instance, in ZR source authors report “These are ORDerly reactions with a reported yield of zero”. I’m afraid that the objects without any known feasible reaction could introduce a bias into the benchmark - instead of answering “Is this reaction feasible” model could answer “Is this product feasible”.

Reviewer_ZHdVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a key but inconsistently evaluated step in retrosynthesis planning: reaction feasibility verification. The authors note that existing feasibility filters are mostly evaluated on negatives constructed by the same procedure that generated them, so it remains untested whether their negative labels agree with chemists' judgments; existing expert evaluations (e.g., expert review of complete routes) also differ in unit of evaluation. The authors therefore introduce the FREA benchmark: an explicit feasibility definition separating completeness (every substantive reactant must be listed; only auxiliaries that contribute no retained heavy atoms may be assumed) from reactivity (reasonable conditions exist under which the listed reactants give isolable products in a single step or credible one-pot operation).

Critical strengths

The empirical finding that source cannot determine feasibility has methodological value. That 79 of 93 zero-yield records are judged feasible and 33 of 289 intended-negative candidates are judged feasible (Table 1, Appendix C.1) directly supports the core argument that using constructed negatives or failed experiments as negative labels introduces label noise, and that independent expert judgment is needed as a reference. Table C.1 further reports per-method "negative purity" (only 77.2% for FT versus about 93% for RR/NC/ET), enabling the later argument that FT's performance decline cannot be attributed to the generation method independently of label noise.

Critical weaknesses

Benchmark size and minority-class counts within sources limit resolution. Of the 751 reactions, ZR has only 14 infeasible, LP only 32 feasible, and GR only 33 feasible (Table G.3), and Appendix F.5 acknowledges wide intervals on ZR and LP; the 282 RP reactions come from eight models, about 35 each, while the Spearman correlations in Table G.1 (e.g., Claude Opus 5's ρ = +0.93 with interval [+0.31, +0.96]) are very wide, and the authors note that the expert ranking reproduces itself at only ρ = 0.86 under resampling. At this scale, conclusions such as "one system ranks models better on RP" are easily driven by individual examples.

Questions for authors
  1. Scale and coverage: with minority classes of only 14/32/33 for ZR/LP/GR, are there plans to expand annotation to support within-source comparisons? Are the model rankings in Table G.1 stable given 282 proposals (about 35 per model), and can leave-one-model-out or leave-one-subset-out sensitivity be reported?

  2. Comparability and label independence: does the possible overlap between the AiZynthFinder filter and FREA (92/93 for ZR, 33/87 for LP) affect how its performance is read in the main text? Since LP was generated by one model generation and evaluated by the next, can a same-generation generate-and-evaluate control be added to test whether the negative result in Table G.5 is robust?

Reviewer_AUdLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces FREA, an expert-annotated benchmark for evaluating reaction feasibility verification. The benchmark contains 751 reactions from four sources: retrosynthesis proposals, zero-yield experimental records, LLM-generated perturbations, and synthetically generated reaction candidates. Each reaction is labeled by chemical experts according to a predefined feasibility criterion. The authors evaluate several verification approaches, including frontier LLMs, forward reaction prediction models, and discriminative classifiers, revealing substantial performance differences across reaction sources. The paper also releases a large corpus of recorded reactions and generated negative candidates and investigates their usefulness for training feasibility verifiers. The results highlight challenges in generalizing reaction feasibility assessment across different reaction distributions.

Critical strengths

The paper addresses a practically relevant problem in reaction feasibility verification and provides an expert-annotated benchmark covering multiple reaction sources. The inclusion of both real-world reaction proposals and generated negative candidates offers a useful resource for examining the limitations of existing verification approaches.

Critical weaknesses
  1. Ambiguous task definition and substantial uncertainty in expert feasibility labels. The paper defines reaction feasibility based on whether reasonable conditions exist under which the proposed reactants can produce the target product. However, reaction conditions are not provided, and the definition allows various auxiliary reagents and reaction procedures to be implicitly assumed. This introduces substantial subjectivity into the feasibility assessment. More importantly, the independent expert annotations achieve only moderate agreement (Cohen's κ = 0.55), with 168 disagreements among 759 independently annotated reactions. Although the authors perform adjudication, the resulting binary labels do not fully capture this underlying uncertainty. The paper should examine how annotation disagreements affect benchmark reliability and the reported model rankings.

  2. Limited benchmark scale and potentially unrepresentative evaluation distribution. Despite being presented as a multi-source benchmark, FREA contains only 751 expert-annotated reactions, with highly uneven feasible/infeasible distributions across its four sources. For instance, the zero-yield subset contains only 14 infeasible reactions, while the generated-reaction subset contains only 33 feasible reactions. Such small subgroup sizes limit the reliability of source-specific comparisons and make fine-grained conclusions difficult to establish. Furthermore, the unweighted average across four differently constructed sources does not necessarily reflect the distribution of reactions encountered in practical synthesis planning.

  3. Insufficient demonstration of practical utility for retrosynthesis planning. Although the paper motivates reaction feasibility verification as an essential component of retrosynthesis planning, its evaluation is largely restricted to reaction-level classification and correlations with expert rankings of retrosynthesis models. It remains unclear whether better performance on FREA translates into improved planning outcomes, such as identifying more feasible synthesis routes or reducing unsuccessful reaction proposals. Without downstream planning experiments or stronger evidence connecting benchmark scores to practical utility, the significance of the proposed benchmark remains insufficiently demonstrated.

  4. Limited demonstrated benefits of the large-scale negative reaction corpus. The paper releases a large corpus of generated negative reactions, but the training experiments provide limited evidence of its practical effectiveness. Although negative supervision improves average AUROC in some settings, it does not consistently improve performance on retrosynthesis proposals, which represent one of the primary intended applications. Furthermore, the reported source-specific improvements do not remain statistically significant after multiple-comparison correction. Consequently, the value of the generated corpus for improving real-world reaction feasibility verification remains uncertain.

  5. Potential confounding effects from source-specific generation artifacts. The benchmark combines retrosynthesis proposals, zero-yield experimental records, LLM perturbations, and synthetically generated reactions, each associated with distinct construction procedures and label distributions. Although expert annotation avoids directly equating generated candidates with infeasible reactions, source-specific structural artifacts may still influence verifier performance. The substantial variations across negative generation methods suggest that models may exploit dataset-specific patterns rather than learning general chemical feasibility. More controlled experiments would help establish whether the benchmark genuinely measures transferable reaction feasibility assessment across different sources.

Questions for authors
  1. Given the relatively low inter-annotator agreement (Cohen's κ = 0.55), how reliable are the expert feasibility labels used as ground truth? Could the authors investigate whether the reported model comparisons and rankings remain consistent when evaluated only on reactions where both experts independently agreed, or when accounting for annotation uncertainty?

  2. What is the practical value of FREA for improving retrosynthesis planning? Although the paper evaluates reaction-level feasibility classification and correlations with expert rankings, it remains unclear whether better benchmark performance translates into improved synthesis planning outcomes. Could the authors provide downstream experiments demonstrating that verifiers performing better on FREA can identify more feasible synthesis routes or improve planning reliability?

3 comments
Reviewer_AUdLOpenReview ↗

Replying to Authors

Dear Authors,

Thank you for your detailed responses and efforts to address my concerns. However, three major concerns remain unresolved:

1. Limited benchmark scale. The current benchmark remains relatively small, raising concerns about its comprehensiveness and generalizability. Although the authors have indicated that additional datasets will be included in the camera-ready version, I believe that such substantial additions should be evaluated during the review process. Relying on future extensions would be unfair to other submissions that have already provided comprehensive experimental evaluations.

2. Insufficient justification for binary reaction feasibility assessment. The proposed benchmark formulates reaction feasibility as a binary classification task, but its scientific significance relative to existing reaction yield prediction tasks remains unclear. Reaction yield provides a more fine-grained and informative measure of reaction outcomes, whereas binary feasibility labels discard valuable information about reaction efficiency. If feasibility is simply determined by whether a reaction succeeds or exceeds a certain yield threshold, it essentially becomes a coarse-grained version of yield prediction. I therefore remain unconvinced that the proposed binary feasibility benchmark offers sufficiently distinct scientific value beyond existing reaction yield prediction benchmarks.

3. Unclear dataset construction rationale and insufficient baseline coverage. I remain unconvinced by the motivation for constructing a small-scale benchmark through an expensive labeling process. A more straightforward and scalable alternative would be to leverage existing reaction databases, such as USPTO, as sources of positive reaction examples and generate large numbers of candidate negative examples through systematic perturbations. While such synthetic negatives would require careful validation to ensure their chemical relevance, this approach could potentially support a substantially larger benchmark at a much lower annotation cost. The authors have not sufficiently justified why their labor-intensive labeling strategy is necessary or what unique scientific value it provides over more scalable alternatives. Furthermore, the baseline evaluation is largely restricted to a subset of SMILES-based models, while several established graph-based approaches, particularly message-passing neural networks, are missing. Given the limited dataset scale, insufficient justification of the annotation strategy, and incomplete baseline coverage, I believe the current work falls considerably short of the comprehensiveness and rigor expected of a dedicated benchmarking study.

Best regards,
Reviewer

AuthorsOpenReview ↗

Rebuttal 2-1

Dear Reviewer AUdL,

Thank you for engaging with our responses. We address the three remaining concerns below. Concerns 2 and 3 raise new questions that were not part of the original review.


C1. Benchmark scale

As in the paper and our earlier responses, we acknowledge that scale limits fine-grained comparisons, but the current results already support the paper's main conclusions, and the paper is careful and frank about what the data do and do not support. Please see our previous response to W2a for the details. We do not ask that the paper be judged on future data: the planned expansion will sharpen fine-grained comparisons, but the main conclusions do not depend on it.


C2. Binary feasibility versus yield prediction

Feasibility verification is an established and still unmet need in retrosynthesis planning, and it answers a different question from yield prediction rather than a coarser version of it.

  1. Feasibility verification is a task retrosynthesis planning already depends on, not one we introduce here. A verifier decides which proposed reactions survive the search, so its errors shape the routes that planning returns. Planning systems already include one: the in-scope filter of Segler et al. [1], the filter policy of AiZynthFinder [2], the explicit feasibility model of Retro-fallback [3] and the scorer ensemble that prunes the search tree in RetroTrim [5], while RetroGFN uses a feasibility model to guide training [4]. Doing this reliably is still open: reactions proposed by generative models are frequently chemically unsound [5], a solved route need not be chemically valid [6], and the literature describes in-silico feasibility evaluation as lacking [6], how to verify a reaction's feasibility as unclear [4], and how to train a high-quality feasibility model as an open research question [7]. FREA measures how well current verifiers make this decision against expert judgment.
  2. Feasibility is not a coarse version of yield, because it is not defined by an outcome or by a threshold on one. A yield measures a single reaction under the conditions actually used, whereas feasibility asks whether any reasonable conditions exist under which the given reactants produce the target. The submission draws this distinction directly: §3 states that the labels do not predict yield and that a zero-yield record can still be labeled feasible, and App. A separates the two tasks. The two therefore come apart in one direction: a successful experiment shows that a transformation works, but a failed one shows only that the conditions tried did not, which is why 79 of the 93 zero-yield records in FREA are judged feasible (§4.2). No threshold on yield recovers this, so yield data cannot supply the infeasible side of the question. In the setting that matters, a yield is not even available: yield benchmarks are built from reactions that were run under chosen conditions, whereas retrosynthesis proposals come without conditions and mostly without an experimental record, yet a verifier still has to accept or reject each one. Asking this binary question also discards nothing at evaluation, since FREA scores every system with a continuous output by AUROC, and reaction efficiency becomes the question at a later stage, once a step is chosen and its conditions are optimized.

References

[1] Segler et al. Planning chemical syntheses with deep neural networks and symbolic AI. Nature, 2018.

[2] Genheden et al. A quick policy to filter reactions based on feasibility in AI-guided retrosynthetic planning. ChemRxiv, 2020.

[3] Tripp et al. Retro-fallback: retrosynthetic planning in an uncertain world. ICLR, 2024.

[4] Gaiński et al. Diverse and feasible retrosynthesis using GFlowNets. Information Sciences, 2025.

[5] Sadowski et al. Trustworthy retrosynthesis: eliminating hallucinations with a diverse ensemble of reaction scorers. arXiv:2510.10645, 2025.

[6] Torren-Peraire et al. Models matter: the impact of single-step retrosynthesis on synthesis planning. Digital Discovery, 2024.

[7] Maziarz et al. Re-evaluating retrosynthesis algorithms with syntheseus. Faraday Discussions, 2025.

AuthorsOpenReview ↗

Rebuttal 2-2

C3a. Expert labels versus generated negatives

This is the question FREA is built to answer. The construction you describe is exactly how we build our corpus, and the paper's central finding is that it does not give a reliable or representative benchmark.

  1. We do construct negatives by large-scale generation, exactly as you describe, and our expert labels show that the resulting labels are not reliable. We propose five construction methods and apply them at corpus scale (§4.1, Figure 1). Their candidates also enter FREA as the GR source. According to our expert labels, 33 of the 289 are in fact feasible (§4.2), and negative purity varies across the methods (§7.2). Labels that come from construction are therefore wrong often enough that they cannot serve as the reference, and building a larger benchmark this way would only repeat the error at a larger scale. As your reply notes, such negatives would need careful validation to ensure their chemical relevance. Expert annotation is that validation.
  2. Even if every label were correct, a benchmark built from one construction can only tell how a verifier does on that construction. Each method breaks a recorded reaction in its own characteristic way, so its candidates share the patterns of that method, whereas the proposals a retrosynthesis model produces fail in their own ways. A system that scores well on one construction has handled that construction, and whether it also handles the reactions a verifier meets in practice is a separate question. FREA is multi-source so that this question can be answered: the proposals of real retrosynthesis models, which are what matters in deployment, sit in the benchmark alongside generated candidates, experimental records and LLM perturbations, and all of them are judged by the same experts under one criterion, which makes results on different kinds of reactions directly comparable. And the answer is that the sources behave very differently. A system near the top on generated candidates can sit near the bottom on retrosynthesis proposals, and the reverse (Table 2). Our training study points the same way, with the gains from learning on generated negatives concentrated outside retrosynthesis proposals (§7.1). A score measured on constructed reactions is therefore not a guide to the setting a verifier is built for, and showing this at all requires a benchmark that holds several kinds of reactions, and that is FREA.

C3b. Baseline coverage

The evaluation is not restricted to SMILES-based models, though a graph-based baseline is indeed missing. Besides the SMILES sequence models and the two LLMs, it includes a gradient-boosted tree on chemical descriptors, the fingerprint-based AiZynthFinder filter and a rule-based baseline (§5, App. E). We will add a message-passing neural network baseline in the camera-ready version.


If our responses have addressed any of your concerns, we would be grateful if you would consider revisiting your assessment. We would of course be glad to take up any suggestions you might have in the camera-ready version, and we strongly believe FREA will be a useful resource for the community.

Thank you again for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.