- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose FREA - a novel expert-curated test and corresponding benchmark for assessing retrosynthesis model ability to distinguish feasible and non-feasible reactions.
The main novelty of the paper is coming from chemical expert curation of this benchmark. The FREA test set was constructed from 4 different sources (both tested on practice and synthetic ones) and further was assessed manually by two experts who also discussed their disagreements and came to a joint decision. The only comment and corresponding question about data construction - it’s clear from Table 1, that synthetic (LP and GR) data sources disbalanced and provide significantly more infeasiable designs, while sources constructed from ORDerly (RP and ZR) provide more feasible designs than infeasible. Can models exploit this disbalance to cheat on the proposed benchmark? Let say, a model can distinguish distribution of ORDerly and synthetic sources, for example if they differ in the size of products/reactants or preference of subfragments in the source, and use it as a strong prior for feasibility/infeasibility decisions.
The experiments part is sufficient enough for this type of work, still leaving several open questions and inconsistencies. The authors compare set open-weight and 2 proprietary LLMs. Obviously, more proprietary LLMs and open-source foundation models for science in benchmark would give more significance to results, but I don't consider this as a drawback in my assessment. The main inconsistency with LLM comparison is the lack of analysis of potential data leaks. For LLMs it’s necessary to provide at least analysis of how many test object was published before each model release/publishing data, if the model has documented training procedure with explicitly stated training dataset it’s necessary to make an analysis on how many benchmark objects or similar ones (for instance, with similar scaffold) was presented during training since this could lead to test set leakage and hence make some models results less reliable.
Also, while the experiment section results analysis is pretty wide and sufficient for benchmark paper, the authors mainly focus on analysis of the results in most cases on low level (comparison of metrics values, confidence intervals) instead of analysing what exactly model lacks, on which types of test objects they make a mistake and what models lack to improve their performance.
Overall, the quality and the significance of the paper is high and raises the important question on feasibility detection in the molecular synthesis domain. The paper is easy-to-read, the text is clear. The only problem with the text is stated above - the frequent mentioning of exact model metrics values instead of high-level analysis, which breaks the paper understanding flow while not providing any useful insights.
Critical strengths
As stated in Summary, the main strength of the paper is the raising of the important question “Do molecular synthesis models determine if reaction is feasible or not”. To my knowledge, none of the previous works made systematic analysis of this question, so the paper provides a significant novelty. Also high-quality expert-curated data also make the paper significant for the area.
Critical weaknesses
None of the stated weaknesses in Summary is critical. Still authors could increase the importance and significance of the paper if they can cover stated inconsistencies from Summary.
Questions for authors
It is not clear for me after reading the text for how many products at least one feasible reaction exists. For instance, in ZR source authors report “These are ORDerly reactions with a reported yield of zero”. I’m afraid that the objects without any known feasible reaction could introduce a bias into the benchmark - instead of answering “Is this reaction feasible” model could answer “Is this product feasible”.
Replying to Authors
Dear Authors,
Thank you for your detailed responses and efforts to address my concerns. However, three major concerns remain unresolved:
1. Limited benchmark scale. The current benchmark remains relatively small, raising concerns about its comprehensiveness and generalizability. Although the authors have indicated that additional datasets will be included in the camera-ready version, I believe that such substantial additions should be evaluated during the review process. Relying on future extensions would be unfair to other submissions that have already provided comprehensive experimental evaluations.
2. Insufficient justification for binary reaction feasibility assessment. The proposed benchmark formulates reaction feasibility as a binary classification task, but its scientific significance relative to existing reaction yield prediction tasks remains unclear. Reaction yield provides a more fine-grained and informative measure of reaction outcomes, whereas binary feasibility labels discard valuable information about reaction efficiency. If feasibility is simply determined by whether a reaction succeeds or exceeds a certain yield threshold, it essentially becomes a coarse-grained version of yield prediction. I therefore remain unconvinced that the proposed binary feasibility benchmark offers sufficiently distinct scientific value beyond existing reaction yield prediction benchmarks.
3. Unclear dataset construction rationale and insufficient baseline coverage. I remain unconvinced by the motivation for constructing a small-scale benchmark through an expensive labeling process. A more straightforward and scalable alternative would be to leverage existing reaction databases, such as USPTO, as sources of positive reaction examples and generate large numbers of candidate negative examples through systematic perturbations. While such synthetic negatives would require careful validation to ensure their chemical relevance, this approach could potentially support a substantially larger benchmark at a much lower annotation cost. The authors have not sufficiently justified why their labor-intensive labeling strategy is necessary or what unique scientific value it provides over more scalable alternatives. Furthermore, the baseline evaluation is largely restricted to a subset of SMILES-based models, while several established graph-based approaches, particularly message-passing neural networks, are missing. Given the limited dataset scale, insufficient justification of the annotation strategy, and incomplete baseline coverage, I believe the current work falls considerably short of the comprehensiveness and rigor expected of a dedicated benchmarking study.
Best regards,
Reviewer