- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper addresses missing byproducts in public reaction data by formulating reaction completion as conditional generation over missing entries of an electron-redistribution (ER) matrix. RxnComp combines autoregressive neural scoring with finite-state constrained beam search intended to enforce double electron conservation, atom/charge conservation, and valence constraints. It reports 0.738 COV-JVR on USPTO-480K and 23.35/82.38 EM@1/EM@5 on RMechDB, outperforming rule-based and sequence baselines. However, the central physical premise of the ER matrix is underspecified and unvalidated, the claimed decoder guarantee is not proven, and the main reference-free evaluation is largely self-referential. The reference-based evaluation, weak supervision, and benchmark coverage are also insufficient, with only USPTO-480K and RMechDB as main benchmarks.
Critical strengths
1、It formulates reaction completion as conditional generation over missing ER-matrix entries, using autoregressive scoring and finite-state constrained decoding, which is more chemically informed than unconstrained SMILES generation.
2、Experiments provide promising evidence: RxnComp achieves a better COV-JVR trade-off in Table 1 and much higher EM@1/EM@5 (23.35/82.38) on RMechDB in Table 2, while ablations show constrained decoding is critical for validity.
Critical weaknesses
1、The paper imposes double conservation of the ER matrix as a hard decoding constraint, but neither the main text nor the appendix provides an exact construction from RDKit reactions to (ER), (EP), and (ΔE), especially regarding diagonal entries, formal charges, radicals, aromaticity, and hypervalent species. Although Appendix A.1 acknowledges that this assumption is idealized and may fail for radicals, charged intermediates, and delocalized systems, it does not quantify how many complete reactions in datasets such as USPTO-480K and RMechDB actually satisfy (ΔE1=0), nor does it analyze whether such violations are widespread, whether they undermine test-set quality, whether they filter out correct byproducts during hard-constrained decoding, or whether they bias metrics like ECR/JVR toward compliance with the method's own constraints rather than chemical correctness. Thus, this core physical premise is both underspecified and unvalidated, potentially affecting the validity of decoding and evaluation.
2、The claimed guarantee of the finite-state decoder is not established, and the key chemical constraint remains undefined. The paper claims that FSM-guided beam search can “guarantee” a completion of the ER matrix that satisfies double conservation, symmetry, boundedness, valence validity, and nonnegative bond order. However, Eq. (2) only provides local reachability conditions--whether the remaining budget can still be filled by the remaining entries at each step. These are necessary but not sufficient for global feasibility. Remaining free variables couple multiple rows and columns, and symmetry means an off-diagonal decision affects two row budgets and two column budgets simultaneously, so local feasibility does not guarantee a globally feasible completion. Second, the paper states that one slack entry per row and column is reserved and later “uniquely” determined by the remaining budgets, but it does not rigorously prove that this construction always preserves symmetry, entry bounds [−K,K], nonnegative bond order, and valence constraints, nor does it quantify closure failures. Moreover, the set Vijchem of chemically admissible values is only described verbally in both the main text and the appendix, without an executable definition covering aromatic bonds, formal charges, radicals, multiple fragments, or hypervalent species.
3、The reference-free evaluation is largely self-referential, and the supplementary evidence in the appendix is limited. The large-scale USPTO-480K evaluation mainly relies on proxy metrics such as ECR and JVR to support claims of “chemical plausibility” and “higher-quality completion candidates.” However, ECR is precisely the matrix property enforced by the FSM decoder, JVR includes ECR, and ACR is directly derived from reactant-minus-main-product composition; thus ECR/JVR in Table 1 mainly verify compliance with the method’s own hard constraints rather than byproduct correctness, and ESR is a weak discriminator for sparse graphs. Although Appendix A.4 adds a direct evaluation on 86 reactions with recorded coproducts, a blinded expert evaluation on 50 reactions, a stratified generalization analysis on 500 reactions, and robustness experiments, these are limited in scale, rely on a single expert without inter-rater agreement, and the generalization analysis still mainly depends on proxy metrics such as COV/JVR. Therefore, the self-referential issue is not fundamentally resolved, and the main large-scale conclusions still lack sufficient independent validation.
4、The paper does not fully report the RMechDB train/validation/test split sizes, splitting criteria, whether splits are separated by reaction family, random seeds, or confidence intervals, nor does it check whether the RMechDB test set overlaps with heuristic training data or whether data leakage exists. The training set is built from over 30k heuristically generated examples, while evaluation relies heavily on proxy metrics such as COV/JVR that are shared with the generation constraints, raising a template-distillation risk. Although the 500-reaction stratified analysis in Appendix A.4 partially addresses out-of-template generalization, it still mainly depends on proxy metrics, is limited in scale, and lacks independent reference validation. The main benchmarks are only USPTO-480K and RMechDB, lacking independent, more reference-complete reaction-completion benchmarks such as CompleteRXN, which makes it difficult to support the stronger claim of a broadly deployable data-curation pipeline.
5、The paper cites CompleteRXN, Zhang et al. (2024), and electron-redistribution related work but does not directly compare with them experimentally or systematically discuss their differences in related work.
Questions for authors
1、Please give a formal, implementable definition of E, including diagonal entries, formal charges, radicals, aromaticity, hypervalent species, and multiple fragments, and explain how ΔE is reconstructed from an RDKit reaction. Before any filtering, what fraction of verified complete reactions actually satisfy ΔE1=0? Appendix A.1 acknowledges that this assumption may fail for radicals, charged intermediates, and delocalized systems, but does not quantify its prevalence or impact in USPTO-480K or RMechDB.
2、Can the authors provide a proof or precise algorithmic argument that Eq. (2) plus the slack-entry closure guarantees a globally feasible symmetric completion under all remaining budget, bound, valence, and nonnegative-bond-order constraints?
3、Vijchem is only described verbally in both the main text and the appendix. Please provide an executable definition covering permitted elements, valence states, radicals, ionic structures, aromatic bonds, and diagonal entries, and explain how these rules handle RMechDB radical reactions.
4、Please report the exact RMechDB train/validation/test split sizes, split criterion, separation by reaction family if any, overlap checks against all heuristic and self-supervised training data, random seeds, and confidence intervals.
5、Can the authors provide independent evidence that high JVR corresponds to chemically correct completion, rather than merely compliance with the method’s own hard constraints?
6、The paper mainly evaluates on USPTO-480K and RMechDB but claims broad deployability for reaction-data curation. Could the authors add evaluation on CompleteRXN or another independent, more reference-complete reaction-completion benchmark?
8 comments
Further Replies to Reviewer wQWn
R3-3. Scope of the Finite-State Decoder Guarantee
We thank the reviewer for this detailed observation. We agree that Eq. (2) provides necessary but not sufficient conditions for global feasibility. However, this does not invalidate our guarantee, because global feasibility is enforced by terminal verification (Algorithm 2, Lines 20–23), rather than Eq. (2) alone.
Our decoder separates two mechanisms:
- Local pruning: Eq. (2) eliminates partial assignments whose residual electron budgets cannot be satisfied, improving search efficiency.
- Terminal acceptance: A completed matrix is returned only after verifying double conservation, symmetry, entry bounds, nonnegative bond orders, and the implemented valence constraints. Otherwise, it is rejected.
Thus, our guarantee concerns the feasibility of returned completions, not the existence of a feasible completion for every input. Our experiments support this distinction: no retained candidate in the evaluated artifacts violates the terminal matrix constraints. Moreover, JVR@1 improves from 1.8% to 87.7% with electron-budget constraints, while greedy decoding already achieves 62.4% coverage, increasing to 84.1% with beam width 10.
We have clarified the respective roles of local pruning and global verification in the revised manuscript.
R3-4. Correctness of the Slack-Entry Closure Mechanism
We clarify that slack-entry determination and global feasibility verification are separate operations.
Once the remaining entries are fixed, the reserved slack entries are determined by the residual electron budgets. Symmetric assignments are enforced, and the resulting matrix is subsequently checked against all global constraints, including entry bounds, nonnegative bond orders, and valence limits.
We do not claim that every locally feasible state necessarily admits a valid closure. Any completion violating these constraints is rejected by the terminal check. If no valid candidate remains, the reaction is counted as uncovered rather than returning an invalid completion.
Consequently, the reaction-level frequency of closure-related failures is bounded above by the uncovered fraction: 15.9% on USPTO at beam width 10. This bound also includes other search failures and should not be interpreted as the exact closure-failure rate. Importantly, no retained candidate in the evaluated artifacts violates the implemented terminal constraints.
We have clarified this closure-and-verification procedure in the revised appendix.
R3-5. Executable Definition of the Chemical Feasibility Mask
We clarify that is an explicit bond-order and valence feasibility mask, rather than an unspecified chemical-validity oracle.
A candidate bond-order change is permitted only when:
- , and the resulting bond order is nonnegative and within the permitted range.
- Bond-order increases respect the element-specific valence limits.
- The remaining electron budgets satisfy the reachability conditions in Eq. (2).
We use fixed integer bond-order-sum limits: 1 for H/halogens, 2 for O, 3 for N, 4 for C, 5 for P, and 6 for S, with 4 as the fallback for other elements. All input elements are permitted.
Regarding special chemical cases, formal charges are checked separately through charge conservation and RDKit sanitization; radical states are not explicitly tracked. Aromatic bonds use integer bond orders, multiple fragments share a common atom-index space, and diagonal entries follow the electron-budget definition in R3-1. These approximations may reject certain charged, radical, or heteroaromatic configurations.
Importantly, we acknowledge these limitations rather than claiming universal chemical validity. RMechDB directly tests the framework on radical reactions, where RxnComp achieves 91.16% coverage and 82.38% EM@5 among covered reactions. We have added the executable constraints and their limitations to the appendix.
Further Replies to Reviewer wQWn
R3-6. Clarification of Reference-Free Evaluation Metrics
We appreciate the concern that ECR/JVR partly measure constraints already enforced by our decoder. We agree that constraint satisfaction should not be conflated with chemical correctness.
However, these metrics represent chemically motivated feasibility requirements, rather than arbitrary properties introduced to favor RxnComp. Atom conservation, charge conservation, and valid bond-order assignments are necessary consistency requirements for the reaction-completion task. They are evaluated identically across all methods.
These checks are not trivially satisfied by learned baselines. For example, ACR@1 is only 0.020 for Molecular Transformer, 0.019 for T5Chem, and 0.118 for Differential-AR, despite training on the same completion data. This demonstrates that conventional sequence generation does not automatically preserve the missing atom inventory.
Nevertheless, we explicitly position ECR/JVR as reference-free feasibility metrics, not measures of experimentally verified coproduct correctness. We provide independent reference-based and expert evaluations in R3-7 to assess the latter. ESR is likewise treated as a descriptive sparsity criterion rather than independent correctness evidence.
R3-7. Independent Validation of Chemical Correctness
To address the concern regarding limited independent validation, we substantially expanded our reference-based and human expert evaluations.
We now report results on three complementary benchmarks with recorded or curated coproduct references:
| Benchmark | N | COV | Top-1 | Top-5 |
|---|---|---|---|---|
| GT84 (annotated USPTO) | 84 | 95% | EM 60.71% | EM 72.62% |
| RMechDB (zero-shot) | 249 | 91.16% | EM 23.35% | EM 82.38% |
| CompleteRXN (zero-shot) | 30,466 | 92.45% | Exact 84.31% / DeepEquiv 89.27% | Exact 87.75% / DeepEquiv 91.84% |
GT84 excludes two training-overlapping reactions from the original 86 annotated reactions. RMechDB evaluates transfer to radical chemistry, while CompleteRXN provides large-scale independent reference-based evaluation on 30,466 USPTO-derived reactions, without training or adaptation on this benchmark. EM@k on RMechDB is computed over covered reactions.
Furthermore, we expanded the blinded expert study from one to five chemists, obtaining an average acceptance rate of 67%, with Fleiss' , indicating very strong inter-rater agreement.
These results provide independent evidence beyond proxy constraint satisfaction. Reference-based evaluation measures recovery of known coproducts, while expert assessment evaluates chemical plausibility. Together, they substantially strengthen the reliability of our evaluation framework.
R3-8. RMechDB Protocol and Data Leakage
We clarify that RMechDB is exclusively used for zero-shot evaluation, with no training or validation split.
- Dataset: 249 evaluation reactions, obtained from 517 native test records through fixed parsing, product-selection, and task-alignment filters.
- Training separation: No RMechDB reactions are used for training, fine-tuning, checkpoint selection, or hyperparameter tuning. Therefore, family-wise train/test splitting within RMechDB is not applicable.
- Overlap verification: We found zero overlaps between the 249 evaluation reactions and our USPTO-derived heuristic/self-supervised training data under complete-reaction, reactants-plus-main-product, canonical-reactant, and full-product matching.
- Structural similarity: Nearest-neighbor Morgan fingerprint Tanimoto similarity averages approximately 0.25, with a maximum of approximately 0.63.
- Reproducibility: Training seed 42; deterministic beam-search inference.
We additionally computed 95% Wilson confidence intervals: COV 91.16% [87.0%, 94.1%], EM@1 23.35% [18.3%, 29.3%], and EM@5 82.38% [76.9%, 86.8%], with EM calculated over covered reactions.
These analyses provide explicit evidence against direct training–evaluation overlap and clarify the statistical uncertainty of our results.
Replying to Reviewer wQWn
Thank you for your positive feedback and valuable suggestions. We believe that all the issues raised can be effectively addressed. Our detailed point-by-point responses are provided below.
R3-1. Formal and Implementable Definition of the ER Representation
We thank the reviewer for this comment. Our ER representation uses integer bond orders, with diagonal entries defined through electron-budget bookkeeping rather than RDKit electron annotations.
For a mapped reaction with N atoms (including explicit hydrogens), let bijR and bijP denote reactant and product bond orders, respectively (zero for unbonded pairs). We define
\begin{cases} b_{ij}^{P}-b_{ij}^{R}, & i\neq j, \cr -\sum_{k\neq i}\Delta E_{ik}, & i=j. \end{cases}$$ Both ER matrices encode bond orders off-diagonal and atom-wise electron budgets on the diagonal, satisfying $E_P=E_R+\Delta E$. **Special chemical cases are handled as follows:** - **Charges and radicals:** Charge conservation is checked separately through CCR; radical states are not explicitly tracked. - **Aromaticity:** Aromatic bond orders are rounded to integers. Unchanged bonds cancel in $\Delta E$, while aromaticity changes may introduce approximation errors. - **Multiple fragments:** All fragments share one atom-index space, with zero bonds between disconnected atoms. - **Hypervalent species:** Approximated through element-specific valence limits (e.g., P and S). - **RDKit reconstruction:** Add explicit hydrogens, align atoms using mapping numbers (with structural-matching fallback), construct bond-order matrices, subtract, and close the diagonal. **Our representation provides a tractable framework for conservation-aware completion, rather than a complete electronic description.** We have clarified these details and limitations in the revised manuscript. ## R3-2. Validity and Limitations of the Double-Conservation Assumption We appreciate the reviewer's concern about the physical validity of double conservation and its potential exclusion of correct coproducts. **First, double conservation is guaranteed by construction.** Diagonal closure ensures $\Delta E\mathbf{1}=0$, while matrix symmetry guarantees zero column sums. Thus, all successfully reconstructed ER matrices satisfy this property, and counting nonzero row sums would not reveal the representation's chemical limitations. **The more relevant question is whether these constraints exclude correct completions.** We address this through reference-based evaluations: - **RMechDB (radical reactions):** 91.16% coverage and 82.38% EM@5 among covered reactions, recovering 187/249 references. - **GT84 (annotated USPTO):** 95% coverage and 72.62% EM@5 among covered reactions. **These results demonstrate substantial compatibility with reference completions, including radical reactions**, despite the representation's approximations. Remaining failures may reflect search, ranking, or representation limitations. **Second, ECR/JVR measure constraint satisfaction, not independent chemical correctness.** We therefore complement these proxy metrics with reference-based recovery and human expert evaluation. Unsuccessful generations count against coverage (COV = 0.841), rather than being treated as valid predictions. Overall, **we do not assume that our simplified ER representation captures every chemical mechanism**. Instead, we assess its practical applicability through coverage, reference recovery, and independent chemical evaluation, as clarified in the revised manuscript.