- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes TREK, an evaluation pipeline for Transition State (TS) conformer generators under both single conformer for reactant and product (Task 1) and multiple conformers (Task 2). The authors claim that current literature only focus on fixed reactant and product while there might be multiple conformers involved during the reaction hence multiple TS must be found by the TS generator. To evaluate it, authors collect multiple reaction pathways for a subset of reactions from Transition1x dataset using xTB and unspecified saddle point optimization method, then compared multiple TS generators for both Task 1 and Task 2. For Task 2, given a set of product conformers and reactant conformers, the authors rank such product-reactant pairs by atom-mapped RMSD and then query TS generator for such "close" pairs. A generation is considered successful when the generated result is within proximity to the xTB-found TS structure. The central benchmark claims are insufficiently supported. The physical reliability of the reference channels and the validity of Task 2 as a measure of channel discovery remain unresolved, limiting the interpretation of the reported model comparisons. I therefore recommend rejection.
Critical strengths
-
The paper identifies a limitation of single-proposal evaluation. Matching one TS for a known endpoint pair does not establish coverage of multiple alternative transition states. The distinction between local generation and reference-set recovery motivates a useful evaluation question.
-
The evaluation protocol accounts explicitly for duplicates and budget constraints. One-to-one matching prevents repeated predictions from inflating reference coverage, while the decomposition into pair-selection, allocation-capacity, and geometric-recovery losses helps interpret failures under a fixed proposal budget.
Critical weaknesses
-
The physical validity of collected reference channels is not clearly established. This benchmark treated xTB-derived TS structures and corresponding channels as golden standard to compare against generations, but the reported DFT audit does no support their physical validity. Consequently, the evaluation results may be off since they may be comparing to wrong references. Moreover, Transition1x contains more than 10,000 reactions where the current dataset only uses 5,000 without specifying the detailed split process. The saddle point optimization process for the benchmark is also missing.
-
Task 2’s connection to practical channel discovery is not adequately justified. Its endpoint ensembles are conditioned on accepted reference records, and credit requires both geometric agreement and membership in the recorded pair–channel relation. Valid discoveries outside that reference support receive no recall credit. Thus, the task measures recovery of a construction-conditioned reference set; the paper does not demonstrate that its model comparisons translate into meaningful channel-discovery performance.
Questions for authors
-
How were the initial 5,000 reactions selected, and why were 1,507 subsequently excluded? On a representative, independently DFT-validated subset, do the channel identities and model rankings remain stable?
-
How did you perform saddle point optimization after found those guesses with xTB?
-
How did those generators perform on the DFT-verfied subset of channels?
9 comments
Response2 to Reviewer Fx4H
The complete terminal-state accounting is:
| Documented terminal state | Reactions |
|---|---|
| At least one retained channel | 3,493 |
| Committed record, no retained channel | 1,474 |
| Missing compact summary record | 33 |
| Total | 5,000 |
The 1,474 include search-budget termination and execution outcomes such as pair-repair/NEB failures. The 33 are a separate record-completeness category with heterogeneous execution-summary states. Consequently, the difference of 1,507 is a construction outcome, not a single chemical rejection criterion.
We verified the source manifests and parsed 876,516 candidate-attempt records, with frozen configuration hashes and terminal-state summaries. We have added the seed/batch provenance, terminal-state table, and gate-specific accounting to Appendix A, and updated the construction diagram to reflect these outcomes accurately.
Q2 + W1: Missing saddle-point optimization and verification protocol
1. From xTB guesses to saddle-point optimization and acceptance checks
We agree that the refinement settings should be explicit. The production records use GFN2-xTB with charge=0 and uhf=0, followed by Sella saddle refinement and directional endpoint checks:
| Stage | Recorded configuration |
|---|---|
| Saddle optimizer | Sella |
| Maximum saddle steps | 40 |
| Saddle convergence setting | fmax=0.1 |
| Initial Sella step parameter | delta0=0.05 |
| Re-diagonalization interval | Every 3 steps |
| Directional endpoint relaxation | tblite-ASE |
| Maximum endpoint-relaxation steps | 50 |
| Endpoint convergence setting | fmax=0.2 |
| Maximum endpoint step | maxstep=0.08 |
| Directional displacement | ±0.18 Å, frozen-code default |
| Significant imaginary-frequency threshold | 100 cm⁻¹ |
| Secondary-frequency threshold | 30 cm⁻¹ |
The c75, c75-H, and c75-M configuration records specify their respective endpoint and reaction-core acceptance criteria. These are checked against the production configuration hashes, and are documented separately rather than compressed into one geometric cutoff.
The higher-level audit is a separate ORCA/B97-3c workflow with OptTS/NumFreq and bidirectional IRC. We have put the production refinement procedure and the higher-level validation procedure side by side in Appendix A/E, making clear how a path-search guess becomes an accepted xTB reference and how selected structures are subsequently examined at the DFT level.
W4: Construction-conditioned Task 2 and the limits of channel-discovery claims
1. Precisely delimit reference-set recovery versus unrestricted channel discovery
TREK evaluates a controlled and practically relevant component of channel search: recovering a reaction's reference set from R/P conformer ensembles under a finite budget. This goes beyond a known-pair local hit because the inference system must select pairs, allocate calls, and cover several targets jointly. All generators receive the same input support and allocation rules, while reference TSs and productive-pair labels remain sealed.
The construction-conditioned support is part of the task definition. It isolates the downstream recovery problem from upstream conformer generation, making model comparisons reproducible. The physical audit adds a complementary test of whether generated initial guesses refine to appropriate endpoint connections.
We have expressed this scope consistently as budgeted reaction-level reference-set recovery, with geometry-family coverage and physical optimization outcomes reported as complementary views. The value of the benchmark is the ability to measure and diagnose this set-level capability, rather than treating a high single-pair hit rate as evidence that the whole reaction ensemble has been recovered.
Response3 to Reviewer Fx4H
2. Sensitivity to pair–channel gating and uncredited out-of-reference candidates
We addressed both issues experimentally in the revised manuscript (Section 3.4, Section 4.3, Appendix D.6, Table 13; Appendix E.4):
- Removing the pair gate and scoring geometry families (Table 13): Removing the pair gate adds 44, 43, 42, 108, and 92 test-set channel matches at 0.50 Å for RobustTSGen, React-OT, TS-DFM, MolGEN, and OA-ReactDiff (raising test channel recall to 89.17%, 87.99%, 88.32%, 86.29%, and 73.60%, respectively). Direct maximum-cardinality matching to the 331 test geometry families (11,174 full-inventory) confirms that a substantial recovery gap remains even after collapsing contextual multiplicity into distinct TS geometries: under gated family matching, RobustTSGen (86.71%), TS-DFM (85.80%), and React-OT (84.89%) remain ahead of MolGEN (76.13%) and OA-ReactDiff (64.65%), while removing the gate raises MolGEN to 89.73% (tying TS-DFM).
- Physical follow-up of uncredited / out-of-reference proposals (Appendix E.4): We audited a stratified sample of 500 uncredited or unknown-pair proposals across the five generators with ORCA/B97-3c OptTS, frequency, and bidirectional IRC calculations. Among the 105 completed numerical IRC chains at the snapshot, 29 initial proposals lay >0.50 Å from every xTB reference of their reaction under ungated scoring, and 14 passed the expected-endpoint connectivity screen after refinement (e.g.,
rxn7940/MolGEN starting at 1.349 Å,rxn1788/React-OT at 0.906 Å, andrxn1825/TS-DFM at 0.784 Å).
We have added these results to Appendix D.6 and E.4, framing Task 2 explicitly as budgeted reference-set recovery over retained ensembles while reporting ungated matching, candidate-deduplicated family coverage (Table 15), and post-optimization connectivity as complementary views of discovery utility.
Response1 to Reviewer Fx4H
We thank Reviewer Fx4H for the thoughtful feedback. The corresponding revisions are highlighted in the revised manuscript.
Q1 + Q3 + W1: Physical reliability of xTB channels and DFT-verified model rankings
1. Independent DFT audit of channel validity and identity
We agree that the initial submission did not present the audit breakdown and denominators clearly enough, and we have expanded both the text and the DFT audit (Section 3.2, Figure 2d, Appendix E.1–E.4):
54/62standard,21/23hydrogen-sensitive, yielding 75/85 = 88.2% strict production pass rate, 95% CI 79.7%–93.5%), 7 pass physical connectivity checks in a separate diagnostic tier (0/15 strict), and 14 relax to alternative basins after IRC.2. Generator performance and ranking stability on the DFT-verified subset
We evaluated the exact same generator predictions against the ORCA/B97-3c optimized reference coordinates on the 473 saddle-verified test reference IDs (107 reactions) and the 89 full-IRC-chain reference IDs (53 reactions), holding predictions, pair gating, and reference IDs fixed (Appendix E.3, Table 16):
Ranking stability depends on the threshold and sample size, which we report explicitly:
We therefore restrict our ranking claims to the tier-level separation at 0.50 Å. Note that Table 16 holds the original reference IDs fixed without merging references that relax to similar DFT geometries, so it specifically measures coordinate-refinement sensitivity rather than DFT-deduplicated channel-identity stability (Appendix E.3).
Q1 + W1: Dataset selection: 5,000 initial reactions and 1,507 exclusions
1. Initial sampling protocol, filtering stages, and attrition accounting
Thank you for requesting the construction provenance. We sampled 5,000 of the 10,000 Transition1x reactions because exhaustive CREST conformer expansion followed by GFN2-xTB path searches and Sella saddle refinement already required 876,516 candidate attempts (175 attempts per reaction). The 5,000-reaction source cohort consists of two non-overlapping fixed-seed batches: 200 reactions with seed 2026080701 and 4,800 with seed 2026080801. Cohort selection did not read generator performance, and the original source partitions were preserved.