acceptodds
Under review as a conference paper at ICLR 2027

TREK: Benchmarking Reaction-Level Transition-State Ensemble Recovery

10 views

Abstract

Accurate transition-state generation for a known reactant–product pair does not establish recovery of the alternatives associated with a reaction. Measuring this broader capability requires a benchmark whose unit is the reaction and whose output is a set. We introduce **TREK** (Transition-state Recovery for Ensemble Kinetics), a benchmark for budgeted TS-ensemble recovery from complete retained reactant and product conformer sets. TREK organizes 3,493 reactions, 10,865 candidate endpoint pairs, and 19,647 xTB-derived proxy channels into two controlled tasks: known-pair one-shot generation and reaction-level recovery with target-blind pair selection and a shared proposal budget. Pair-aware one-to-one scoring measures distinct reference recovery, while a stagewise decomposition attributes the recovery gap to selection, capacity, and geometry/diversity. Across five generators, leading models achieve 94.0–95.6% pair-hit rates on held-out reactions at 0.50 Å, alongside 74.3–76.1% complete-reference-set recovery in the ensemble task. TREK further identifies a tolerance-dependent shift from geometric precision to output capacity, quantifies the coverage gained per additional proposal, and isolates 38 shared zero-hit reactions enriched in internal endpoint rearrangement. Frozen DFT and bidirectional IRC audits provide complementary evidence for individual reference candidates and conformationally distinct connections. TREK establishes a common experimental basis for developing generators that recover reaction-level structural diversity, connecting accurate individual proposals to budget-efficient ensemble coverage.

Primary area:
datasets and benchmarks
Keywords:
AI for science, transition state generation, conformer ensembles, reaction kinetics, molecular generative models, benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Fx4HOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes TREK, an evaluation pipeline for Transition State (TS) conformer generators under both single conformer for reactant and product (Task 1) and multiple conformers (Task 2). The authors claim that current literature only focus on fixed reactant and product while there might be multiple conformers involved during the reaction hence multiple TS must be found by the TS generator. To evaluate it, authors collect multiple reaction pathways for a subset of reactions from Transition1x dataset using xTB and unspecified saddle point optimization method, then compared multiple TS generators for both Task 1 and Task 2. For Task 2, given a set of product conformers and reactant conformers, the authors rank such product-reactant pairs by atom-mapped RMSD and then query TS generator for such "close" pairs. A generation is considered successful when the generated result is within proximity to the xTB-found TS structure. The central benchmark claims are insufficiently supported. The physical reliability of the reference channels and the validity of Task 2 as a measure of channel discovery remain unresolved, limiting the interpretation of the reported model comparisons. I therefore recommend rejection.

Critical strengths
  1. The paper identifies a limitation of single-proposal evaluation. Matching one TS for a known endpoint pair does not establish coverage of multiple alternative transition states. The distinction between local generation and reference-set recovery motivates a useful evaluation question.

  2. The evaluation protocol accounts explicitly for duplicates and budget constraints. One-to-one matching prevents repeated predictions from inflating reference coverage, while the decomposition into pair-selection, allocation-capacity, and geometric-recovery losses helps interpret failures under a fixed proposal budget.

Critical weaknesses
  1. The physical validity of collected reference channels is not clearly established. This benchmark treated xTB-derived TS structures and corresponding channels as golden standard to compare against generations, but the reported DFT audit does no support their physical validity. Consequently, the evaluation results may be off since they may be comparing to wrong references. Moreover, Transition1x contains more than 10,000 reactions where the current dataset only uses 5,000 without specifying the detailed split process. The saddle point optimization process for the benchmark is also missing.

  2. Task 2’s connection to practical channel discovery is not adequately justified. Its endpoint ensembles are conditioned on accepted reference records, and credit requires both geometric agreement and membership in the recorded pair–channel relation. Valid discoveries outside that reference support receive no recall credit. Thus, the task measures recovery of a construction-conditioned reference set; the paper does not demonstrate that its model comparisons translate into meaningful channel-discovery performance.

Questions for authors
  1. How were the initial 5,000 reactions selected, and why were 1,507 subsequently excluded? On a representative, independently DFT-validated subset, do the channel identities and model rankings remain stable?

  2. How did you perform saddle point optimization after found those guesses with xTB?

  3. How did those generators perform on the DFT-verfied subset of channels?

9 comments
AuthorsOpenReview ↗

Response1 to Reviewer Fx4H

We thank Reviewer Fx4H for the thoughtful feedback. The corresponding revisions are highlighted in the revised manuscript.

Q1 + Q3 + W1: Physical reliability of xTB channels and DFT-verified model rankings

1. Independent DFT audit of channel validity and identity

We agree that the initial submission did not present the audit breakdown and denominators clearly enough, and we have expanded both the text and the DFT audit (Section 3.2, Figure 2d, Appendix E.1–E.4):

  1. Complete breakdown of the 100-candidate random audit (Appendix E.1): Among 100 uniformly sampled xTB candidates, 96 / 100 (96.0%, Wilson 95% CI 90.2%–98.4%) converge to a legitimate ORCA/B97-3c first-order saddle with a single significant imaginary mode (only 4 fail OptTS/NumFreq). All 96 saddles complete bidirectional IRC: 75 pass strict endpoint-commitment criteria in the two production profiles (54/62 standard, 21/23 hydrogen-sensitive, yielding 75/85 = 88.2% strict production pass rate, 95% CI 79.7%–93.5%), 7 pass physical connectivity checks in a separate diagnostic tier (0/15 strict), and 14 relax to alternative basins after IRC.
  2. Expansion to all 591 test-set reference inputs (Appendix E.3): Submitting all 591 test-set xTB references to the same ORCA/B97-3c workflow yields 473 / 591 (80.0%) verified first-order DFT saddles at the frozen snapshot, with a median xTB-to-DFT coordinate shift of only 0.138 Å (P90 = 0.392 Å), and 89 references with completed bidirectional IRC + terminal-minimum chains across 53 test reactions. Given the available computational resources and time budget, we report the completed results at this frozen snapshot. We will continue the remaining verification and update the manuscript upon completion.
  3. Basin-resolved multi-channel identity (Appendix E.2): The 8-reaction multi-round pilot canonicalizes 96 complete DFT/IRC certificates into 11 distinct basin-to-basin channels (with 2–3 distinct channels per reaction in four reactions), establishing conformationally distinct connections at the DFT/IRC level.

2. Generator performance and ranking stability on the DFT-verified subset

We evaluated the exact same generator predictions against the ORCA/B97-3c optimized reference coordinates on the 473 saddle-verified test reference IDs (107 reactions) and the 89 full-IRC-chain reference IDs (53 reactions), holding predictions, pair gating, and reference IDs fixed (Appendix E.3, Table 16):

Generator473 IDs @0.20 Å (xTB → DFT)473 IDs @0.50 Å (xTB → DFT)89 IDs @0.20 Å (xTB → DFT)89 IDs @0.50 Å (xTB → DFT)
RobustTSGen52.01 → 38.27%84.14 → 76.32%56.18 → 41.57%93.26 → 87.64%
React-OT58.35 → 63.42%82.66 → 77.17%61.80 → 70.79%87.64 → 80.90%
TS-DFM54.12 → 50.53%83.72 → 76.11%58.43 → 50.56%91.01 → 80.90%
MolGEN49.05 → 47.15%71.04 → 65.33%65.17 → 55.06%88.76 → 85.39%
OA-ReactDiff40.17 → 47.15%61.52 → 59.62%53.93 → 62.92%84.27 → 80.90%

Ranking stability depends on the threshold and sample size, which we report explicitly:

  1. On the 473-ID set at 0.50 Å, the tier separation is preserved: the leading three generators (React-OT 77.17%, RobustTSGen 76.32%, TS-DFM 76.11%) remain within ~1 pp of one another and clearly separated from MolGEN (65.33%) and OA-ReactDiff (59.62%). On the smaller 89-ID subset (53 reactions), all five models achieve 80.90%–87.64% at 0.50 Å without wide tier separation.
  2. At 0.20 Å, rankings shift with training-target alignment: pure DFT-trained React-OT (58.35% → 63.42%) and OA-ReactDiff (40.17% → 47.15%) improve when scored against DFT coordinates, whereas RobustTSGen (fine-tuned on Transition1x-xTB; Appendix C) drops from 52.01% to 38.27% (moving from third to fifth).

We therefore restrict our ranking claims to the tier-level separation at 0.50 Å. Note that Table 16 holds the original reference IDs fixed without merging references that relax to similar DFT geometries, so it specifically measures coordinate-refinement sensitivity rather than DFT-deduplicated channel-identity stability (Appendix E.3).

Q1 + W1: Dataset selection: 5,000 initial reactions and 1,507 exclusions

1. Initial sampling protocol, filtering stages, and attrition accounting

Thank you for requesting the construction provenance. We sampled 5,000 of the 10,000 Transition1x reactions because exhaustive CREST conformer expansion followed by GFN2-xTB path searches and Sella saddle refinement already required 876,516 candidate attempts (175 attempts per reaction). The 5,000-reaction source cohort consists of two non-overlapping fixed-seed batches: 200 reactions with seed 2026080701 and 4,800 with seed 2026080801. Cohort selection did not read generator performance, and the original source partitions were preserved.

AuthorsOpenReview ↗

Response2 to Reviewer Fx4H

The complete terminal-state accounting is:

Documented terminal stateReactions
At least one retained channel3,493
Committed record, no retained channel1,474
Missing compact summary record33
Total5,000

The 1,474 include search-budget termination and execution outcomes such as pair-repair/NEB failures. The 33 are a separate record-completeness category with heterogeneous execution-summary states. Consequently, the difference of 1,507 is a construction outcome, not a single chemical rejection criterion.

We verified the source manifests and parsed 876,516 candidate-attempt records, with frozen configuration hashes and terminal-state summaries. We have added the seed/batch provenance, terminal-state table, and gate-specific accounting to Appendix A, and updated the construction diagram to reflect these outcomes accurately.

Q2 + W1: Missing saddle-point optimization and verification protocol

1. From xTB guesses to saddle-point optimization and acceptance checks

We agree that the refinement settings should be explicit. The production records use GFN2-xTB with charge=0 and uhf=0, followed by Sella saddle refinement and directional endpoint checks:

StageRecorded configuration
Saddle optimizerSella
Maximum saddle steps40
Saddle convergence settingfmax=0.1
Initial Sella step parameterdelta0=0.05
Re-diagonalization intervalEvery 3 steps
Directional endpoint relaxationtblite-ASE
Maximum endpoint-relaxation steps50
Endpoint convergence settingfmax=0.2
Maximum endpoint stepmaxstep=0.08
Directional displacement±0.18 Å, frozen-code default
Significant imaginary-frequency threshold100 cm⁻¹
Secondary-frequency threshold30 cm⁻¹

The c75, c75-H, and c75-M configuration records specify their respective endpoint and reaction-core acceptance criteria. These are checked against the production configuration hashes, and are documented separately rather than compressed into one geometric cutoff.

The higher-level audit is a separate ORCA/B97-3c workflow with OptTS/NumFreq and bidirectional IRC. We have put the production refinement procedure and the higher-level validation procedure side by side in Appendix A/E, making clear how a path-search guess becomes an accepted xTB reference and how selected structures are subsequently examined at the DFT level.

W4: Construction-conditioned Task 2 and the limits of channel-discovery claims

1. Precisely delimit reference-set recovery versus unrestricted channel discovery

TREK evaluates a controlled and practically relevant component of channel search: recovering a reaction's reference set from R/P conformer ensembles under a finite budget. This goes beyond a known-pair local hit because the inference system must select pairs, allocate calls, and cover several targets jointly. All generators receive the same input support and allocation rules, while reference TSs and productive-pair labels remain sealed.

The construction-conditioned support is part of the task definition. It isolates the downstream recovery problem from upstream conformer generation, making model comparisons reproducible. The physical audit adds a complementary test of whether generated initial guesses refine to appropriate endpoint connections.

We have expressed this scope consistently as budgeted reaction-level reference-set recovery, with geometry-family coverage and physical optimization outcomes reported as complementary views. The value of the benchmark is the ability to measure and diagnose this set-level capability, rather than treating a high single-pair hit rate as evidence that the whole reaction ensemble has been recovered.

AuthorsOpenReview ↗

Response3 to Reviewer Fx4H

2. Sensitivity to pair–channel gating and uncredited out-of-reference candidates

We addressed both issues experimentally in the revised manuscript (Section 3.4, Section 4.3, Appendix D.6, Table 13; Appendix E.4):

  1. Removing the pair gate and scoring geometry families (Table 13): Removing the pair gate adds 44, 43, 42, 108, and 92 test-set channel matches at 0.50 Å for RobustTSGen, React-OT, TS-DFM, MolGEN, and OA-ReactDiff (raising test channel recall to 89.17%, 87.99%, 88.32%, 86.29%, and 73.60%, respectively). Direct maximum-cardinality matching to the 331 test geometry families (11,174 full-inventory) confirms that a substantial recovery gap remains even after collapsing contextual multiplicity into distinct TS geometries: under gated family matching, RobustTSGen (86.71%), TS-DFM (85.80%), and React-OT (84.89%) remain ahead of MolGEN (76.13%) and OA-ReactDiff (64.65%), while removing the gate raises MolGEN to 89.73% (tying TS-DFM).
  2. Physical follow-up of uncredited / out-of-reference proposals (Appendix E.4): We audited a stratified sample of 500 uncredited or unknown-pair proposals across the five generators with ORCA/B97-3c OptTS, frequency, and bidirectional IRC calculations. Among the 105 completed numerical IRC chains at the snapshot, 29 initial proposals lay >0.50 Å from every xTB reference of their reaction under ungated scoring, and 14 passed the expected-endpoint connectivity screen after refinement (e.g., rxn7940/MolGEN starting at 1.349 Å, rxn1788/React-OT at 0.906 Å, and rxn1825/TS-DFM at 0.784 Å).

We have added these results to Appendix D.6 and E.4, framing Task 2 explicitly as budgeted reference-set recovery over retained ensembles while reporting ungated matching, candidate-deduplicated family coverage (Table 15), and post-optimization connectivity as complementary views of discovery utility.

Reviewer_ETtGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a new benchmark for evaluating transition-state generation of whole reactions and not just individual reactant–product pairs. Existing transition state generative models only answer the question whether we can generate a correct transition state given a pair of reactant and product. However, one reaction may have various reactant and product conformations such that there may be multiple transition states. This is a many-to-many problem, and performing well on specific reactant–product pairs does not mean a model can perform on all alternatives associated with the reaction. The TREK benchmark aims to address this with a much more thorough set of information about each reaction including a set of possible outputs. It evaluates existing models on the original task and a new task, i.e. how many different transition states that one model can retrieve given the complete conformer set and finite sampling budget.

Critical strengths

The newly defined problem is a reasonable and valuable one. The statistical reports are organized and fine-grained. The evaluation criterion is clean and transferrable, e.g. the selector and allocator are fixed for all models and blind to the answer. The error decomposition is exact. An interesting negative result on “new coordinate of transition state doesn’t refer to useful diversity”, i.e. the popular diversity metrics like pairwise RMSD spread and uniqueness are not an alternative for how many ground truth targets are actually covered.

Critical weaknesses
  • Some standard baselines are missing in the benchmarking, e.g. racerTS, NEB, GSM, etc.
  • There are multiple saddle points between each pair of endpoints based on the xTB-computed transition states. However, xTB is very coarse. In appendix E.2, the DFT calculations show something opposite, i.e. there exists only one transition state between a pair of endpoints. This makes one question the the claim of “capacity bottleneck”
  • Around 70% of reactions only have one pair of reactant-product conformers. Task 2 is not a hard one in this setting.
  • The task seems easier than the realistic setting as the model’s input is already filtered, i.e. the conformations that cannot find a transition states are removed before evaluation.
  • The ground truth targets are calculated using xTB. In 100 DFT-validated samples, only 75 of them are correct. So around a fourth of the ground truth targets might be wrong. Most of the models are trained using DFT-based data but the evaluation is using xTB-based data. There may be a mismatch here. Note that 37 out of the 38 reactions that all five models fail on, 37 are in the training set. These failures suggest that xTB-based references may differ from the original DFT-based targets.
  • If the models are discovering some new transition states that are not in the labelled set, they are not able to get the reward. As a result, this benchmark is encouraging the models to reproduce the results of a coarse-grained method, not discover something new.
  • The test set only has 109 reactions, which is not enough.
  • Some reactions have as many as 43 answers. However, the model only has 24 chances to generate transition states so it is not possible to capture all ground truth targets.
Questions for authors
  • The generative process of React-OT is fixed. How does it generate different structures? How does a different model create such generation difference? If the repeated structures are merged, what is the effective sampling budget for React-OT in practice?
  • What is the criterion for identifying different transition state structures? How many answers fall within the evaluation threshold and how are these cases handled when measuring coverage?
  • What if we replace the input with unfiltered conformations?
  • Can you evaluate more reactions using DFT such that we can understand better if one pair of endpoints correspond to more transition states?
8 comments
AuthorsOpenReview ↗

Response to Reviewer ETtG

We thank Reviewer ETtG for the thoughtful feedback. The corresponding revisions are highlighted in the revised manuscript.

W1. Missing standard baselines (racerTS, NEB, GSM)

1. Baseline inclusion criteria and possible supplementary comparisons

Thank you for recommending conventional pathway-search baselines. We added ORCA NEB-TS calculations on the exact same 30 productive endpoint pairs used in our generator audit. NEB-TS was initialized from DFT-optimized reactant and product endpoints without a reference TS guess, whereas the five generators' frozen proposals entered local ORCA/B97-3c saddle optimization directly:

MethodInput requirementFirst-order-saddle checks passed (/ 30)Pass rate
ORCA NEB-TSOptimized R/P endpoints + iterative pathway search16 / 3053.3%
RobustTSGen + local OptTSDirect neural proposal from R/P27 / 3090.0%
React-OT + local OptTSDirect neural proposal from R/P27 / 3090.0%
MolGEN + local OptTSDirect neural proposal from R/P27 / 3090.0%
OA-ReactDiff + local OptTSDirect neural proposal from R/P25 / 3083.3%
TS-DFM + local OptTSDirect neural proposal from R/P22 / 3073.3%

On the same 30 pairs, direct neural proposals achieve substantially higher first-order saddle convergence (73.3%–90.0%) than de novo NEB-TS pathway searches (53.3%), while avoiding the iterative multi-image force evaluations required during NEB path construction (at the frozen snapshot, 9 NEB-TS chains and 2–5 generator chains in this 30-pair slice had finished full bidirectional IRC + terminal optimization queues, alongside 593 completed generator IRC chains across Task 1 and Task 2-core).

For racerTS, the input conditions differ materially: it requires a TS structure and reaction-center information, whereas TREK supplies only R/P endpoint ensembles and seals the reference TS structures. This distinction is already discussed in Section 2. Providing a reference TS to racerTS would turn the experiment into a TS-initialized conformer-expansion task, not the R/P-to-TS recovery task evaluated here. We have made the baseline-inclusion criterion explicit and added the NEB-TS comparison to the revised paper.

Q4 + W2 + W5: Physical validity of xTB channels, multi-saddle claims, and DFT mismatch

1. DFT validation of same-pair multiplicity and the capacity-bottleneck claim

Two distinct notions of multiplicity are involved here. Reaction-level multiplicity does not require every canonical endpoint pair to have several DFT saddles. Appendix E.2 reports 11 canonical DFT/IRC connections across the pilot reactions; rxn2556 has three connections, and rxn6192, rxn6194, and rxn8551 each have two. These connections belong to distinct canonical endpoint-basin pairs while sharing the mapped net reaction topology within each reaction. They directly support conformationally distinct reaction-level pathways.

The pilot therefore does not show that every pair has at most one possible TS; it shows the connections found and validated in that search. We agree that xTB proxy multiplicity should be distinguished from multiple DFT-certified saddles for a single canonical pair, and have stated this distinction prominently in Section 3.2.

The capacity term is also precisely defined: it counts reference targets that the planned pair-specific slots cannot cover, even with perfect geometry. It is a combinatorial property of the benchmark task. Our added family matching and candidate-deduplication analyses further separate this slot capacity from effective geometric sampling. Together, these analyses preserve the reaction-level motivation while giving the capacity result its correct interpretation.

2. Impact of xTB–DFT mismatch, the 75/100 audit, and the 37/38 failure pattern

The statement that one quarter of the production references are incorrect does not follow from the reported 75/100 statistic. The composition of the audit, already given in Appendix E.1, is:

Audit groupCandidatesStrictly accepted
Standard profile6254
Hydrogen-sensitive profile2321
Diagnostic profile150
Entire audit10075

Seven diagnostic candidates pass the profile-aware physical checks but lie outside that audit's strict inventory. Thus, the full mixed audit gives 75/100 strict passes, while its two strict-profile groups give 75/85 (88.24%). We have shown the group composition in the main text and Figure 2d so that these outcomes are not conflated.

AuthorsOpenReview ↗

Response2 to Reviewer ETtG

We also directly tested the reference-level mismatch concern by optimizing all 591 test-set xTB references with ORCA/B97-3c: 473 / 591 (80.0%) converged to first-order saddles with a median xTB-to-DFT shift of 0.138 Å (P90 = 0.392 Å). Replacing xTB coordinates with DFT coordinates on these 473 fixed IDs (Appendix E.3, Table 16) shows that at 0.50 Å, the leading three models remain within ~1 pp of each other (77.17%, 76.32%, 76.11%) and ahead of MolGEN (65.33%) and OA-ReactDiff (59.62%). At 0.20 Å, score shifts (-13.74 to +6.98 pp) are sensitive to the reference level and partially reflect training targets: pure DFT-trained React-OT (58.35% → 63.42%) and OA-ReactDiff (40.17% → 47.15%) improve against DFT coordinates, whereas the Transition1x-xTB fine-tuned RobustTSGen checkpoint drops from 52.01% to 38.27%. We now report these DFT-coordinate scores alongside xTB scores in Appendix E.3.

Finally, 37/38 shared-failure reactions belonging to the source-training split matches the base rate: 3,317 / 3,493 reactions (95.0%) in TREK belong to the training split, showing no statistical enrichment (odds ratio 0.84, Holm-adjusted p=1.0p = 1.0; Table 12). However, the reviewer's intuition regarding larger xTB-to-DFT shifts on these hard reactions is supported by our supplementary ORCA audit of the 38-reaction cohort: their xTB-to-DFT reference shift has a larger median of 0.226 Å (P90 = 0.562 Å, vs. 0.138 Å on test), combined with significantly larger internal endpoint rearrangement (ΔD=0.679\Delta_D = 0.679 vs. 0.540A˚0.540\text{Å}, p=6.84×10−5p = 6.84\times 10^{-5}; best-model Task 1 RMSD 0.6100.610 vs. 0.072A˚0.072\text{Å}). Notably, when the generators' initial guesses on these 38 zero-hit reactions are subjected to local ORCA optimization, at least one generator proposal converges to a first-order DFT saddle in all 38/38 reactions (with 18/38 having completed full IRC chains at the snapshot), confirming that geometric misses on large-rearrangement reactions can reflect both reference shift and transport difficulty.

W3 + W7. Task 2 difficulty, multi-pair prevalence, and test-set size

1. Performance stratification by number of productive endpoint pairs

One candidate pair does not make Task 2 a single-target problem, and multi-pair reactions show a steep additional drop. Across TREK, 2,854 of 5,525 productive pairs support multiple geometry families (4,633 support multiple proxy channels), so even single-pair reactions require covering multiple TS alternatives rather than achieving one local hit. Stratifying the 109 test reactions into the 68 single-candidate-pair reactions (∣Ei∣=1|\mathcal{E}_i|=1) and 41 multi-candidate-pair reactions (∣Ei∣≥2|\mathcal{E}_i|\ge 2), alongside the 92 reactions with zero selection and capacity losses (ui=si=∣Ci∣u_i=s_i=|\mathcal{C}_i|, 382 channels; Table 14), reveals clear separation at 0.50 Å channel recall:

CohortRobustTSGenTS-DFMReact-OTMolGENOA-ReactDiff
Single-candidate-pair test reactions (68)93.09%91.87%91.06%93.90%82.52%
Multi-candidate-pair test reactions (41)73.62%73.62%73.33%49.57%40.58%
Zero selection & capacity loss ($u_i=s_i=\mathcal{C}_i$, 92)94.24%92.41%91.88%

In the 41 multi-pair reactions—where pair selection and cross-pair budget allocation are active—recall drops to 73.33%–73.62% for the top three models and 40.58%–49.57% for OA-ReactDiff and MolGEN. Even in the 92 reactions where every reference is addressable without selection or slot-capacity loss, recovery remains incomplete (72.51%–94.24%), and within-pair candidate deduplication (W8.2) further separates single-shot precision from multi-modal diversity. We have added these stratified results to Section 4.3 and Appendix D.6 (Table 14).

2. Test-set representativeness and statistical uncertainty

We address sample-size uncertainty at the reaction level, preserving the dependence among a reaction's pairs and targets. The 109 reactions are the retained source-test partition; the construction did not select them based on generator performance. Appendix D.2 reports 2,000 whole-reaction bootstrap replicates and paired comparisons with Holm correction.

For example, original-protocol test recall at 0.50 Å is 81.7% [75.1, 88.0] for RobustTSGen and 80.7% [74.1, 87.1] for React-OT. Accordingly, the paper treats the leading methods as a statistically unresolved group under that metric, rather than asserting a decisive ordering. The new family results are presented as point estimates on the same test cohort.

The complementary 3,493-reaction analysis establishes the inventory-wide pattern and is labeled separately as descriptive. The new near-duplicate-controlled result appears in both cohorts: MolGEN reaches 73.41% test family recall and 76.23% full-inventory family recall. We have presented cohort definitions and uncertainty alongside the principal results.

AuthorsOpenReview ↗

Response3 to Reviewer ETtG

Q3 + W4: Filtered endpoint ensembles versus realistic unfiltered inputs

1. Evaluation with unfiltered endpoint conformers

The key distinction is between retaining endpoint conformers and revealing productive endpoint pairs. TREK retains conformers supported by accepted records, but exposes their full Cartesian product to the selector. The test set therefore contains 372 candidate pairs, including 189 unknown combinations; the full inventory contains 10,865 candidate pairs, including 5,340 unknown combinations. The selector receives neither productive-pair labels nor reference TS structures.

Thus, pair selection remains a genuine inference task within the retained endpoint support. An all-unfiltered-conformer input is a different, broader stress test; a direct performance comparison for that setting is not included in the completed experiments. We have made the retained-input definition explicit in Section 3.1 and distinguished it from unrestricted conformer search.

2. Definition of the task setting and its scope

TREK is designed to isolate reaction-level TS recovery from a specified conformational input, just as Task 1 isolates TS generation given a known pair. This controlled construction allows the same input support, target-blind selector, and allocation rule to be used for every generator. It makes the change from a local prediction problem to a set-recovery problem measurable.

Expanding the input with additional conformers can be evaluated as a separate input-support experiment, while preserving the same scoring machinery. We have described this separation clearly: the present benchmark evaluates selection, allocation, and generation over retained ensembles; conformer generation is an upstream component.

Q2 + W6: Incomplete reference coverage and criteria for distinct transition states

1. Identity/deduplication rules and close-target overlap at the scoring threshold

TREK distinguishes contextual proxy identity from geometry-family identity. A proxy retains endpoint-family information, TS geometry family, atom mapping, and acceptance profile. Different contexts can therefore share one geometry family. This gives 591 contextual channels and 331 families on Test, and 19,647 channels and 11,174 families on All.

Structural scoring uses all-atom RMSD with proper alignment and equal-element permutation, at 0.20 and 0.50 Å, followed by maximum-cardinality one-to-one assignment. Each proposal and each target can receive at most one assignment. When scoring families directly, distance is minimized over admissible family members; with the pair gate enabled, members must belong to the proposal's recorded pair relation.

The reviewer's question also exposes a separate issue: the scoring neighborhoods of distinct targets can overlap, allowing near-coincident candidates to fill different slots. By construction, collapsing the 19,647 contextual channels into 11,174 geometry families merges 8,473 channels (43.1%; 260 / 591 = 44.0% on Test) that share a TS geometry family across different contexts. To quantify how overlapping tolerance neighborhoods affect scoring, we apply 0.05/0.10 Å within-pair candidate deduplication and rerun direct family matching (detailed in W8.2 below). We have stated the identity, matching, and deduplication rules separately in Section 3.4 and Appendix B.

2. Distinguishing reference-set recovery from open-ended TS discovery

We agree that geometric reference recovery and useful TS discovery should be assessed separately. TREK leaves unknown endpoint pairs unlabeled; absence of a reference match is not a negative chemical label.

We tested the practical consequence by tracing audit-selected unmatched or unknown-pair outputs through quantum-chemical refinement. Among 500 selected candidates, 105 numerical chains were complete at the audit snapshot. Of these, 29 initial candidates were farther than 0.50 Å from every xTB reference of their reaction under ungated scoring. Fourteen subsequently passed the expected-endpoint connectivity screen after optimization and bidirectional IRC. Three documented examples are:

Reaction / generatorInitial minimum ungated RMSD to xTB references
rxn7940 / MolGEN1.349 Å
rxn1788 / React-OT0.906 Å
rxn1825 / TS-DFM0.784 Å

These are useful initial guesses despite an initial geometric miss. The examples provide exactly the complementary evidence the reviewer requests: a reference-recovery score and a downstream physical outcome can differ. We have included both in the revised evaluation, labeling the 14 cases as endpoint-consistent optimization outcomes rather than newly identified independent channels.

Reviewer_Qhj1OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

TREK evaluates whether a model can recover the set of transition-state structures associated with reactant and product conformer ensembles. It compares methods under a shared proposal budget and separates errors in pair selection, proposal allocation, and geometry prediction. The results show that good performance on known endpoint pairs can still leave much of a reaction's transition-state ensemble unrecovered.

Critical strengths
  1. The benchmark asks a useful question: does a model recover the range of transition states for a reaction, rather than produce a good structure for a supplied pair? Using a shared proposal budget makes this comparison more informative than comparing geometry errors alone.
  2. The error decomposition helps explain where coverage is lost. It distinguishes choosing the wrong pairs from assigning too few proposals or predicting poor geometries. The gap between known-pair performance and full-reaction recovery is a useful finding.

Earlier datasets such as RGD1 already contain multiple pathways, and the paper discusses related work including racerTS. TREK's contribution is the evaluation problem and budgeted comparison, rather than the observation that reactions can have multiple transition states.

Critical weaknesses
  1. The proposal budget is useful for defining a benchmark, but its practical meaning needs more explanation. Recall is still rising at the largest budget shown in Section 4.3. The paper explicitly controls proposal counts rather than runtime, so this is not an unfair comparison within its stated protocol. However, it leaves open whether the remaining coverage gap persists when inexpensive extra proposals are allowed, and whether the methods compare similarly at equal runtime. A longer budget curve and timing information would help interpret the result.
  2. The choice of reference target affects what counts as a failure. The references are xTB structures, whereas the evaluated generators have different training histories. The higher-level audit in Section 3.2 supports the physical plausibility of the references, but does not show how much of the strict-threshold error comes from differences between levels of theory. There is also a distinction between recovering contextual channels and distinct geometries: Section 3.2 groups the channels into fewer geometry families. A family-level comparison would help show how much structural diversity is actually missed.

The pair-aware matching rule raises a related question. A proposal can only receive credit through a recorded endpoint-pair relation, even if its geometry would match a reference associated with another pair. Please explain how often this restriction affects the results. It would also help to explain in the main text how repeated proposals are generated for each model, including where their diversity comes from.

The presentation could be easier to follow. A short worked example connecting endpoint conformers, productive pairs, channels, and geometry families before the formal definitions would make the protocol more accessible.

I lean toward acceptance. The benchmark exposes a limitation that pairwise geometry scores can hide, but the budget and reference choices need clearer interpretation.

Questions for authors
  1. Does recall approach a plateau at larger proposal budgets? What is the cost of increasing the budget for each generator, and how are distinct repeated proposals obtained?
  2. How much of the strict-threshold error could reflect the difference between xTB references and the models' training targets? Could the existing higher-level audit help assess this without requiring a new large-scale reference set?
  3. Does the main conclusion change when scoring geometry families rather than contextual channels, or when measuring the effect of the endpoint-pair matching gate?
7 comments
AuthorsOpenReview ↗

Response to Reviewer Qhj1

We thank Reviewer Qhj1 for the thoughtful feedback. The corresponding revisions are highlighted in the revised manuscript.

Q1 + W1: Budget saturation, computational cost, and practical meaning of the proposal budget

1. Extended budget–recall curve and saturation

Thank you for this suggestion. We extended the full-inventory experiment to M=32 for React-OT and RobustTSGen. The following results use the same Top-8 selector, round-robin allocation, and channel-scoring protocol over 3,493 reactions and 19,647 reference channels. M is the reaction-shared top-up budget.

GeneratorThresholdM=8M=16M=32Gain, 16 → 32
React-OT0.20 Å49.94%53.80%56.11%+2.31 pp
RobustTSGen0.20 Å50.03%53.77%56.02%+2.25 pp
React-OT0.50 Å74.06%82.02%84.89%+2.87 pp
RobustTSGen0.50 Å75.24%83.39%85.33%+1.94 pp

The extended curves show pronounced diminishing returns. At 0.50 Å, the M=8→16 gains are 7.96 and 8.15 percentage points for React-OT and RobustTSGen; doubling the top-up budget from 16 to 32 yields only 2.87 and 1.94 additional points. Recovery remains incomplete, with 15.11% and 14.67% of the reference inventory unrecovered at M=32. Thus, increasing the budget improves assignment coverage, but does not remove the recovery gap over the tested range.

Together with our analysis of each generator's native sampling mechanism in W1.2 and W3.2 below, these extended curves show why nominal call counts and distinct geometric sampling must be separated: for deterministic ODE defaults (React-OT, RobustTSGen), the small M=16→32 gains arise from additional near-identical outputs matching closely spaced contextual references rather than newly explored conformations. We have incorporated the M=32 results into Section 4.3 and Appendix D.4 (Table 10).

2. Per-model runtime and the repeated-proposal mechanism

The source of repeated-proposal diversity depends on the executed inference configuration. We audited all five model implementations and performed controlled GPU probes for React-OT and RobustTSGen.

GeneratorSource of variation in the evaluated configuration
React-OTFixed R/P-midpoint initialization and ODE inference
RobustTSGenFixed R/P-midpoint initialization and ODE inference
TS-DFMRandomness in distance-to-coordinate reconstruction
MolGENRandom-noise sampling
OA-ReactDiffRandom-noise sampling

For React-OT and RobustTSGen, the probes detected zero random-function calls and no CPU/CUDA RNG-state changes. Repeated runs with the same seed and batch size still differed slightly: the largest observed deviations were approximately 0.00254 Å and 0.00190 Å, respectively. These are consistent with numerical nondeterminism, rather than independent random initializations. The probe outputs were used only for this mechanism check.

We also recovered measured generation-stage timings from the official Task 1 production logs, rather than relying only on nominal call counts. All five models were evaluated on the same 3,493 reactions and 5,525 productive pairs in 32 shards per model, with matching input hashes, no generation failures, and H100 80 GB GPU hardware. The recorded elapsed time divided by the number of generated proposals gives:

GeneratorFull-inventory mean (s/proposal)Matched single-pair subset, batch=1 (s/proposal)
React-OT0.1490.209
RobustTSGen0.1500.208
TS-DFM2.1292.151
MolGEN1.6201.623
OA-ReactDiff5.6708.138

The matched subset contains the same 2,428 single-pair reactions for every model and helps reduce differences arising from batching. These measurements include model generation, adapter communication, and in-loop result checks, but exclude model loading, initial input preparation, and final output writing. They are timings from historical production runs, not a new standardized exclusive-GPU latency test; the sum of shard times also does not represent the end-to-end elapsed time of a parallel multi-GPU campaign. They show that the cost per proposal varies substantially across generators, while React-OT and RobustTSGen have closely matched observed costs.

AuthorsOpenReview ↗

Response 2 to Reviewer Qhj1

In terms of allocated calls, M=16 gives 63,791 proposal slots per model over the full inventory, whereas M=32 gives 119,679 (+16 calls per reaction). The Task 2 scores were computed by selecting budgeted proposal sequences from previously generated reservoirs, which were constructed through different reuse and supplementary-generation procedures across models. Consequently, the full reservoir-generation time is not the runtime of the scored M=16 subset. The Task 1 measurements provide empirical per-proposal cost context, but they do not constitute a measured equal-wall-clock Task 2 comparison; we therefore do not assume that model rankings remain unchanged under equal-time budgets. We have added this runtime summary and its timing boundaries to the revised experimental setup and Appendix C.

In computational demand, M=16 allocates 63,791 generator-call slots per model over the full inventory; M=32 allocates 119,679 under the same allocation rule (+16 calls per reaction). Beyond call counts, per-call cost scales with each model's native solver steps (Appendix C): 10 midpoint ODE function evaluations (NFE) for React-OT and RobustTSGen, 20 midpoint steps plus L-BFGS reconstruction for TS-DFM, 50 dopri5 steps for MolGEN, and 150 diffusion steps with 5× resampling for OA-ReactDiff. We have added the sampling-mechanism analysis and allocated-call counts to Section 4.1, Appendix C.1, and Appendix D.4.

Q2 + Q3 + W2: xTB versus higher-level references; contextual channels versus geometry families

1. Cross-level reference mismatch and strict-threshold sensitivity

We addressed this directly by submitting all 591 test-set xTB references to ORCA/B97-3c optimization: 473 / 591 (80.0%) converged to verified first-order saddles, exhibiting a modest median xTB-to-DFT coordinate shift of 0.138 Å (P90 = 0.392 Å). On these 473 fixed reference IDs (across 107 test reactions), we kept model predictions, pair gating, and matching rules fixed and replaced only the xTB reference coordinates with the optimized DFT coordinates (Appendix E.3, Table 16):

Generator[email protected] Å: xTB → DFTShift (pp)[email protected] Å: xTB → DFTShift (pp)
RobustTSGen52.01 → 38.27%-13.7484.14 → 76.32%-7.82
React-OT58.35 → 63.42%+5.0782.66 → 77.17%-5.49
TS-DFM54.12 → 50.53%-3.5983.72 → 76.11%-7.61
MolGEN49.05 → 47.15%-1.9071.04 → 65.33%-5.71
OA-ReactDiff40.17 → 47.15%+6.9861.52 → 59.62%-1.90

This directly quantifies how reference theory levels affect evaluation:

  1. At 0.50 Å, all models decrease by 1.9–7.8 pp, but the tier separation is preserved: React-OT (77.17%), RobustTSGen (76.32%), and TS-DFM (76.11%) remain within ~1 pp of each other and well ahead of MolGEN (65.33%) and OA-ReactDiff (59.62%), as the 0.50 Å tolerance radius exceeds the 0.138 Å median xTB-to-DFT shift.
  2. At the strict 0.20 Å threshold, score changes are substantial (-13.74 to +6.98 pp) and reorder the models: because 0.138 Å is comparable to the 0.20 Å cutoff, training-target differences become prominent—pure DFT-trained React-OT (+5.07 pp) and OA-ReactDiff (+6.98 pp) improve against DFT coordinates, whereas the Transition1x-xTB fine-tuned RobustTSGen checkpoint drops by 13.74 pp (from third to fifth).

Because this experiment holds the 473 reference IDs fixed without merging references that converge to similar DFT structures, it specifically isolates coordinate-level sensitivity. A separate comparison on the 89 full-IRC-chain reference IDs (53 reactions) is also reported in Section 3.2 and Appendix E.3 (Table 16).

2. Family-level coverage and interpretation of structural diversity

Following this suggestion, we ran direct maximum-cardinality matching to geometry families, separately from channel matching. The test set contains 591 contextual channels and 331 geometry families across 109 reactions. At 0.50 Å:

GeneratorChannel gatedChannel ungatedFamily gatedFamily ungated
RobustTSGen81.73%89.17%86.71%90.94%
React-OT80.71%87.99%84.89%88.22%
TS-DFM81.22%88.32%85.80%89.73%
MolGEN68.02%86.29%76.13%89.73%
OA-ReactDiff58.04%73.60%64.65%77.34%

Family scoring increases recovery in this comparison, while a gap remains even without the pair gate. The effect is not restricted to the 109-reaction test set: over all 3,493 reactions, gated family recalls are 86.34%, 84.69%, 82.88%, 77.62%, and 60.35%, respectively, against 11,174 families. The full-inventory analysis is reported separately from the source-test evaluation.

AuthorsOpenReview ↗

Response3 to Reviewer Qhj1

To further diagnose how each model's native sampling mechanism contributes to family coverage (see W1.2 and W3.2), we evaluated within-pair 0.05 Å candidate deduplication prior to family matching (Appendix D.6, Table 15). Here the model ordering reverses: under their official deterministic ODE defaults, RobustTSGen and React-OT retain only 277 and 279 unique test representatives across 275 selected pairs (48.94% and 47.73% deduplicated family recall), and TS-DFM's coordinate-reconstruction noise retains 303 representatives (48.94% recall). By contrast, explicit stochastic sampling in MolGEN generates 1,614 unique test representatives and retains 73.41% family recall (76.23% on All). We have added both Table 13 and Table 15 to the revised paper, as they separately measure slot-allocated neighborhood coverage and multi-sample geometric diversity.

Q1 + Q3 + W3: Pair-aware matching restriction and repeated-proposal diversity

1. Ablation of pair-aware matching versus geometry-only matching

We quantified the gate's effect using the same predictions. At 0.50 Å on the test set:

GeneratorAdditional channel matches without the gateReactions affected
RobustTSGen4412
React-OT4315
TS-DFM4215
MolGEN10828
OA-ReactDiff9231

Directly answering Q3: At 0.50 Å, MolGEN gains +18.27 pp on channels (68.02% → 86.29%, reducing its gap to the leading group from ~13.7 pp to ~2.9 pp) and reaches 89.73% on ungated geometry families (tying TS-DFM and exceeding React-OT's 88.22%; Table 13). This occurs because stochastic proposals from MolGEN and OA-ReactDiff more frequently land near references associated with another recorded pair of the same reaction. We have added both gated and ungated results across channels and families in Section 3.4, Section 4.3, and Appendix D.6 (Table 13).

2. How each model produces and deduplicates repeated proposals

Thank you for highlighting this distinction. Exact-coordinate deduplication alone does not remove numerical near-duplicates. One-to-one matching prevents repeated credit to the same reference, but nearly coincident candidates can occupy different matching slots when reference tolerance neighborhoods overlap.

We therefore retain candidate representatives in their actual proposal order within each pair. A subsequent candidate is merged when its distance to a retained representative is at most epsilon. Representative selection uses no reference labels. We examine epsilon=0.05 and 0.10 Å and rerun family matching after candidate merging; this is ordered representative selection, not transitive clustering.

At M=16, from 2,019 outputs over 275 selected test pairs, RobustTSGen, React-OT, and TS-DFM retain only 277, 279, and 303 representatives at 0.05 Å (276, 277, and 291 at 0.10 Å), whereas MolGEN and OA-ReactDiff retain 1,614 and 1,054 (1,126 and 862 at 0.10 Å). We have added these effective-budget counts and deduplicated family recalls to Section 4.3 and Appendix B / D.6 (Table 15), explicitly distinguishing slot-level neighborhood coverage from distinct geometric exploration.

W4. Worked example and protocol accessibility 1. One worked example covering conformers, productive pairs, channels and families

We agree and have introduced the existing rxn4245 example before the formal definitions. Two reactant and two product conformers produce four candidate pairs: two have recorded TS support and two remain unknown. The productive pair R2–P2 has five contextual xTB proxy targets, of which three are illustrated in Figure 2b. Geometry families then group targets by the frozen TS-geometry identity, separately from their endpoint context.

We have used rxn1825 in Figure 3 to continue the explanation: allocating several proposals to a pair, obtaining a local hit, and recovering the reaction's complete reference set are separate steps. A short matching example also shows why one-to-one assignment and candidate near-duplicate control serve different purposes. This provides the requested progression: endpoint conformers → candidate/productive pairs → contextual channels → geometry families → set recovery.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.