- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CAG (Calibration for Autoregressive Guidance) to guide reaction prediction models towards specific user-specified reaction types during generation. The primary contribution of the paper is the guidance mechanism, validated on USPTO-190 and USPTO-50k.
Critical strengths
- The method overall seems to work (e.g. on USPTO-50k the average performance does increase)
- The case against classifier-free guidance is fairly well-argued.
Critical weaknesses
-
Baselines: a few things stood out to me.
First, one obvious baseline is sampling larger numbers of reactions and filtering by constraint satisfaction. Table 3 seems to show this working fairly well: e.g. the row around line 1872 is a generator with reaction filtering and seems to be the best performing method in the whole table, including CAG. That seems to undercut a bit part of the paper's narrative.
Second, there are likely other simpler methods which might accomplish similar thing, e.g. stochastic sampling with a scaled temperature (probably followed by filtering).
So overall it would be good to show that the complexity of CAG is justified.
-
Reaction validity: a potential consequence of the guidance is producing invalid reactions. Figure 5a seems to show that increasing adherence comes with lower validity. So I'm concerned that the extra "solved reactions" are actually just invalid ones, which wouldn't actually solve the problem.
Questions for authors
From what I wrote above, my biggest questions are:
- Does this actually perform better than baselines like reaction filtering
- Are the new reactions found by CAG actually valid (sorry if it's in the paper and I missed it)
More generally, I would love to see the writing be improved a bit. In particular I think the arguments in §3.1-3.2 could be shortened and made clearer (giving more high-level intuition).
4 comments
Q1, improving the writing, and references.
Q1. Do we outperform baselines like reaction filtering
Our method complements (positively) search-level mechanisms like hard filtering (F) and value function guidance (G) rather than competing with them. On the USPTO-50k checkpoint, CAG alone matches the filtering row the reviewer points to on adherent solve while returning five times as many adhered routes per target, and adding CAG to filtering raises adherent solve further (Table 5). On the stronger USPTO-Full checkpoint, now reported separately in Table 6, CAG improves every search-level mechanism it combines with on the share of requested steps, and of whole routes whose class matches the request. On this stronger checkpoint, solve rate declines with CAG, since it fails to solve more complex targets within the time budget given that it's a slower method. The row the reviewer cites comes from the USPTO-Full runs of the submitted Table 5: we found a labelling error in the classifier's training data and reran that whole block, which is what Table 6 now reports.
Improved writing
We have rewritten §3.1 and §3.2 around the high-level argument the reviewer asks for, moving the formal statements to the appendix. The rewritten sections now say, in plain terms, (i) that guidance only changes the generator's output if it voices its preference as confidently as the generator does, (ii) that this is hardest for properties decided at a few specific token positions, which we now name substructural and contrast with global properties such as Tanimoto similarity in a new figure (Figure 2), and (iii) that reaching the required confidence on such a property needs a prohibitive number of samples per (product, disconnection) because sharp class boundaries block the interpolation a network would normally rely on. We have additionally added Figures 3 and 4 to the Methods section (Section 3) to illustrate both how the training corpus does not contain the prohibitive number of samples needed to achieve the desired confidence (Figure 3), and the mechanisms of CAG (Figure 4). We have also applied minor edits to the other sections of the paper (Abstract, Intro, Limitations and Discussion, and Conclusion) to better reflect our clarified claims.
Score
If the reviewer finds our answers and updated manuscript satisfactory, we hope they would consider raising their score accordingly. We remain happy to address any further concerns during the discussion period.
References
[1] P. Schwaller, R. Petraglia, V. Zullo, V. H. Nair, R. A. Haeuselmann, R. Pisoni, C. Bekas, A. Iuliano, T. Laino. Predicting retrosynthetic pathways using transformer-based models and a hyper-graph exploration strategy. Chemical Science 11(12), 3316-3325, 2020. (introduces round-trip accuracy, and argues no single metric is sufficient on its own)
[2] F. Hastedt, R. M. Bailey, K. Hellgardt, S. N. Yaliraki, E. A. del Rio Chanona, D. Zhang. Investigating the reliability and interpretability of machine learning frameworks for chemical retrosynthesis. Digital Discovery 3, 1194-1212, 2024. (benchmarks how often single-step predictions are chemically infeasible)
[3] J. Choe, S. Chen, Y. Jung. Assessing the extrapolation capability of template-free retrosynthesis models. Digital Discovery 5(6), 2591, 2026. DOI 10.1039/D6DD00072J. (round-trip scores are bounded by the forward model and can penalise chemically reasonable out-of-distribution predictions; the authors resort to manual inspection to assess plausibility)
[4] S. Xie, R. Yan, J. Guo, Y. Xia, L. Wu, T. Qin. Retrosynthesis Prediction with Local Template Retrieval. AAAI 2023, 5330-5338.
[5] Z. Zhong, J. Song, Z. Feng, T. Liu, L. Jia, S. Yao, M. Wu, T. Hou, M. Song. Root-aligned SMILES: a tight representation for chemical reaction prediction. Chemical Science 13(31), 9023-9034, 2022. DOI 10.1039/d2sc02763a.
Rebuttal acknowledged
Thank you for these changes- I think the new experiments strengthen the work and resolves most of my concerns. My initial score was 3, and although I view the paper more favourably, the small overall effect size means I don't want to raise my score to 4 (the max possible score).
We thank the reviewer for taking the time to go through the revised manuscript, and are glad the new experiments resolved most of their concerns. We understand the reservation about the overall effect size. We remain available to answer any further questions.
New baselines, validity, and improved writing
We thank the reviewer for their insightful suggestions and answer their questions below.
W1. Sampling larger number of reactions
We have run the reviewer's suggestion and report it in a new appendix section "D.3. Sampling more candidates from the generator" (lines 1774-1820, Table 3). We hold the generator, the checkpoint and the data of the main steering experiment fixed and change only the beam width, sweeping omega in {5, 10, 20} over the 640 products of USPTO-190-Steps against all 11 requested classes (7,040 pairs). We vary the beam width because in this generator the requested count is bounded by the beam width and number of augmentations [5].
A wider beam enlarges the candidate pool without making any individual candidate more likely to carry the requested class. Per-sample adherence is flat across the sweep (0.090, 0.089, 0.088), while the pool and class diversity grow (from 16.4 to 78., and 0.345 to 0.632 respectively), and the rank of the first adherent samples grows (5.9 to 18.9). A planner that expands only the leading candidates of each call therefore benefits less from the adherent samples at a beam size as high as 20. CAG at lambda = 2 instead raises the per-sample rate itself to 0.146 and reaches a higher class diversity, 0.692, from the omega = 10 pool.
W2. Simpler methods accomplishing similar things (e.g. stochastic sampling with a scaled temperature)
We have also run this baseline: stochastic sampling from the generator with temperature scaling. We report our results in a new appendix section "D.4 Stochastic Sampling with Scaled Temperature" (lines 1821-1861, Table 4). The main takeaways from these results is that changing the temperature scaling in stochastic sampling leads to a wider selection pool, without targeting a specific property value. This is reflected in the adherence, which is essentially equal to that of the beam search sampler (0.09 for all temperatures), while the achievable class diversity (ACD) is higher (from 0.495 for beam search to 0.619 for stochastic sampling with temperature T=2.), matching also an increase in the average candidate pool per (product, requested class) pair from 36 to 53 samples out of the 100 requested. CAG (at lambda=2) outperforms both the unguided beam search and the stochastic sampler with higher temperature with an adherence of 0.146 and an ACD of 0.692, at a comparable round-trip validity (0.223 against 0.250 for T=2.).
W3+Q2 validity of the solved guided reactions
Evaluating the validity (i.e. plausibility) of a computational retrosynthetic prediction is a known challenge in the field [1, 2, 3]. The best proxy (round-trip accuracy, using a forward model to predict the product back from the proposed reactants) is biased for favoring the standard reactant set for each product since it was trained on a single reaction per product. This is why we report, in Fig. 5b, both round-trip accuracy and the rxn-insight name identification. On the reviewer's question on whether guidance recovers valid precursors at all, we point to the result reported in Fig. 5a, which states that out of the 287 products where the unguided baseline finds no round-trip correct samples, CAG finds at least one round-trip valid precursor for 231 of them, in addition to tying with RetroKNN [4], one of the best template-based methods available, on achievable class diversity filtered by round-trip validity.
In the multi-step setting, the reviewer's concern is that the extra targets CAG solves are solved with worse chemistry than the ones the unguided model already reaches. We therefore split CAG's adherent routes into the targets it shares with the unguided model and the targets only CAG solves, and compare both against the unguided model's own adherent routes. All numbers below are per-step rates within adherent solved routes: round-trip is the share of steps whose product appears in the forward model's top-10 predictions, and specific RXN-Insight name the share of steps assigned a name other than OtherReaction. Brackets are 95% intervals obtained by resampling the targets with replacement (10,000 draws).
On both proxies, the targets CAG alone solves come out slightly ahead of the targets it shares with the unguided model, so the extra solves are not degraded.