- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Route-Instruction Grounding and Steering (RIGS) as a steering adapter to improve sampling frequency of specific retrosynthetic precursors according to a language guidance input. This is done with a frozen Qwen3 embedding layer for the language input, their two-stage RIGS steering adapter aligning the language embeddings and steering generation, and then generation with GraphDiT.
Critical strengths
The authors show that, compared to matched reference and matched shuffled results, performance is generally better by around 10-20 percentage points, but with a large variance, so instruction following does work. Increasing the precursor training pool isn’t monotonic, with the best results at top30. They show that their stage A alignment does improve performance over just stage B interventions, but primarily in the avoided set. They also compare their RIGS generation-time guidance to reranking afterward, with RIGS P@1 performance better than reranking, but N@1 performance worse. They finish by showing that their results generalize, to varying degrees, to other datasets.
Critical weaknesses
The main contribution of RIGS seems to be its steering performance, but it is unclear that RIGS performs better than a simple reranking. Additionally, the evaluation metric used to benchmark RIGS is lacking completeness and may be skewing results. The preferred set used to evaluate performance contains a set of reasonable preferred reactions, but it doesn’t contain all possible paths that should be preferred, so it’s possible to generate a precursor that follows the language guidance instructions, but is penalized for not being in the preferred set. Generalization, particularly to the Pistachio dataset, is also somewhat lacking.
Questions for authors
- Why did you choose to interfere at generation-time if the primary goal is to rerank, given that RIGS doesn’t show coverage improve?
- You claim that coverage is not compromised. In addition to the HIT@100 in table 3, it would be helpful to include total percentage of preferred candidates rather than asking it it just contains a single preferred candidate.
- You include a comparison to simple reranking in table 3, which is helpful, but the comparison to RIGS doesn’t seem fair. The reranking employed is much simpler than RIGS, and may not be accurately representing a fair comparison. For a perfect reranking, P@1 should be equal to HIT@100, rather than the 22.23 to 64.80 difference. Further, reranking for P@1 does even worse than the baseline. Ablations targeting this reranking, perhaps focusing on P@1 performance rather than N@1 performance which the current reranking seems to be doing, would be very helpful.
- The preferred set is a set of at most 100 precursors, but how well does this span the space of all precursors that fit the guidance? In particular, the RDKit oracle used for reranking, which should provide perfect ranking, achieves a P@1 of 26.99 in table 16 instead of equaling the HIT@100 of 64.80. Since this is the case, how do you rationalize using this constructed set of 100 precursors when there are many other valid precursors that will be penalized in evaluation, which may be skewing benchmark results?
- Why is the performance on the Pistachio set so much worse than USPTO?
6 comments
Thank you for your response. However, there are still some unanswered questions, primarily pertaining to reranking and the RDKit oracle.
(1) Although RIGS can improve control with little change in Hit@100, this can also theoretically be done with a reranking. Specifically, reranking can’t affect coverage, but RIGS doesn’t seem to affect coverage either. Choosing to steer at generation time when coverage isn’t affected is still unclear, unless RIGS does substantially better than reranking, but this is also unclear.
(2/3) Reranking is, as you say, also a challenging problem. However, it doesn’t seem to be fairly compared to. There is still the huge discrepancy between the RDKit oracle going from 64.80 for Hit@100 to 26.99 for P@1, the magnitude of which remains unaddressed. Although enumerating through all precursor sets is impractical, it is by itself concerning that the oracle goes from 64.80 to 26.99, when an appropriately matched objective and evaluation should have Hit@100 be similar to P@1. This suggests a deep mismatch between the preferred set evaluation and RDKit property objectives. Additionally, the scalar head reranking reported in (Table 3) is trained off of RDKit, not the precursor sets. So, even with perfect optimization of the RDKit properties, this method of reranking achieves 26.99 with the oracle, which is lower than the RIGS reporting, purely because of the evaluation mismatch. Notably, the reranking decreases RDKit regret substantially, unlike RIGS, indicating that the reranking does work if evaluated on the metric it’s trained on. In essence, reranking is much less aligned to the evaluated metric than RIGS and doesn’t constitute a fair comparison of RIGS with reranking.
(4) Although Pistachio is a different reaction set, how much different is it than USPTO? Do they both cover similar classes of reactions, and are the precursor sets still applicable? The performance on Pistachio is substantially worse, to the point of barely improving P@1 at all. Can you quantify the overlap, or otherwise identify why the generalization is so poor, especially considering that RIGS is trained on Pistachio data as well?
I choose to maintain my rating.
Thank you for your detailed follow-up and for clarifying your remaining concerns. We address these points below with additional analyses and explanations.
(1) Generation-time control beyond Hit@100
RIGS changes instruction-conditioned proposal probabilities. Hit@100 only records whether a preferred candidate appears in 100 draws, so similar scores can coexist with different probability allocations. Table 1 reproduces central quantiles from the frequency-ranked Top-10 table in Section 2 of our first-round response, providing direct-property evidence beyond preferred-set membership.
Table 1. Direct-property quantiles of frequency-ranked Top-10 candidates.
We directly scored up to ten distinct valid, frequency-ranked precursor sets for each of 963 paired products, including outputs outside the preferred inventory. Each product was scored on its assigned property and guidance direction and contributed equal total weight, shared equally among its candidates. Entries are Q25 / Q50 (median) / Q75; arrows indicate the requested direction.
\hline \text{Property and direction} & \text{Reference} & \text{RIGS} \\\\ \hline \text{Procurement burden}\\ \downarrow & 10.00\\,/\\,10.69\\,/\\,13.00 & 10.00\\,/\\,10.69\\,/\\,13.18 \\\\[3pt] \text{Procurement burden}\\ \uparrow & 10.00\\,/\\,10.69\\,/\\,12.30 & 10.00\\,/\\,10.69\\,/\\,13.04 \\\\[3pt] \text{Complexity difference}\\ \downarrow & 0.000\\,/\\,0.416\\,/\\,0.872 & -0.138\\,/\\,0.325\\,/\\,0.794 \\\\[3pt] \text{Complexity difference}\\ \uparrow & -0.005\\,/\\,0.412\\,/\\,0.888 & 0.030\\,/\\,0.421\\,/\\,0.926 \\\\[3pt] \text{Halogen atom count}\\ \downarrow & 0\\,/\\,1\\,/\\,2 & 0\\,/\\,0\\,/\\,1 \\\\[3pt] \text{Halogen atom count}\\ \uparrow & 0\\,/\\,1\\,/\\,2 & 1\\,/\\,1\\,/\\,2 \\\\[3pt] \text{Ring count}\\ \downarrow & 1\\,/\\,2\\,/\\,3 & 1\\,/\\,2\\,/\\,3 \\\\[3pt] \text{Ring count}\\ \uparrow & 1\\,/\\,2\\,/\\,2 & 1\\,/\\,2\\,/\\,3 \\\\ \hline \end{array}$$ Under lower-complexity and lower-halogen guidance, the Top-10 medians decrease from 0.416 to 0.325 and from 1 to 0, respectively. These changes demonstrate instruction-aligned shifts across the returned alternatives, evaluated directly on their properties rather than solely through preferred-set membership. These probabilities matter when a planner repeatedly requests candidate expansions under a finite sampling budget. For example, a region with 1% probability under the unguided generator has a 63.4% chance of appearing in 100 independent draws. Even a perfect reranker cannot recover it from a pool in which it never appears. If instruction conditioning raises that probability to 20%, just ten independent draws give an 89.3% chance of reaching the region. Our setting addresses a practical need in multistep synthesis planning: a single-step generator should respond to route-specific requirements expressed in free-form language. RIGS learns how these instructions favor or discourage precursor alternatives for the same product and transfers this grounding into the proposal distribution itself. This provides a trainable single-step component for more selective multistep search, with the potential to reduce the effective branching factor under limited sampling budgets.
Generation-Time Steering, Reranking, and Preferred-Set Evaluation
We thank the reviewer for recognizing the instruction-following gains and Stage A’s suppression of conflicting alternatives. We clarify generation-time steering, LLM selection, and evaluation beyond reference membership.
1. Why steer at generation time
Coverage asks whether a relevant alternative appears; steering changes how frequently it is proposed. RIGS can therefore improve instruction control with little change in Hit@100. In multistep planning, single-step proposals supply search expansions [1]. Concentrating proposals on compatible alternatives can reduce the effective branching factor when the planner prioritizes or limits expansions using this distribution.
2. Reranking and candidate-pool composition
A perfect preferred-membership selector would attain P@1 = Hit@100 on a fixed pool. Reliable reranking remains a challenging research problem in retrosynthetic planning [2,3]. Our appendices assess complementary requirements using RDKit validity and descriptor utility, relaxed round-trip accuracy and coverage, SCScore-based complexity reduction, procurement burden, and reaction-class ambiguity. Forward consistency offers partial evidence of chemical feasibility [4].
We additionally compare Reference, Reference + LLM, RIGS, and RIGS + LLM using four direct property evaluators: procurement burden, synthetic complexity difference, halogen count, and ring count. These assess instruction satisfaction beyond offline Top100 membership. Compared with Reference + LLM, RIGS + LLM reduces mean procurement burden from 9.86 to 8.93 under lower-burden guidance, increases halogen count from 2.75 to 3.17 under higher-halogen guidance, and increases ring count from 3.04 to 3.38 under higher-ring guidance. Steering changes the proposed distribution; LLM selection assesses completed candidates.
Separately, we compute product-weighted empirical quantiles for frequency-ranked Top-1 and Top-10 outputs on 963 paired Reference–RIGS products. Under lower-complexity guidance, all five quantiles decrease for both rankings: the Top-1 median falls from 0.401 to 0.157 and Q75 from 0.860 to 0.613. Under lower-halogen guidance, both rankings’ medians decrease from 1 to 0 and Q75 from 2 to 1. Zero-halogen output fractions increase from 38.14% to 55.93% for Top-1 and from 31.57% to 51.55% for Top-10.
Top-10 quantiles. Each line gives Reference → RIGS, with values ordered Q10/Q25/Q50/Q75/Q90; arrows beside metrics indicate the requested direction.
These results complement Hit@100 by showing changes across the returned alternatives, beyond the appearance of a single favorable candidate.
3. Preferred-set coverage
The finite inventory provides a reproducible multi-reference benchmark of known instruction-compatible alternatives. Exhaustively enumerating all compatible precursor sets is impractical. Reference, Matched, and Shuffled use the same frozen inventory, enabling paired comparisons of instruction-dependent control. At 65M Top30, Matched improves P@1 over Shuffled by 18.52 percentage points. Direct property evaluation additionally credits improvements outside this inventory.
4. Performance on Pistachio
Our templates [5] are extracted from USPTO. Differences in Pistachio’s reaction distribution may reduce their coverage and applicability, limiting planning performance. Extending our data pipeline to extract and curate templates from more diverse reaction sources is an important direction for future work.
References
[1] Chen, B., Li, C., Dai, H., & Song, L. (2020). Retro*: Learning Retrosynthetic Planning with Neural Guided A* Search. ICML, PMLR 119, 1608–1616.
[2] Lin, M. H., Tu, Z., & Coley, C. W. (2022). Improving the performance of models for one-step retrosynthesis through re-ranking. Journal of Cheminformatics, 14, 15.
[3] Li, J., Fang, L., & Lou, J.-G. (2023). RetroRanker: leveraging reaction changes to improve retrosynthesis prediction through re-ranking. Journal of Cheminformatics, 15, 58.
[4] Torren-Peraire, P., et al. (2024). Models Matter: the impact of single-step retrosynthesis on synthesis planning. Digital Discovery, 3, 558–572.
[5] Yu, K., et al. (2024). Double-Ended Synthesis Planning with Goal-Constrained Bidirectional Search. Advances in Neural Information Processing Systems, 37, 112919–112949.