- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Provenance-Trace Policy Optimization (PTPO), an RL fine-tuning method for agents performing multi-step retrosynthesis planning. Unlike conventional outcome-based RL, which primarily rewards successful route completion, PTPO introduces finer-grained credit assignment using intermediate information from the synthesis search process. It constructs reaction-level advantages based on the AND–OR search structure, employs matched-context comparisons to account for differences in candidate actions across trajectories, and uses provenance tracing to assign credit to earlier tool interactions. Experiments on standard retrosynthesis benchmarks demonstrate substantial improvements in planning success rates, particularly under limited search budgets.
Critical strengths
-
Meaningful problem formulation and effective credit assignment. The main contribution is constructing reaction-level advantages from the AND–OR search structure, extending binary outcome rewards to intermediate decisions. Matched-context comparisons further address differences in available candidate actions across agent trajectories. This is a reasonable and valuable approach to improving RL training for retrosynthesis planning.
-
Strong benchmark performance and broader applicability. The method achieves substantial improvements in solve rates on established benchmarks. Moreover, provenance-based credit assignment is potentially applicable to tool-using agents beyond retrosynthesis.
Critical weaknesses
-
Simplistic reward design and limited assessment of synthesis quality. The local reward relies on heuristic precursor-state scores, particularly the arbitrarily assigned value of 0.5 for OPEN branches, which may affect the learning outcome. Furthermore, the terminal reward still primarily depends on whether a complete synthesis route is found, without adequately considering route quality. Consequently, the resulting local rewards may not accurately reflect reaction quality.
-
Limited gains in synthesis capability and dependence on search depth. The performance improvement diminishes as the search budget increases, suggesting that PTPO primarily improves search efficiency rather than enabling the agent to solve fundamentally more challenging synthesis problems. Moreover, the local reward estimation strongly depends on sufficiently deep and broad exploration during training. The ablation results also suggest that local advantages contribute more to performance than provenance tracing, despite the latter being emphasized in the paper.
Questions for authors
-
How sensitive is PTPO to the heuristic reward assigned to OPEN branches (0.5)? Would alternative values materially affect performance?
-
How does the effectiveness of local advantage estimation depend on the depth and breadth of exploration during training?
2 comments
Rebuttal-Part 2
Q1: How sensitive is PTPO to the heuristic reward assigned to OPEN branches (0.5)? Would alternative values materially affect performance?
Thank you for raising this concern. We clarify that, In our formulation, places an unresolved precursor midway between DEAD () and SOLVED (). To assess its effect, we conducted one-at-a-time sweeps on USPTO-190 with all other settings fixed, including the hyperparameter showing the largest overall variation.
| 0.00 | 0.25 | 0.50 | 0.75 | |
|---|---|---|---|---|
| Pass@1 (%) | 54.21 | 63.15 | 67.89 | 65.26 |
| 0.000 | 0.125 | 0.250 | 0.375 | |
|---|---|---|---|---|
| Pass@1 (%) | 46.50 | 64.73 | 67.89 | 66.31 |
The largest performance drops occur at endpoints that effectively remove a mechanism: treats unresolved and dead precursors identically, while removes the turn-level objective. Away from these endpoints, performance varies more moderately. In particular, and achieve 63.15% and 65.26%, respectively, compared with 67.89% at . We kindly refer the reviewer to the newly added Appendix F.8, “Effect of Hyper-parameters,” in the revised manuscript for additional sensitivity analyses.
Q2: How does the effectiveness of local advantage estimation depend on the depth and breadth of exploration during training?
- Thank you for this important question. During training, exploration affects local advantage estimation through both precursor resolution and the availability of matched comparisons. Shallow search may leave more precursors unresolved, reducing differentiation between reaction scores. Exploring alternative reactions across rollouts can provide comparison support, but only when distinct reactions are selected under the same candidate view; broader exploration therefore does not automatically yield stronger local supervision.
- Importantly, complete successful routes are not required: trajectories with the same terminal reward can still resolve different precursor subproblems. When supported comparisons are insufficient, the minimum-support gate withholds the turn-level objective. Under the default training configuration, this objective is active in 83.2% of optimizer steps, indicating that local supervision is regularly available. We kindly refer the reviewer to Appendix F.10 (Figures 11 and 12) for the corresponding training diagnostics.
Rebuttal-Part 1
We thank the reviewer for the time and effort during the review process. Regarding the reviewer's comments, we provide the following responses.
We thank the reviewer's in-depth comments. We address the two points in turn.
(1) The precursor scores encode a certification ordering, and the OPEN value is pinned by construction.
(2) Template-based reaction selection and evaluation of route similarity.
The above results show that PTPO achieves the lowest TED among the agentic RL systems, indicating that its gains extend beyond planning success to closer agreement with reference synthesis trees. It also outperforms every search baseline except PDVN, ranking second overall.
References
[1] S. Genheden, O. Engkvist, and E. Bjerrum. Clustering of synthetic routes using tree edit distance. Journal of Chemical Information and Modeling, 2021.
(1) On the narrowing margin. Indeed, this is a common characteristic of budgeted retrosynthesis planning: as N increases, repeated attempts allow methods to recover from early failures, causing success rates to converge. To better characterize planning efficiency before saturation, we report the predictor-call budgets required to reach matched success rates in a complementary evaluation on ChEMBL-1000:
As shown in the table above, across common success levels, RetroAgent requires 1.39–2.21 times as many predictor calls as PTPO; PTPO reaches 85% success within 347 calls, whereas RetroAgent does not within 500, demonstrating better limited-budget efficiency.
(2) On the dependence on exploration. Local estimation requires sufficient precursor resolution and matched comparisons. PTPO handles limited evidence by assigning OPEN an intermediate value and withholding the turn-level objective when support falls below Mmin. We refer the reviewer to our response to Q2 and Appendix F.10 (Figures 11 and 12) for detailed exploration ablations and local-supervision coverage diagnostics, respectively.
(3) On the relative contribution of provenance tracing. Yes, removing PT advantages costs 21.4 points while removing provenance tracing costs 5.6. Note that provenance tracing is not an independent mechanism but a redistribution inside the local channel, returning credit to the turn that produced the selected candidate, so it acts on top of the local signal rather than in place of it.