acceptodds
Under review as a conference paper at ICLR 2027

Provenance-Trace Policy Optimization for Agentic Retrosynthesis Planning

19 views

Abstract

Agentic reinforcement learning (RL) offers a promising approach to retrosynthetic planning. Existing methods train LLM agents to select reactions and explore synthesis routes through tool-mediated interaction, using outcome-based rewards to encourage successful planning. However, trajectory-level rewards do not explicitly distinguish how individual search decisions contribute to planning. Equally rewarded trajectories may resolve different precursor subgoals, while candidate discovery can support reaction selections in later turns. To address this challenge, we propose Provenance-Trace Policy Optimization (PTPO), which converts structured search evidence into local supervision for search decisions. Specifically, PTPO aggregates terminal precursor assessments using the AND–OR synthesis structure and computes local advantages by comparing decisions made for the same molecule under identical ordered candidate views. Through provenance tracing, PTPO further propagates these advantages to the earlier turns that revealed the selected candidates. These local advantages complement trajectory-level supervision through separate turn- and trajectory-level PPO objectives, with each advantage channel normalized independently. Credit estimation reuses the sampled trajectories without a learned critic or additional rollouts. Experiments show that a 4B agent trained with PTPO achieves robust planning performance. On both in-distribution and out-of-distribution retrosynthesis benchmarks, PTPO consistently outperforms outcome-only reward RL agents across all evaluated search budgets.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Agentic Reinforcement Learning, Retrosynthetic Planning, Provenance-Based Credit Assignment, AND–OR Search

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_9AvTOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Provenance-Trace Policy Optimization (PTPO), an RL fine-tuning method for agents performing multi-step retrosynthesis planning. Unlike conventional outcome-based RL, which primarily rewards successful route completion, PTPO introduces finer-grained credit assignment using intermediate information from the synthesis search process. It constructs reaction-level advantages based on the AND–OR search structure, employs matched-context comparisons to account for differences in candidate actions across trajectories, and uses provenance tracing to assign credit to earlier tool interactions. Experiments on standard retrosynthesis benchmarks demonstrate substantial improvements in planning success rates, particularly under limited search budgets.

Critical strengths
  1. Meaningful problem formulation and effective credit assignment. The main contribution is constructing reaction-level advantages from the AND–OR search structure, extending binary outcome rewards to intermediate decisions. Matched-context comparisons further address differences in available candidate actions across agent trajectories. This is a reasonable and valuable approach to improving RL training for retrosynthesis planning.

  2. Strong benchmark performance and broader applicability. The method achieves substantial improvements in solve rates on established benchmarks. Moreover, provenance-based credit assignment is potentially applicable to tool-using agents beyond retrosynthesis.

Critical weaknesses
  1. Simplistic reward design and limited assessment of synthesis quality. The local reward relies on heuristic precursor-state scores, particularly the arbitrarily assigned value of 0.5 for OPEN branches, which may affect the learning outcome. Furthermore, the terminal reward still primarily depends on whether a complete synthesis route is found, without adequately considering route quality. Consequently, the resulting local rewards may not accurately reflect reaction quality.

  2. Limited gains in synthesis capability and dependence on search depth. The performance improvement diminishes as the search budget NN increases, suggesting that PTPO primarily improves search efficiency rather than enabling the agent to solve fundamentally more challenging synthesis problems. Moreover, the local reward estimation strongly depends on sufficiently deep and broad exploration during training. The ablation results also suggest that local advantages contribute more to performance than provenance tracing, despite the latter being emphasized in the paper.

Questions for authors
  1. How sensitive is PTPO to the heuristic reward assigned to OPEN branches (0.5)? Would alternative values materially affect performance?

  2. How does the effectiveness of local advantage estimation depend on the depth and breadth of exploration during training?

2 comments
AuthorsOpenReview ↗

Rebuttal-Part 1

We thank the reviewer for the time and effort during the review process. Regarding the reviewer's comments, we provide the following responses.

W1: Simplistic reward design and limited assessment of synthesis quality.

We thank the reviewer's in-depth comments. We address the two points in turn.

(1) The precursor scores encode a certification ordering, and the OPEN value is pinned by construction.

  • We would like to clarify that these scores reflect precursor resolution during search. SOLVED is 1, DEAD is 0, and a reaction is scored by the minimum over its precursors, reflecting the AND condition of the search graph. The resulting pooled score is an empirical, search-dependent assessment rather than an intrinsic measure of reaction quality.
  • The local advantage is centered within a matched candidate view. When all compared reactions have identical pooled scores, their centered reaction advantages are zero.
  • We kindly refer the reviewer to our response to Q1, where we report the effect of varying ρopen\rho_{\mathrm{open}} on USPTO-190.

(2) Template-based reaction selection and evaluation of route similarity.

  • The LLM policy does not generate reactions; it selects RDChiral-verified candidates from a template-based single-step model. Thus, template grounding is enforced by the action space, while the reward ranks admissible routes. Following [1], we assess route quality using tree edit distance and route size against USPTO-190 references on the 79 targets solved by all systems, enabling a paired comparison:
SystemTED ↓Tree nodes
Reference routes0.0017.79
Retro*11.5119.16
Retro*-010.3118.76
Retro*+9.7417.04
Retro*+-010.6918.08
EG-MCTS13.6525.60
PDVN9.1614.34
Retro-R1 (V1)10.3217.45
RetroAgent9.9418.48
PTPO (ours)9.3116.76

The above results show that PTPO achieves the lowest TED among the agentic RL systems, indicating that its gains extend beyond planning success to closer agreement with reference synthesis trees. It also outperforms every search baseline except PDVN, ranking second overall.

References

[1] S. Genheden, O. Engkvist, and E. Bjerrum. Clustering of synthetic routes using tree edit distance. Journal of Chemical Information and Modeling, 2021.

W2: Limited gains in synthesis capability and dependence on search depth.

(1) On the narrowing margin. Indeed, this is a common characteristic of budgeted retrosynthesis planning: as NN increases, repeated attempts allow methods to recover from early failures, causing success rates to converge. To better characterize planning efficiency before saturation, we report the predictor-call budgets required to reach matched success rates in a complementary evaluation on ChEMBL-1000:

Success Rate (%)PTPO (calls)RetroAgent (calls)Ratio
607101.43×
7013191.46×
7523321.39×
8052901.73×
82871922.21×
85347Not reached within 500—

As shown in the table above, across common success levels, RetroAgent requires 1.39–2.21 times as many predictor calls as PTPO; PTPO reaches 85% success within 347 calls, whereas RetroAgent does not within 500, demonstrating better limited-budget efficiency.

(2) On the dependence on exploration. Local estimation requires sufficient precursor resolution and matched comparisons. PTPO handles limited evidence by assigning OPEN an intermediate value and withholding the turn-level objective when support falls below Mmin⁡M_{\min}. We refer the reviewer to our response to Q2 and Appendix F.10 (Figures 11 and 12) for detailed exploration ablations and local-supervision coverage diagnostics, respectively.

(3) On the relative contribution of provenance tracing. Yes, removing PT advantages costs 21.4 points while removing provenance tracing costs 5.6. Note that provenance tracing is not an independent mechanism but a redistribution inside the local channel, returning credit to the turn that produced the selected candidate, so it acts on top of the local signal rather than in place of it.

AuthorsOpenReview ↗

Rebuttal-Part 2

Q1: How sensitive is PTPO to the heuristic reward assigned to OPEN branches (0.5)? Would alternative values materially affect performance?

Thank you for raising this concern. We clarify that, In our formulation, ρopen=0.5\rho_{\mathrm{open}}=0.5 places an unresolved precursor midway between DEAD (00) and SOLVED (11). To assess its effect, we conducted one-at-a-time sweeps on USPTO-190 with all other settings fixed, including the hyperparameter showing the largest overall variation.

ρopen\rho_{\mathrm{open}}0.000.250.500.75
Pass@1 (%)54.2163.1567.8965.26
λpt\lambda_{\mathrm{pt}}0.0000.1250.2500.375
Pass@1 (%)46.5064.7367.8966.31

The largest performance drops occur at endpoints that effectively remove a mechanism: ρopen=0\rho_{\mathrm{open}}=0 treats unresolved and dead precursors identically, while λpt=0\lambda_{\mathrm{pt}}=0 removes the turn-level objective. Away from these endpoints, performance varies more moderately. In particular, ρopen=0.25\rho_{\mathrm{open}}=0.25 and 0.750.75 achieve 63.15% and 65.26%, respectively, compared with 67.89% at 0.50.5. We kindly refer the reviewer to the newly added Appendix F.8, “Effect of Hyper-parameters,” in the revised manuscript for additional sensitivity analyses.

Q2: How does the effectiveness of local advantage estimation depend on the depth and breadth of exploration during training?

  • Thank you for this important question. During training, exploration affects local advantage estimation through both precursor resolution and the availability of matched comparisons. Shallow search may leave more precursors unresolved, reducing differentiation between reaction scores. Exploring alternative reactions across rollouts can provide comparison support, but only when distinct reactions are selected under the same candidate view; broader exploration therefore does not automatically yield stronger local supervision.
  • Importantly, complete successful routes are not required: trajectories with the same terminal reward can still resolve different precursor subproblems. When supported comparisons are insufficient, the minimum-support gate withholds the turn-level objective. Under the default training configuration, this objective is active in 83.2% of optimizer steps, indicating that local supervision is regularly available. We kindly refer the reviewer to Appendix F.10 (Figures 11 and 12) for the corresponding training diagnostics.
Reviewer_ZgHvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes an RL algorithm to fine-tune an LLM agent for retrosynthesis, which adds a turn-level advantage to the usual trajectory-level one. The turn-level advantage compares reactions selected for the same molecule under the same candidate list across rollouts. Reactions whose precursors were solved at the end of the rollout are given credit, which is then passed to the earlier turn that revealed the candidate. The authors demonstrate that this set up leads to empirical improvements validated through ablations.

Critical strengths
  • The paper identifies a clear gap in the fine-tuning of agentic retrosynthetic models, which is then exploited in the proposed method.
  • The granular RL-based algorithm shows clear improvements over the non fine-tuned version of the system, leading to performance matching that of SOTA models.
Critical weaknesses
  • (W1) The method's novelty is ambiguous, especially without comparison to the closest baselines: GiGPO [1] is highly similar to the current method: both methods combine step-level and trajectory-level group relative advantages to train the policy. In fact, the main differences are the AND–OR precursor score used to compare reactions and the provenance trace, which gives the smallest gain in the ablation (Table 2: −5.6 Pass@1 without it, −2.3 points at N=500). While GiGPO is cited in the introduction, it is not included as a baseline in the empirical study, and the authors do not discuss its similarity to their own approach. Positioning relative to this baseline would clarify this paper's contribution.

  • (W2) The current experimental set up does not isolate how much the turn-level advantage helps: The ablation that compares No PT to the full method (Table 2) reports a gain of 21.4 points in Pass@1. This comparison might put the No PT variant at a disadvantage. Groups where all rollouts get the same reward have a trajectory-level advantage of 0. These groups appear to be kept in No PT, because the filter keeps a group if any turn-level advantage is nonzero (Alg. 1), even though PT is "computed but unused" in this variant (Sec. 4.4). Fig. 3(b) shows that such groups make up 82% of the groups after dynamic sampling in the last 100 training steps, so they dilute the learning signal. A No PT variant with a filter that removes zero-advantage groups would show the gain that comes from the turn-level advantage alone. Additionally, the authors could add the turn-level advantage to a strong baseline like RetroAgent [2] to show that its benefit translates to comparable training setups.

  • (optional - W3) Comments and questions to improve the empirical evaluation and paper presentation as a whole:

    • (a) The setup of the search-based baselines is not specified: which single-step model and checkpoint each method uses, and whether the authors ran these methods themselves or took the numbers from earlier work. It is also unclear how Pass@1 is computed for these methods, since they usually explore the search tree from the target to purchasable starting materials in a single pass. Please clarify how these methods were run, to improve the interpretation of the results.
    • (b) The authors do not discuss why PTPO's advantage shrinks as the budget grows. On USPTO-190, PDVN outperforms PTPO at N ≥ 300, and PTPO's margin over RetroAgent drops from 12.32 points at N = 50 to 2.53 points at N = 500 (1.60 points on ChEMBL-1000). A few lines explaining why PTPO's gains flatten at larger budgets would be helpful.
    • (c) The empirical results include no assessment of the quality or plausibility of the routes. Relying only on whether a route connecting the target to purchasable starting materials can be found is a heavily critiqued metric in retrosynthesis [3,4]. I recommend adding metrics that compare the recovered routes against the reference routes of USPTO-190 (e.g., top-k route accuracy [4] or route similarity [5]), and if possible the round-trip accuracy of the intermediate steps [6].
    • (d) The target-level planning efficiency plot (Fig. 2) adds only one result that Table 1 does not already give: the success rate with fewer than 50 single-step model calls. It also shows one checkpoint, one dataset (ChEMBL-1000), no error bars, and no search-based baselines. Could the authors rescope the plot to focus on this low-budget range? In its current form, most of it is redundant with Table 1.
    • (e) The numbers in Table 1 and Fig. 2 do not agree. On ChEMBL-1000, the curves in Fig. 2 level off at 77.60% (PTPO), 70.30% (Retro-R1) and 70.20% (RetroAgent), but Table 1 reports 86.00%, 82.42% and 84.40% at N = 500. The text says that Fig. 2 uses "selected checkpoints" (l. 368). Which checkpoints are these, and how were they selected?
    • (f) The authors do not specify how they choose the checkpoint used for evaluation (no validation set is mentioned), or how the manually set parameter values are chosen (e.g., ρ_open = 0.5, δ = 0.6, λ_tr = 0.3, λ_pt = 0.25, α_L = 0.35, α_N = 0.25, M_min = 64), and no sweep is provided. In addition, each variant seems to be trained only once, so the reported standard deviations do not show how much the results change between training runs.
Questions for authors

Questions

  • A matched-context group needs the same molecule with the same ordered candidate list, and at least two different selected reactions, out of only K = 8 rollouts per target. Could the authors report, per training step, the fraction of turns (and of reaction selections) with a nonzero PT advantage, and how often the turn-level objective is switched off by the M_min = 64 threshold?

References

[1] L. Feng, Z. Xue, T. Liu, and B. An. Group-in-group policy optimization for LLM agent training. NeurIPS, 2025.

[2] Y. Zhu, J. Gan, X. Sun, F. Sun, Y. Shi, M. M. Islam, C. Shang, W. Gao, C. W. Coley, Y. Sun, and W. Wang. RetroAgent: Harnessing LLMs to search over structured memory for agentic retrosynthesis planning. arXiv:2607.14512, 2026.

[3] K. Maziarz, A. Tripp, G. Liu, M. Stanley, S. Xie, P. Gaiński, P. Seidl, and M. Segler. Re-evaluating retrosynthesis algorithms with Syntheseus. arXiv:2310.19796, 2023.

[4] S. Genheden and E. Bjerrum. PaRoutes: towards a framework for benchmarking retrosynthesis route predictions. Digital Discovery, 1(4):527–539, 2022.

[5] S. Genheden, O. Engkvist, and E. Bjerrum. Clustering of synthetic routes using tree edit distance. Journal of Chemical Information and Modeling, 61(8):3899–3907, 2021.

[6] P. Schwaller, R. Petraglia, V. Zullo, V. H. Nair, R. A. Haeuselmann, R. Pisoni, C. Bekas, A. Iuliano, and T. Laino. Predicting retrosynthetic pathways using transformer-based models and a hyper-graph exploration strategy. Chemical Science, 11(12):3316–3325, 2020.

12 comments
AuthorsOpenReview ↗

Rebuttal-Part 1

We sincerely appreciate the reviewer for the positive feedback and valuable comments. Regarding the comments of the reviewer, we provide the following responses.

W1: PTPO’s relationship to GiGPO requires clearer discussion and empirical comparison.

Thank you for this valuable comment. The main differences are as follows.

  • Comparison context. GiGPO groups actions by a shared anchor state. PTPO groups reaction selections by the same molecule and ordered candidate view, requiring at least two distinct selected reactions. This permits local comparisons across trajectories whose surrounding search graphs differ, without requiring their complete search states to match.

  • Local evidence. GiGPO compares discounted returns within anchor-state groups. PTPO instead scores reactions using the minimum precursor assessment, pooling evidence by molecule–reaction pair and comparing selections within matched candidate views. Both signals depend on subsequent search; they differ in the evidence aggregated and the context used for comparison.

  • Value pool and baseline. GiGPO's single group serves as both. PTPO separates them: value is pooled over every occurrence of the same reaction in the rollout group, while the baseline ranges only over alternatives offered on the same candidate menu.

  • Optimization. GiGPO combines episode and step advantages before clipped optimization. PTPO maintains separately normalized objectives, using a trajectory-level ratio for episode advantages and a turn-level ratio for PT advantages, with a minimum-support gate for the local objective.

We adapted GiGPO to retrosynthesis planning and trained it under the same backbone and planning environment:

  • GiGPO implementation details. We define the anchor state as a hash of the canonical partial AND–OR graph, including all committed reactions, molecule OPEN/SOLVED and expansion statuses, the OPEN frontier, ordered candidate caches, and the remaining budget in five-turn buckets. Session-local identifiers, graph insertion order, free-text rationales, and timestamps are excluded. Thus, turns are grouped only when they share the same partial route, frontier, candidate observations, and budget bracket.

  • We implement A=AE+ωASA=A^{E}+\omega A^{S} with ω=1.0\omega=1.0 and discounted returns with γ=0.99\gamma=0.99. Within each anchor group, step-level advantages are computed by subtracting the mean discounted return, without standard-deviation scaling. GiGPO and PTPO use the same Qwen3-4B-Instruct-2507 backbone, predictor, environment, terminal reward, optimizer, K=8K=8 rollouts per target, 600 training steps, episode limits, and evaluation protocol.

MethodPass@1N=50N{=} 50N=100N{=}100N=200N{=}200
Base (no RL)6.10±2.1018.90±1.6018.90±1.6018.90±1.60
GiGPO48.32±1.5556.27±1.4165.39±1.2574.21±1.50
PTPO (ours)67.89±1.9070.74±1.4280.32±1.0984.74±1.34

As shown above, PTPO consistently outperforms GiGPO across all reported budgets, including gains of 19.57 percentage points in Pass@1 and 10.53 points at N=200N=200. PTPO exploits precursor-level search outcomes to compare reaction choices under matched candidate views and traces credit to the queries that revealed them. Its separate trajectory- and turn-level objectives support learning from both overall planning outcomes and intermediate search progress.

References

[1] L. Feng, Z. Xue, T. Liu, and B. An. Group-in-group policy optimization for LLM agent training. NeurIPS, 2025.

AuthorsOpenReview ↗

Rebuttal-Part 2

W2: The No PT ablation may underestimate the baseline’s performance by retaining zero-advantage groups, making the contribution of turn-level supervision unclear.

Thank you for this careful observation. In the No PT variant in Table 2, PT advantages are computed but not optimized, while the dynamic-sampling filter retains a group when terminal rewards differ or any PT advantage is nonzero (Section 3.3). Reward-constant groups retained by this rule provide no EP gradient. To isolate the turn-level contribution, we retrained No PT using an EP-only filter that retains a group only when terminal rewards differ, with all other settings unchanged (600 training steps, Qwen3-4B-Instruct-2507, K=8K=8). Results on USPTO-190 are shown below:

VariantPass@1N=50N{=} 50N=100N{=}100N=200N{=}200
No PT (Table 2)46.50±0.8048.20±0.6059.80±1.6067.90±1.60
No PT, EP-only filter52.10±1.1456.20±2.6067.10±1.1078.10±1.30
PTPO67.89±1.9070.74±1.4280.32±1.0984.74±1.34

EP-only filtering raises No PT’s Pass@1 from 46.50% to 52.10%, reducing the PTPO gap from 21.39 to 15.79 points. PTPO still leads by 15.79 points in Pass@1 and 6.64–14.54 points across N=50N=50–200200. These results address the concern about retaining zero-advantage groups and demonstrate the substantial contribution of turn-level supervision under the corrected comparison.

W3(a): The search-based baseline configurations and Pass@1 evaluation protocols are insufficiently specified, limiting the interpretability of the comparisons.

To clarify, the single-step model used by PTPO is specified in Section 4.1 (“Single-step Model”). Following RetroAgent, we use the same template-based MLP with pretrained DESP weights to evaluate learned search capability under the same reaction predictor. The search-based baseline results in Table 1 were taken from RetroAgent, which in turn reports them from Retro-R1. Following RetroAgent and Retro-R1, Pass@1 measures success on the first attempt without retry: the first trajectory for agent methods and the published no-regret evaluation for search baselines, which retains only one of the kk proposed reactions at each state.

W3(b): PTPO’s performance advantage decreases as the search budget increases, and the reasons for this trend are not adequately discussed.

Thank you for raising this concern. PTPO primarily improves early reaction selection, yielding its largest gains at low budgets. This is central to the task, which seeks successful synthesis routes under limited predictor-call budgets rather than unconstrained search. As NN increases, additional rollouts and PDVN’s broader value-guided search narrow the gap on USPTO-190. Nevertheless, PTPO outperforms PDVN at N=500N=500 on the out-of-distribution ChEMBL-1000 benchmark (86.00% vs. 80.60%), supporting efficient and generalizable planning.

W3 (c) The empirical results include no assessment of the quality or plausibility of the routes.

Thank you for this valuable comment. Route quality deserves direct measurement, and we add one against the USPTO-190 reference routes.

  • On plausibility: every step in a PTPO route is selected from candidates returned by a template-based single-step model, each an RDChiral-verified application of a USPTO reaction template, so the policy selects among template-grounded transformations rather than inventing them. This is also why we adopt this template-based single-step model: it gives a common, template-grounded reaction space and enables fair comparison with prior work.

  • On similarity: following your suggestion, we report TED[5] to the reference and route size on the 79 USPTO-190 targets solved by all systems for paired comparison. PTPO, RetroAgent, and Retro-R1 each use a budget of 500 unique single-step model calls per target (Table 1); classical search planners are rerun for 500 search iterations with expansion width 50 and the same building-block set:

SystemTED ↓Tree nodes
Reference routes0.0017.79
Retro*11.5119.16
Retro*-010.3118.76
Retro*+9.7417.04
Retro*+-010.6918.08
EG-MCTS13.6525.60
PDVN9.1614.34
Retro-R1 (V1)10.3217.45
RetroAgent9.9418.48
PTPO (ours)9.3116.76

PTPO attains the lowest TED among the agentic RL systems (against RetroAgent and Retro-R1) and improves on every search baseline except PDVN, which reaches 9.16.

AuthorsOpenReview ↗

Rebuttal-Part 3

W3(d)/(e): Figure 2 overlaps with Table 1, while differences in the reported results and checkpoint selection require clarification.

Thank you for the suggestions. we have revised the manuscript and rescoped Figure 2 to focus on the low-budget range of 0–50 single-step model calls. It reports first-trajectory success (Pass@1), with our RetroAgent reproduction reaching 70.20%, whereas Table 1 quotes RetroAgent’s published results (73.82±0.49% Pass@1 and 84.40±0.38% at N=500N=500). The discrepancy reflects different result sources and evaluation protocols: Figure 2 evaluates a single trajectory, while Table 1’s budgeted columns report cumulative success across successive rollouts. PTPO uses its final checkpoint, while RetroAgent and Retro-R1 use their released checkpoints.

W3(f): Checkpoint selection and the training procedures for different variants are insufficiently described, and no hyperparameter sensitivity analysis is provided.

  • First, no validation- or test-based checkpoint selection is performed. PTPO and all variants are trained from the same initialization for a fixed 600 steps and evaluated using their final checkpoints.

  • Second, to evaluate the remaining choices, we vary one hyperparameter at a time on USPTO-190 while keeping all other settings fixed. The results are summarized below.

Sensitivity to ρopen\rho_{\mathrm{open}}

ρopen\rho_{\mathrm{open}}0.000.250.500.75
Pass@1 (%)54.2163.1567.8965.26

Sensitivity to δ\delta

δ\delta0.300.450.600.75
Pass@1 (%)62.6368.9467.8964.73

Sensitivity to λtr\lambda_{\mathrm{tr}}

λtr\lambda_{\mathrm{tr}}0.000.150.300.45
Pass@1 (%)62.2863.6867.8966.31

Sensitivity to λpt\lambda_{\mathrm{pt}}

λpt\lambda_{\mathrm{pt}}0.0000.1250.2500.375
Pass@1 (%)46.5064.7367.8966.31

Sensitivity to Mmin⁡M_{\min}

Mmin⁡M_{\min}163264128
Pass@1 (%)66.8465.8467.8952.63

Sensitivity to αL\alpha_L

αL\alpha_L0.150.250.350.45
Pass@1 (%)60.5267.8967.8966.31

Overall, the sweeps support the default configuration as a practical starting point, with further tuning guided by the observed sensitivity of individual hyperparameters. We have added these results and practical tuning guidelines to Sections F.8 and F.9 of the revised manuscript, respectively (Figure 10).

Q1: Strict candidate-view matching may limit the availability of local supervision, but its coverage and the frequency of disabling the turn-level objective are not reported.

Thank you for raising this concern. We now report turn-level supervision coverage, computed over LLM-generated turns, and the frequency of disabling the local objective over the PTPO training run:

MetricPTPO
Mean fraction of turns with APT≠0A^{\mathrm{PT}}\neq 023.8%
Optimizer steps with 0<M<640<M<6413.0%
Optimizer steps with the local objective active (M≥64M\geq 64)83.2%

The minimum-support threshold is applied to the total number of supported turns across each 128-trajectory optimizer batch. Thus, relatively sparse turn-level supervision still provides sufficient accumulated support to enable the local objective in 83.2% of optimizer steps. Most disabled steps contain nonzero PT advantages but do not reach Mmin⁡=64M_{\min}=64. We have added the coverage curve, supported-turn distribution, and accompanying statistical descriptions to Section F.10 of the revised manuscript (Figures 11 and 12).

Reviewer_wycdOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PTPO addresses credit assignment in RL-trained retrosynthesis agents, where terminal rewards provide limited feedback on intermediate decisions. It uses terminal precursor states and the AND–OR search structure to compare reaction choices under matched candidate views, then traces the resulting advantages to earlier tool calls that supplied those candidates. Local and trajectory-level signals are optimized through separate PPO objectives, reusing existing rollouts without a learned critic.

The authors train a Qwen3-4B agent and evaluate it on USPTO-190 and ChEMBL-1000. Reported Pass@1 reaches 67.89% and 78.04%, exceeding RetroAgent by 14.63 and 4.22 percentage points. Ablations examine candidate matching, provenance tracing, and optimization granularity. Gains are strongest at low search budgets, while PDVN remains stronger at high budgets on USPTO-190.

Critical strengths
  1. The local supervision is well aligned with the task structure. Failure to complete a synthesis route does not imply that every reaction choicewas poor. PTPO uses precursor completion to distinguish local decisions and propagatesfeedback to the queries that supplied the candidates. This extracts informationoverlooked by terminal rewards without requiring additional rollouts or a learned value network.

  2. The performance gains are substantial. Compared with RetroAgent usingthe same backbone, PTPO improves first-rollout success rates by 14.63 and 4.22 percentage points on the two benchmarks. Removing candidate matching or provenance tracing also noticeably reduces performance. These results support the method’s practical value for retrosynthetic planning.

Critical weaknesses
  1. The reaction score does not distinguish a poorreaction from poor downstream search. Whether a precursor is eventually solved depends on both the selected reaction and subsequent search decisions and effort. Candidate matching ensures identicalavailable options but does not control these downstream factors. The score may therefore be a useful training heuristic,but its interpretation as an accurate assessment of decision quality remains insufficiently justified.

  2. The justification for provenance tracing is weaker than that for scoring reactionchoices. A good reaction choice does not necessarily imply that querying that molecule earlier was a good decision. Conversely, a querythat returns useful candidates may receive negative feedback becauseof a subsequent selection error. Propagating selection advantages directly to query turns still conflates the contributions of acquiring information and using it.

Questions for authors
  • Why require identical candidate views when computing thecomparison baseline, while pooling evidence across candidate views when estimating reaction values? Could the latter reintroduce the contextual differences that the former attempts to control?
2 comments

We appreciate the reviewer for the constructive comments and positive feedback on our paper. Regarding the reviewer's comments, we provide the following responses.

W1. Reaction-level credit may be affected by downstream search outcomes.

  • We thank the reviewer for the in-depth comments. We acknowledge that interpreting QAND(m,r)Q_{\mathrm{AND}}(m,r) as an accurate assessment of intrinsic reaction quality is too strong, and we have revised Sec. 3.2 accordingly. Its intended interpretation is more limited: within a matched context, AORA^{\mathrm{OR}} compares pooled, search-dependent assessments of reaction choices.
  • Downstream resolution is part of QπQ^\pi under a bounded budget. Matching controls candidate availability and ordering, but not the surrounding graph, remaining budget, or subsequent decisions. When no distinct alternative is selected under the same view, AOR=0A^{\mathrm{OR}}=0, preventing unsupported reaction-level credit. To examine whether this control is substantive, we progressively tightened the matching criteria, with the results summarized below:
Baseline for the local comparisonPass@1 (%)
None (trajectory outcome only, λpt=0\lambda_{\mathrm{pt}}=0)46.50±0.80
Molecule only (g(e)=meg(e)=m_e)54.39±2.65
Molecule and ordered candidate view (PTPO)67.89±1.90

These results support the benefit of candidate-view matching, which improves Pass@1 by 13.50 points over molecule-only grouping in this setting. Appendix F.7 further shows that 32.4% of PTPO’s and 50.4% of RetroAgent’s committed reactions in the audited solved rollouts remain uncertified yet share the terminal reward. This illustrates the limited granularity of terminal rewards and motivates supplementing them with intermediate search evidence.

W2. Provenance tracing may conflate information acquisition with candidate selection.

  • We thank the reviewer for this observation. The trace links selected candidates to their discovery turns and transfers discounted selection credit, serving as a practical credit-assignment heuristic rather than an independent estimate of query value. Thus, the transferred signal reflects the realized downstream usefulness of a query under the current policy, rather than its standalone information value.
  • Two design choices restrict this transfer. First, a producer turn receives credit only from its linked supported selections, averaged including zero advantages. Second, temporal discounting and λtr=0.3\lambda_{\mathrm{tr}}=0.3 scale the transferred contribution.

We kindly refer the reviewer to the provenance-tracing ablation in Table 2, summarized below:

VariantPass@1 (%)N=100N=100 (%)
w/o provenance tracing (λtr=0\lambda_{\mathrm{tr}}=0)62.28±2.6075.79±0.91
PTPO67.89±1.9080.32 ±1.09

These gains support provenance tracing as an effective auxiliary training signal for planning.

Q1. Pooling evidence across candidate views may undermine matched-context comparisons.

(1) The comparison set and evidence pool serve different purposes.

  • The baseline bgb_g averages reaction scores, weighted by occurrence, over selections in g=(m,σ)g=(m,\sigma), comparing choices under the same ordered candidate view.
  • QAND(m,r)Q_{\mathrm{AND}}(m,r) averages outcomes of eligible molecule–reaction occurrences. With K=8K=8, view-specific estimates may have few observations; pooling increases sample support within one target’s rollout group without mixing targets.

(2) Candidate matching restricts the comparison set even with pooled scores.

  • Fixed precursors do not imply equal resolution probabilities: outcomes depend on the search graph, remaining budget, and subsequent decisions. QAND(m,r)Q_{\mathrm{AND}}(m,r) is therefore a pooled, search-dependent empirical assessment.
  • With g(e)=meg(e)=m_e, the baseline can include alternatives absent from the current menu; g(e)=(me,σe)g(e)=(m_e,\sigma_e) restricts it to same-view selections. Pooling shares reaction evidence without changing this restriction on which selections enter the comparison baseline.
  • In Table 2, molecule-only grouping achieves 54.39% Pass@1 and 82.28% at N=500N=500, versus PTPO’s 67.89% and 87.58%. These gains support the benefit of candidate-view matching under the pooled scoring scheme.

Thank you for the thoughtful response. The additional experiments have addressed my main concerns and strengthened the paper substantially. I will therefore raise my score.

Comments on the paper

2 comments
Area_Chair_ZbhnOpenReview ↗

Please check the authors' rebuttal ASAP.

Dear reviewers,

As the discussion period will end very soon, please check the authors' rebuttal and see if your concerns have been addressed. Thank you.

-AC

AuthorsOpenReview ↗

Summary of Revisions

Dear Area Chair and Reviewers,

Thank you for your constructive feedback. We have revised the manuscript to incorporate the additional experiments and clarifications discussed in our responses. The main revisions are summarized below.

  1. GiGPO comparison and implementation. We evaluated GiGPO using the same backbone and planning environment as PTPO. Appendix F.4 documents GiGPO’s anchor-state definition and dynamic-sampling filter. Figures 8–9 report the fraction of turns sharing an anchor and the number of anchor groups per target containing distinct actions over 600 training steps.

  2. Corrected ablation of turn-level supervision. Following the reviewers’ suggestion, we retrained the No-PT baseline with an EP-only filter that retains rollout groups only when terminal rewards differ. The corrected baseline achieves 52.10% Pass@1, giving PTPO a gain of 15.79 percentage points. We updated Table 2, Section 4.3, and Appendix F.9 accordingly.

  3. Route similarity and paired evaluation. We report tree edit distance (TED), size-normalized TED, and standard deviations across targets, along with the route-selection rules and search budgets. We expanded the comparison from the 79 targets solved by all systems to the 155 solved by both PTPO and RetroAgent. On this broader set, mean TED to the reference routes is 17.21 for PTPO and 23.80 for RetroAgent (Appendices F.3 and F.6, Tables 4 and 6).

  4. Interpretation and coverage of local supervision. We have clarified in Section 3.2 how downstream search affects reaction scores, why PTPO compares reactions selected for the same molecule under the same ordered candidate list, and how provenance tracing assigns credit to the turns that revealed the selected reactions. Appendix F.11 now reports the fraction of turns receiving nonzero local advantages and shows that the turn-level objective is active in 83.2% of optimizer steps.

  5. Evaluation protocol and hyperparameter sensitivity. We refocused Figure 2 on the first 50 predictor calls and clarified how its single-rollout evaluation differs from the cumulative success reported in Table 1. Appendices F.9–F.10 report sensitivity analyses and tuning guidance for the unresolved-precursor score, the provenance-tracing and turn-level weights, and the minimum-support threshold.

As the discussion deadline approaches, we would greatly appreciate any feedback you may have. If any points remain unclear, we would be very happy to provide additional clarification and engage in further discussion. Thank you again for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.