- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TSG-Diff, a tree-structured GRPO framework for reinforcement-learning fine-tuning of protein diffusion models. The method reuses intermediate denoising checkpoints and propagates normalized terminal rewards through the resulting trees to construct segment-level advantages. The application to all-atom protein design is potentially valuable, and the use of independent structural evaluators is a positive aspect of the experimental evaluation.
Critical strengths
none
Critical weaknesses
- TreeGRPO has already introduced tree-structured diffusion rollouts with shared-prefix reuse, reward propagation from terminal leaves, edge-level credit assignment, and GRPO-based policy updates. These mechanisms closely correspond to the primary methodological contributions claimed by TSG-Diff.
- Despite emphasizing efficient exploration under a fixed rollout budget, the paper does not provide sufficiently detailed empirical comparisons of reward versus cumulative denoising NFEs, wall-clock time, and reward-evaluation cost. Matched-budget comparisons, training curves, and multi-seed statistical analysis are necessary to substantiate the central claims.
- The paper overlooks several important prior works on tree-search-guided diffusion generation and reward-guided fine-tuning, particularly PepTune[1] and TR2-D2[2].
[1] Tang S, Zhang Y, Chatterjee P. Peptune: De novo generation of therapeutic peptides with multi-objective-guided discrete diffusion[J]. ArXiv, 2025: arXiv: 2412.17780 v4.
[2] Tang S, Zhu Y, Tao M, et al. Tr2-d2: Tree search guided trajectory-aware fine-tuning for discrete diffusion[J]. ArXiv, 2025: arXiv: 2509.25171 v1.
Questions for authors
- How does TSG-Diff establish methodological novelty over existing tree-search-based diffusion optimization methods, particularly TreeGRPO, TR2-D2, and PepTune?
5 comments
Title: Author Response To Reviewer (2/5)
All methods use the same BoltzGen initialization, training complexes paired by seed, 12 denoising steps, a learning rate of , 400 single-complex training updates, and the same banded interface-contact reward. Chain GRPO, the trajectory-credit tree, both TSG-Diff variants, and the TreeGRPO port score four leaves per update; the small- and large-search TR2-D2-style adaptations score 6 and 96, respectively. Thus, matching training updates does not equalize cumulative rollout NFEs, reward evaluations, optimizer steps, or wall-clock time.
Frozen policies are evaluated on the same 120 held-out complexes. Because this evaluation uses the training reward function, it measures generalization of the optimization objective; independent predicted structural metrics are reported separately in Tables 1a and 3c.
Table 2a. Frozen-policy reward and cumulative computational cost across training checkpoints. Rewards are reported as mean ± s.e. across training seeds; parentheses in the checkpoint columns indicate cumulative rollout NFEs. The 100-, 200-, and 400-update columns use seeds 45–47 (n = 3), while the final two columns report 400-update rewards for seeds 42–44 and all six seeds, respectively. For seeds 42–44, chain GRPO, the trajectory-credit tree, and TSG-Diff (λ = 1) use earlier checkpoints re-scored with the same evaluation protocol. Reward-evaluation counts refer to individual scored leaves, rather than batched calls to the reward function.
| Method | NFEs / reward evaluations / optimizer steps per update | Seconds per update (estimated GPU-h at 400) | Reward-evaluation time per update, ms [kernel; wall] | 100 updates | 200 updates | 400 updates | 400 updates, seeds 42–44 | 400 updates, six seeds |
|---|---|---|---|---|---|---|---|---|
| Chain GRPO | 48.0 / 4 / 12 | 6.45 (0.72) | [0.12; 0.86] | 0.538 ± 0.008 (4,800) | 0.511 ± 0.019 (9,600) | 0.531 ± 0.008 (19,200) | 0.547 ± 0.015 | 0.539 ± 0.008 |
| Tree, trajectory-level credit | 40.0 / 4 / 12 | 7.88 (0.88) | [0.12; 0.86] | 0.524 ± 0.012 (3,972) | 0.529 ± 0.019 (7,997) | 0.532 ± 0.003 (16,001) | 0.558 ± 0.029 | 0.545 ± 0.014 |
| TSG-Diff (λ = 1) | 40.0 / 4 / 12 | 7.91 (0.88) | [0.12; 0.86] | 0.539 ± 0.015 (3,972) | 0.546 ± 0.006 (7,997) | 0.553 ± 0.016 (16,001) | 0.559 ± 0.010 | 0.556 ± 0.008 |
| TSG-Diff (λ = 0.5) | 40.0 / 4 / 12 | 7.88 (0.88) | [0.12; 0.86] | 0.543 ± 0.012 (3,972) | 0.571 ± 0.008 (7,997) | 0.573 ± 0.007 (16,001) | 0.581 ± 0.014 | 0.577 ± 0.008 |
| TreeGRPO port | 29.5 / 4 / 2 | 3.81 (0.42) | [0.12; 0.86] | 0.526 ± 0.001 (4,060) | 0.531 ± 0.002 (7,400) | 0.535 ± 0.012 (11,800) | 0.547 ± 0.010 | 0.541 ± 0.007 |
| TR2-D2-style, small search | 63.2 / 6 / 12 | 7.96 (0.88) | [0.47; 3.4] | 0.519 ± 0.005 (6,325) | 0.529 ± 0.007 (12,647) | 0.532 ± 0.011 (25,300) | 0.542 ± 0.009 | 0.537 ± 0.006 |
| TR2-D2-style, large search | 587.5 / 96 / 12 | 12.04 (1.34) | [1.6; 14] | 0.536 ± 0.002 (59,269) | 0.557 ± 0.007 (117,469) | 0.558 ± 0.012 (235,008) | 0.566 ± 0.008 | 0.562 ± 0.006 |
Update times are measured medians. Training GPU-hours are estimates obtained as median seconds per update × 400 / 3,600 for single-GPU runs, rather than measured end-to-end totals. Rollout NFEs exclude the forward and backward computations used for optimization and the cost of policy evaluation.
Reward-evaluation timings profile the training reward on the median-size complex, with 77 design atoms and 592 target atoms. Entries report [GPU kernel time; fastest observed wall-clock time on a shared GPU], accounting for each method's reward-call schedule. The first five methods make one batched call on four leaves per update; the TR2-D2-style adaptations make 3 + 1 and 12 + 1 batched calls.
Table 2b. Paired reward differences after 400 updates across six seeds (42–47). Values are mean differences with 95% t-intervals over paired training-seed differences on the fixed 120-complex evaluation panel. Bold intervals exclude zero. The checkpoint provenance described above also applies here. These intervals characterize variation across training seeds on this panel, rather than uncertainty from sampling new evaluation complexes.
| Comparison | Mean reward difference [95% CI] | Seeds agreeing with mean sign |
|---|---|---|
| TSG-Diff (λ = 0.5) − chain GRPO | +0.038 [+0.019, +0.057] | 6/6 |
| TSG-Diff (λ = 1) − chain GRPO | +0.017 [+0.008, +0.026] | 6/6 |
| TSG-Diff (λ = 1) − tree, trajectory-level credit | +0.011 [−0.007, +0.029] | 5/6 |
| TSG-Diff (λ = 1) − TreeGRPO port | +0.015 [+0.008, +0.022] | 6/6 |
| TSG-Diff (λ = 0.5) − TreeGRPO port | +0.036 [+0.026, +0.046] | 6/6 |
Title: Author Response To Reviewer (3/5)
The updated results support three conclusions:
-
TSG-Diff improves reward over chain GRPO with fewer NFEs but slower updates. At 400 updates, both λ settings outperform chain GRPO across all six seeds, with paired 95% CIs excluding zero. Rollout NFEs decrease by 16.7% at the same 1,600 reward evaluations, but update times increase by 22–23% (7.88–7.91 versus 6.45 seconds).
-
TSG-Diff improves reward over the TreeGRPO port at higher cost. Both paired 95% CIs exclude zero, but rollout NFEs increase by 35.6% and update time roughly doubles, demonstrating a reward–cost tradeoff.
-
Component-level evidence remains limited. The same-tree segment-credit comparison is inconclusive. Although λ = 0.5 yields higher mean reward than λ = 1 (0.577 versus 0.556), no paired CI is reported, so its advantage is not statistically established.
W3. The paper overlooks several important prior works on tree-search-guided diffusion generation and reward-guided fine-tuning, particularly PepTune and TR2-D2.
We agree that PepTune and TR2-D2 are important prior work and will expand the related-work discussion to clarify their relationship to TSG-Diff. We distinguish their search and learning mechanisms from the empirical comparisons, which address both controlled adaptations and complete released pipelines.
PepTune’s main practical limitation for our setting is the recurring cost of inference-time search. Its Monte Carlo Tree Guidance uses batched expansion and property scoring to optimize peptide SMILES without updating the generator during guidance. Consequently, subsequent guided-generation runs repeat search and scoring. Its reported setting uses 128 iterations × 50 children, corresponding to 6,400 candidate rollouts per search run, although a run can deliver multiple designs. Receptor-bound structural evaluation also requires an additional structure-generation or docking stage. TSG-Diff instead updates a coordinate-generating policy, allowing training-time exploration to be amortized over subsequent generation. This distinction does not by itself establish lower total cost.
TR2-D2 introduces replay-buffer construction, refresh, and off-policy weighting costs. It uses MCTS to curate trajectories for weighted denoising cross-entropy on masked discrete diffusion. Its behavior depends on the search budget and buffer-refresh schedule; its peptide experiments report a reward–diversity tradeoff when changing refresh frequency. Applying this objective to continuous coordinates requires an adaptation. TSG-Diff instead uses fresh policy rollouts and segment advantages without maintaining a replay buffer across updates. Both approaches learn policies that can subsequently generate without search, so this benefit is not unique to TSG-Diff.
TreeGRPO couples generated-leaf count with reward-evaluation expenditure in its published algorithm. However, it scores every generated leaf and places branching within a contiguous stochastic window. TSG-Diff provides a different allocation mechanism: resume stored checkpoints and select leaves for scoring independently of the number generated. This permits separate control of generation and scoring budgets, although unscored branches still incur denoising cost. These are methodological distinctions whose benefits require empirical comparison.
Our tree is a rollout structure rather than a search (expansion nodes and scored leaves are drawn uniformly, with no value-guided selection as in PepTune and TR2-D2). To separate algorithmic effects from differences in pretrained models and molecular representations, we use two protocols:
- A — Controlled comparison on BoltzGen: common initialization, reward, and evaluation, with 400 matched training updates and explicit reporting of NFEs, reward evaluations, and wall-clock time. These runs do not match cumulative compute. A separate inference-time experiment compares PepTune-style guidance with flat sampling at equal scored-leaf counts.
- B — Complete-pipeline comparison: the released PepTune and TR2-D2 implementations and TSG-Diff are evaluated on eight common targets. We report delivered designs, eligibility for the common structural evaluator, conditional quality metrics, and compute costs. This protocol includes differences in pretrained models, representations, and cohort sizes.
Controlled post-training results. TSG-Diff (λ = 0.5) achieves the highest six-seed mean reward in Table 2a: 0.577 ± 0.008, versus 0.562 ± 0.006 and 0.537 ± 0.006 for large- and small-search TR2-D2-style fine-tuning. Compared with large search, it uses approximately 93% fewer rollout NFEs, 96% fewer reward evaluations, and 35% shorter updates. The numerical reward gain (+0.015) lacks a paired confidence interval, leaving its statistical uncertainty unresolved.
Title: Author Response To Reviewer (1/5)
Dear Reviewer PySA,
We sincerely appreciate your valuable feedback and provide point-wise responses below. Key changes are highlighted in blue in the revised manuscript.
We agree that tree-structured diffusion rollouts, prefix reuse, and leaf-to-edge credit propagation are established by TreeGRPO. Our contribution is the particular combination of checkpoint-resumed suffix expansion, within-tree and condition-wide advantage normalization, and parent-relative checkpoint-segment credit, V(v)−V(parent(v)), blended with terminal-leaf advantage. TSG-Diff also separates candidate generation from reward evaluation: each tree generates 1+LN candidate trajectories, of which K leaves are selected for scoring. These choices define our formulation for all-atom protein diffusion; internal ablations alone cannot establish an advantage over TreeGRPO.
We therefore compare TSG-Diff with a BoltzGen port of TreeGRPO using the same initialization, banded interface-contact reward, evaluation protocol, and common PPO clipping and KL settings. The port follows the subtree-mean credit calculation in the released implementation. The comparison matches 400 training updates and 1,600 training reward evaluations per run, while explicitly reporting differences in rollout NFEs, optimizer steps, and training time.
Table 1a. Comparison after 400 training updates: frozen-policy reward, AF2-Multimer ipTM, and computational cost. Reward is mean ± s.e. across seeds 42–47, evaluated on 120 held-out complexes using the training reward function. AF2-Multimer ipTM is an independent predicted interface-confidence score, reported as mean ± s.e. across seeds 45–47 of the per-policy mean over 32 test complexes × 3 designs, using single-sequence input, one model, and three recycles with the native receptor and designed peptide. For seeds 42–44, chain GRPO and TSG-Diff (λ = 1) use earlier checkpoints of the same recipe, re-scored in the common evaluation harness. The other runs were trained for this comparison. NFEs count rollout denoiser calls, with the average per update in parentheses. GPU-hours are estimated from median seconds per update × 400 / 3,600, with one job per GPU, excluding the first 10 updates and checkpoint writes from the timing measurement.
TSG-Diff improves frozen-policy reward over the TreeGRPO port at matched updates and reward evaluations, but requires more compute. Across six seeds, paired gains are +0.015 (95% CI: [+0.008, +0.022]) for λ = 1 and +0.036 ([+0.026, +0.046]) for λ = 0.5, with all seeds favoring TSG-Diff (Table 2b). Estimated training cost increases from 0.42 to 0.88 GPU-hours. Structural evidence is weaker: λ = 0.5 achieves the highest numerical AF2-Multimer ipTM (0.134 ± 0.001; Table 1a), but λ = 1 does not improve over the base policy, and paired ipTM confidence intervals are unavailable. These algorithm-level comparisons cannot isolate effects of rollout construction, normalization, credit assignment, or optimizer steps. The segment-credit ablation remains inconclusive (+0.011; 95% CI: [−0.007, +0.029]), and neither two-level normalization nor leaf selection is isolated. Released peptide-pipeline comparisons are discussed under W3.
Thank you for the suggestion. We now report frozen-policy reward at 100, 200, and 400 updates, rollout NFEs, reward-evaluation counts, update times, scoring microbenchmarks, and paired reward differences across six seeds, distinguishing denoising savings from wall-clock efficiency.