acceptodds
Under review as a conference paper at ICLR 2027

From Chains to Trees: Efficient Exploration and Credit Assignment in Protein Diffusion

Abstract

Diffusion models provide strong generative priors for conditional protein design, but steering them toward downstream objectives remains challenging. Reinforcement learning (RL) post-training is crucial for aligning generative models with specific targets, but its prohibitive computational cost remains a major barrier to widespread adoption. In protein design, this leads to two problems: (1) rollout budgets are spent repeatedly resampling similar prefixes; (2) terminal rewards provide sparse supervision for long denoising trajectories. We propose TSG-Diff, an RL framework that organizes reverse diffusion as a branching tree over intermediate checkpoint nodes. Improves training efficiency by recasting the denoising process as a search tree. Starting from a small set of initial rollouts, TSG-Diff expands selected checkpoints to diverse continuations while reusing partial trajectories. Terminal rewards on final designs are converted into group-relative leaf advantages and then propagated through the tree to produce segment-level credit for intermediate denoising decisions. We instantiate TSG-Diff for protein design with a composite reward combining binding affinity, interface confidence, and structural regularization. Our approach provides more informative supervision under a fixed rollout budget, and empirical results show improved optimization of binding-related objectives while preserving structural plausibility and naturalness. The code can be found at this anonymous link: https://anonymous.4open.science/r/TSG-Diff-F223

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Tree-structured reinforcement learning, Credit assignment, Protein design

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PySAOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TSG-Diff, a tree-structured GRPO framework for reinforcement-learning fine-tuning of protein diffusion models. The method reuses intermediate denoising checkpoints and propagates normalized terminal rewards through the resulting trees to construct segment-level advantages. The application to all-atom protein design is potentially valuable, and the use of independent structural evaluators is a positive aspect of the experimental evaluation.

Critical strengths

none

Critical weaknesses
  • TreeGRPO has already introduced tree-structured diffusion rollouts with shared-prefix reuse, reward propagation from terminal leaves, edge-level credit assignment, and GRPO-based policy updates. These mechanisms closely correspond to the primary methodological contributions claimed by TSG-Diff.
  • Despite emphasizing efficient exploration under a fixed rollout budget, the paper does not provide sufficiently detailed empirical comparisons of reward versus cumulative denoising NFEs, wall-clock time, and reward-evaluation cost. Matched-budget comparisons, training curves, and multi-seed statistical analysis are necessary to substantiate the central claims.
  • The paper overlooks several important prior works on tree-search-guided diffusion generation and reward-guided fine-tuning, particularly PepTune[1] and TR2-D2[2].

[1] Tang S, Zhang Y, Chatterjee P. Peptune: De novo generation of therapeutic peptides with multi-objective-guided discrete diffusion[J]. ArXiv, 2025: arXiv: 2412.17780 v4.

[2] Tang S, Zhu Y, Tao M, et al. Tr2-d2: Tree search guided trajectory-aware fine-tuning for discrete diffusion[J]. ArXiv, 2025: arXiv: 2509.25171 v1.

Questions for authors
  • How does TSG-Diff establish methodological novelty over existing tree-search-based diffusion optimization methods, particularly TreeGRPO, TR2-D2, and PepTune?
5 comments
AuthorsOpenReview ↗

Title: Author Response To Reviewer (1/5)

Dear Reviewer PySA,

We sincerely appreciate your valuable feedback and provide point-wise responses below. Key changes are highlighted in blue\color{blue}{\text{blue}} in the revised manuscript.


W1: TreeGRPO has already introduced tree-structured diffusion rollouts with shared-prefix reuse, reward propagation from terminal leaves, edge-level credit assignment, and GRPO-based policy updates. These mechanisms closely correspond to the primary methodological contributions claimed by TSG-Diff.

We agree that tree-structured diffusion rollouts, prefix reuse, and leaf-to-edge credit propagation are established by TreeGRPO. Our contribution is the particular combination of checkpoint-resumed suffix expansion, within-tree and condition-wide advantage normalization, and parent-relative checkpoint-segment credit, V(v)−V(parent⁡(v))V(v)-V(\operatorname{parent}(v)), blended with terminal-leaf advantage. TSG-Diff also separates candidate generation from reward evaluation: each tree generates 1+LN1+LN candidate trajectories, of which KK leaves are selected for scoring. These choices define our formulation for all-atom protein diffusion; internal ablations alone cannot establish an advantage over TreeGRPO.

We therefore compare TSG-Diff with a BoltzGen port of TreeGRPO using the same initialization, banded interface-contact reward, evaluation protocol, and common PPO clipping and KL settings. The port follows the subtree-mean credit calculation in the released implementation. The comparison matches 400 training updates and 1,600 training reward evaluations per run, while explicitly reporting differences in rollout NFEs, optimizer steps, and training time.

Table 1a. Comparison after 400 training updates: frozen-policy reward, AF2-Multimer ipTM, and computational cost. Reward is mean ± s.e. across seeds 42–47, evaluated on 120 held-out complexes using the training reward function. AF2-Multimer ipTM is an independent predicted interface-confidence score, reported as mean ± s.e. across seeds 45–47 of the per-policy mean over 32 test complexes × 3 designs, using single-sequence input, one model, and three recycles with the native receptor and designed peptide. For seeds 42–44, chain GRPO and TSG-Diff (λ = 1) use earlier checkpoints of the same recipe, re-scored in the common evaluation harness. The other runs were trained for this comparison. NFEs count rollout denoiser calls, with the average per update in parentheses. GPU-hours are estimated from median seconds per update × 400 / 3,600, with one job per GPU, excluding the first 10 updates and checkpoint writes from the timing measurement.

MethodFrozen reward ↑ (120 evaluation complexes)AF2-Multimer ipTM ↑ (32 test complexes)Cumulative rollout NFEs ↓ (per update)Training reward evaluations ↓Optimiser steps per updateTraining GPU-hours ↓
TreeGRPO, BoltzGen port (w = 2, k = 2)0.541 ± 0.0070.126 ± 0.00211,800 (29.5)1,60020.42
TSG-Diff (λ = 1)0.556 ± 0.0080.128 ± 0.00216,001 (40.0)1,600120.88
TSG-Diff (λ = 0.5), Algorithm 1 as written0.577 ± 0.0080.134 ± 0.00116,001 (40.0)1,600120.88
Reference: chain GRPO0.539 ± 0.0080.129 ± 0.00119,200 (48.0)1,600120.72
Reference: base policy, no fine-tuning0.5230.129----

TSG-Diff improves frozen-policy reward over the TreeGRPO port at matched updates and reward evaluations, but requires more compute. Across six seeds, paired gains are +0.015 (95% CI: [+0.008, +0.022]) for λ = 1 and +0.036 ([+0.026, +0.046]) for λ = 0.5, with all seeds favoring TSG-Diff (Table 2b). Estimated training cost increases from 0.42 to 0.88 GPU-hours. Structural evidence is weaker: λ = 0.5 achieves the highest numerical AF2-Multimer ipTM (0.134 ± 0.001; Table 1a), but λ = 1 does not improve over the base policy, and paired ipTM confidence intervals are unavailable. These algorithm-level comparisons cannot isolate effects of rollout construction, normalization, credit assignment, or optimizer steps. The segment-credit ablation remains inconclusive (+0.011; 95% CI: [−0.007, +0.029]), and neither two-level normalization nor leaf selection is isolated. Released peptide-pipeline comparisons are discussed under W3.


W2: Despite emphasizing efficient exploration under a fixed rollout budget, the paper does not provide sufficiently detailed empirical comparisons of reward versus cumulative denoising NFEs, wall-clock time, and reward-evaluation cost. Matched-budget comparisons, training curves, and multi-seed statistical analysis are necessary to substantiate the central claims.

Thank you for the suggestion. We now report frozen-policy reward at 100, 200, and 400 updates, rollout NFEs, reward-evaluation counts, update times, scoring microbenchmarks, and paired reward differences across six seeds, distinguishing denoising savings from wall-clock efficiency.

AuthorsOpenReview ↗

Title: Author Response To Reviewer (2/5)

All methods use the same BoltzGen initialization, training complexes paired by seed, 12 denoising steps, a learning rate of 2×10−52\times10^{-5}, 400 single-complex training updates, and the same banded interface-contact reward. Chain GRPO, the trajectory-credit tree, both TSG-Diff variants, and the TreeGRPO port score four leaves per update; the small- and large-search TR2-D2-style adaptations score 6 and 96, respectively. Thus, matching training updates does not equalize cumulative rollout NFEs, reward evaluations, optimizer steps, or wall-clock time.

Frozen policies are evaluated on the same 120 held-out complexes. Because this evaluation uses the training reward function, it measures generalization of the optimization objective; independent predicted structural metrics are reported separately in Tables 1a and 3c.

Table 2a. Frozen-policy reward and cumulative computational cost across training checkpoints. Rewards are reported as mean ± s.e. across training seeds; parentheses in the checkpoint columns indicate cumulative rollout NFEs. The 100-, 200-, and 400-update columns use seeds 45–47 (n = 3), while the final two columns report 400-update rewards for seeds 42–44 and all six seeds, respectively. For seeds 42–44, chain GRPO, the trajectory-credit tree, and TSG-Diff (λ = 1) use earlier checkpoints re-scored with the same evaluation protocol. Reward-evaluation counts refer to individual scored leaves, rather than batched calls to the reward function.

MethodNFEs / reward evaluations / optimizer steps per updateSeconds per update (estimated GPU-h at 400)Reward-evaluation time per update, ms [kernel; wall]100 updates200 updates400 updates400 updates, seeds 42–44400 updates, six seeds
Chain GRPO48.0 / 4 / 126.45 (0.72)[0.12; 0.86]0.538 ± 0.008 (4,800)0.511 ± 0.019 (9,600)0.531 ± 0.008 (19,200)0.547 ± 0.0150.539 ± 0.008
Tree, trajectory-level credit40.0 / 4 / 127.88 (0.88)[0.12; 0.86]0.524 ± 0.012 (3,972)0.529 ± 0.019 (7,997)0.532 ± 0.003 (16,001)0.558 ± 0.0290.545 ± 0.014
TSG-Diff (λ = 1)40.0 / 4 / 127.91 (0.88)[0.12; 0.86]0.539 ± 0.015 (3,972)0.546 ± 0.006 (7,997)0.553 ± 0.016 (16,001)0.559 ± 0.0100.556 ± 0.008
TSG-Diff (λ = 0.5)40.0 / 4 / 127.88 (0.88)[0.12; 0.86]0.543 ± 0.012 (3,972)0.571 ± 0.008 (7,997)0.573 ± 0.007 (16,001)0.581 ± 0.0140.577 ± 0.008
TreeGRPO port29.5 / 4 / 23.81 (0.42)[0.12; 0.86]0.526 ± 0.001 (4,060)0.531 ± 0.002 (7,400)0.535 ± 0.012 (11,800)0.547 ± 0.0100.541 ± 0.007
TR2-D2-style, small search63.2 / 6 / 127.96 (0.88)[0.47; 3.4]0.519 ± 0.005 (6,325)0.529 ± 0.007 (12,647)0.532 ± 0.011 (25,300)0.542 ± 0.0090.537 ± 0.006
TR2-D2-style, large search587.5 / 96 / 1212.04 (1.34)[1.6; 14]0.536 ± 0.002 (59,269)0.557 ± 0.007 (117,469)0.558 ± 0.012 (235,008)0.566 ± 0.0080.562 ± 0.006

Update times are measured medians. Training GPU-hours are estimates obtained as median seconds per update × 400 / 3,600 for single-GPU runs, rather than measured end-to-end totals. Rollout NFEs exclude the forward and backward computations used for optimization and the cost of policy evaluation.

Reward-evaluation timings profile the training reward on the median-size complex, with 77 design atoms and 592 target atoms. Entries report [GPU kernel time; fastest observed wall-clock time on a shared GPU], accounting for each method's reward-call schedule. The first five methods make one batched call on four leaves per update; the TR2-D2-style adaptations make 3 + 1 and 12 + 1 batched calls.

Table 2b. Paired reward differences after 400 updates across six seeds (42–47). Values are mean differences with 95% t-intervals over paired training-seed differences on the fixed 120-complex evaluation panel. Bold intervals exclude zero. The checkpoint provenance described above also applies here. These intervals characterize variation across training seeds on this panel, rather than uncertainty from sampling new evaluation complexes.

ComparisonMean reward difference [95% CI]Seeds agreeing with mean sign
TSG-Diff (λ = 0.5) − chain GRPO+0.038 [+0.019, +0.057]6/6
TSG-Diff (λ = 1) − chain GRPO+0.017 [+0.008, +0.026]6/6
TSG-Diff (λ = 1) − tree, trajectory-level credit+0.011 [−0.007, +0.029]5/6
TSG-Diff (λ = 1) − TreeGRPO port+0.015 [+0.008, +0.022]6/6
TSG-Diff (λ = 0.5) − TreeGRPO port+0.036 [+0.026, +0.046]6/6
AuthorsOpenReview ↗

Title: Author Response To Reviewer (3/5)

The updated results support three conclusions:

  1. TSG-Diff improves reward over chain GRPO with fewer NFEs but slower updates. At 400 updates, both λ settings outperform chain GRPO across all six seeds, with paired 95% CIs excluding zero. Rollout NFEs decrease by 16.7% at the same 1,600 reward evaluations, but update times increase by 22–23% (7.88–7.91 versus 6.45 seconds).

  2. TSG-Diff improves reward over the TreeGRPO port at higher cost. Both paired 95% CIs exclude zero, but rollout NFEs increase by 35.6% and update time roughly doubles, demonstrating a reward–cost tradeoff.

  3. Component-level evidence remains limited. The same-tree segment-credit comparison is inconclusive. Although λ = 0.5 yields higher mean reward than λ = 1 (0.577 versus 0.556), no paired CI is reported, so its advantage is not statistically established.


W3. The paper overlooks several important prior works on tree-search-guided diffusion generation and reward-guided fine-tuning, particularly PepTune and TR2-D2.

We agree that PepTune and TR2-D2 are important prior work and will expand the related-work discussion to clarify their relationship to TSG-Diff. We distinguish their search and learning mechanisms from the empirical comparisons, which address both controlled adaptations and complete released pipelines.

PepTune’s main practical limitation for our setting is the recurring cost of inference-time search. Its Monte Carlo Tree Guidance uses batched expansion and property scoring to optimize peptide SMILES without updating the generator during guidance. Consequently, subsequent guided-generation runs repeat search and scoring. Its reported setting uses 128 iterations × 50 children, corresponding to 6,400 candidate rollouts per search run, although a run can deliver multiple designs. Receptor-bound structural evaluation also requires an additional structure-generation or docking stage. TSG-Diff instead updates a coordinate-generating policy, allowing training-time exploration to be amortized over subsequent generation. This distinction does not by itself establish lower total cost.

TR2-D2 introduces replay-buffer construction, refresh, and off-policy weighting costs. It uses MCTS to curate trajectories for weighted denoising cross-entropy on masked discrete diffusion. Its behavior depends on the search budget and buffer-refresh schedule; its peptide experiments report a reward–diversity tradeoff when changing refresh frequency. Applying this objective to continuous coordinates requires an adaptation. TSG-Diff instead uses fresh policy rollouts and segment advantages without maintaining a replay buffer across updates. Both approaches learn policies that can subsequently generate without search, so this benefit is not unique to TSG-Diff.

TreeGRPO couples generated-leaf count with reward-evaluation expenditure in its published algorithm. However, it scores every generated leaf and places branching within a contiguous stochastic window. TSG-Diff provides a different allocation mechanism: resume stored checkpoints and select KK leaves for scoring independently of the number generated. This permits separate control of generation and scoring budgets, although unscored branches still incur denoising cost. These are methodological distinctions whose benefits require empirical comparison.

Our tree is a rollout structure rather than a search (expansion nodes and scored leaves are drawn uniformly, with no value-guided selection as in PepTune and TR2-D2). To separate algorithmic effects from differences in pretrained models and molecular representations, we use two protocols:

  • A — Controlled comparison on BoltzGen: common initialization, reward, and evaluation, with 400 matched training updates and explicit reporting of NFEs, reward evaluations, and wall-clock time. These runs do not match cumulative compute. A separate inference-time experiment compares PepTune-style guidance with flat sampling at equal scored-leaf counts.
  • B — Complete-pipeline comparison: the released PepTune and TR2-D2 implementations and TSG-Diff are evaluated on eight common targets. We report delivered designs, eligibility for the common structural evaluator, conditional quality metrics, and compute costs. This protocol includes differences in pretrained models, representations, and cohort sizes.

Controlled post-training results. TSG-Diff (λ = 0.5) achieves the highest six-seed mean reward in Table 2a: 0.577 ± 0.008, versus 0.562 ± 0.006 and 0.537 ± 0.006 for large- and small-search TR2-D2-style fine-tuning. Compared with large search, it uses approximately 93% fewer rollout NFEs, 96% fewer reward evaluations, and 35% shorter updates. The numerical reward gain (+0.015) lacks a paired confidence interval, leaving its statistical uncertainty unresolved.

Reviewer_R74ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TSG-Diff for RL fine-tuning of protein diffusion models. It builds trees by resuming generation from saved intermediate states. This allows multiple candidates to share denoising prefixes. Terminal rewards are normalized within each tree and across trees. The resulting leaf advantages are averaged over subtrees. Differences between parent and child values provide segment advantages, which are combined with trajectory advantages for PPO updates. The method is applied to BoltzGen with rewards for binding affinity, interface confidence, and structural preservation. Experiments cover peptide binder design and side-chain packing. The results show improvements over the base model on several computational metrics. Ablations examine reward terms, normalization, prefix reuse, and segment credit.

Critical strengths
  1. The evaluation goes beyond the training reward. The authors explicitly recognize that Boltz-derived scores are coupled to the optimization objective. They therefore include Rosetta, ESMFold, and several external structure predictors. This makes the reported improvements more convincing than gains measured only by the reward heads.
  2. The ablations separate two important components. Table 3 compares segment credit with trajectory credit under the same tree sampler. It also includes independent chains. These controls provide useful evidence that both credit assignment and prefix reuse contribute to performance.
Critical weaknesses
  1. The benefit of the proposed changes over TreeGRPO is not established. TreeGRPO already uses branching diffusion rollouts, shared prefixes, and tree credit assignment. This paper uses a different sampling strategy, two levels of advantage normalization, and a different credit propagation rule. Table 3 supports these components relative to simpler alternatives. However, it does not show whether the proposed changes improve on TreeGRPO.
  2. Important training details remain unclear. Equation (6) requires transition probability ratios. Table 6 reports a reverse-transition noise level, but the paper does not specify the transition distribution or how its log probabilities are computed. This makes the PPO update difficult to reproduce. Also, Algorithm 1 and Appendix D select K leaves per tree, while Algorithm 2 uses all leaves. The value of K is not reported. These choices affect reward normalization, credit assignment, and the policy update.
  3. The affinity reward needs further justification. Section 3.4 and Appendix B.1 use a learned binding logit to reward peptide binding. However, the standard Boltz-2 affinity module is designed for small molecules binding to proteins. The paper does not explain how this module is applied to peptides or whether a separate peptide affinity model is used. This leaves the suitability of a central reward term unclear.
Questions for authors
  1. Can you compare TSG-Diff with TreeGRPO using the same BoltzGen model and rewards under matched denoising compute? Please also report the number of reward evaluations. Such a comparison would help establish the benefit of the proposed algorithmic changes.
  2. What stochastic transition distribution is used during training? How are the transition log probabilities in Equation (6) computed? Does training use K selected leaves or all leaves from each tree? Please report K and the leaf selection rule, and reconcile Algorithms 1 and 2.
  3. Which model provides the affinity reward for peptide binders? If it is the standard Boltz-2 affinity module, how are peptides represented, and what supports its use here? If a separate model is used, please describe its training and validation.
6 comments
AuthorsOpenReview ↗

Title: Author Response To Reviewer (1/6)

Dear Reviewer R74Z,

We sincerely appreciate your valuable feedback and provide point-wise responses below. Key changes are highlighted in blue\color{blue}{\text{blue}} in the revised manuscript.


W1: The benefit of the proposed changes over TreeGRPO is not established. TreeGRPO already uses branching diffusion rollouts, shared prefixes, and tree credit assignment. This paper uses a different sampling strategy, two levels of advantage normalization, and a different credit propagation rule. Table 3 supports these components relative to simpler alternatives. However, it does not show whether the proposed changes improve on TreeGRPO.

We agree that the original ablations do not establish whether TSG-Diff improves on TreeGRPO. We have therefore added a direct comparison with a BoltzGen port of TreeGRPO, reporting both performance and computational cost.

TreeGRPO already establishes tree-structured diffusion rollouts, shared-prefix reuse, and leaf-to-edge credit propagation. Our methodological contribution concerns the combination of checkpoint-resumed suffix expansion, within-tree and condition-wide advantage normalization, and parent-relative checkpoint-segment credit, V(v)−V(parent⁡(v))V(v)-V(\operatorname{parent}(v)), blended with terminal-leaf advantage. TSG-Diff also separates candidate generation from reward evaluation: each tree generates 1+LN1+LN candidate trajectories, of which KK leaves are selected for scoring. These distinctions define the proposed method; their benefits require empirical evaluation.

The comparison uses the same BoltzGen initialization, training complexes paired by seed, banded interface-contact reward, 12-step denoising schedule, learning rate of 2×10−52\times10^{-5}, evaluation protocol, and PPO clipping and KL settings. The TreeGRPO port follows the subtree-mean credit calculation in its released implementation, with w=2w=2 and k=2k=2. All trained methods below receive 400 training updates and 1,600 terminal reward evaluations.

Table 1. Performance and computational cost after 400 training updates.

MethodHeld-out reward ↑AF2-Multimer ipTM ↑Cumulative rollout NFEsTraining reward evaluationsOptimizer steps per updateEstimated training GPU-h
TreeGRPO, BoltzGen port0.541 ± 0.0070.126 ± 0.00211,8001,60020.42
TSG-Diff, λ=1\lambda=10.556 ± 0.0080.128 ± 0.00216,0011,600120.88
TSG-Diff, λ=0.5\lambda=0.50.577 ± 0.0080.134 ± 0.00116,0011,600120.88
Reference: chain GRPO0.539 ± 0.0080.129 ± 0.00119,2001,600120.72
Reference: base policy0.5230.129————

Reward is mean ± s.e. across six training seeds (42–47), evaluated on 120 held-out complexes using the training reward function. AF2-Multimer ipTM is an independent predicted interface-confidence metric, reported as mean ± s.e. across three seeds (45–47), with each policy evaluated on 32 test complexes and three designs per complex. Evaluation uses single-sequence input, one model, and three recycles with the native receptor and designed peptide. For seeds 42–44, chain GRPO and TSG-Diff with λ=1\lambda=1 use earlier checkpoints of the same training recipe, re-scored under the common protocol; the other runs were trained for this comparison. The λ=0.5\lambda=0.5 setting corresponds to Algorithm 1 as written.

Rollout NFEs count denoiser calls during trajectory generation and exclude optimization and policy-evaluation costs. Training GPU-hours are estimated as median seconds per update × 400 / 3,600 for single-GPU runs; timing excludes the first ten updates and checkpoint writes. They are not measured end-to-end totals.

TSG-Diff improves held-out reward over TreeGRPO by +0.015+0.015 (λ=1\lambda=1) and +0.036+0.036 (λ=0.5\lambda=0.5), with paired 95% tt-CIs of [+0.008,+0.022][+0.008,+0.022] and [+0.026,+0.046][+0.026,+0.046] across six seeds on a fixed panel.

W2 Important training details remain unclear. Equation (6) requires transition probability ratios. Table 6 reports a reverse-transition noise level, but the paper does not specify the transition distribution or how its log probabilities are computed. This makes the PPO update difficult to reproduce. Also, Algorithm 1 and Appendix D select K leaves per tree, while Algorithm 2 uses all leaves. The value of K is not reported. These choices affect reward normalization, credit assignment, and the policy update.

We thank the reviewer for identifying these omissions and inconsistencies. We specify the reverse-transition distribution and the score used in the policy update, report KK alongside the tree configuration, and reconcile Algorithms 1 and 2. We also provide additional analyses of the reliability of checkpoint supervision below.

AuthorsOpenReview ↗

Title: Author Response To Reviewer (2/6)

Following Eq. (1), the state is st=(c,t,zt)\mathbf s_t=(\mathbf c,t,\mathbf z_t), and the action is at=zt−1\mathbf a_t=\mathbf z_{t-1}. To specify the reverse-transition implementation, we introduce additional notation: μθ,t\boldsymbol\mu_{\theta,t} for the transition mean, νt\nu_t for its standard deviation, hh for the reverse-step scale, η\eta for the reverse-transition noise multiplier, and ϵnum\epsilon_{\mathrm{num}} for a numerical safeguard. These implementation-level definitions supplement the policy notation in Eq. (1). For t=T,…,1t=T,\ldots,1, with σt>σt−1\sigma_t>\sigma_{t-1}, the implemented training policy is

πθ(at∣st)=N(μθ,t,νt2I),\pi_\theta(\mathbf{a}_t\mid\mathbf{s}_t) = \mathcal{N}\left(\boldsymbol{\mu}_{\theta,t},\nu_t^2\mathbf{I}\right),

where

μθ,t=zˉt+h(σt−1−σt)zˉt−x^θ(zˉt,σt,c)max⁡(σt,ϵnum),νt=ηmax⁡!(σt2−σt−12,ϵnum).\boldsymbol{\mu}_{\theta,t}=\bar{\mathbf{z}}_t+h(\sigma_{t-1}-\sigma_t)\frac{\bar{\mathbf{z}}_t-\hat{\mathbf{x}}_{\theta}(\bar{\mathbf{z}}_t,\sigma_t,\mathbf{c})}{\max(\sigma_t,\epsilon_{\mathrm{num}})},\qquad\nu_t=\eta\sqrt{\max\\!\left(\sigma_t^2-\sigma_{t-1}^2,\epsilon_{\mathrm{num}}\right)}.

Here, zˉt\bar{\mathbf{z}}_t denotes zt\mathbf{z}_t centered using the valid-atom mask, x^θ\hat{\mathbf{x}}_{\theta} denotes the predicted clean structure, and ϵnum=10−12\epsilon_{\mathrm{num}}=10^{-12}. We introduce the symbols hh and η\eta here for the parameters named “reverse-step scale” and “reverse-transition noise level” in Table 6; the original table does not assign them symbols. Thus, η\eta is a noise multiplier, whereas the transition standard deviation νt\nu_t also depends on the noise schedule σt\sigma_t. The safeguard ϵnum\epsilon_{\mathrm{num}} is distinct from both the advantage-normalization constant ϵnorm\epsilon_{\mathrm{norm}} in Eq. (3) and the PPO clipping parameter ϵ\epsilon in Eq. (6). The sampled action is stored before centering for the next denoiser evaluation.

For each stored transition in trajectory gg, we additionally introduce ℓtg(θ)\ell_t^g(\theta) for the implemented coordinate-averaged score and r~tg(θ)\widetilde r_t^g(\theta) for the corresponding surrogate ratio. With DD denoting the total number of atom-coordinate slots, including padding, the implementation computes

=-\frac{ \sum_{d=1}^{D} \left(a_{t,d}^g-\mu_{\theta,t,d}^g\right)^2 }{ 2D(\nu_t^2+\epsilon_{\mathrm{num}}) }, \qquad \widetilde r_t^g(\theta) =\exp\!\left[ \ell_t^g(\theta)-\ell_t^g(\theta_{\mathrm{old}}) \right].$$ Both scores use the same stored state–action pair. The reduction averages over all $D$ atom-coordinate slots, including padding, without applying an atom mask. This is a **coordinate-normalized surrogate**, whereas the text below Eq. (6) currently describes an exact importance ratio. Because the old and current policies share the transition variance, the Gaussian normalization constants cancel. However, the division by $D$ remains: ignoring numerical regularization, $$\widetilde{r}_t^g(\theta)=\left[\frac{\pi_{\theta}(\mathbf{a}_t^g\mid\mathbf{s}_t^g)}{\pi_{\theta_{\mathrm{old}}}(\mathbf{a}_t^g\mid\mathbf{s}_t^g)}\right]^{1/D}.$$ We revise Eq. (6) and its accompanying definition to specify this surrogate and its use inside the clipping operator. Regarding leaf selection, after constructing each tree, the implementation uniformly selects $K$ terminal leaves without replacement, before reward evaluation. These selected leaves determine reward normalization, checkpoint-value estimation, credit assignment, and the policy update. Algorithm 2 incorrectly describes evaluating all terminal leaves. We amend it to include the same selection step as Algorithm 1 and Appendix D. The released configuration uses $$(M,N,L,K,T)=(3,2,3,6,50).$$ Following Appendix D, each tree contains $1+LN=7$ candidate terminal leaves. After tree construction, we uniformly select $K=6$ leaves without replacement for reward evaluation, normalization, credit propagation, and policy updates. Thus, $$G_{\mathrm{cand}}=M(1+LN)=21, \qquad G_{\mathrm{eff}}=MK=18$$ per conditioning input. We use the composite reward defined in Eq. (5), combining affinity, interface confidence, and structural regularization. The paper’s objective uses the composite reward in Eq. (5), repeated in Eq. (8), which combines affinity, interface confidence, and structural regularization. For the additional analyses below, we introduce two auxiliary reward protocols, labeled **Setting A** and **Setting B** only in this response. Setting A uses a banded interface-contact reward; Setting B uses the released lDDT-to-native reward. These analyses examine training and credit-estimation behavior under the specified auxiliary rewards. They do not evaluate the full composite objective in Eq. (5), and their results should not be interpreted as performance measurements under that objective.
AuthorsOpenReview ↗

Title: Author Response To Reviewer (3/6)

Table 2a. Tree configurations and auxiliary-reward training comparisons. Setting A results are frozen training rewards on 120 complexes after 400 updates, reported as mean ± standard error across seeds 45–47. The base-policy reward is 0.523. Differences are relative to the Setting A baseline, with 95% tt-intervals over seeds. “—” denotes an unmeasured result. Figure 4 in Appendix F.1 of the revised manuscript presents metric-versus-NFE curves at four budgets: 5K, 10K, 15K, and 20K cumulative denoiser calls, comparing chain GRPO with TSG-Diff at λ=1\lambda=1 and λ=0.5\lambda=0.5..

SettingMMNNLLKKLeaves/treeScored/updateTTRewardDifference [95% interval]
Baseline3236718500.563 ± 0.013-
Fewer scored leaves323276500.528 ± 0.019−0.035 [−0.058, −0.012]
Intermediate scored-leaf count3234712500.534 ± 0.015−0.029 [−0.049, −0.009]
All-leaf training3237721500.559 ± 0.013−0.004 [−0.006, −0.002]
More expansions per round, fewer rounds3326718500.546 ± 0.010−0.017 [−0.033, −0.001]
Fewer expansions per round, more rounds3166718500.539 ± 0.009−0.024 [−0.040, −0.008]

For checkpoint supervision, we retain the notation of Eq. (4):

\qquad V(v)=\frac{1}{n_v} \sum_{u\in\operatorname{Desc}(v)}A_{\mathrm{leaf}}(u), \qquad n_v>0.$$ Here, $\operatorname{Desc}(v)$ contains the selected leaves descending from $v$, and $A_{\mathrm{leaf}}$ is the two-level normalized advantage from Eq. (3). Both $n_v$ and $V(v)$ depend on the selected leaves. The scalar $K$, which specifies the number of scored leaves per tree, is distinct from the checkpoint schedule $\mathcal K=\{\kappa_0,\ldots,\kappa_Q\}$. Selecting six leaves per tree does not imply six descendants per checkpoint. Value-error statistics are restricted to nodes with at least one selected descendant ($n_v>0$) and nonzero conditional reward variance ($\sigma_R(v)>0$). The original chain contributes a candidate leaf, but uniform selection of $K<7$ leaves without replacement does not guarantee that this leaf is selected; $K=7$ selects all candidate leaves. Support therefore depends on both the generated tree and the selected leaves. To examine descendant-sampling uncertainty, we conduct a separate diagnostic under the frozen base policy. We retain two auxiliary reward protocols: **Setting A** uses the banded interface-contact reward, and **Setting B** uses the lDDT-to-native reward. These labels identify the diagnostic rewards only. Neither diagnostic evaluates the full composite objective in Eq. (5). We distinguish the normalized $V(v)$ in Eq. (4) from the reward-space estimator $$\widehat V_K^{(R)}(v) =\frac{1}{n_v} \sum_{u\in\operatorname{Desc}(v)}r(u), \qquad n_v>0,$$ where $r(u)$ is the auxiliary reward under Setting A or B. This estimator averages rewards before advantage normalization. Table 2b reports its discrepancies from the continuation reference and the associated credit-agreement metrics. It does not directly measure the bias and variance of the normalized $V(v)$ used for training. We compare $K\in\{2,4,6,7\}$ using $N=2$, $L=3$, and $T=50$. Here, $K=6$ is the six-leaf scoring budget, and $K=7$ scores all seven candidate leaves per tree. Each setting uses uniform leaf selection without replacement. The diagnostic includes all three expansion rounds and preserves dependencies arising from shared ancestry. Table 2a reports training comparisons with $M=3$, whereas Table 2b summarizes reward-space estimates from two frozen root chains per complex. The latter does not directly evaluate the across-tree normalized checkpoint estimator used in the $M=3$ training configuration. For each of 120 complexes, we use two frozen root chains and draw approximately 20 continuations from each chain's three non-terminal checkpoint states. The diagnostic evaluates 200 trees per complex resampled with the training code's expansion rule. For each evaluated tree and checkpoint $v$, $V_{\mathrm{ref}}^{(R)}(v)$ is the unweighted mean reward of continuations from $v$'s state that the tree did not use. The estimate $\widehat V_K^{(R)}(v)$ is the unweighted mean reward of the selected leaves at or below $v$, and $\sigma_R(v)$ denotes the conditional reward standard deviation at that checkpoint. Equal weighting of included descendants does not make shared-ancestry samples independent. The approximately 20 continuations describe the initial pool, rather than the number remaining in every reference after excluding continuations used by the evaluated tree. We use this finite-sample reference as a comparator, not as a known population value. Scoring all seven candidate leaves provides the all-leaf comparison, but does not replace the additional continuation reference.
Reviewer_E1uoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a heuristic rollout method and credit assignment by reformulating reverse diffusion as a branching tree over predefined intermediate checkpoints, named TSG-Diff. They apply TSG-Diff to protein design to satisfy composite reward functions, including binding affinity, interface confidence, and structural regularization. Empirical results show that TSG-Diff outperforms existing approaches on co-design and packing tasks.

Critical strengths
  1. The presentation is clear and easy to follow. In particular, the motivation for tree reformulation and the sparse supervision that encourages the design of TSG-Diff are well framed.
  2. Reformulating the reverse diffusion as a branching tree is interesting to me. The heuristic design of the tree-structured rollout and checkpoint-segment credit assignment is straightforward but effective.
Critical weaknesses
  1. My main concern with this paper is the lack of comparison with related work [1] & [2], which introduces similar ideas on credit assignment for GRPO. Could the authors differentiate their work from this one? If you can provide more experimental results, it would be better.
  2. In Sec. 3.3, how do we ensure the reliability of the intermediate supervision produced by checkpoint-segment credit propagation with limited samples and uneven branching? Specifically, the authors never report the value of K, so the reliability of V(v)—an average over the selected descendants—cannot be assessed. Could the authors report K and the bias/variance of V(v) at that K against a large-K reference?
  3. This work emphasizes that computational efficiency is a central motivation and provides Appendix D to show the better use of the denoising budget. Please provide compute-matched metric-versus-NFE curves at three or more budget points for chain GRPO and TSG-Diff.
Questions for authors
  1. Search-based methods such as MCTS expand paths sequentially, but these sequential dependencies batch poorly on diffusion inference and severely bottleneck online RL efficiency. I am unclear on the issues with MCTS. IUC, batched MCTS can also generate search trees in parallel and achieve reward-driven exploration. Could you explain your opinion further?
5 comments
AuthorsOpenReview ↗

Title: Author Response To Reviewer (1/5)

Dear Reviewer E1uo,

We sincerely appreciate your valuable feedback and provide point-wise responses below. Key changes are highlighted in blue\color{blue}{\text{blue}} in the revised manuscript.


W1: My main concern with this paper is the lack of comparison with related work TreeRPO & TreeGRPO, which introduces similar ideas on credit assignment for GRPO. Could the authors differentiate their work from this one? If you can provide more experimental results, it would be better.

We thank the reviewer for highlighting these closely related methods and explain the algorithmic differences more precisely.

TreeRPO constructs trees of language-model continuations, recursively averages child rewards, and computes advantages within sibling groups. TSG-Diff instead averages the normalized advantages of selected descendant leaves and combines within-tree and across-tree normalization. This distinction affects how uneven subtrees are weighted.

TreeGRPO combines deterministic ODE steps with branching inside stochastic SDE windows and propagates leaf advantages to individual edges. TSG-Diff initializes complete trajectories, selects stored intermediate checkpoints, and generates additional suffixes from those checkpoints. Its credit rule combines a checkpoint-segment value difference with the terminal leaf advantage:

At=λ[V(v)−V(pa⁡(v))]+(1−λ)Aleaf(u),A_t = \lambda\bigl[V(v)-V(\operatorname{pa}(v))\bigr]+(1-\lambda)A_{\mathrm{leaf}}(u),

where V(v)V(v) averages the advantages of selected leaves below checkpoint vv, and uu is the leaf belonging to the sampled trajectory. Together with two-level normalization, this defines the specific method we seek to evaluate.

We compare TSG-Diff with a TreeRPO-style diffusion adaptation and a BoltzGen port of TreeGRPO. The TreeRPO adaptation follows the released implementation’s credit calculation. The TreeGRPO port implements its branching and backup rules. All methods use the same initialization, banded interface-contact reward, evaluation protocol, and common PPO clipping and KL settings. We match 400 training updates and 1600 training reward evaluations per run; denoising compute and optimizer steps differ.

Table 1a. Performance and training cost at a matched reward-evaluation budget. All methods use 400 training updates and 1,600 training reward evaluations per run. Denoising compute and optimizer steps differ.

MethodHeld-out reward ↑AF2-Multimer ipTM ↑Cumulative rollout NFEsTraining reward evaluationsEstimated training time (GPU-h)
Chain GRPO0.539 ± 0.0080.129 ± 0.00119,2001,6000.72
TreeRPO-style credit (diffusion adaptation)0.535 ± 0.0120.121 ± 0.00313,7001,6000.91
TreeGRPO (BoltzGen port)0.541 ± 0.0070.126 ± 0.00211,8001,6000.42
Tree with trajectory-level credit0.545 ± 0.0140.132 ± 0.00616,0011,6000.88
TSG-Diff, λ=1\lambda=10.556 ± 0.0080.128 ± 0.00216,0011,6000.88
TSG-Diff, λ=0.5\lambda=0.50.577 ± 0.0080.134 ± 0.00116,0011,6000.88

Reward is mean ± s.e. across six training seeds on 120 held-out complexes. ipTM uses three seeds, 32 complexes, and three designs per complex. GPU-hours are estimated from median update time, excluding evaluation and other overheads.

At the matched reward-evaluation budget, TSG-Diff (λ=0.5\lambda=0.5) achieves the highest mean reward and ipTM, exceeding TreeRPO and TreeGRPO by 0.042 and 0.036 in reward. Against trajectory-level credit, it gains 0.032 in reward and 0.002 in ipTM at identical reported compute. Compared with TreeGRPO, however, it uses 35.6% more NFEs, 12 versus two optimizer steps per update, and roughly twice the training time; equal-compute performance remains unresolved.

Segment credit alone remains inconclusive: λ=1\lambda=1 improves reward over trajectory-level credit by 0.011. Paired ipTM confidence intervals are unavailable. The TreeRPO and TreeGRPO comparisons therefore assess complete methods without isolating individual components.


W2: In Sec. 3.3, how do we ensure the reliability of the intermediate supervision produced by checkpoint-segment credit propagation with limited samples and uneven branching? Specifically, the authors never report the value of K, so the reliability of V(v)—an average over the selected descendants—cannot be assessed. Could the authors report K and the bias/variance of V(v) at that K against a large-K reference?

We thank the reviewer for highlighting the missing sampling specification and the uncertainty in checkpoint-level supervision. We clarify the leaf-selection procedure, report additional reliability diagnostics, and distinguish what these diagnostics establish from what remains unmeasured.

The released configuration uses

(M,N,L,K,T)=(3,2,3,6,50).(M,N,L,K,T)=(3,2,3,6,50).

AuthorsOpenReview ↗

Title: Author Response To Reviewer (2/5)

Following Appendix D, each tree contains 1+LN=71+LN=7 candidate terminal leaves. After constructing each tree, we uniformly select K=6K=6 leaves without replacement, before reward evaluation. These selected leaves are used for reward normalization, checkpoint-value estimation, credit propagation, and policy updates. Thus, each conditioning input produces M(1+LN)=21M(1+LN)=21 candidate leaves, of which MK=18MK=18 are scored. We have corrected Algorithm 2 and its introductory paragraph to match Algorithm 1 and Appendix D. Both algorithms now specify uniform checkpoint sampling with replacement and uniform selection of KK terminal leaves without replacement. Only the selected leaves determine reward normalization, checkpoint values, credit assignment, and policy updates.

Crucially, KK denotes the number of selected leaves per tree, rather than the number supporting every checkpoint. Using the notation of Eq. (4),

nv=∣Desc⁡K(v)∣,V(v)=1nv∑u∈Desc⁡K(v)Aleaf(u),nv>0,n_v=|\operatorname{Desc}_K(v)|,\qquad V(v)=\frac{1}{n_v}\sum_{u\in\operatorname{Desc}_K(v)}A_{\mathrm{leaf}}(u),\qquad n_v > 0,

where Desc⁡K(v)\operatorname{Desc}_K(v) contains the selected leaves descending from checkpoint vv, and AleafA_{\mathrm{leaf}} is the normalized advantage defined in Eq. (3). Uneven branching produces different support sizes across checkpoints, and shared ancestry introduces dependencies among descendants. Consequently, neither K=6K=6 nor advantage normalization alone guarantees reliable supervision at every checkpoint.

We report two complementary analyses below. Both use auxiliary rewards: Setting A uses a banded interface-contact reward, and Setting B uses the released lDDT-to-native reward. These labels apply only to the additional analyses. Neither setting evaluates the full composite objective in Eq. (5), which combines affinity, interface confidence, and structural regularization.

First, we examine training sensitivity to the leaf budget and branching configuration under Setting A. Table 2a reports frozen-policy evaluation rewards on 120 complexes after 400 updates, as mean ± standard error across seeds 45–47. The base-policy reward is 0.523. All configurations use T=50T=50, and differences are relative to the baseline, with 95% tt-intervals over seeds.

Table 2a. Training comparisons under auxiliary Setting A.

Configuration(M,N,L)(M,N,L)KKLeaves/treeScored/inputRewardDifference [95% interval]
Baseline(3, 2, 3)67180.563 ± 0.013—
Fewer scored leaves(3, 2, 3)2760.528 ± 0.019−0.035 [−0.058, −0.012]
Intermediate leaf budget(3, 2, 3)47120.534 ± 0.015−0.029 [−0.049, −0.009]
All leaves scored(3, 2, 3)77210.559 ± 0.013−0.004 [−0.006, −0.002]
More expansions per round, fewer rounds(3, 3, 2)67180.546 ± 0.010−0.017 [−0.033, −0.001]
Fewer expansions per round, more rounds(3, 1, 6)67180.539 ± 0.009−0.024 [−0.040, −0.008]

These comparisons show sensitivity to both the scoring budget and tree structure, but they do not directly measure checkpoint-value estimation error.

Second, we examine descendant-sampling uncertainty under the frozen base policy, using N=2N=2, L=3L=3, and T=50T=50 and varying K∈{2,4,6,7}K\in\{2,4,6,7\}. Each setting uses uniform leaf selection without replacement. This diagnostic includes all three expansion rounds and preserves dependencies arising from shared ancestry. Table 2a reports training comparisons with M=3M=3, whereas Table 2b summarizes reward-space estimates from two frozen root chains per complex. The latter does not directly evaluate the across-tree normalized checkpoint estimator used in the M=3M=3 training configuration.

The measured quantity is the reward-space estimator

=\frac{1}{n_v} \sum_{u\in\operatorname{Desc}_K(v)}r(u), \qquad n_v>0,$$ where $r(u)$ is the auxiliary reward. This differs from $V(v)$ in Eq. (4), which averages normalized advantages. For each of 120 complexes, we use two frozen root chains and draw approximately 20 continuations from each chain's three non-terminal checkpoint states. The diagnostic evaluates 200 trees per complex resampled with the training code's expansion rule. For each evaluated tree and checkpoint $v$, $V_{\mathrm{ref}}^{(R)}(v)$ is the unweighted mean reward of continuations from $v$'s state that the tree did not use. The estimate $\widehat V_K^{(R)}(v)$ is the unweighted mean reward of the selected leaves at or below $v$, and $\sigma_R(v)$ denotes the conditional reward standard deviation at that checkpoint. Equal weighting of included descendants does not make shared-ancestry samples independent.
AuthorsOpenReview ↗

Title: Author Response To Reviewer (3/5)

The approximately 20 continuations describe the initial pool, rather than the number remaining in every reference after excluding continuations used by the evaluated tree. We use this finite-sample reference as a comparator, not as a known population value. Scoring all seven candidate leaves provides the all-leaf comparison, but does not replace the additional continuation reference.

Across repeated estimates, we measure

Bias⁡K(R)(v)=E[V^K(R)(v)]−Vref(R)(v).\operatorname{Bias}_K^{(R)}(v) = \mathbb{E}[\widehat{V}_K^{(R)}(v)] - V_{\mathrm{ref}}^{(R)}(v).

Var⁡K(R)(v)=E![(V^K(R)(v)−E[V^K(R)(v)])2].\operatorname{Var}_K^{(R)}(v) = \mathbb{E}\\!\left[\left(\widehat{V}_K^{(R)}(v) - \mathbb{E}[\widehat{V}_K^{(R)}(v)]\right)^2\right].

Table 2b reports standardized discrepancies from the finite continuation reference. If that reference is fixed across repeated estimates, the squared error standard deviation equals the estimator variance; if the reference is recomputed, the error variance also depends on the reference variance and its covariance with the estimator. The table does not separately quantify reference uncertainty, so its error SD is not interpreted as an isolated estimate of the variance of V^K(R)(v)\widehat V_K^{(R)}(v). Statistics are conditional on nv>0n_v>0 and σR(v)>0\sigma_R(v)>0. The variance-equivalent effective sample count is 1/s21/s^2, where ss is the standardized error SD. It summarizes precision relative to independent continuations and is not a literal count of independent leaves.

Rows pool the three checkpoints with equal node weights and 2,000 bootstrap resamples over complexes. Value-error statistics are restricted to nodes with at least one selected descendant (nv>0n_v>0) and nonzero conditional reward variance (σR(v)>0\sigma_R(v)>0). The original chain contributes a candidate leaf, but uniform selection of K<7K<7 leaves without replacement does not guarantee that this leaf is selected; K=7K=7 selects all candidate leaves. Support therefore depends on both the generated tree and the selected leaves. The table is conditional on supported, nonzero-variance nodes; it does not quantify the frequency of unsupported or excluded nodes. Reference credit is rur_u minus the mean of unused continuations from the leaf's branch state. It is compared with A(u)−V^K(R)(v)A(u)-\widehat V_K^{(R)}(v) using sign agreement over nonzero pairs and Pearson correlation over all pairs.

Table 2b. Reward-space estimation and segment-credit diagnostics under the frozen base policy. Bias and error SD are standardized by σR(v)\sigma_R(v). Brackets indicate 95% bootstrap intervals over complexes.

SettingKKMean descendants (effective samples)Standardized biasStandardized error SDCredit sign agreementCredit Pearson correlation
B21.7 (1.3)−0.04 [−0.11, +0.04]0.87 [0.81, 0.93]0.740.56
B42.7 (2.1)−0.03 [−0.08, +0.03]0.69 [0.65, 0.74]0.790.73
B64.1 (4.3)−0.02 [−0.06, +0.02]0.48 [0.45, 0.51]0.840.83
B74.7 (4.3)−0.02 [−0.07, +0.03]0.48 [0.45, 0.51]0.840.83
A21.7 (1.4)+0.02 [−0.06, +0.10]0.86 [0.77, 0.95]0.730.49
A42.7 (2.1)+0.01 [−0.05, +0.07]0.69 [0.62, 0.76]0.790.66
A64.1 (3.8)+0.00 [−0.04, +0.05]0.51 [0.45, 0.57]0.840.75
A74.7 (2.5)+0.01 [−0.05, +0.06]0.63 [0.56, 0.70]0.810.70

Increasing KK from 2 to 6 reduces the reported estimation error and improves credit agreement under both auxiliary rewards. At K=6K=6, the standardized bias is −0.02 in Setting B and approximately zero in Setting A, with both confidence intervals including zero. The standardized error SD is 0.48 and 0.51, respectively; these describe discrepancies from the finite continuation reference. Credit sign agreement is 0.84 in both settings. Scoring all seven leaves yields unchanged metrics at the displayed precision in Setting B and higher error SD and lower credit agreement in Setting A; it therefore does not show a consistent improvement in these diagnostics.

These results provide evidence about reward-space estimation under the two auxiliary rewards. They do not directly establish the bias and variance of the normalized V(v)V(v) used for training under the full composite objective. Addressing that part of the reviewer’s request requires an additional evaluation of Eq. (4) using both the production normalization and fixed reference normalization. The latter would separate descendant-sampling error from variability introduced by estimating normalization statistics from the selected leaves.


W3: This work emphasizes that computational efficiency is a central motivation and provides Appendix D to show the better use of the denoising budget. Please provide compute-matched metric-versus-NFE curves at three or more budget points for chain GRPO and TSG-Diff.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.