acceptodds
Under review as a conference paper at ICLR 2027

Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning

Abstract

Reinforcement learning with verifiable rewards has become a scalable post-training paradigm for large language models. However, training agents over long-horizon interactions presents two challenges: using only terminal rewards makes turn-level credit assignment sparse and high-variance, while repeated updates on a fixed rollout batch introduce policy staleness as the current policy drifts from the behavior policy. Previous work has used learned critics, process reward models, and on-policy distillation to provide denser credit, but these methods need a value model, process labels, or a stronger external teacher, extra components that increase training complexity. Importance sampling and clipping restrict policy drift, but they do not specify a finite target that stops sequential updates from overshooting. In this work, we introduce TreeOPSD, a critic-free, tree-aware posterior self-distillation framework for long-horizon agentic reinforcement learning. By treating each interaction turn as a distinct tree node, TreeOPSD branches alternative continuations from visited turn-level prefixes and recursively backs up their terminal rewards into implicit turn-level credit. It then converts each credit into a closed-form, bounded likelihood-ratio teacher derived from outcome rewards relative to the frozen behavior policy. Matching the student policy to this teacher mitigates policy drift under sequential mini-batch optimization with a finite target that stops sequential updates from overshooting. This matching recovers the policy gradient direction at the behavior policy. Empirically, TreeOPSD improves agentic capabilities across several competitive benchmarks and model scales, especially for long-horizon tasks, e.g., raising the average accuracy of Qwen3-30B-A3B-Thinking-2507 by 4.97 points over GRPO on the Deep Research setting.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning, LLM agents, credit assignment, on-policy distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hGNBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper propose TreeOPSD, a method that constructs a delicate student and teacher distribution to do self-distillation in tree sampling. TreeOPSD first devide the policy optimization into target construction and distribution matching. For the student distribution, TreeOPSD constructs a Bernoulli distrubtion of posterior on the sampling probability between πθ\pi_{\theta} and πold\pi_{old}. For the target teacher distribution, TreeOPSD construct a MPO style objective and derive the teacher distribution, which has good properties shown in Thm1 and Thm2.

Critical strengths
  1. TreeOPSD's mathematical derivation is delicate and complete.
  2. TreeOPSD achieves significant improvement in deep research related benchmark.
  3. The sampling budget of TreeOPSD and other baselines is strictly controlled to ensure fair comparison.
Critical weaknesses
  1. In other works, OPSD's teacher distribution is always condition on some privileged information, but TreeOPSD's teacher distribution is constructed based on KL regularized policy update. The naming of "TreeOPSD" might be misleading.
Questions for authors
  1. The derived gradient weight is like a "soft clip" based on the credit VV. However, some policy optimization methods, like the CSIPO from "MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention", abandon the clip on policy update (but preserve the clip on importance sampling weight) and achieve better performance compared with optimization method with clip. When will the clip be effective and when will it not?
1 comment
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for the careful examination and provide our response below.


W1: The naming of TreeOPSD

Thank you for pointing this out. We find that recent self-distillation methods that condition the same model on privileged context or feedback go by several different names and acronyms (such as OPSD [1-2], SDPO [3], and SDFT [4]) without a unified naming convention yet, and the overlap with our acronym is a coincidence. Specifically, while our training also uses on-policy rollouts, "OP" in TreeOPSD (Tree-aware Outcome-Posterior Self-Distillation) stands for Outcome-Posterior to highlight our core mechanism: relying solely on verifiable outcome rewards backed up on the rollout tree to construct an explicit posterior target, rather than conditioning on privileged text hints. Meanwhile, "SD" stands for Self-Distillation, as the teacher distribution is derived entirely from the policy model's own rollouts rather than an external teacher. To avoid any ambiguity from this acronym coincidence, we have spelled out the full name of TreeOPSD at its first mention in the revised manuscript and polished the related exposition.

Q1: Mechanistic trade-offs of clipping (PPO vs. CISPO) and how TreeOPSD bridges them

Thanks for raising CISPO [5]. We agree with your reading: TreeOPSD acts as a "soft clip", but around a credit-dependent target ratio ri⋆(Vi)r_i^\star(V_i) rather than a fixed range [1−ϵ,1+ϵ][1-\epsilon, 1+\epsilon]. In PPO/GRPO, clipping the policy update provides a hard stop that prevents the policy from drifting too far across multiple updates on the same rollout batch. However, as MiniMax-M1 [5] shows, zeroing out gradients outside [1−ϵ,1+ϵ][1-\epsilon, 1+\epsilon] hurts when the tokens that reach the boundary first are the most informative ones (e.g., rare reasoning tokens).

CISPO makes the opposite trade-off by removing the clip on the policy update and clipping only the importance sampling weight. This bounds the weight magnitude while keeping every token's gradient active across all updates. However, it also removes the stopping point: as πθ\pi_\theta moves in the direction of ViV_i, the weight never shrinks. For Vi>0V_i > 0, it grows up to (1+ϵhighIS)Vi(1+\epsilon^{\text{IS}}_{\text{high}})V_i and keeps pushing indefinitely, which can lead to over-optimization across repeated updates.

TreeOPSD combines the strengths of both methods: for each turn, it sets a target ratio ri⋆(Vi)r_i^\star(V_i) and scales the update by the remaining gap between ri⋆(Vi)r_i^\star(V_i) and the turn's geometric-mean ratio rθ,ir_{\theta,i}. Like CISPO, because this bounded weight is computed at the turn level without hard token clipping, individual informative tokens are never zeroed out early. Like PPO, it retains an explicit stopping point as the weight smoothly decays to 00 at ri⋆(Vi)r_i^\star(V_i) (and reverses sign upon overshoot), making this "soft clip" most effective across multi-step updates (E>1E>1) on long-horizon tasks where unstopped pushing would over-optimize noisy or shared-prefix credits. Conversely, it offers little extra benefit under single-step on-policy updates (E=1E=1, where rθ,i=1r_{\theta,i}=1 and TreeOPSD reduces to standard policy gradient by Theorem 2), and can be overly conservative if the target radius ρ\rho is set too small (as shown at ρ=0.10\rho=0.10 in Table 2b). We have added this comparison in Appendix C.4 (Figure 6).

References

[1] Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. ICML 2026.

[2] Rethinking On-Policy Self-Distillation for Thinking Models. arXiv 2026.

[3] Reinforcement Learning via Self-Distillation. ICML 2026.

[4] Self-Distillation Enables Continual Learning. ICML 2026.

[5] MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv 2025.


Thank you again for the supportive review. We would be glad to answer any further questions, and we hope the clarifications strengthen your confidence in our work.

Reviewer_54B6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TreeOPSD, addressing two challenges in RL training: sparse credit assignment and policy staleness. TreeOPSD constructs shared-prefix rollout trees. Terminal rewards are recursively backed up to obtain turn-level credits. The method converts the credit into bounded Bernouli posterior targets and optimize the student policy using a forward-KL distillation objective.

The authors theoretically show that the proposed objective recovers the tree policy-gradient direction at the behavior policy. Experiments on multi-hop QA and deep research benchmarks demonstrate improvements over GRPO, GSPO and tree-based policy-gradient methods.

Critical strengths
  1. The work provides a theoretically consistent connection between tree credit assignment and posterior distillation.

  2. The proposed posterior distillation objective provides an interesting alternative to directly optimizing fixed advantages.

Critical weaknesses
  1. The proposed framework relies on branching alternative contributions from intermediate prefixes to estimate turn-level credits. Although the initialize-then-expand sampling strategy avoids the exponential complexity of fully expanding a trajectory tree, there remains an important trade-off between branching budget, credit-assignment quality, and trajectory horizon.

Specifically, the method uses only a small number of branching anchors per initial trajectory. As the interaction horizon increases, this sparse branching strategy may provide insufficient information for accurately estimating the contributions of intermediate decisions, particularly those early in the trajectory whose outcomes depend on many subsequent actions.

  1. The performance of the experimental results are relatively low. For example, Qwen3-30b trained with TreeOPSD achieves 48.16% on HotPotQA. This result is substantially below publicly reported results: 55.3% from Vicuna-13b trained with AGILE [1], 50.2% from Qwen2.5-14b trained with Tree-GRPO [2], 55.0% from Qwen-2.5-7b trained with R1-Searcher [3].

  2. (minor) The full expansion of the method acronym (TreeOPSD) is missing.

[1] AGILE: A Novel Reinforcement Learning Framework of LLM Agents, NeurIPS 2024

[2] Tree Search for LLM Agent Reinforcement Learning, ICLR 2026

[3] R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning

Questions for authors

see weaknesses.

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part I)

We thank the reviewer for the constructive feedback and believe our responses below can address the reviewer's concerns.


W1: Trade-off between branching budget, credit-assignment quality, and trajectory horizon

We agree that the trade-off between sampling budget and credit-estimation granularity is real, and it forms a Pareto frontier shared by all tree rollout RL methods (including Tree-GRPO [2], ARPO [6], and AT$^2$PO [7]) rather than a limitation specific to TreeOPSD. Because tree acquisition is orthogonal to our core contribution in policy optimization (Section 2.2 and Appendix C.1), we adopt the established uniform branching scheme to match flat GRPO's rollout budget, and the consistent gains over GRPO in both our work and prior tree RL methods confirm that sparse branching already provides effective intermediate supervision. Regarding how TreeOPSD behaves under this trade-off on long horizons, we highlight three points:

  1. Recursive backup concentrates outcome evidence on early turns. Under Eq. 5, every branch anchor sampled downstream of a turn adds a terminal leaf to that turn's subtree backup, and under uniform anchor selection each leaf provides an unbiased continuation rollout from that prefix. The first turn of each initial trajectory therefore aggregates all N+1N+1 leaves of its subtree whenever Tm>1T_m>1, whereas only the turns after the last anchor rely on a single leaf (as every turn does under GRPO). The early decisions highlighted by the reviewer are actually where the tree accumulates the most outcome evidence.
  2. TreeOPSD is built to tolerate noisy credit. When a backed-up credit ViV_i is noisy due to a small branching budget or a long horizon, a fixed-credit policy gradient repeatedly applies the same noisy coefficient across all sequential mini-batch updates. TreeOPSD limits this sensitivity in two ways: the target ratio is strictly bounded at every depth (Lemma B.1 and Eq. 12), and the gradient weight decays to 00 once rθ,ir_{\theta,i} reaches ri⋆r_i^\star.
  3. Adaptive rollout allocation can push this Pareto frontier in a plug-and-play manner. Because Theorems 1 and 2 condition on any acquired rollout tree (Appendices B.1 and C.1), our uniform sampler can be directly replaced by predictive rollout allocators such as TRACE [5], which concentrate the branching budget on high-uncertainty intermediate prefixes to increase effective credit resolution without extra rollout cost. To verify this, we plug TRACE into both TreePO and TreeOPSD on Qwen3-4B-Thinking-2507 following the Multi-Hop QA setting in Table 1. As shown in Supplementary Table 3, incorporating TRACE consistently improves both methods under the same rollout budget, and TreeOPSD + TRACE achieves the best performance across all benchmarks.

We have added these new results and discussions in Appendices C.5 and D.7 of the revised manuscript.

Supplementary Table 3: Performance comparison of composing TreePO and TreeOPSD with TRACE on Multi-Hop QA under the settings in Table 1 (Qwen3-4B-Thinking-2507).

MethodHotpot2WikiMuSiQueBamboogleAvg
TreePO (Uniform)47.2549.9022.0554.4043.40
TreeOPSD (Uniform)48.1652.0022.7653.6044.13
TreePO + TRACE47.9450.6822.8055.2044.16
TreeOPSD + TRACE48.9752.8423.4656.0045.32

W2: Absolute performance on HotpotQA compared with [1, 2, 3]

We thank the reviewer for this comment and for pointing out these strong results [1–3]. We also kindly clarify that in Table 1, 48.16% on HotpotQA is from Qwen3-4B-Thinking-2507, while Qwen3-30B-A3B-Thinking-2507, an MoE model with ~3B active parameters, reaches 48.89%.

Our primary goal in this work is not to engineer state-of-the-art leaderboard scores on specific benchmarks, but to cleanly isolate and verify the algorithmic effectiveness of TreeOPSD in addressing sparse credit assignment and policy staleness. We therefore intentionally use a minimal R1-Zero-style RLVR pipeline without task-specific engineering for a fair comparison among GRPO, GSPO, TreePO, and TreeOPSD, which makes our training setup fundamentally different from [1–3]:

  1. AGILE [1] (55.3% with Vicuna-13B, their no-advice ablation): TreeOPSD follows a zero-cold-start RLVR setup directly from the base model. In contrast, AGILE first collects 10,240 GPT-4 and GPT-3.5 trajectories for SFT warm-start before RL (Appendix B.3 of [1]) and equips the agent with an external expert advice-seeking module.
AuthorsOpenReview ↗

Rebuttal by Authors (Part II)

  1. R1-Searcher [3] (55.0% with Qwen2.5-7B): R1-Searcher performs multi-rollout difficulty filtering on training queries, trains jointly on HotpotQA and 2WikiMultiHopQA in a two-stage RL curriculum with BGE-large retrieval (up to 8 retrievals), and reports its HotpotQA numbers on a 500-question subset. Following Search-R1's environment [4], we train on unfiltered HotpotQA alone in a single stage (at most 4 retrieval turns with E5 retrieval) and evaluate on the complete 7,405-question HotpotQA validation split.
  2. Tree-GRPO [2] (50.2% with Qwen2.5-14B): Tree-GRPO shares a similar pipeline to ours, but uses a 14B dense model (Qwen2.5-14B, over 4$\times$ our active parameters) trained for 180 steps with batch size 512 (92,160 prompts). Our models are trained for at most 80 steps with batch size 256 (at most 20,480 prompts, roughly 4.5×4.5\times fewer) and select checkpoints by the 4-benchmark average rather than HotpotQA alone. Even so, on the 4-benchmark average, our 3B-active Qwen3-30B-A3B (45.93% Avg, with 50.88% on 2Wiki and 59.20% on Bamboogle) outperforms Tree-GRPO's 14B dense model (45.25% Avg: 50.2 Hotpot, 50.5 2Wiki, 25.9 MuSiQue, 54.4 Bamboogle).

Notably, even among the works cited by the reviewer, Tree-GRPO's 50.2% on a 14B dense model is substantially lower than R1-Searcher's 55.0% on a 7B model. This gap shows that absolute scores across different training and retrieval pipelines are not directly comparable, and that controlled comparisons under matched budgets and environments (Tables 1 and 3) provide the most reliable evaluation.

W3: Full expansion of the method acronym (TreeOPSD)

Thank you for pointing out this omission. TreeOPSD stands for Tree-aware Outcome-Posterior Self-Distillation, and we have added the full name of TreeOPSD at its first mention in the revised manuscript and polished the related exposition.

References

[1] AGILE: A Novel Reinforcement Learning Framework of LLM Agents. NeurIPS 2024.

[2] Tree Search for LLM Agent Reinforcement Learning. ICLR 2026.

[3] R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning. arXiv 2025.

[4] Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. COLM 2025.

[5] TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning. EMNLP 2026.

[6] Agentic Reinforced Policy Optimization. ICLR 2026.

[7] AT$^2$PO: Agentic Turn-based Policy Optimization via Tree Search. ACL 2026.


Finally, thanks again for your time and helpful comments. We hope all concerns are well addressed, and we would be grateful if you could reconsider the evaluation of our work.

Reviewer_6fVBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This work presents TreeOPSD, a critic free tree aware posterior self distillation method for long horizon LLM agent RLVR. It builds rollout trees by branching continuations from shared prefixes, recursively backs terminal rewards to get implicit turn level credit, then maps credit into closed form bounded Bernoulli teachers. Optimizing via forward KL matching yields finite update targets: gradients vanish at target and reverse sign upon overshoot, mitigating policy staleness across sequential mini batch updates. On multi hop QA and deep research agent benchmarks across multiple model sizes, TreeOPSD outperforms GRPO, GSPO and its TreePO ablation, with larger gains on long horizon settings.

Critical strengths
  1. The innovation lies in how to transform the turn-level credibility obtained from tree-based backups into a closed-form, finite objective, optimize it using posterior KL matching, dynamically correct for policy drift, and prevent overshoot during multiple rounds of iteration.
  2. Careful ablation with TreePO, which isolates the contribution of posterior KL matching from tree backup credit assignment.
Critical weaknesses
  1. The current version is confusing. The core effect of dynamic residual driven gradient adjustment could conceivably be realized via hand designed adaptive weighting. This raises a concern that the Bernoulli distribution, forward KL matching and commutativity focused theoretical apparatus may serve as excessive mathematical packaging, rather than being strictly necessary for the observed empirical gains. It would be helpful to disentangle whether performance improvements stem from the spring style residual mechanism itself or incidental regularization introduced by this specific formulation.
  2. Empirical scope is limited to multi hop QA and web research agents. Generalization to environments with few shared trajectory prefixes remains untested.
Questions for authors
  1. The proposed Bernoulli forward KL formulation also brings implicit gradient regularization effects. It would be valuable to add a minimal ablation implementing the spring like residual update without Bernoulli / KL machinery, together with training time diagnostics such as frequency of gradient sign reversal and measured policy drift statistics, to further validate the claimed mechanism.
  2. Empirical validations for composing TreeOPSD with alternative tree sampling strategies are absent.
  3. High barriers to implementation: It’s not simply a matter of replacing the loss function; it requires significant changes to the entire rollout tree pipeline.
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part I)

We thank the reviewer for the valuable feedback and believe the responses below sufficiently address all concerns.


W1 & Q1: Necessity of the Bernoulli Forward-KL formulation, backup commutativity, and spring ablation

Thank you for raising this point and ablation design. By the Fenchel–Young identity (Eq. 25), each turn's loss in TreeOPSD (Eq. 13) equals TreePO's fixed-credit surrogate plus a single proximal penalty on the turn's mean log-ratio sθ,is_{\theta,i}: τDKL(Ber⁡(qi)∥Ber⁡(σ(sθ,i)))=−Visθ,i+τlog⁡cosh⁡(sθ,i/2)+const,\tau D_{\mathrm{KL}}\bigl(\operatorname{Ber}(q_i)\Vert \operatorname{Ber}(\sigma(s_{\theta,i}))\bigr) = -V_i s_{\theta,i} + \tau\log\cosh(s_{\theta,i}/2) + \text{const}, which yields the per-token ascent weight Γi(sθ,i)=Vi−τ(σ(sθ,i)−12)=Vi−τ2tanh⁡(sθ,i/2)\Gamma_i(s_{\theta,i}) = V_i - \tau\bigl(\sigma(s_{\theta,i}) - \frac{1}{2}\bigr) = V_i - \frac{\tau}{2}\tanh(s_{\theta,i}/2). Thus, the "spring-style residual" and "implicit gradient regularization" are not two separate effects, but the same restoring term (τ/2)tanh⁡(sθ,i/2)(\tau/2)\tanh(s_{\theta,i}/2).

More importantly, the Bernoulli Forward-KL and affine-teacher formulation is not excessive mathematical packaging. Without Bernoulli or KL machinery, the most direct way to hand-design a spring-like residual update is to subtract a linear restoring force proportional to turn drift from the fixed credit: Γilin(sθ,i)=Vi−κsθ,i\Gamma_i^{\text{lin}}(s_{\theta,i}) = V_i - \kappa s_{\theta,i}, which vanishes at si⋆=Vi/κs_i^\star = V_i/\kappa and reverses sign past it. With initial stiffness κ=τ/4\kappa = \tau/4, this hand-crafted Linear Spring is the exact first-order Taylor expansion of TreeOPSD around the behavior policy (sθ,i=0s_{\theta,i}=0). While TreeOPSD reduces to this first-order approximation under small drift, Γilin\Gamma_i^{\text{lin}} grows without bound in ∣sθ,i∣|s_{\theta,i}| when a rare token or exploratory tool call experiences a sharp probability shift across mini-batches. This produces oversized restoring gradients even on uninformative turns (Vi≈0V_i \approx 0) and destabilizes off-policy updates. In contrast, mapping drift through the Bernoulli posterior σ(sθ,i)∈(0,1)\sigma(s_{\theta,i})\in(0,1) smoothly saturates the restoring force at τ2\frac{\tau}{2}, ensuring global gradient boundedness ∣Γi(sθ,i)∣≤(12+ρ)τ|\Gamma_i(s_{\theta,i})| \le (\frac{1}{2}+\rho)\tau for all rθ,i∈(0,∞)r_{\theta,i}\in(0,\infty) (Theorem 2(ii)).

Meanwhile, on a rollout tree, each internal turn ii is shared by multiple downstream branches. For training efficiency, rather than repeating turn ii across every branch, we update it once using a single parent teacher qiq_i built from its backed-up credit ViV_i (Eq. 5). For this single parent update to yield the same unbiased gradient as aggregating all descendant branches and recover the tree policy gradient of Eq. 6 at πθold\pi_{\theta_{\text{old}}}, teacher construction and divergence matching must commute with tree backup (Theorem 1). Pairing the affine teacher qi=12+Vi/τq_i = \frac{1}{2} + V_i/\tau with Forward KL guarantees this commutativity because qiq_i is affine in ViV_i and the KL gradient is linear in qiq_i. Without it, a nonlinear bounded teacher such as qilog=σ(Vi/τ′)q_i^{\text{log}} = \sigma(V_i/\tau') suffers from Jensen's-inequality bias whenever sibling outcomes diverge, distorting parent updates relative to Eqs. 5–6 and degrading BrowseComp-Plus accuracy throughout training (Section 4.3, Figure 5c–d and Appendix D.2, Figure 9c).

Following your suggestion, we compare TreePO, the first-order Linear Spring, and TreeOPSD on Qwen3-4B-Thinking-2507 under the Multi-Hop QA setting of Table 1. As shown in Supplementary Table 1, Linear Spring improves over TreePO on average (43.84% vs. 43.40%), confirming that the finite-target spring mechanism itself is the primary driver of the gain, while TreeOPSD further outperforms Linear Spring (44.13% vs. 43.84%), verifying the benefit of global sigmoid saturation. Training-time diagnostics across the E=4E=4 optimizer blocks further confirm the claimed dynamics: from Block 1 (on-policy, ∣sθ,i∣=0|s_{\theta,i}|=0) to Block 4, TreePO's fixed weight ViV_i applies no damping or sign reversal, allowing mean turn drift ∣sθ,i∣|s_{\theta,i}| to reach 0.148 with 14.7% of active turns pushed past ri⋆r_i^\star. In contrast, TreeOPSD damps 66.5% of active turns approaching ri⋆r_i^\star (0<Γi/Vi<10 < \Gamma_i/V_i < 1) and reverses the gradient sign on the 9.2% of turns that move past ri⋆r_i^\star (Γi/Vi<0\Gamma_i/V_i < 0), holding mean Block-4 drift down to 0.109 while capping restoring gradients on tail turns (∣sθ,i∣>2|s_{\theta,i}| > 2) where Linear Spring exceeds τ/2\tau/2. We have added these results and diagnostics in Appendix D.5.

AuthorsOpenReview ↗

Rebuttal by Authors (Part II)

Supplementary Table 1: Ablation study comparing TreeOPSD with its first-order Linear Spring approximation (without Bernoulli/KL) on Multi-Hop QA under the settings in Table 1 using Qwen3-4B-Thinking-2507.

MethodAscent Weight Γi\Gamma_iHotpot2WikiMuSiQueBamboogleAvg
TreePOViV_i47.2549.9022.0554.4043.40
Linear SpringVi−τ4sθ,iV_i - \frac{\tau}{4}s_{\theta,i}47.8251.4022.5553.6043.84
TreeOPSDVi−τ(σ(sθ,i)−12)V_i - \tau(\sigma(s_{\theta,i})-\frac{1}{2})48.1652.0022.7653.6044.13

W2: Generalization to environments with few shared prefixes

TreeOPSD does not rely on independent rollouts sharing prefixes by chance, as Stage-2 expansion explicitly branches from visited turn boundaries (Eq. 3). To test generalization beyond search-based QA to a domain where shared prefixes are sparse across long trajectories, we evaluate on Multi-Turn Function Calling (BFCL v4) [1] with Qwen3-4B-Thinking-2507 following the 80%/20% train/test split and avg@4 evaluation protocol of TRACE [2], training for 50 steps (128 prompts per step) under the same rollout budget (G=8G=8 for flat methods and M=4,N=2M=4, N=2 for tree methods). Unlike information-seeking QA, BFCL requires executing sequences of stateful API calls over up to 40 turns, so N=2N=2 branches per root trajectory yield sparse branching across the 40-turn horizon. As shown in Supplementary Table 2, TreeOPSD consistently outperforms GRPO, GSPO, and TreePO across all four subsets (+2.19 over TreePO and +3.92 over GRPO on Avg), as recursive backup propagates downstream branch outcomes to earlier turns while bounded targets stabilize updates on unbranched segments. We have added these results in Appendix D.6.

Supplementary Table 2: Performance comparison on multi-turn Function Calling (BFCL v4) with Qwen3-4B-Thinking-2507.

MethodBaseMiss-FuncMiss-ParamLongAvg
ReAct34.4420.1428.7921.7426.28
GRPO54.4428.4741.6734.2439.70
GSPO53.8929.1742.4234.2439.93
TreePO56.1130.5643.1835.8741.43
TreeOPSD58.3332.6445.4538.0443.62

Q2: Composing TreeOPSD with alternative tree samplers

Because Theorems 1 and 2 condition on any acquired rollout tree (Appendices B.1 and C.1), TreeOPSD is orthogonal to how branching anchors are chosen and can be paired directly with alternative tree samplers such as TRACE [2], which uses a lightweight predictor to allocate branches to high-uncertainty intermediate prefixes rather than sampling anchors uniformly. To verify this, we plug TRACE into both TreePO and TreeOPSD on Qwen3-4B-Thinking-2507 following the Multi-Hop QA setting in Table 1. As shown in Supplementary Table 3, incorporating TRACE consistently improves both methods under the same rollout budget, while the gain of TreeOPSD over TreePO is preserved (+0.73 under uniform branching and +1.16 under TRACE). We have added these results in Appendix D.7.

Supplementary Table 3: Performance comparison of composing TreePO and TreeOPSD with TRACE on Multi-Hop QA under the settings in Table 1 (Qwen3-4B-Thinking-2507).

MethodHotpot2WikiMuSiQueBamboogleAvg
TreePO (Uniform)47.2549.9022.0554.4043.40
TreeOPSD (Uniform)48.1652.0022.7653.6044.13
TreePO + TRACE47.9450.6822.8055.2044.16
TreeOPSD + TRACE48.9752.8423.4656.0045.32

Q3: Implementation simplicity and pipeline integration

TreeOPSD does not require custom modifications to the rollout pipeline itself: it uses the standard two-stage initialize-then-expand tree sampler (Section 2.2) shared across recent tree-based agentic RL methods (such as Tree-GRPO [3], ARPO [4], AT$^2$PO [5], and TRACE [2]), whose open-source rollout implementations can be directly reused. Once a rollout tree is collected, computing the backed-up turn credits ViV_i (Eq. 5) is a simple one-pass bottom-up weighted average. Beyond this, TreeOPSD is indeed a drop-in, 5-line loss replacement using standard binary cross-entropy with logits:

import torch.nn.functional as F

def treeopsd_loss(logp, logp_old, mask, V, tau):
    """One prompt tree; rows = unique actor turns (shared prefixes counted once).
    logp, logp_old: [n_turns, T] token log-probs under pi_theta / frozen pi_theta_old
    mask:           [n_turns, T] 1 on policy-generated tokens (observations masked)
    V:              [n_turns] backed-up turn credit (Eq. 5); tau = (K-1) / (rho * sqrt(K))
    """
    L_i = mask.sum(-1)
    s = ((logp - logp_old) * mask).sum(-1) / L_i       # Eq. 8: mean log-ratio = student logit
    q = 0.5 + V / tau                                  # Eq. 11: frozen soft label
    bce = F.binary_cross_entropy_with_logits(s, q, reduction="none")  # KL(Ber(q) || Ber(sigmoid(s))) + const
    return tau * (L_i * bce).sum() / L_i.sum()         # Eq. 13
AuthorsOpenReview ↗

Rebuttal by Authors (Part III)

Moreover, with the rapid development of modern coding agents, we believe adapting this self-contained module into existing RLVR pipelines requires minimal engineering effort.

References

[1] The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. ICML 2025.

[2] TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning. EMNLP 2026.

[3] Tree Search for LLM Agent Reinforcement Learning. ICLR 2026.

[4] Agentic Reinforced Policy Optimization. ICLR 2026.

[5] AT$^2$PO: Agentic Turn-based Policy Optimization via Tree Search. ACL 2026.


Finally, we express sincere gratitude for your valuable review and suggestions, which have helped improve the quality and presentation of our work. We hope our clarifications and new experiments address your concerns, and we would appreciate it if you could reconsider the evaluation of our paper.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global Response by Authors

We thank all the reviewers for taking the time to read our manuscript carefully and for providing constructive and insightful feedback. We are encouraged by the positive comments, such as:

  • Theoretically consistent and complete formulation connecting tree credit assignment with posterior self-distillation (Reviewers 54B6, hGNB).
  • Closed-form, finite turn-level targets that dynamically correct off-policy drift and prevent overshoot across sequential updates, offering an effective alternative to fixed-advantage optimization (Reviewers 6fVB, 54B6).
  • Careful experimental design and strictly controlled sampling budgets, with the TreePO ablation cleanly isolating the contribution of posterior KL matching from tree credit assignment (Reviewers 6fVB, hGNB).
  • Significant improvement on Deep Research benchmarks (Reviewer hGNB).

Meanwhile, we have worked carefully to address the reviewers' concerns and questions, and have also polished the overall presentation of the manuscript (with revisions directly addressing the reviewers' comments highlighted in blue in the revised PDF). Below we summarize our main clarifications, theoretical analyses, and additional experiments, with detailed point-by-point responses provided in the individual replies:

  • Theoretical analysis and ablation against a first-order Linear Spring without Bernoulli/KL machinery, along with training-time drift and sign-reversal diagnostics (Supplementary Table 1, Appendix D.5, Reviewer 6fVB).
  • Mechanistic comparison of off-policy clipping trade-offs across PPO/GRPO, CISPO (MiniMax-M1), and TreeOPSD's credit-calibrated target projection (Appendix C.4, Reviewer hGNB).
  • Generalization to stateful multi-turn Function Calling (BFCL v4) under sparse shared prefixes (Supplementary Table 2, Appendix D.6, Reviewer 6fVB).
  • Analysis of the branching-budget vs. horizon trade-off on early turns, and plug-and-play composition with adaptive rollout allocation (TRACE) (Supplementary Table 3, Appendices C.5 and D.7, Reviewers 6fVB, 54B6).
  • Detailed comparison of training and evaluation protocols against AGILE, R1-Searcher, and Tree-GRPO on HotpotQA (Reviewer 54B6).
  • Clarifications on the full expansion and naming of TreeOPSD (Reviewers 54B6, hGNB), as well as its 5-line drop-in loss implementation (Reviewer 6fVB).

We hope our responses adequately address the reviewers' concerns. We would be very happy to clarify any remaining questions and look forward to further discussion.

AuthorsOpenReview ↗

A gentle follow-up on our rebuttal and revised manuscript

Dear Reviewers,

Thank you again for your time and constructive feedback on our submission. We posted our point-by-point responses and uploaded the revised manuscript several days ago, and with the discussion period drawing to a close, we wanted to gently follow up in case you have a chance to take a look. Please let us know if our revisions and clarifications address your concerns, or if there are any remaining questions we can clarify.

Thank you very much for your time and consideration.

Best regards,

The Authors

AuthorsOpenReview ↗

Summary of the Rebuttal

Dear Chairs and Reviewers,

Thank you for your time and constructive feedback. As the discussion period comes to a close, to assist with the final assessment, we briefly summarize our core contribution and our main responses to the questions raised in the reviews.


Core contribution. We propose TreeOPSD, a critic-free framework for long-horizon agentic RL that reformulates policy optimization into target construction and distribution matching. By backing up sparse terminal rewards on shared-prefix rollout trees, TreeOPSD constructs a closed-form target teacher policy at the turn level and aligns the evolving student policy with this target via distribution matching. Without a learned critic, process labels, or an external teacher, this framework achieves fine-grained credit assignment while dynamically controlling off-policy updates across sequential mini-batches to prevent overshoot, consistently outperforming flat and tree-based RL baselines under matched budgets.


Below we summarize our main responses to the questions raised in the reviews:

Q1: Is the Bernoulli Forward-KL formulation necessary, or could a hand-crafted spring rule suffice? (Reviewer 6fVB)

A hand-crafted Linear Spring is in fact the exact first-order Taylor expansion of TreeOPSD around the behavior policy. Our new ablation and training diagnostics (Supplementary Table 1, Appendix D.5) show that this Linear Spring already outperforms fixed-credit TreePO, confirming the value of a finite-target restoring force, while TreeOPSD improves further because sigmoid saturation prevents oversized gradients under large off-policy drift and affine Forward KL keeps shared-turn updates unbiased under tree backup.

Q2: How does TreeOPSD perform when branching is sparse over long horizons, and does it work with other tree samplers? (Reviewers 6fVB, 54B6)

While rollout budget and credit granularity form a Pareto frontier shared by all tree-based RL methods, sparse branching is already sufficient to provide effective fine-grained credit assignment over trajectory-level baselines like GRPO. TreeOPSD explicitly branches from visited turns, where recursive backup concentrates outcome evidence on early decisions and bounded targets limit sensitivity to noisy credit (Appendix C.5). New experiments confirm consistent gains on 40-turn stateful function calling (BFCL v4) despite sparse shared prefixes (Supplementary Table 2, Appendix D.6), and show that plugging in an active rollout allocator further pushes this frontier under the same budget (Supplementary Table 3, Appendix D.7).

Q3: How does TreeOPSD compare with PPO/GRPO clipping and CISPO? (Reviewer hGNB)

PPO and GRPO hard-clip updates at a fixed interval, which prematurely drops rare informative tokens, whereas CISPO preserves all token gradients by clipping only detached importance weights, which removes the stopping point and risks over-optimization. TreeOPSD bridges the two: its turn-level weight keeps all tokens active while smoothly decaying to zero at a credit-dependent target across multi-step updates, and reduces to standard policy gradient under single-step updates (Appendix C.4).

Q4: Why are the absolute HotpotQA scores lower than those in AGILE, R1-Searcher, and Tree-GRPO? (Reviewer 54B6)

Our training and evaluation settings are largely different from those in the cited works, as we intentionally adopt a minimal zero-cold-start RLVR setup on the full validation split without task-specific engineering to cleanly isolate algorithmic effects. In contrast, the cited methods rely on GPT-4/3.5 SFT warm-start trajectories (AGILE), query filtering with multi-dataset curricula and smaller evaluation subsets (R1-Searcher), or a larger 14B dense backbone trained on far more prompts (Tree-GRPO, which our 3B-active MoE model still surpasses on the four-benchmark average).

Q5: Method naming, presentation clarity, and implementation effort. (Reviewers 6fVB, 54B6, hGNB)

We spelled out TreeOPSD (Tree-aware Outcome-Posterior Self-Distillation) in the revised manuscript to clarify that Outcome-Posterior relies solely on verifier outcomes rather than privileged text hints, streamlined the proofs and exposition in Section 3 and Appendix B, and showed that TreeOPSD reuses standard open-source tree samplers as a five-line binary cross-entropy loss replacement.

Thank you again for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.