- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper propose TreeOPSD, a method that constructs a delicate student and teacher distribution to do self-distillation in tree sampling. TreeOPSD first devide the policy optimization into target construction and distribution matching. For the student distribution, TreeOPSD constructs a Bernoulli distrubtion of posterior on the sampling probability between and . For the target teacher distribution, TreeOPSD construct a MPO style objective and derive the teacher distribution, which has good properties shown in Thm1 and Thm2.
Critical strengths
- TreeOPSD's mathematical derivation is delicate and complete.
- TreeOPSD achieves significant improvement in deep research related benchmark.
- The sampling budget of TreeOPSD and other baselines is strictly controlled to ensure fair comparison.
Critical weaknesses
- In other works, OPSD's teacher distribution is always condition on some privileged information, but TreeOPSD's teacher distribution is constructed based on KL regularized policy update. The naming of "TreeOPSD" might be misleading.
Questions for authors
- The derived gradient weight is like a "soft clip" based on the credit . However, some policy optimization methods, like the CSIPO from "MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention", abandon the clip on policy update (but preserve the clip on importance sampling weight) and achieve better performance compared with optimization method with clip. When will the clip be effective and when will it not?
Rebuttal by Authors
We thank the reviewer for the careful examination and provide our response below.
Thank you for pointing this out. We find that recent self-distillation methods that condition the same model on privileged context or feedback go by several different names and acronyms (such as OPSD [1-2], SDPO [3], and SDFT [4]) without a unified naming convention yet, and the overlap with our acronym is a coincidence. Specifically, while our training also uses on-policy rollouts, "OP" in TreeOPSD (Tree-aware Outcome-Posterior Self-Distillation) stands for Outcome-Posterior to highlight our core mechanism: relying solely on verifiable outcome rewards backed up on the rollout tree to construct an explicit posterior target, rather than conditioning on privileged text hints. Meanwhile, "SD" stands for Self-Distillation, as the teacher distribution is derived entirely from the policy model's own rollouts rather than an external teacher. To avoid any ambiguity from this acronym coincidence, we have spelled out the full name of TreeOPSD at its first mention in the revised manuscript and polished the related exposition.
Thanks for raising CISPO [5]. We agree with your reading: TreeOPSD acts as a "soft clip", but around a credit-dependent target ratio ri⋆(Vi) rather than a fixed range [1−ϵ,1+ϵ]. In PPO/GRPO, clipping the policy update provides a hard stop that prevents the policy from drifting too far across multiple updates on the same rollout batch. However, as MiniMax-M1 [5] shows, zeroing out gradients outside [1−ϵ,1+ϵ] hurts when the tokens that reach the boundary first are the most informative ones (e.g., rare reasoning tokens).
CISPO makes the opposite trade-off by removing the clip on the policy update and clipping only the importance sampling weight. This bounds the weight magnitude while keeping every token's gradient active across all updates. However, it also removes the stopping point: as πθ moves in the direction of Vi, the weight never shrinks. For Vi>0, it grows up to (1+ϵhighIS)Vi and keeps pushing indefinitely, which can lead to over-optimization across repeated updates.
TreeOPSD combines the strengths of both methods: for each turn, it sets a target ratio ri⋆(Vi) and scales the update by the remaining gap between ri⋆(Vi) and the turn's geometric-mean ratio rθ,i. Like CISPO, because this bounded weight is computed at the turn level without hard token clipping, individual informative tokens are never zeroed out early. Like PPO, it retains an explicit stopping point as the weight smoothly decays to 0 at ri⋆(Vi) (and reverses sign upon overshoot), making this "soft clip" most effective across multi-step updates (E>1) on long-horizon tasks where unstopped pushing would over-optimize noisy or shared-prefix credits. Conversely, it offers little extra benefit under single-step on-policy updates (E=1, where rθ,i=1 and TreeOPSD reduces to standard policy gradient by Theorem 2), and can be overly conservative if the target radius ρ is set too small (as shown at ρ=0.10 in Table 2b). We have added this comparison in Appendix C.4 (Figure 6).
References
[1] Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. ICML 2026.
[2] Rethinking On-Policy Self-Distillation for Thinking Models. arXiv 2026.
[3] Reinforcement Learning via Self-Distillation. ICML 2026.
[4] Self-Distillation Enables Continual Learning. ICML 2026.
[5] MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv 2025.
Thank you again for the supportive review. We would be glad to answer any further questions, and we hope the clarifications strengthen your confidence in our work.