- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Symphony, an RL method that replaces the importance sampling ratio in standard PPO and TRPO with (default ). The motivation stems from heavy-tailed ratios that occur in high-dimensional control tasks due to the multiplicative combination of probability ratios across dimensions. This method dampens large ratios by reweighting per-sample gradients with , while leaving the gradient unchanged when . The paper provides a binomial-residual approximation bound and a second-moment bound. Experiments on MJX and JAXAtari evaluate clipped and KL-penalized variants. Symphony outperforms standard PPO on Humanoid and Walker2D (high-dimensional tasks), while maintaining comparable performance on low-dimensional MJX tasks and Atari (discrete actions, where no product of ratios arises).
Critical strengths
This is a simple, well-motivated method with clear empirical patterns. The results support the motivation and algorithm design, with gains appearing only where per-dimension ratio compounding is expected to matter (high-dimensional continuous control) and disappearing in low-dimensional tasks.
The empirical results are carefully reported across multiple seeds.
Critical weaknesses
The empirical results only support the authors' claims under one training configuration, which does not include learning-rate decay or other commonly used PPO techniques. As a result, PPO peaks near 7,500 on Humanoid and then degrades to 3,750. With an appropriate configuration, such as an SPO-style pipeline, the ranking of standard PPO and Symphony reverses.
The paper lacks sufficient baselines and discussion of related work. As far as I know, RL methods used in LLM training have already attempted to address unbounded importance sampling ratios, such as GSPO[1].
The ablation results do not adequately explain the claims. The order ablation shows that , where and importance weighting is essentially removed, performs as well as .
Questions for authors
Do the gains hold across MJX tasks under a standard PPO pipeline (reward scaling, learning-rate decay)?
If is applied only to samples with and , are the gains preserved?
Could the authors discuss the differences between Symphony and GSPO[1] ?
[1] Zheng, Chujie, et al. "Group Sequence Policy Optimization." arXiv preprint arXiv:2507.18071 (2025).
4 comments
I thank the authors for the detailed response. The pipeline ablation and the GSPO discussion address my concerns, and I am raising my rating. I still suggest applying only to samples with and : Section 3.4 shows that all active high-ratio transitions have , so this one-line ablation would show whether the gain comes from attenuating these samples or from the curvature for . The Dual-Clip comparison does not settle this, since Dual-Clip only activates when the ratio exceeds 3.
We thank Reviewer VskW for revising their score upward and are thrilled with their positive assessment. To address the remaining concern, we ran the suggested targeted ablation on Walker2d on 3 configurations: the SPO-selected setting, the PPO-selected setting, and the Dual-Clip PPO-selected setting. A detailed discussion is provided in Appendix F (page 63), written in full. To save a click, please refer to the table below:
| Full Symphony | Tail-Only Symphony | Difference (Tail − Full) | |
|---|---|---|---|
| SPO-selected | 3866.97 ± 1307.96 | 2367.09 ± 371.67 | −1499.88 |
| PPO-selected | 3236.74 ± 1031.44 | 3073.11 ± 1176.57 | −163.63 |
| Dual-Clip-selected | 4173.45 ± 158.58 | 2197.01 ± 812.30 | −1976.45 |
We find that attenuating only the active high-ratio, negative-advantage region may not be sufficient to reproduce Symphony's behavior across the tested configurations.
Closing note: We are once again grateful to Reviewer VskW for increasing their score. We address the remaining concern with the suggested ablation. We would be grateful if the reviewer would consider a further upward revision of their score.
Thanks for your response. I will maintain my score.
We thank Reviewer VskW for their evaluation of our manuscript and for recognizing the gains in high-dimensional tasks. We address their concerns as follows.
Standard PPO pipeline & ranking reversal: Our SPO-style pipeline (Section 3.3, Table 16) already includes CleanRL-style running reward scaling, linear LR decay, and value clipping, and replacing its non-standard 7-layer actor with a standard 3-layer MLP (2x64 Tanh, Table 3) restores Symphony's lead (2663.3±1283.0 vs. 1619.1±374.9 for PPO and 902.0±47.1 for SPO).
Applying fn only to A<0andr>1+ϵ : We have not ablated applying fn solely to A<0 and r>1+ϵ, because Symphony (Clipped) (Eq. 5) deliberately curves the active r∈[1−ϵ,1+ϵ] region for A>0 as well, and it outperforms asymmetric A<0 truncation (Dual-Clip PPO) in Section 3.2 (6949.63 vs. 5924.86 on Humanoid; 2951.55 vs. 2738.74 on Walker2d).
On a related note, Section 3.4 (page 10, "High-ratio tail suppression") directly analyzes these active r>1+ϵ transitions—showing that 100% of them occur with A<0 and that Symphony consistently suppresses their tail leverage (Figure 5d).
Symphony vs. GSPO: GSPO targets sequence-level LLM RL by taking a sequence-length geometric mean (πθ(y∣x)/πold(y∣x))1/∣y∣ (scaling local gradients by 1/∣y∣) with standard linear clipping, Symphony operates per MDP transition with fixed order n and rescales fn(r)=nr1/n−(n−1) so fn′(1)=1 preserves exact local policy-gradient magnitude while introducing intrinsic negative curvature (fn′′(1)=−(n−1)/n<0) inside the active region.
Order ablation at n=64 : As shown in Table 1 (page 6), taking n→∞ yields fn(r)=n(r1/n−1)+1→1+logr, which removes ratio reweighting (r1/n→1) while retaining the unweighted policy-gradient signal ∇θlogπθ(a∣s) at O(1) variance, and on Ant (Table 2 Part II, Figure 7), n=64 drops return (2174.04 vs. 2440.52 at n=4), which confirms the bias–stability tradeoff of excessive damping.
Closing note: We once again thank the reviewer for their evaluation, and we are confident that we have addressed their concerns. We would be grateful if the reviewer could revisit their recommendation.