acceptodds
Under review as a conference paper at ICLR 2027

Trust Regions Sell, But Who's Buying? Generalized Power-Law Surrogates for Clipped and KL-Constrained Reinforcement Learning

29 views

Abstract

Standard proximal (e.g., PPO) and trust-region (e.g., TRPO) methods stabilize reinforcement learning but optimize linear importance-ratio surrogates that do not account for the curvature of the surrogate under large likelihood-ratio displacements. We introduce Symphony, a generalized family of RL objectives derived from the \(n\)-th root likelihood ratio, \((\pi'/\pi)^1/n\). By isolating higher-order geometric contributions via a binomial expansion, Symphony unifies computationally efficient first-order clipped surrogates with KL-constrained natural gradient updates. We analyze its local geometry, showing that this power-law transformation preserves the Fisher Information Metric direction at the origin while damping ratio-driven gradient growth away from it. Evaluations across the MJX Continuous Control benchmark show substantial gains in sample efficiency and asymptotic stability, alongside competitive, environment-dependent Atari performance on visual control tasks in JAXAtari. Notably, under extended-horizon evaluations on Humanoid (two billion transitions under a shared configuration), clipped Symphony demonstrates positive gains over PPO. In a controlled 10M-step Humanoid evaluation within our common implementation, where PPO and SPO configurations were independently selected by mean screening AUC, Symphony achieves higher final returns than both PPO and SPO under each baseline's selected settings, while demonstrating decisive sample efficiency gains over SPO. Experiments across varying geometric orders and clipping ranges confirm that Symphony provides resilience across relaxed clipping boundaries (\(\epsilon=0.4\)) and ratio drift, offering a curvature-aware foundation for scalable policy optimization.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Policy Optimization, Policy Gradient Methods, Proximal Policy Optimization (PPO), Trust Region Methods, KL Divergence, Likelihood Ratios

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_VskWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Symphony, an RL method that replaces the importance sampling ratio rr in standard PPO and TRPO with fn(r)=nr1/n−(n−1)f_n(r)=n r^{1/n}-(n-1) (default n=2n=2). The motivation stems from heavy-tailed ratios that occur in high-dimensional control tasks due to the multiplicative combination of probability ratios across dimensions. This method dampens large ratios by reweighting per-sample gradients with r1/nr^{1/n}, while leaving the gradient unchanged when r=1r=1. The paper provides a binomial-residual approximation bound and a second-moment bound. Experiments on MJX and JAXAtari evaluate clipped and KL-penalized variants. Symphony outperforms standard PPO on Humanoid and Walker2D (high-dimensional tasks), while maintaining comparable performance on low-dimensional MJX tasks and Atari (discrete actions, where no product of ratios arises).

Critical strengths

This is a simple, well-motivated method with clear empirical patterns. The results support the motivation and algorithm design, with gains appearing only where per-dimension ratio compounding is expected to matter (high-dimensional continuous control) and disappearing in low-dimensional tasks.

The empirical results are carefully reported across multiple seeds.

Critical weaknesses

The empirical results only support the authors' claims under one training configuration, which does not include learning-rate decay or other commonly used PPO techniques. As a result, PPO peaks near 7,500 on Humanoid and then degrades to 3,750. With an appropriate configuration, such as an SPO-style pipeline, the ranking of standard PPO and Symphony reverses.

The paper lacks sufficient baselines and discussion of related work. As far as I know, RL methods used in LLM training have already attempted to address unbounded importance sampling ratios, such as GSPO[1].

The ablation results do not adequately explain the claims. The order ablation shows that n=64n=64, where r1/n≈1r^{1/n}\approx 1 and importance weighting is essentially removed, performs as well as n=2n=2.

Questions for authors

Do the gains hold across MJX tasks under a standard PPO pipeline (reward scaling, learning-rate decay)?

If fnf_n is applied only to samples with A<0A<0 and r>1+εr>1+\varepsilon, are the gains preserved?

Could the authors discuss the differences between Symphony and GSPO[1] ?

[1] Zheng, Chujie, et al. "Group Sequence Policy Optimization." arXiv preprint arXiv:2507.18071 (2025).

4 comments

We thank Reviewer VskW for their evaluation of our manuscript and for recognizing the gains in high-dimensional tasks. We address their concerns as follows.

Standard PPO pipeline & ranking reversal: Our SPO-style pipeline (Section 3.3, Table 16) already includes CleanRL-style running reward scaling, linear LR decay, and value clipping, and replacing its non-standard 7-layer actor with a standard 3-layer MLP (2x64 Tanh, Table 3) restores Symphony's lead (2663.3±1283.02663.3 \pm 1283.0 vs. 1619.1±374.91619.1 \pm 374.9 for PPO and 902.0±47.1902.0 \pm 47.1 for SPO).

Applying fnf_n only to A<0andr>1+ϵA < 0 \quad \text{\textbf{\textit{and}}} \quad r > 1 + \epsilon : We have not ablated applying fnf_n solely to A<0A < 0 and r>1+ϵr > 1 + \epsilon, because Symphony (Clipped) (Eq. 5) deliberately curves the active r∈[1−ϵ,1+ϵ]r \in [1-\epsilon, 1+\epsilon] region for A>0A > 0 as well, and it outperforms asymmetric A<0A < 0 truncation (Dual-Clip PPO) in Section 3.2 (6949.636949.63 vs. 5924.865924.86 on Humanoid; 2951.552951.55 vs. 2738.742738.74 on Walker2d).

On a related note, Section 3.4 (page 10, "High-ratio tail suppression") directly analyzes these active r>1+ϵr > 1 + \epsilon transitions—showing that 100% of them occur with A<0A < 0 and that Symphony consistently suppresses their tail leverage (Figure 5d).

Symphony vs. GSPO: GSPO targets sequence-level LLM RL by taking a sequence-length geometric mean (πθ(y∣x)/πold(y∣x))1/∣y∣(\pi_\theta(y\vert{}x)/\pi_{\text{old}}(y\vert{}x))^{1/\vert{}y\vert{}} (scaling local gradients by 1/∣y∣1/\vert{}y\vert{}) with standard linear clipping, Symphony operates per MDP transition with fixed order nn and rescales fn(r)=nr1/n−(n−1)f_n(r) = nr^{1/n}-(n-1) so fn′(1)=1f_n'(1)=1 preserves exact local policy-gradient magnitude while introducing intrinsic negative curvature (fn′′(1)=−(n−1)/n<0f_n''(1)=-(n-1)/n < 0) inside the active region.

Order ablation at n=64n=64 : As shown in Table 1 (page 6), taking n→∞n \to \infty yields fn(r)=n(r1/n−1)+1→1+log⁡rf_n(r) = n(r^{1/n}-1)+1 \to 1 + \log r, which removes ratio reweighting (r1/n→1r^{1/n} \to 1) while retaining the unweighted policy-gradient signal ∇θlog⁡πθ(a∣s)\nabla_\theta \log \pi_\theta(a\vert s) at O(1)\mathcal{O}(1) variance, and on Ant (Table 2 Part II, Figure 7), n=64n=64 drops return (2174.042174.04 vs. 2440.522440.52 at n=4n=4), which confirms the bias–stability tradeoff of excessive damping.


Closing note: We once again thank the reviewer for their evaluation, and we are confident that we have addressed their concerns. We would be grateful if the reviewer could revisit their recommendation.

I thank the authors for the detailed response. The pipeline ablation and the GSPO discussion address my concerns, and I am raising my rating. I still suggest applying fnf_n only to samples with A<0A<0 and r>1+ϵr>1+\epsilon: Section 3.4 shows that all active high-ratio transitions have A<0A<0, so this one-line ablation would show whether the gain comes from attenuating these samples or from the curvature for A>0A>0. The Dual-Clip comparison does not settle this, since Dual-Clip only activates when the ratio exceeds 3.

We thank Reviewer VskW for revising their score upward and are thrilled with their positive assessment. To address the remaining concern, we ran the suggested targeted ablation on Walker2d on 3 configurations: the SPO-selected setting, the PPO-selected setting, and the Dual-Clip PPO-selected setting. A detailed discussion is provided in Appendix F (page 63), written in full. To save a click, please refer to the table below:

Full SymphonyTail-Only SymphonyDifference (Tail − Full)
SPO-selected3866.97 ± 1307.962367.09 ± 371.67−1499.88
PPO-selected3236.74 ± 1031.443073.11 ± 1176.57−163.63
Dual-Clip-selected4173.45 ± 158.582197.01 ± 812.30−1976.45

We find that attenuating only the active high-ratio, negative-advantage region may not be sufficient to reproduce Symphony's behavior across the tested configurations.


Closing note: We are once again grateful to Reviewer VskW for increasing their score. We address the remaining concern with the suggested ablation. We would be grateful if the reviewer would consider a further upward revision of their score.

Thanks for your response. I will maintain my score.

Reviewer_FeWtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a kind of policy optimization surrogates that replaces the linear importance ratio term with a generalized power law transformation. The proposed transformation preserves the standard policy gradient at the behavior policy while moderating the contribution of large importance ratios during subsequent optimization. The authors incorporate this construction into clipped and KL penalized objectives.

The theoretical analysis provides a binomial decomposition and local approximation bounds as well. The analysis also characterizes how gradient second moments depend on importance ratios.

Empirical validation includes six continuous control tasks implemented in MJX and seven Atari games implemented in JAXAtari. The experiments compare Symphony with PPO and SPO and also include additional comparisons with independently tuned baselines and alternative implementations. The reported results demonstrate improved returns and training stability on several continuous control tasks, particularly during extended training, while improvements on Atari are more modest and the relative performance depends on the implementation.

Critical strengths

A critical strength of Symphony is its mathematically simple formulation as a PPO variant. The power law transformation preserves the policy gradient at the behavior policy while moderating the growth of gradient weights at large importance ratios. It integrates directly into PPO’s clipped objective within the existing optimization framework. The reported results on a series of continuous control tasks, especially Humanoid and Walker2D, provide evidence of improved stability during extended training. This supports acceptance because the method offers an accessible extension of a widely used algorithm with demonstrated practical benefits on some challenging continuous control tasks.

Critical weaknesses
  1. Insufficient evidence for the proposed mechanism and its additional stabilization benefit.

The controlled comparison with PPO supports a benefit of the modified surrogate within the authors’ implementation. However, the extent to which this benefit arises from suppressing extreme ratio contributions remains unclear. Standard PPO already clips large ratios for positive advantages, while Dual Clip PPO (Ye et al., 2020) directly addresses extreme ratios for negative advantages. A comparison against Dual Clip PPO under matched settings would therefore be particularly informative. Reporting gradient contributions separately for positive and negative advantages would further clarify the proposed mechanism. The extended training comparisons also use a shared configuration and do not establish superiority over baselines tuned specifically for the full training horizon. These gaps limit confidence in the additional stabilization benefit provided by the power law transformation over existing treatments of extreme importance ratios.

  1. Limited evidence for the robustness and breadth of the empirical gains.

The substantial gains over PPO in Table 2 are concentrated in three MJX locomotion tasks. Across the seven Atari games, Symphony does not demonstrate a clear performance advantage over PPO. The Atari results therefore leave the practical benefit of the proposed surrogate in discrete visual control insufficiently established. The additional evaluation with independently tuned baselines strengthens the evidence for continuous control but is restricted to Humanoid. Section 3.2 also reports a reversal in the ranking of methods under an alternative implementation. These findings limit confidence in the breadth and robustness of the empirical advantage. Further evaluation across diverse tasks with appropriately tuned baselines would help establish the conditions under which the proposed surrogate consistently improves performance.

Questions for authors
  1. What theoretical or empirical evidence favors the proposed power law family over other simple smooth transformations that satisfy (f(1)=f'(1)=1) and attenuate extreme importance ratios? Is there evidence that the specific functional form provides a meaningful advantage over these alternatives?

  2. How much of Symphony’s improvement over PPO under matched clipping settings can be attributed to reduced gradient contributions from extreme importance ratios? What diagnostics support this explanation, particularly for samples with negative advantages that remain active under standard PPO clipping?

  3. Is there existing empirical evidence that Symphony provides meaningful performance gains in RLHF using PPO, or should its relevance to this setting be understood as a direction for future validation?

  4. Can the authors provide a comparison with Dual Clip PPO under matched training budgets and comparable hyperparameter tuning? Such a comparison would clarify whether smooth power law attenuation offers a meaningful benefit over directly truncating extreme contributions from samples with negative advantages.

  5. What evidence supports the persistence of Symphony’s advantage when baselines are tuned for the full training horizon? How should the extended training gains be interpreted in light of the shorter budgets used for independent baseline tuning and the reported reversal in method rankings across implementations?

5 comments

We thank Reviewer FeWt for their positive assessment and for pointing out evidence that supports acceptance. We address the concerns raised below.

Proposed power law surrogate: We want to emphasize that we never claim that the proposed transformation is superior to other solutions that satisfy f(1)=1f(1) = 1, f′(1)=1f'(1) = 1, and f(r)∼r1/nf(r) \sim r^{1/n}. For our work, we start with f(r)=ar1/n+bf(r) = ar^{1/n} + b and then we solve for f(1)=1f(1) = 1 and f′(1)=1f'(1) = 1. There may be other transformations that provide better results and therefore should be considered future work. For extreme importance ratios, the submitted manuscript already provides importance ratios in the diagnostic plots for every evaluation discussed in it: Appendix D.

Improvement in matched settings: The revised manuscript includes Section 3.4 (page 10) and Figure 5(d) (bottom row, page 10) to address this. Empirically, from the revised manuscript, we notice that Symphony consistently suppresses tail leverage.

RLHF as future work: Yes, Symphony on RLHF should be considered as future work.

Dual-Clip PPO and Walker2d: Section 3.2 (page 9) in the revised manuscript now provides dedicated Dual-Clip PPO runs. We also understand the need for a benchmark other than Humanoid, so we run experiments on Walker2d using the same 36-configuration sweep on 4 seeds for 5M screening steps, followed by 10M-step confirmation runs. Empirical results from the manuscript are as follows:

SymphonyDual-Clip PPO
Humanoid6949.63±205.566949.63 \pm 205.565924.86±168.875924.86 \pm 168.87
Walker2d2951.55±714.242951.55 \pm 714.242738.74±1057.332738.74 \pm 1057.33

Clarification on Scope and Full-Horizon Baseline Tuning: To clarify, our manuscript does not claim that Symphony outperforms baselines tuned for the full training horizons. For reported ranking reversals across different implementations, with new evidence from Walker2d, we observe that all baselines perform competitively within our implementation (4827−53824827 - 5382), while only SPO performs well (3366±3643366 \pm 364) in its pipeline; the others do not (2149−22592149 - 2259). To build an understanding of the plausible reasons for Humanoid, we present Humanoid sensitivity ablation in Section 3.3. The results from the revised manuscript are:

Reference (7-layer)Constant LRRaw RewardsShallow Actor (2×642 \times 64)
PPO982.4±141.9982.4 \pm 141.91771.6±539.01771.6 \pm 539.0529.2±23.1529.2 \pm 23.11619.1±374.91619.1 \pm 374.9
SPO833.2±77.9833.2 \pm 77.9842.3±48.2842.3 \pm 48.2507.2±22.7507.2 \pm 22.7902.0±47.1902.0 \pm 47.1
Symphony (Clip.)916.0±214.2916.0 \pm 214.21078.4±109.01078.4 \pm 109.0514.3±10.4514.3 \pm 10.42663.3±1283.0\mathbf{2663.3} \pm \mathbf{1283.0}

To understand how long-horizon runs differ from shorter-horizon runs, we refer the reviewer to FAQ 11 (page 70).


Closing note: We once again thank the reviewer for their insights, which helped us strengthen the manuscript. We can confidently say that we addressed the reviewer's concern. We would be grateful if the reviewer could revisit their assessment of our work.

AuthorsOpenReview ↗

Request for clarification

There appears to be a discrepancy between your written assessment and the overall score. You explicitly state that “This supports acceptance because the method offers an accessible extension of a widely used algorithm”, yet assign a score of 2, whereas acceptance corresponds to 4. We would appreciate clarification on the reasoning behind this discrepancy.

Thanks for the response and the additional experiments. I like the motivation behind Symphony, and that’s what I wanted to highlight in my review. I still think the theoretical case for choosing the power-law surrogate needs more depth (Question 1), and I see the method as an incremental contribution. The new results also leave me unsure how consistently the gains hold across tasks and implementations (Weakness 2). I’ll keep my score of 2.

Reviewer_euXfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper tries to address the issue of how on-policy reinforcement learning methods that use linear importance-ratio surrogates, such as PPO, might not account for the curvature of the surrogate well under large likelihood ratios. This may be amplified in multi-dimensional continuous control due to the dimensionality curse of importance sampling. They derive a transformation of the importance ratio f_n(r) = nr^{1/n} - (n-1) and show it has nice properties. After constructing a clipped and KL-constrained algorithm based on this transformation, they ablated their Symphony algorithm against PPO on several control tasks.

Critical strengths

The proposed power-law transformation of the importance ratio preserves first-order policy-gradient agreement, addresses curvature at large importance-ratios, and is straightforward to implement. The paper provides a robust analysis of the properties of the transformation.

The paper provides broad empirical improvements in control tasks. The method shows better performance than PPO on challenging environments, and the long-horizon experiments provide evidence that the transformation may lead to better training stability.

Critical weaknesses

The paper argues that PPO's surrogate is prone to fail at large likelihood-ratio displacements. However, there are several PPO variants, including Dual-Clip PPO, that limits the gradient contribution from large likelihood-ratio, negative advantage samples. There still could be advantages of the Symphony algorithm, but the improvements have not been compared to similar existing variants.

The particular power-law transformation does not seem very motivated, though it does exhibit good properties. There are other concave functions that may satisfy similar properties. Also, there are no good heuristics for the choice of n, and ablations show that n can vary depending on the task.

Questions for authors

The power law transformation satisfies first-order agreement and dampens the second gradient, but what justifies this transformation over other mappings? Does this offer a stronger guarantee than other possible transformations?

The paper's motivation is that PPO's surrogate may be unstable from large importance ratios. There have been attempts to alleviate this issue, such has Dual-Clip PPO. Could the authors provide comparisons against other PPO variants that specifically try to address this issue?

4 comments

We thank Reviewer euXf for their positive assessment of our paper's soundness, clarity, and empirical stability gains. We have rewritten Sections 3.2 (page 9) and 3.4 (page 10), and address both questions concisely below:

Comparisons against Dual-Clip PPO: As requested, we screen Dual-Clip PPO (c=3.0c=3.0) across 36 configurations over 4 seeds for 5 million steps, and then confirm on fresh seeds over 10M steps. Symphony (Clipped) outperforms tuned Dual-Clip PPO on Humanoid (6949.63±205.566949.63 \pm 205.56 vs. 5924.86±168.875924.86 \pm 168.87) and matches it on Walker2d (2951.55±714.242951.55 \pm 714.24 vs. 2738.74±1057.332738.74 \pm 1057.33). In both cases, Symphony inherited Dual-Clip PPO's hyperparameter tuning. Figures 4(e) and 4(f) on page 9 provide the plot, and Table 13 (page 49) and Table 15 (page 50) provide per-seed performance.

Other possible transformations: There are other transformations which satisfy (i) f(1)=1f(1) = 1, (ii) f′(1)=1f'(1) = 1, and (iii) f(r)∼r1/nf(r) \sim r^{1/n}. However, for this work, we obtain fn(r)=nr1/n−(n−1)f_n(r) = n r^{1/n} - (n-1) by solving f(r)=ar1/n+bf(r) = ar^{1/n} + b subject to (i) and (ii). Since we are not aware of what other transformations might exist subject to the presented constraints, we will refrain from claiming a stronger guarantee than other transformations. This should be considered as future work.


Closing note: We once again thank the reviewer for their positive evaluation of the presentation and soundness. We are confident that we have addressed the concerns and would be grateful if the reviewer could revisit their recommendation.

Reviewer_euXfOpenReview ↗

Transformation

Hi. Thank you for the results; they show that your method does seem effective and provide benefits that Dual-Clip PPO does not. I have some clarifications about my issue with the transformation. I am not sure if the (iii) condition f(r) ~ r^{1/n} is well motivated. Preserving on-policy gradient requires the first two. On the other hand, other concave transformations can also exhibit some local curvature. Consider the logarithmic family: gn(r)=1+nn−1log⁡((n−1)r+1n).g_n(r)=1+\frac{n}{n-1}\log\left(\frac{(n-1)r+1}{n}\right). It is strictly increasing and concave and satisfies:

g_n''(1)=-\frac{n-1}{n}.$$ It matches Symphony's value, first, and second derivative at r = 1 for every n. It also helps decrease the influence of the high importance ratios of the policy gradient, which satisfies overall main motivation. Overall, I would appreciate some further justification other than the three conditions you stated for choosing the power law form over other transformations.

We thank Reviewer euXf for recognizing the benefits that Symphony brings over Dual-Clip PPO. Regarding the remaining clarification, we have already addressed it in the manuscript, between lines 161 and 162. However, we thank the reviewer for requesting further clarification and hope the paragraphs below address it.

Kakade and Langford performance difference lemma: Equation 1 (page 3) is the starting point of Symphony, which is where we ask, "How can we evaluate the performance of a future policy π′\pi' when its state-visitation distribution dπ′d_{\pi'} is not directly available under the current policy π\pi?"

Importance sampling used in TRPO and PPO: We turn our attention to what TRPO and PPO propose, which is ∫A(s,a)π′(da∣s)=∫π′(a∣s)π(a∣s)A(s,a)π(da∣s)=∫rπ′(s,a)A(s,a)π(da∣s)\int A(s, a) \pi'(da \mid s) = \int \frac{\pi'(a \mid s)}{\pi(a \mid s)} A(s, a) \pi(da \mid s) = \int r_{\pi'}(s, a) A(s, a) \pi (da \mid s). The gradients of this should be ∫rπ′(s,a)A(s,a)∇log⁡π′(a∣s)π(da∣s)\int r_{\pi'}(s, a) A(s, a) \nabla \log \pi'(a \mid s) \pi (da \mid s).

Damping heavy-tailed ratio excursion: We notice that in ∫rπ′(s,a)A(s,a)∇log⁡π′(a∣s)π(da∣s)\int r_{\pi'}(s, a) A(s, a) \nabla \log \pi'(a \mid s) \pi (da \mid s) we can use rπ′(s,a)r_{\pi'}(s, a) to damp heavy-tailed ratio excursion. Since PPO tries to clip it and SPO proposes something they call ϵ\epsilon-aligned (definition 5.1, page 6 of the SPO paper), we ask the question, "What if we try to dial down rπ′(s,a)r_{\pi'}(s, a) instead and introduce curvature damping even before we hit the clipping?" This is also what Figure 1 (page 1) of the manuscript shows, where we dampen the ratios even before they hit clipping.

Conclusion: This is why we propose f(r)∼r1/nf(r) \sim r ^ {1/n} as a source of motivation, and we are not claiming this as an important condition for a surrogate to perform universally best. There may be other conditions and surrogates, such as the reviewer's proposed surrogate gng_n. Therefore, the existence of different surrogates that may or may not satisfy our conditions, or of different conditions, should be considered strictly future work.


Closing note: We once again thank the reviewer for their very positive assessment. Again, we want to clarify that different surrogates satisfy different conditions, such as the suggested gng_n. Still, we are not claiming superiority over those surrogates, nor are we claiming that our proposed conditions, f(1)=1f(1) = 1, f′(1)=1f'(1) = 1, and f(r)∼r1/nf(r) \sim r^{1/n}, are the universal optimum. We request that the reviewer revisit their score, as we believe we have effectively addressed all their concerns. The only remaining concern in the follow-up message was seeking the motivation behind f(r)∼r1/nf(r) \sim r^{1/n}. Although we believe that Section 2 (page 3) of the original manuscript addresses this concern effectively, we are open to suggestions for revising that section to remove any ambiguity.

We are happy to entertain any further follow-ups from Reviewer euXf!

AuthorsOpenReview ↗

Update in the revised manuscript

We have updated Section 2 (page 3, lines 158–161) in the revised manuscript to narrow and clarify our theoretical framing. Rather than framing the transformation in general terms, we now explicitly restrict our construction to affine power-law maps, f(r)=ar1/n+bf(r)=ar^{1/n}+b, guided by three design criteria: objective-value preservation, first-order policy-gradient alignment, and sublinear attenuation of extreme importance ratios. We also clarify that we do not claim uniqueness or superiority over other smooth transformations.

We hope this helps mitigate concerns about the motivation for f(r)∼r1/nf(r) \sim r^{1/n}.

Best, The authors

Reviewer_UURgOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Symphony, a family of power-law policy surrogates fn(r)=nr1/n−(n−1)f_n(r)=nr^{1/n}-(n-1) that preserves the policy gradient at the behavior policy while attenuating large importance-ratio contributions. The basic algebra and conditional approximation bounds appear correct, but the connection between these results and the practical algorithm remains insufficiently established.

Critical strengths
  1. Simple and practical formulation. The power-law surrogate is straightforward to implement within a PPO-style training pipeline.

  2. Sound core identities. The first-order gradient agreement at the behavior policy and the binomial residual decomposition are correctly established.

  3. Improved theoretical precision. The manuscript explicitly distinguishes its conditional approximation bound from a monotonic-improvement guarantee for the practical algorithm.

Critical weaknesses
  1. Incomplete gradient analysis. Equation (6) covers only the surrogate gradient. The implemented KL penalty yields [r1/nA−β(r−1)]∇θlog⁡πθ[r^{1/n}A-\beta(r-1)]\nabla_\theta\log\pi_\theta, reintroducing a linear ratio term into the full update.

  2. Limited variance guarantee. The bounded-score assumption is not established for the Gaussian policies used. Reducing the ratio exponent alone does not guarantee lower gradient variance than PPO.

  3. Unenforced assumptions. Neither clipping nor small KL ensures the uniform ratio neighborhood required by the residual bound. Its applicability to actual training trajectories remains unclear.

  4. Heuristic curvature substitution. Positive-advantage curvature can be offset by negative-advantage contributions. The proposed reduction of the KL coefficient lacks a full-policy justification.

Questions for authors
  1. How does the second-moment analysis extend to the full implemented KL gradient, which includes the term −β(r−1)∇θlog⁡πθ-\beta(r-1)\nabla_\theta\log\pi_\theta?

  2. Can you establish moment conditions applicable to the Gaussian policies used experimentally, and clarify whether any variance ordering relative to PPO is guaranteed?

  3. What ensures that practical updates satisfy the neighborhood assumptions required by the residual bound, given that neither clipping nor small KL guarantees uniformly bounded likelihood ratios?

  4. Can the proposed KL-coefficient calibration be justified after accounting for both positive and negative advantages under policy normalization?

1 comment

We are grateful to Reviewer UURg for their evaluation and value the positives they noted, such as simplicity, sound core identities, and publicly available code and logs. To address concerns regarding Symphony (KL), we have revised the manuscript to remove any ambiguity. New additions in Section 2 are highlighted in blue text, while Sections 3.2–3.4 and Appendix B and E were rewritten in full.

Full implemented KL gradient: We thank the reviewer for pointing this out, and we have added the requested gradient to Section 2 (page 6) of the revised manuscript. The requested gradient is as follows: Tr(E[g^n,KL g^n,KL⊤])≤2Tr(E[g^n g^n⊤])+2β2E[(rθ−1)2∥∇θlog⁡πθ∥22].\text{Tr}(\mathbb E[\hat g_{n, KL} \, \hat g_{n, KL}^\top]) \le 2 \text{Tr} (\mathbb E[\hat g_n \, \hat g_n^\top]) + 2 \beta^2 \mathbb E[(r_\theta - 1)^2 \Vert \nabla_\theta \log \pi_\theta \Vert_2^2]. Here, g^n,KL=(rθ1/nAπ−β(rθ−1))∇θlog⁡πθ\hat g_{n, KL} = (r_\theta^{1/n} A^\pi - \beta (r_\theta -1)) \nabla_\theta \log \pi_\theta. For Symphony (Clipped), the results remain the same as before because β=0\beta = 0.

Gaussian moments & variance ordering: We have added the Gaussian LqL_q-moment bound (via Hölder's inequality for clamped log⁡σ∈[−20,2]\log \sigma \in [-20, 2]) and an explicit variance-ordering clarification in Section 2 (pages 5–6) and Corollary B.5(1) (Appendix B.5). As shown in Table 20 (page 61), no pointwise variance ordering over PPO holds across iterates (Symphony has higher early-exploration second moments at 1M–9M steps before dropping 3.1×3.1\times below PPO at 10M); Equation (6) bounds the tail-growth exponent in rθr_\theta rather than implying uniform variance dominance.

Neighborhood assumptions vs. practical updates: We revised Section 2 (page 6) and added Corollary B.5(3) (Appendix B.5, page 26) to remove the uniform ratio-bound assumption: for n=2n=2, C2,ρ‾≡1C_{2,\overline{\rho}} \equiv 1 holds globally without any ratio cap (∣L2(π′)−L2(π′)∣≤2Amax⁡1−γ\vert \mathcal{L}_2(\pi') - L_2(\pi')\vert \le \frac{2A_{\max}}{1-\gamma}), and for general n≥2n \ge 2, a tail-splitting decomposition bounds the residual unconditionally via Jensen's inequality. Empirically, Figures 11 (page 36) through 26 (page 54) show importance-ratio statistics (mean, min, max, and 99th percentile) for each evaluation, confirming that clipping and negative curvature keep practical updates concentrated near unity.

KL-coefficient calibration: Short answer: yes. As shown in Section 2 (page 6), under normalized advantages (E[Aπ]=0\mathbb{E}[A^\pi] = 0) and local score symmetry (E[Aπψψ⊤]=0\mathbb{E}[A^\pi \psi\psi^\top] = 0), the positive- and negative-advantage quadratic terms cancel in expectation at θold\theta_{\text{old}}. Away from θold\theta_{\text{old}}, large ratio excursions (r≫1r \gg 1) occur along Aπ>0A^\pi > 0 expansion where negative curvature acts as an adaptive brake, so λsym=0.5\lambda_{\text{sym}} = 0.5 serves as a directional local calibration for positive-advantage expansion rather than a symmetric second-order equivalence.


Closing note: We are confident that the revised manuscript addresses the issues with the gradient analysis. Please revisit the scores and recommendations.

Reviewer_shmaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Symphony, a modified policy optimization surrogate for on-policy RL. Instead of using the standard likelihood ratio r directly, Symphony applies the transformation

f_n(r) = n r^(1/n) - (n-1)

and combines it with either PPO-style clipping or a KL penalty.

The main motivation is that likelihood ratios can become very large in high-dimensional continuous control because the joint action probability is a product across action dimensions. The authors argue that this can create large gradient contributions and unstable updates. Symphony makes the dependence on the ratio sublinear while preserving the policy gradient at r=1.

The paper provides:

  • a derivation of the surrogate and its residual using a binomial expansion;
  • analysis showing that the local policy-gradient direction is preserved;
  • a second-moment argument showing reduced dependence on large ratios;
  • experiments on six MJX continuous-control tasks and seven JAXAtari tasks;
  • ablations over transformation order, clipping threshold, learning rate, and KL penalty;
  • a tuned Humanoid comparison against PPO and SPO;
  • an implementation-sensitivity study using an SPO-style pipeline.

The main empirical result is that Symphony is substantially more stable than PPO on some long-horizon continuous-control tasks, especially Humanoid and Walker2D. On Atari, performance is mostly comparable to PPO rather than clearly better. The implementation study also shows that method rankings can reverse depending on the surrounding training pipeline.

Critical strengths
  • Simple modification with strong results in the main continuous-control setting.

    Symphony only changes the surrogate objective and is easy to integrate into a PPO-style implementation. Despite being simple, the long-horizon gains on Humanoid and Walker2D are large. The most convincing part is that PPO reaches reasonable performance and later degrades, while Symphony remains stable.

  • Good effort on experimental controls.

    The paper does more than compare against one fixed PPO configuration. The main benchmarks use multiple seeds and uncertainty estimates, there are several useful ablations, and PPO/SPO receive separate hyperparameter searches in the Humanoid study.

    I also appreciate that the authors report the negative implementation-sensitivity result rather than hiding it. This makes the paper more credible.

Critical weaknesses
  • It is still unclear how broadly the main stability result transfers beyond the authors' primary training setup.

    The strongest gains come from very long continuous-control runs using constant learning rates and a shared implementation. In these experiments PPO often learns reasonably well early, but then degrades later, while Symphony remains stable. This is a meaningful stress test, but it is not the same as showing that Symphony is generally better than a well-tuned PPO pipeline.

    The tuned Humanoid experiment helps address this concern, since PPO and SPO each receive their own hyperparameter search. Under the PPO-selected setting, Symphony has a clearly better final return. However, the AUC difference against PPO has a confidence interval that includes zero. So I think the evidence is stronger for improved late-stage stability than for improved sample efficiency.

    More importantly, the implementation-sensitivity experiment shows a very large ranking reversal. Symphony performs well in the authors' main pipeline but poorly in the SPO-style pipeline, while SPO shows the opposite behavior. This suggests the surrogate interacts strongly with implementation choices such as learning-rate scheduling, reward normalization, architecture, value-loss clipping, optimizer details, or action handling. The paper acknowledges this, but does not identify which of these factors are responsible. Because of this, it is currently difficult to know under what conditions Symphony should be expected to work well.

    I think a small targeted ablation here would improve the paper a lot. It is probably not necessary to test every implementation choice, but isolating one or two major factors, for example constant vs decaying learning rate or raw vs normalized rewards, would make the practical scope of the method much clearer.

  • The proposed variance-reduction mechanism is plausible, but the current evidence does not directly establish it.

    The mathematical argument clearly shows that the ratio-dependent gradient multiplier changes from r to r^(1/n), so very large likelihood ratios are compressed. This is useful and I agree with the basic intuition. However, Equation (6) is a bound on the ratio-dependent part of the gradient second moment under a bounded score-function/advantage assumption. This is not the same as proving that the full stochastic policy-gradient estimator has lower variance in the actual Gaussian policy setting used in the experiments.

    There is also an asymmetry that could be discussed more clearly: for r > 1, the transform reduces the ratio contribution, but for r < 1, r^(1/n) is larger than r. So the effect on the full gradient distribution is not simply "all gradients become smaller." It depends on the ratio distribution, advantage signs, clipping, and the score function.

    The bandit experiment is supportive, but it measures dispersion across gradient coordinates within a seed, not minibatch-level gradient estimator variance. The continuous-control diagnostics also show smaller ratio and KL excursions, but again this is indirect evidence for the proposed mechanism.

    I think the paper would be stronger with direct measurements of actor-gradient statistics during training. For example, comparing PPO and Symphony on minibatch gradient-norm variance, second moments, 95th/99th percentile gradient norms, or the fraction of gradient magnitude contributed by high-ratio samples would connect the theory much more directly to the observed stability gains.

  • A smaller theoretical issue is that the "uniqueness" of the transform is somewhat overstated.

    The transform is unique after assuming the specific parametric form a r^(1/n) + b together with the conditions f(1)=1 and f'(1)=1. But many other nonlinear functions could also satisfy these local conditions and introduce negative curvature. I think the paper should state the uniqueness claim more narrowly, because otherwise it sounds like this transformation is uniquely implied by the general design requirements.

Questions for authors
  1. Can the second-moment result be stated under assumptions that better match the Gaussian policy used in the experiments? In particular, is the uniform boundedness assumption necessary?

  2. Do you have direct measurements of stochastic actor-gradient variance or gradient tails for PPO vs Symphony during Humanoid or Walker2D training? This would help validate the main proposed mechanism.

  3. Can the authors identify which parts of the SPO-style pipeline cause the large performance reversal? Even a limited ablation over learning-rate decay, reward normalization, or architecture would be informative.

  4. Does the baseline-specific tuning result also hold on another task such as Walker2D? This would make the result less dependent on Humanoid.

  5. For PPO, should the main claim be improved late-stage stability rather than improved sample efficiency? The final-return difference is significant in the tuned experiment, while the AUC difference is not clearly separated from zero.

1 comment

We thank the reviewer shma for their thoughtful review, and are grateful that they found our manuscript credible. We address the concerns below:

Uniform L∞L_\infty boundedness assumption: The uniform L∞L_\infty assumption is not necessary, and we have updated Section 2 (page 5, revised in blue text) and added Appendix B.5 (Corollary B.5 (1), page 23).

Actor-gradient variance: We completely understand that this will help validate the main proposed mechanism, so the revised manuscript now includes a dedicated discussion of this, along with the plots. This discussion can be found in Section 3.4 (page 10), Figure 5 (bottom row, page 10), and Appendix E.10 (page 60). These runs are on Humanoid using the PPO-selected configuration. Four plots compare second moment, requested tail norms, minibatch covariance trace, and high-ratio contribution against training steps.

Ablation of SPO-style pipeline: Just like the actor-gradient variance, we now have a dedicated discussion in the revised manuscript. This discussion can be found in Section 3.3 (page 9), Figure 5 (top row, page 10), Table 3 (page 9), and Appendix E.11 (page 61). The key takeaway we found is that Symphony has superior performance on a shallow actor for Humanoid.

Baselines specific tuning Walker2d: We include baseline-specific Walker2d runs in the revised manuscript. This update can be found in Section 3.2 (page 9) and Figure 4 (bottom row, page 9).

Late-stage stability vs. improved sample efficiency: We appreciate this nuanced distinction, and we agree on keeping the claim narrow and tied strictly to what the statistics support. We frame our empirical gains in terms of sample efficiency and fixed-budget returns rather than making a broader claim of universal late-stage stability across arbitrary training pipelines or time horizons. Our revised manuscript reflects this scope.


Additional concern regarding uniqueness: As stated in Section 2 (page 3, "Solving f(r)=ar1/n+bf(r) = a r^{1/n} + b subject to (i) and (ii) uniquely yields..."), uniqueness is intended strictly within the affine power-law family ar1/n+ba r^{1/n} + b rather than across all concave functions. We have removed the word "uniquely" from that paragraph, as well as the opening of Appendix B.5 (page 23), in the revised manuscript to remove any ambiguity.

There may be other concave transformations satisfying conditions (i), (ii), and (iii). For this work, we found fn(r)=nr1/n−(n−1)f_n(r) = n r^{1/n} - (n-1) to be a solution.


Closing note: We would be extremely grateful if the reviewer could reconsider their scores for our submission in light of the clarifications and additional results above. We are confident that these changes have substantially strengthened the manuscript.

We once again thank Reviewer shma for their time, efforts, and consideration.

Best,

The authors

Comments on the paper

2 comments

Dear Reviewers,

Thank you for your contributions to the review process. As the author–reviewer discussion period ends on November 18, I would like to kindly remind everyone to review the authors' rebuttals and engage in the discussion.

In particular, please indicate whether your main concerns have been adequately addressed and clarify any remaining issues that may affect your final assessment.

Your feedback will be very helpful for the subsequent reviewer–AC discussion and meta-review process.

Thank you for your time and effort!

AuthorsOpenReview ↗

Request for Reviewer Follow Up

Dear Reviewers and AC rvTu,

We are thankful to the reviewers for their reviews and to AC rvTu for their efforts to facilitate our discussions with the reviewers. We are thrilled to have addressed VskW's concerns and are glad that, based on our rebuttal and amendments to the manuscript, they have revised their decision. Along these lines, we have also incorporated their suggested ablation in our revision. We are also glad that Reviewer euXf took part in the discussion. Although we have addressed their major concerns, we don't know whether our recent follow-up clarification regarding f(r)∼r1/nf(r) \sim r^{1/n} addresses their concern.

We kindly request that Reviewer shma and Reviewer FeWt engage in a discussion and let us know whether their concerns are resolved. Since today is the last day of the author-reviewer discussion, we have limited time for follow-up messages and may not be able to address any remaining concerns adequately.

Once again, we are thrilled to have received positive assessments and to have our critical strengths recognized in Symphony, our manuscript, and the shared code and CSV files.

Best, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.