acceptodds
Under review as a conference paper at ICLR 2027

Revisiting Amari's -Divergence to improve clipped surrogate PPO

Abstract

Proximal Policy Optimization (PPO) is one of the most popular policy gradient algorithms due to its widespread practicality and scalability. However, its performance is known to be highly sensitive to increasing the clipping threshold and policy network depth. To address this sensitivity, we introduce ALDIPPO, which replaces the clip with an -divergence penalty on the importance ratio. This makes the penalty stiffer for samples outside the range, which PPO rejects by switching their gradient off, wasting the rollout budget spent to collect them. We devise several online update rules for based on on-policy batch statistics, and show that ALDIPPO beats PPO from toy MiniGrid environments to standard MuJoCo tasks, and generalizes better in the Procgen benchmark.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Policy Optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_sDPtOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper revisits the clipping mechanism of PPO and proposes ALDIPPO, which replaces the clipped surrogate with an advantage-weighted alpha-divergence penalty on the importance ratio. The parameter alpha controls the skewness of the penalty, and the paper proposes several online rules that adapt alpha using batch statistics such as advantage moments, signal-to-noise ratio, and critic explained variance. The paper also provides monotonic-improvement and sample-complexity analyses. Experiments on MiniGrid, MuJoCo, and Procgen show improved ratio control and generally better robustness than PPO, particularly with deeper policies and larger clipping thresholds.

Critical strengths
  1. The paper provides a clear analysis of why PPO clipping may fail to control importance ratios under shared parameterization. The stationary-ratio analysis and the interpretation of alpha as controlling the skewness of the penalty provide an intuitive connection between PPO-style trust-region control and the alpha-divergence family.

  2. The mechanism-level experiments are well aligned with the motivation. In particular, the MiniGrid experiments directly examine importance-ratio deviation under different network depths and clipping thresholds instead of relying only on downstream return. As the policy network becomes deeper, PPO exhibits increasingly large ratio deviations, whereas ALDIPPO remains substantially more controlled.

  3. The paper includes substantial theoretical and empirical analysis. It provides monotonic-improvement and sample-complexity results and evaluates several adaptive alpha rules across MiniGrid, eight MuJoCo tasks, and Procgen. In the challenging deep-policy MuJoCo setting, ALDIPPO substantially improves over PPO and is competitive with or better than SPO.

Critical weaknesses
  1. The novelty boundary is relatively narrow because both the motivating problem and the use of alpha-divergence in PPO have substantial prior work. The fact that PPO clipping does not strictly constrain importance ratios has already been explicitly studied by prior work such as Truly PPO, while SPO already replaces clipping with a non-zero-gradient penalty and demonstrates advantages for deep policies. More importantly, Xu et al. (2023) already proposed an alpha-divergence-based PPO method, which the paper itself identifies as its closest prior work. Therefore, the main incremental contribution appears to be the online adaptation of alpha and the specific per-sample formulation, rather than introducing alpha-divergence into PPO itself.

  2. The central claimed contribution, adaptive alpha, is not adequately isolated experimentally. The experiments compare the proposed adaptive rules mainly against PPO and SPO, where SPO corresponds to the particular fixed choice alpha = 2. However, there is no systematic comparison against several fixed alpha values, a validation-selected best fixed alpha, or the closest alphaPPO baseline. This is particularly important because the paper's monotonic-improvement theorem holds for any alpha sequence, including a constant alpha, and the paper itself notes that the theorem does not establish that changing alpha improves return. Therefore, the current experiments do not clearly demonstrate whether the gain comes from adaptive alpha or simply from using a more suitable fixed penalty geometry than alpha = 2.

  3. Some of the robustness claims appear to depend on deliberately difficult settings and additional tuning. The strongest MuJoCo results are obtained with seven-layer policies, a regime already known to be problematic for PPO. In the standard two-layer setting, the results are considerably more mixed, and PPO outperforms all penalty-based methods on some tasks such as Walker2d and HalfCheetah. In addition, the original alpha-SNR rule collapses at epsilon = 0.5 on Procgen and requires a lower alpha ceiling and retuned gate constants. This weakens the claim that adaptive alpha itself provides robust hyperparameter adaptation.

  4. There is a nontrivial gap between the theoretical algorithms and the practical implementation. The monotonic-improvement theorem assumes bounded importance ratios, while the paper acknowledges that shared parameters can cause practical ratios to leave the stationary interval. The sample-complexity theorem analyzes a projected trust-ball algorithm with a constant penalty weight and a specific unbiased sampler, rather than the practical ALDIPPO update using advantage-dependent weighting and standard PPO-style optimization. Therefore, the theoretical results are informative but do not directly establish guarantees for the implemented algorithm. Moreover, although the paper states that an anonymized code archive is attached, I could not find or inspect the concrete implementation code in the materials available to me, which further limits verification of the practical algorithm.

Questions for authors
  1. Xu et al. (2023) already apply alpha-divergence to PPO, while SPO corresponds to a fixed alpha = 2 penalty. Could the authors clarify more precisely what constitutes the main methodological novelty of ALDIPPO beyond these prior approaches?

  2. Could the authors provide a controlled comparison between the proposed adaptive alpha rules and multiple fixed alpha values, including a validation-selected best fixed alpha and ideally alphaPPO? Without this comparison, it is difficult to determine whether the improvement is caused by online adaptation itself.

  3. The monotonic-improvement theorem holds for constant alpha as well as adaptive alpha. Is there theoretical or empirical evidence that the proposed batch statistics reliably predict which alpha value is preferable at a given update?

  4. How should the theoretical guarantees be interpreted for the practical implementation, given that the sample-complexity analysis uses projection, a constant penalty weight, and a special unbiased sampler that are not used in the reported practical algorithm?

2 comments

We thank the reviewer for the detailed and thoughtful review. We address the weaknesses and questions:

Weaknesses

  1. Our starting point is the nonstationarity of on-policy learning (Section 3.2): rollouts can be far from earlier ones, and advantages are nonstationary, so the penalty should adapt to the batch. No prior method chooses the divergence from the data or adapts it as the policy learns. PPO's KL penalty is fixed; SPO coincides with using a fixed α=2\alpha=2, so our method is a strict generalization of it; Xu et al. grid-search one α\alpha per environment before training (Gaussian policies only); Truly PPO keeps the clip and adds a rollback term. Our per-sample penalty, weighted by ∣A^∣/ϵ|\hat{A}|/\epsilon, sets the stationary ratio by ϵ\epsilon for every α\alpha (Proposition 3.1), so α\alpha can change without changing the trust region. The rules are derived from the penalty (Theorem B.1, Proposition B.2), and Theorems 4.1–4.2 cover the whole family. The preferable α\alpha does depend on the problem (Table 7, Figure 12).

  2. In Table 7 we add fixed α∈1,2,3,4\alpha \in \\{1,2,3,4\\}, a validation-selected α\alpha, α\alphaPPO with α∈0,0.5\alpha \in \\{0, 0.5\\} (bb chosen as in Xu et al., Table 8) and KL-penalty PPO. The best fixed α\alpha changes across environments: 2–3 on HumanoidStandup, 3 on Humanoid, 4 on Walker2d and Ant. αB3\alpha^{\mathcal{B}3} scores 0.992 without tuning, against 0.982 for the best fixed α=4\alpha = 4. The average α\alpha for αB3\alpha^{\mathcal{B}3} lies between 1.2 and 1.9, between fixed α=1\alpha = 1 and 22, yet it beats both on three tasks and ties on the fourth, so the gain comes from varying α\alpha. α\alphaPPO and KL-penalty PPO reach median normalised scores of 0.018–0.095, with KL far above SPO (refer to Figure 11).

  3. Deep policies and large ϵ\epsilon are harder settings, but natural: RL post-training of billion-parameter language models uses very deep policies, where the clip failure we study matters most. We establish this objective in standard RL with guarantees, and mention LLM fine-tuning as direct future work. With two-layer policies, PPO is only best on Walker2d and HalfCheetah (refer to Table 6). Procgen at ϵ=0.5\epsilon = 0.5 is a stress test: at 2.5×2.5\times the usual radius, far more samples leave the clip range (~80% of PPO's already do at ϵ=0.2\epsilon = 0.2, Figure 14). A single recalibration of the αSNR\alpha^{\mathrm{SNR}} gate and ceiling (refer to Table 3) turns them into stable updates (refer to Table 2).

  4. Part (a) of Theorem 4.1 needs only bounded ratios (Assumption 1, p.16), while Part (b) needs the stationary interval (Assumption 2), which shared parameters can violate. We measured how often this happens on the four MuJoCo environments. We show in Figure 13 that the mean ∣w−1∣|w-1| is 0.08–0.10 against 1.35–1.57 for PPO. In Figure 14, we show that only ~15% of samples exceed ϵ\epsilon against ~80% of PPO.

Questions

1–2. See weaknesses 1–2.

  1. Theorem 4.1 provides a monotonic improvement guarantee for every α\alpha. With deep policies and nonstationary learning, the critic, the rollout distribution and the task difficulty change together, so the best rule cannot be established in general. We do not claim a superior per-update selector; we observe that the preferable α\alpha depends on the environment (Table 7, Figure 12), and αB3\alpha^{\mathcal{B}3} matches the best fixed α\alpha without tuning.

  2. The three techniques referred to are the standard ingredients of the analyses we build on. The sampler is used in Algorithm 3 of (Agarwal et al. 2021), which makes the batch surrogate unbiased. The constant weight is TRPO's constant penalty, recovered by Lemma C.4; it makes the penalty curvature (a0/ϵ)Fρ(a_0/\epsilon)F_\rho for every α\alpha, so the update is an inexact natural-gradient step (Lemma D.2, Eq. (31)). The projection is ProjW\mathrm{Proj}_W of Agarwal et al. (Algorithm 4) in parameter units; it keeps ratios in [1−cwϵ,1+cwϵ][1 - c_w\epsilon, 1 + c_w\epsilon], giving strong concavity uniformly in α\alpha. Empirical algorithms like TRPO also drop its penalty, and the ratios still stay in this interval (refer to Figures 13–14). The new element is PPO-style multi-step optimisation of one batch of a nonlinear surrogate; its maximizer is biased, and the Lipschitz Hessian (Assumption 5) bounds this bias by O(1/N)\mathcal{O}(1/N) (Lemma D.3, Eq. (61)), giving the NPG rate O(εˉ−3)\mathcal{O}(\bar\varepsilon^{-3}) of Liu et al. (2020) instead of the policy gradient one O(εˉ−4)\mathcal{O}(\bar\varepsilon^{-4}). Please see Appendix G for details of ALDIPPO algorithm.

We look forward to further discussion from the reviewer in the remaining time of the discussion period. If there are no further questions, we appreciate if the score is updated to reflect this.

Thank you for your comprehensive response. I have raised my score to 3.

Reviewer_pvhAOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits PPO’s clipped objective and argues that when samples in a batch interact through shared parameters, the clip does not actually bound the importance ratio. The authors replace the clip with an advantage-weighted α-divergence penalty while keeping ε as the trust-region radius. The parameter α controls the skewness of the penalty, and they propose update rules that adapt α online from batch statistics. They provide a monotone improvement guarantee and a sample-complexity analysis for multi-epoch updates. Experiments on MiniGrid, MuJoCo with deep policies, and Procgen show bounded ratio deviations, lower sensitivity to ε, and performance matching or improving PPO/SPO.

Critical strengths
  • The analysis of why PPO’s clip fails under shared parameters is concrete and convincing, and Proposition 3.1 plus the figure formalize the issue well.
  • The theory is fairly complete: it gives the stationary ratio of the α-divergence penalty, a monotone improvement guarantee, and an O(ε^{-3}) sample complexity, with a clear contrast to PPO/SPO.
Critical weaknesses
  • The adaptive-α rules are mainly based on Taylor expansions and proxies. The theory only guarantees improvement for any α within the interval, but does not strongly characterize the convergence or stability of the adaptive rules themselves. α^SNR also needs retuning at ε=0.5, suggesting limited robustness.
  • Although the experiments cover several benchmarks, they mainly compare PPO/SPO and lack direct comparisons with KL-penalty PPO and TRPO-style trust-region methods. Computational overhead and hyperparameter sensitivity are also not sufficiently reported.
Questions for authors
  1. How stable are the adaptive-α rules under nonstationary training and noisy advantage estimates? Are there failure cases or sensitivity analyses?
  2. Can the authors report wall-clock overhead, the number of additional hyperparameters, and a direct comparison with KL-penalty PPO?
  3. Are the constants involving |α−2| in Theorem 4.2 still tight under adaptive α? Please provide an intuitive explanation.
3 comments

We thank the reviewer for the careful reading and constructive suggestions. We address the weaknesses and questions:

Weaknesses

  1. We agree that the rules come from Taylor expansions and batch proxies, and that our guarantees do not characterise their dynamics. Theorem 4.1 holds for any α∈[αmin⁡,αmax⁡]\alpha \in [\alpha_{\min}, \alpha_{\max}], and every rule is clipped to this range, so each adaptive update inherits it. For stability, we added the αt\alpha_t trajectories (Figure 12), the ratios they produce (Figures 13–14) and a sensitivity analysis (Table 9); see Q1. We see Procgen at ϵ=0.5\epsilon = 0.5 as a stress test rather than a lack of robustness (Q1); the batch-direction rules use the same hyperparameters at both ϵ\epsilon.

  2. We added KL-penalty PPO with β=3\beta = 3, the best fixed value found by Schulman et al. (2017), the α\alphaPPO of Xu et al. (2023) and constant α∈1,3,4\alpha \in \\{1, 3, 4\\} on the four MuJoCo tasks of the main text (Table 7, Figure 11). TRPO is a natural-gradient method that uses the inverse Fisher Information matrix via conjugate gradient, plus a line search. None of the compared methods, ours included, uses this inverse. Therefore, it is not really comparable to the other methods studied in the paper.

Questions

  1. All rules stay in [αmin⁡,αmax⁡][\alpha_{\min}, \alpha_{\max}] (Figure 12): αB3\alpha^{\mathcal{B}3} rises slowly, αB2\alpha^{\mathcal{B}2} moves highest and αSNR\alpha^{\mathrm{SNR}} settles in each environment. The rules keep the ratios bounded (please refer to Figures 13–14 in Appendix F.4), where we plot mean ∣w−1∣|w-1| over environment steps and fraction of samples with ∣w−1∣>ϵ|w-1| > \epsilon over 4 MuJoCo tasks.

    On Procgen, the batch statistics must be complemented by the gate constants (κs,cs,κe)(\kappa_s, c_s, \kappa_e) and the range. We choose (20,0.1,1)(20, 0.1, 1), in all the experiements except Procgen at ϵ=0.5\epsilon = 0.5. At 2.5×2.5\times the usual radius, far more samples fall outside the clip range and PPO switches off their gradient. To our knowledge, prior work, SPO included, has not evaluated this setting. A single recalibration, (40,0.03,10)(40, 0.03, 10) with αmax⁡=2\alpha_{\max} = 2 (Table 3), lets these samples enter a stable update.

    Each increase in the EMA weight β∈0,0.9,0.99\beta \in \\{0, 0.9, 0.99\\} shrinks the change in α\alpha between updates, while the return shows no consistent dependence on β\beta (Table 9). The best fixed α\alpha changes across environments, while the rules need no tuning (Table 7).

  2. With the same network, rollout and gradient steps, the α\alpha step adds about 0.8 ms to a 1.3 s update; every method is within run-to-run variation of PPO, and the rollout takes about 90% of the time (refer to Tables 10–11). Extra hyperparameters: the range for αB1\alpha^{\mathcal{B}1}; the range and β\beta for αB2\alpha^{\mathcal{B}2} and αB3\alpha^{\mathcal{B}3}; the range, β\beta and gate constants for αSNR\alpha^{\mathrm{SNR}}. We use [1,4][1, 4] and β=0.9\beta = 0.9 throughout. KL-penalty PPO scores 0.087 against 0.992 for αB3\alpha^{\mathcal{B}3} (refer to Table 7 and Figure 11), so a constant-weight KL penalty does not control the update.

  3. Under adaptive αt\alpha_t they are bounded by cmax⁡=max⁡(2−αmin⁡,αmax⁡−2)=2c_{\max} = \max(2 - \alpha_{\min}, \alpha_{\max} - 2) = 2 (p.6 l.323–324). This is not tight: the mean ∣αt−2∣|\alpha_t - 2| is 0.26 for αSNR\alpha^{\mathrm{SNR}}, 0.44 for αB3\alpha^{\mathcal{B}3}, 0.75 for αB2\alpha^{\mathcal{B}2} and 1.04 for αB1\alpha^{\mathcal{B}1}, which is the only rule reaching the worst case. Intuitively, ∣α−2∣=∣fα′′′(1)∣|\alpha - 2| = |f_\alpha'''(1)| is the skew of the penalty relative to SPO's symmetric χ2\chi^2, and it enters only the model bias, Cprox(α)≤c1+G3Gω2∣α−2∣/(2μF)C_{\mathrm{prox}}(\alpha) \le c_1 + G^3 G_\omega^2 |\alpha - 2|/(2\mu_F) (Lemma D.2). The guarantee thus holds for every α\alpha at the same O(εˉ−3)\mathcal{O}(\bar\varepsilon^{-3}) rate, with ∣α−2∣|\alpha - 2| entering only as a constant.

We look forward to further discussion from the reviewer in the remaining time of the discussion period. If there are no further questions, we appreciate if the score is updated to reflect this.

Dear Reviewer pvhA,

As today is the last day of the discussion period, we would like to kindly follow up on our rebuttal.

We want to make sure that we have clarified and addressed all of your concerns, and we would be happy to answer any further questions you may have. If our response has resolved your concerns, we would be grateful if you could take it into account in your final assessment.

Thank you again for your time and constructive feedback.

Thanks for the authors' detailed rebuttal. I maintain my positive evaluation.

Reviewer_T3LQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduce ALDIPPO, which replaces the clip with an α\alpha-divegence penalty on the importance ratio to address the sensitivity in PPO. They introduce their method by a vivid motivating example, monotone improvement, global convergence and sample complexity claims are well proved. The experimental results show the best α\alpha-family will outperform PPO.

Critical strengths
  1. Well established motivating example to illustrate the failure of PPO and introduce the ALDIPPO.
  2. Solid theoretical and experimental results. (But have some issue on supporting your claim, see critical weakness)
Critical weaknesses
  1. Inconsistent notations: loss function in eq.(3),(4) and (7) are not consistent, which causing confusion on the Fig 1 about the shape of curve (the curve open upward or downward).
  2. I believe your theoretical results donesn't clearly show the improvement by introducing the adaptive α\alpha, we know that PPO and SPO may both have value enhancement property, so what is the gain of your proposed method theoretically?
  3. I didn't find the clear algorithm procedure illustrating your method, there are some vague descriptions in Remark 1 and algorithm paragraph above theorem 4.2, they are scattered and hard to follow, for example, i don't know how to select α\alpha in eq.(15), even if using method in Remark 1, but i still don't know for each iteration kk, the data you used to update α\alpha is the same batch to update θ^k+1\hat{\theta}_{k+1}? Or they are different? As a result, i suggest there should be a complete and compact algorithm procedure to clearly claim your idea.
Questions for authors
  1. I can't find the definition of Aˉ,αmin\bar{A}, \alpha_{min} and αmax\alpha_{max}, so update of α\alpha will less than αmin\alpha_{min} when Aˉ>0\bar{A}>0 , will that contradicts the fact α∈[αmin,αmax]\alpha \in[\alpha_{min} , \alpha_{max}]?
  2. Derivation of α\alpha update rules is vague,how does clip operator, EMA operator work in remark 1? How does δ\delta works? I believe the formula is wrong for there be no δ\delta in your update rule.
  3. How do you select α\alpha updating scheme given the 4 types in practice?
  4. I believe your main contribution is applying α\alpha-divergence family to generalize the SPO's setting, however, your paper pay a lot of attention to beat PPO method. My concern is how to show your advantage versus the SPO, why the proposed method outperforms it in theoretical or experimental aspects. It should be the key and i think the current manuscript is lack of evidence.
2 comments

We thank the reviewer for their attention to detail. We address the weaknesses and questions:

Weaknesses

  1. Eq. (3) and Eq. (4) are maximization objectives (the latter with a KL penalty), and Eq. (7) is a minimization objective; Figure 1 shows the clipped penalty. We use the notation by Schulman et al. (2017) and, to clarify, we make the direction of each objective explicit in the revised text.

  2. We do not claim that adapting α\alpha improves the return in theory. The theory guarantees the whole family: Theorem 4.1 holds for every α∈[αmin⁡,αmax⁡]\alpha \in [\alpha_{\min}, \alpha_{\max}] and any sequence αt\alpha_t, including one chosen online (p.6 l.273–274). SPO's χ2\chi^2 penalty is the special case α=2\alpha = 2. PPO has none: its clip does not bound the ratio under shared parameters (Section 3.1), and Theorem 4.2 fails for the clipped objective, which is constant past the clip (p.6 l.321–322). So adapting α\alpha is safe; the reason to adapt comes from Proposition B.2 (the preferable α\alpha rises with advantage noise) and from experiments where no constant α\alpha is best everywhere (Q4).

  3. We agree that an algorithm would help explain our methods better, and have added Algorithms 1 and 2 to Appendix G.

Questions

  1. Aˉ\bar{A}, ∣A∣‾\overline{|A|} and σA\sigma_A are the mean, mean absolute value and standard deviation of A^\hat{A} over the batch B\mathcal{B}, now defined at first use (p.4 l.203–205). [αmin⁡,αmax⁡][\alpha_{\min}, \alpha_{\max}] is the admissible range, [1,4][1, 4] on MuJoCo and Procgen and [1,3][1, 3] on MiniGrid (Table 5). The reviewer is right that (12a) falls below αmin⁡\alpha_{\min}; this was a typo on our end. As in the submitted code, αtB1=clip(αmin⁡−2Aˉ/(ϵ,∣A∣‾))\alpha^{\mathcal{B}1}_t = \mathrm{clip}\big(\alpha_{\min} - 2\bar{A}/(\epsilon\\,\overline{|A|})\big). We have fixed it and made it explicit in Algorithm 2. Figure 12 shows αt\alpha_t changing with the batch statistics within this range.

  2. In Appendix B, we choose α\alpha to reduce the batch penalty Ψ(α)=EB[∣A^∣fα(w)]\Psi(\alpha) = \mathbb{E}_{\mathcal{B}}[|\hat{A}| f_\alpha(w)], strictly convex in α\alpha. A Newton step gives αB=52−2M3M4\alpha^{\mathcal{B}} = \frac{5}{2} - \frac{2M_3}{M_4} with Mk=EB[∣A^∣(w−1)k]M_k = \mathbb{E}_{\mathcal{B}}[|\hat{A}|(w-1)^k] (Theorem B.1, Eq. (10)), which we use as a direction from αmin⁡\alpha_{\min}. As ww is unknown before the update, each rule replaces wi−1w_i - 1 by a proxy δi\delta_i, so Mk=EB[∣A^∣δk]M_k = \mathbb{E}_{\mathcal{B}}[|\hat{A}|\delta^k]:

    • αB1\alpha^{\mathcal{B}1}: δi=ϵ,sgn,A^i\delta_i = \epsilon\\,\mathrm{sgn}\\,\hat{A}_i gives M3/M4=Aˉ/(ϵ,∣A∣‾)M_3/M_4 = \bar{A}/(\epsilon\\,\overline{|A|}), hence (12a) has no δ\delta;
    • αB2\alpha^{\mathcal{B}2}: δi=ϵtanh⁡(A^i/σA)\delta_i = \epsilon\tanh(\hat{A}_i/\sigma_A), with Mktanh⁡=EB[∣A^∣δik]M_k^{\tanh} = \mathbb{E}_{\mathcal{B}}[|\hat{A}|\delta_i^k], now stated in Remark 1;
    • αB3\alpha^{\mathcal{B}3}: δi=ϵ(A^i−Aˉ)/m2\delta_i = \epsilon(\hat{A}_i - \bar{A})/\sqrt{m_2} with unweighted moments gives M3/M4=g1/(ϵg2)M_3/M_4 = g_1/(\epsilon g_2), with skewness g1g_1 and kurtosis g2g_2.

    The clip keeps α\alpha in [αmin⁡,αmax⁡][\alpha_{\min}, \alpha_{\max}], where Theorem 4.1 holds. The EMA zt=βzt−1+(1−β)xtz_t = \beta z_{t-1} + (1-\beta)x_t, β=0.9\beta = 0.9, is a convex combination, so it stays in range; it smooths α\alpha, while the return mostly does not depend on β\beta (Table 9). αtSNR=αmin⁡+(αmax⁡−αmin⁡)(1−qt)\alpha^{\mathrm{SNR}}_t = \alpha_{\min} + (\alpha_{\max} - \alpha_{\min})(1 - q_t) maps a reliability score qt∈[0,1]q_t \in [0,1] (Eq. (11)) to α\alpha, following Proposition B.2.

  3. We don't claim there is one optimal rule. In high-dimensional settings, the critic, the rollout distribution and the task difficulty change together, so the best rule cannot be established in general. For example, αB3\alpha^{\mathcal{B}3} achieves the highest median normalized score for continuous control tasks (0.934 in Table 1 and 0.992 in Table 7) and αSNR\alpha^{\mathrm{SNR}} achieves the highest mean normalized for Procgen at ϵ=0.5\epsilon = 0.5 (0.194, Table 2) and at ϵ=0.2\epsilon = 0.2, αB1\alpha^{\mathcal{B}1} is highest (0.313, Table 4), showing that the preferable rule depends on many confounding factors.

  4. Our goal is to show that no constant α\alpha, SPO's α=2\alpha = 2 included, solves all environments. With constant α∈1,2,3,4\alpha \in \\{1,2,3,4\\} (Table 7), the best α\alpha is 2–3 on HumanoidStandup, 3 on Humanoid and 4 on Walker2d and Ant. αB3\alpha^{\mathcal{B}3} matches it without tuning (0.992, against 0.982 for α=4\alpha = 4 and 0.841 for SPO). One of our rules is the best method on 6 of 8 MuJoCo tasks (refer to Table 1), 6 of 8 Procgen games at ϵ=0.5\epsilon = 0.5 (refer to Table 2) and 14 out of 16 at ϵ=0.2\epsilon = 0.2 (refer to Table 4). Theoretically, on the stationary interval, the leading term of Theorem 4.1(b) is the same for every α\alpha (p.6 l.270–273), so the theory does not single out α=2\alpha = 2.

We look forward to further comments from the reviewer in the remaining time of the discussion period. If there are no further questions, we appreciate if the score is updated to reflect this.

I appreciate your comprehensive response. I have raised my score to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.