- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper revisits the clipping mechanism of PPO and proposes ALDIPPO, which replaces the clipped surrogate with an advantage-weighted alpha-divergence penalty on the importance ratio. The parameter alpha controls the skewness of the penalty, and the paper proposes several online rules that adapt alpha using batch statistics such as advantage moments, signal-to-noise ratio, and critic explained variance. The paper also provides monotonic-improvement and sample-complexity analyses. Experiments on MiniGrid, MuJoCo, and Procgen show improved ratio control and generally better robustness than PPO, particularly with deeper policies and larger clipping thresholds.
Critical strengths
-
The paper provides a clear analysis of why PPO clipping may fail to control importance ratios under shared parameterization. The stationary-ratio analysis and the interpretation of alpha as controlling the skewness of the penalty provide an intuitive connection between PPO-style trust-region control and the alpha-divergence family.
-
The mechanism-level experiments are well aligned with the motivation. In particular, the MiniGrid experiments directly examine importance-ratio deviation under different network depths and clipping thresholds instead of relying only on downstream return. As the policy network becomes deeper, PPO exhibits increasingly large ratio deviations, whereas ALDIPPO remains substantially more controlled.
-
The paper includes substantial theoretical and empirical analysis. It provides monotonic-improvement and sample-complexity results and evaluates several adaptive alpha rules across MiniGrid, eight MuJoCo tasks, and Procgen. In the challenging deep-policy MuJoCo setting, ALDIPPO substantially improves over PPO and is competitive with or better than SPO.
Critical weaknesses
-
The novelty boundary is relatively narrow because both the motivating problem and the use of alpha-divergence in PPO have substantial prior work. The fact that PPO clipping does not strictly constrain importance ratios has already been explicitly studied by prior work such as Truly PPO, while SPO already replaces clipping with a non-zero-gradient penalty and demonstrates advantages for deep policies. More importantly, Xu et al. (2023) already proposed an alpha-divergence-based PPO method, which the paper itself identifies as its closest prior work. Therefore, the main incremental contribution appears to be the online adaptation of alpha and the specific per-sample formulation, rather than introducing alpha-divergence into PPO itself.
-
The central claimed contribution, adaptive alpha, is not adequately isolated experimentally. The experiments compare the proposed adaptive rules mainly against PPO and SPO, where SPO corresponds to the particular fixed choice alpha = 2. However, there is no systematic comparison against several fixed alpha values, a validation-selected best fixed alpha, or the closest alphaPPO baseline. This is particularly important because the paper's monotonic-improvement theorem holds for any alpha sequence, including a constant alpha, and the paper itself notes that the theorem does not establish that changing alpha improves return. Therefore, the current experiments do not clearly demonstrate whether the gain comes from adaptive alpha or simply from using a more suitable fixed penalty geometry than alpha = 2.
-
Some of the robustness claims appear to depend on deliberately difficult settings and additional tuning. The strongest MuJoCo results are obtained with seven-layer policies, a regime already known to be problematic for PPO. In the standard two-layer setting, the results are considerably more mixed, and PPO outperforms all penalty-based methods on some tasks such as Walker2d and HalfCheetah. In addition, the original alpha-SNR rule collapses at epsilon = 0.5 on Procgen and requires a lower alpha ceiling and retuned gate constants. This weakens the claim that adaptive alpha itself provides robust hyperparameter adaptation.
-
There is a nontrivial gap between the theoretical algorithms and the practical implementation. The monotonic-improvement theorem assumes bounded importance ratios, while the paper acknowledges that shared parameters can cause practical ratios to leave the stationary interval. The sample-complexity theorem analyzes a projected trust-ball algorithm with a constant penalty weight and a specific unbiased sampler, rather than the practical ALDIPPO update using advantage-dependent weighting and standard PPO-style optimization. Therefore, the theoretical results are informative but do not directly establish guarantees for the implemented algorithm. Moreover, although the paper states that an anonymized code archive is attached, I could not find or inspect the concrete implementation code in the materials available to me, which further limits verification of the practical algorithm.
Questions for authors
-
Xu et al. (2023) already apply alpha-divergence to PPO, while SPO corresponds to a fixed alpha = 2 penalty. Could the authors clarify more precisely what constitutes the main methodological novelty of ALDIPPO beyond these prior approaches?
-
Could the authors provide a controlled comparison between the proposed adaptive alpha rules and multiple fixed alpha values, including a validation-selected best fixed alpha and ideally alphaPPO? Without this comparison, it is difficult to determine whether the improvement is caused by online adaptation itself.
-
The monotonic-improvement theorem holds for constant alpha as well as adaptive alpha. Is there theoretical or empirical evidence that the proposed batch statistics reliably predict which alpha value is preferable at a given update?
-
How should the theoretical guarantees be interpreted for the practical implementation, given that the sample-complexity analysis uses projection, a constant penalty weight, and a special unbiased sampler that are not used in the reported practical algorithm?
2 comments
Thank you for your comprehensive response. I have raised my score to 3.
We thank the reviewer for the detailed and thoughtful review. We address the weaknesses and questions:
Weaknesses
Our starting point is the nonstationarity of on-policy learning (Section 3.2): rollouts can be far from earlier ones, and advantages are nonstationary, so the penalty should adapt to the batch. No prior method chooses the divergence from the data or adapts it as the policy learns. PPO's KL penalty is fixed; SPO coincides with using a fixed α=2, so our method is a strict generalization of it; Xu et al. grid-search one α per environment before training (Gaussian policies only); Truly PPO keeps the clip and adds a rollback term. Our per-sample penalty, weighted by ∣A^∣/ϵ, sets the stationary ratio by ϵ for every α (Proposition 3.1), so α can change without changing the trust region. The rules are derived from the penalty (Theorem B.1, Proposition B.2), and Theorems 4.1–4.2 cover the whole family. The preferable α does depend on the problem (Table 7, Figure 12).
In Table 7 we add fixed α∈1,2,3,4, a validation-selected α, αPPO with α∈0,0.5 (b chosen as in Xu et al., Table 8) and KL-penalty PPO. The best fixed α changes across environments: 2–3 on HumanoidStandup, 3 on Humanoid, 4 on Walker2d and Ant. αB3 scores 0.992 without tuning, against 0.982 for the best fixed α=4. The average α for αB3 lies between 1.2 and 1.9, between fixed α=1 and 2, yet it beats both on three tasks and ties on the fourth, so the gain comes from varying α. αPPO and KL-penalty PPO reach median normalised scores of 0.018–0.095, with KL far above SPO (refer to Figure 11).
Deep policies and large ϵ are harder settings, but natural: RL post-training of billion-parameter language models uses very deep policies, where the clip failure we study matters most. We establish this objective in standard RL with guarantees, and mention LLM fine-tuning as direct future work. With two-layer policies, PPO is only best on Walker2d and HalfCheetah (refer to Table 6). Procgen at ϵ=0.5 is a stress test: at 2.5× the usual radius, far more samples leave the clip range (~80% of PPO's already do at ϵ=0.2, Figure 14). A single recalibration of the αSNR gate and ceiling (refer to Table 3) turns them into stable updates (refer to Table 2).
Part (a) of Theorem 4.1 needs only bounded ratios (Assumption 1, p.16), while Part (b) needs the stationary interval (Assumption 2), which shared parameters can violate. We measured how often this happens on the four MuJoCo environments. We show in Figure 13 that the mean ∣w−1∣ is 0.08–0.10 against 1.35–1.57 for PPO. In Figure 14, we show that only ~15% of samples exceed ϵ against ~80% of PPO.
Questions
1–2. See weaknesses 1–2.
Theorem 4.1 provides a monotonic improvement guarantee for every α. With deep policies and nonstationary learning, the critic, the rollout distribution and the task difficulty change together, so the best rule cannot be established in general. We do not claim a superior per-update selector; we observe that the preferable α depends on the environment (Table 7, Figure 12), and αB3 matches the best fixed α without tuning.
The three techniques referred to are the standard ingredients of the analyses we build on. The sampler is used in Algorithm 3 of (Agarwal et al. 2021), which makes the batch surrogate unbiased. The constant weight is TRPO's constant penalty, recovered by Lemma C.4; it makes the penalty curvature (a0/ϵ)Fρ for every α, so the update is an inexact natural-gradient step (Lemma D.2, Eq. (31)). The projection is ProjW of Agarwal et al. (Algorithm 4) in parameter units; it keeps ratios in [1−cwϵ,1+cwϵ], giving strong concavity uniformly in α. Empirical algorithms like TRPO also drop its penalty, and the ratios still stay in this interval (refer to Figures 13–14). The new element is PPO-style multi-step optimisation of one batch of a nonlinear surrogate; its maximizer is biased, and the Lipschitz Hessian (Assumption 5) bounds this bias by O(1/N) (Lemma D.3, Eq. (61)), giving the NPG rate O(εˉ−3) of Liu et al. (2020) instead of the policy gradient one O(εˉ−4). Please see Appendix G for details of ALDIPPO algorithm.
We look forward to further discussion from the reviewer in the remaining time of the discussion period. If there are no further questions, we appreciate if the score is updated to reflect this.