acceptodds
Under review as a conference paper at ICLR 2027

Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of Divergence

Abstract

Reinforcement learning has become the foundational paradigm for enhancing the reasoning capabilities of LLMs. Yet the dominant approach, Proximal Policy Optimization (PPO), relies on a geometrically degenerate clipping proxy that cannot adapt its resistance to policy shifts, leading to premature clipping and degraded exploration. Subsequent variants alleviate these issues through fragmented and heuristic modifications. We propose Generalized Proximal Policy Optimization (GPPO), a principled and unified framework for understanding and designing LLM RL algorithms through the lens of divergence. We establish a generalized -divergence trust-region theory with a monotonic performance improvement guarantee, and show that different -divergences induce distinct geometries and policy update dynamics. We further introduce novel estimator topologies that provide more faithful divergence estimates than the conventional single-sample Monte Carlo estimation. Under this framework, PPO and recent variants such as DAPO, DPPO, and RIPO emerge as special cases, establishing a principled two-dimensional design space of DivergenceEstimator. Extensive experiments on eight competition-level math and coding benchmarks show that both dimensions are critical. The best-performing GPPO instantiation surpasses standard GRPO by up to 50% and 14% on math and coding, delivering SOTA performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning with Verifiable Rewards, LLM RL, RLVR

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_75QQOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposed GPPO, a framework to unify PPO-like algorithms from a perspective of (divergence, estimation). By comparing many variants and existing RL algorithms on Qwen3-8B-Base, it shows the effectiveness of proposed framework.

Critical strengths

the paper tried to design/understand PPO-like algorithms at a higher abstraction level: by designing the f-divergence and estimators, it provides a unified lens to analysize PPO-like RL algorithms.

Critical weaknesses
  • Novelty and Insights: the novelty remains limited and the new insights from the paper remains unclear to me. From my understanding, DPPO and many other prior works have already established most of the foundations for this (divergence, estimators) view, including the TV/KL divergence with MC/Binary/Top-K estimators. Extending from TV/KL to f-divergence is meaningful, but a bit incremental. New insights are also expected beyond the unifed perspective. The zero/non-zero curvature seems plausible, but it's not convincing enough.
  • The Reliability of Empical Claims: although I respect the authors' effort, especially for the many algorihtms in evaluation, the reliability of the causal claims remains unclear to me. The concern mainly comes from hyperparameters of δ\delta. Because Aotmic and Binary estimators share the same δ\delta with MC, I'm not sure whether the performance gap is due to estimator choice, or hyperparameter choice. For example, for TV divergence, atomic estimation is 0.5∗∣π−μ∣0.5 * |\pi - \mu|, while Binary estimation is ∣π−μ∣|\pi - \mu|, they only differ in a constant scale. From Equation 19 and 20, the first term of Binary estimation is exactly the atomic estimation, so there may be a consistent shift between atomic and binary estimations especially when f(r)f(r) is always positive.
Questions for authors
  • The atomic estimation is confusing. For KL, f(r)=−log⁡rf(r) = - \log r, so the atomic estimation is −μlog⁡r- \mu \log r, which is always positive when r<1r < 1 and always negative when r>1r > 1. Given δ>0\delta > 0, does this mean KL with atomic estimation only mask those r<1r<1 tokens? What's the exact implementations?
  • Line 307 is confusing too. Given MC estimation for KL is −log⁡r- \log r, the proximal region should be f(r)≤δf(r) \leq \delta, which should be −log⁡r≤δ-\log r \leq \delta. So why we should contrain −δ≤log⁡r≤δ-\delta \leq \log r \leq \delta?
  • As noted in DPPO, the original TRPO bound (Theorem 1) is no longer valid in modern LLM RL due to γ=1\gamma =1 and group-based advantage estimation. The revised performance bound in DPPO is recommanded for mathematical rigor.
  • What's the meaning of notations like {$ r>0: f(r) \leq \delta $} for proximal regions?
  • The notations in Equation 10 and 11 are confusing. why Rf^(δ)\hat{R_f}(\delta) does not take rr as input, but using clip⁡Rf^(δ)(r)\operatorname{clip}_{\hat{R_f}(\delta)} (r)?
  • To be honest, I don't understand the curvature related claims in Section 3.2. What's the meaining of resistance in this section, and why adaptive resistance is important? I highly recommand rewrite this section to make it easier to follow.
  • Evaluations under various models will make the empicial results more solid. Sometimes conclusions may change for different models. So far there is only one evaluated model.
Reviewer_5PxhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Generalized Proximal Policy Optimization (GPPO), a framework that organizes clipping-based reinforcement learning for large language models along two axes: the divergence-generating function and the method used to estimate the divergence. It interprets PPO and several recent variants as instances of this framework and introduces probability-aware clipping through Atomic and Binary estimators. The paper also presents a generalized f-divergence trust-region performance bound and argues that divergence geometry explains differences in policy update dynamics. Experiments compare four divergences and three estimators using Qwen3-8B-Base on mathematical reasoning and code generation benchmarks. Reducing premature clipping of useful low-probability actions is practically relevant. However, the value of the proposed theoretical perspective depends on whether it correctly explains the implemented updates and provides reliable design guidance; the current KL instantiation is inconsistent with the stated framework.

Critical strengths
  1. A useful organization of probability-aware clipping methods. Section 3 provides a clear divergence-by-estimator design space for comparing several clipping rules. Section 3.3 also highlights an important distinction between a large change in the probability ratio and a small change in probability mass for a rare action. This motivates a potentially useful mechanism for preserving exploration and is the main conceptual strength of the paper.

  2. A systematic empirical comparison of design choices. Table 1 compares four divergences and three estimators across both mathematical reasoning and code generation tasks. This provides useful evidence that the choice of clipping configuration can materially affect performance and makes the proposed design space worth investigating empirically.

Critical weaknesses
  1. The theoretical framework contains unresolved inconsistencies. The KL generating function provides only a lower clipping bound, so the two-sided DAPO interval in Section 3.4 does not follow. Sample clipping also does not enforce the divergence constraint used in the theoretical analysis, and the TV threshold in Section 4.1 conflicts with Equation 14.

  2. The claimed mechanisms are insufficiently validated. In Equation 11, divergence affects updates only through clipping boundaries, leaving the benefit attributed to curvature unclear. Under behavior-policy sampling, the expectation of Atomic estimation uses squared action probabilities and does not equal the full divergence, so its claimed greater fidelity remains unproven.

Questions for authors
  1. What exact function was used in the KL experiments: −log⁡r-\log r, ∣log⁡r∣|\log r|, −log⁡r+r−1-\log r+r-1, or another function? Please provide the exact lower and upper clipping bounds for all MC, Atomic, and Binary configurations.

  2. What explicit relationship between the surrogate gain and the divergence penalty is required for monotonic improvement? How is this relationship checked or enforced during practical training?

  3. For configurations with identical final clipping intervals but different generating-function curvatures, are the gradient updates different in any way? If they are not, how should the geometric interpretation be revised?

  4. Can you report the bias, variance, and clipping-decision error rates of Atomic, MC, and Binary relative to the full-vocabulary divergence? This would help distinguish improved estimation quality from a relaxation of the constraint.

1 comment

Thank you for the clarifications and additional experiments; they address several concerns and strengthen the empirical evidence. However, my main concern about the mechanistic explanation remains. The revised interpretation clarifies how the clipping rules operate, but it still does not explain why they improve exploration. Since this explanation is central to the paper's contribution, I am keeping my rating at 2.

Reviewer_HgApOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Generalized Proximal Policy Optimization (GPPO), a framework that organizes proximal policy optimization methods along two dimensions: divergence choice and estimator topology. It derives a generalized ff-divergence trust-region performance bound and formulates clipping rules using Monte Carlo, Atomic, and Binary estimators, interpreting several existing algorithms as special cases. Experiments with Qwen3-8B-Base on eight mathematical reasoning and code generation benchmarks compare these design choices, with the KL–Atomic configuration achieving the strongest reported performance.

Critical strengths
  • This paper addresses a meaningful limitation of fixed ratio clipping in LLM RL. The relative probability ratio can hide the absolute probability change which can have different distributional effects for common and rare tokens.
  • The divergence ×\times estimator framework organizes multiple proximal policy optimization methods and enables a structural view to explore different clipping rules. This work also provides experiments on a group of divergences and estimators, showing their interaction effects.
  • The results on mathematical reasoning and code generation are promising.
Critical weaknesses
  • I notice that the constraint f(r)≤δf(r)\leq\delta implies only r≥e−δr\geq e^{-\delta} given f(r)=−log⁡rf(r)=-\log r, but the paper gives two-sided interval. This discrepancy requires clarification, especially because KL–Atomic is the best-performing configuration. Additionally, Under a∼μa\sim\mu, the expectation of μ(a∣s)f(r)\mu(a|s)f(r) generally differs from the full divergence. As a result, a smaller second moment does not establish a better bias–variance trade-off or more accurate divergence estimation. The paper should distinguish a probability-aware contribution constraint from an estimator of the full divergence.
  • The performance bound concerns the maximum state-wise divergence between complete policy distributions, whereas GPPO applies clipping based on sampled-token proxies. The paper does not establish that these clipping rules enforce the conditions required by the bound. Moreover, a small divergence constraint alone does not guarantee improvement unless the surrogate gain sufficiently offsets the divergence-dependent error term.
  • Entropy and training-score curves are consistent with improved exploration, but the gain is not sufficiently attributed. Analyses of clipping rates and probability changes across token-probability groups, alongside independent training seeds and uncertainty estimates, would make the conclusions more convincing.
  • This paper's contribution is more like a theoretical framework that provides a unified view to past clipping skills, but it lacks a sufficient discussion and positioning to prior work. BandPO[1] already derives probability-aware clipping intervals from general ff-divergence constraints and contains the same general Binary formulation. RIPO[2] also covers the χ2\chi^2–Atomic instance. The paper should discuss these overlaps explicitly and distinguish its contributions in unification, generalization, and new combinations from previously established mechanisms.

[1] Li, Yuan, et al. "BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning." arXiv preprint arXiv:2603.04918 (2026).

[2] Cai, Zhicheng, et al. "Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization." arXiv preprint arXiv:2607.10169 (2026).

Questions for authors
  • Could the authors clearly explain how we can get $ e^{-\delta}\le r\le e^\delta $ in KL-MC/KL-Atomic?
  • Could the authors clarify under what conditions the proposed sampled-token clipping objective guarantees monotonic policy improvement? The practical algorithm does not appear to enforce the divergence constraint of maximum state-wise divergence between full policy distributions.
  • Bounding divergence alone does not ensure a nonnegative performance lower bound unless the surrogate improvement offsets the divergence-dependent error term. How are these conditions satisfied by GPPO in practice?

Comments on the paper

2 comments

Thanks for the clarification and the additional experiments. They make the mathematical reasoning results more convincing, but I do have one follow up question: does solving Eq.(9) ensure the sufficient gain in Q3? Or is there additional conditions needed?

The authors basically addressed my technical concerns. I'm open to raise my score to 3, but with many preconditions. I didn't see an updated pdf, so please make the following updates (and I expect AC will check them).

  • Theoretical Bound to LLM RL: the theory justification should be tailored to undiscounted finite-horizon settings, with group-wise advantage definition instead of token-wise advantage definition. Can refer to DPPO for this.
  • A proper claim on the role of constraint scale: I strongly suggest a hyperparameter-tuning experiment to rule out the potential impact of different hyperparameters, especially for Atomic and Binary. As discussed, TV-Atomic and TV-Binary induce identical proximal regions. And basically, Binary has a larger magnitude than Atomic, which should be considered for hyperparamter selection.
  • A limitation section is required: although the authors claim it's a unified and generalized framework, it obviously cannot cover many commonly used techniques. For example, top-k estimator and multi-sample MC estimator.
  • All above promised modifications.

Overall, I don't think the proposed framework is significant, the novelty and new insights are kind of limited from my view. Although this doesn't necessarily lead to a reject, it requires more solid experiments to make readers learn enough.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.