- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposed GPPO, a framework to unify PPO-like algorithms from a perspective of (divergence, estimation). By comparing many variants and existing RL algorithms on Qwen3-8B-Base, it shows the effectiveness of proposed framework.
Critical strengths
the paper tried to design/understand PPO-like algorithms at a higher abstraction level: by designing the f-divergence and estimators, it provides a unified lens to analysize PPO-like RL algorithms.
Critical weaknesses
- Novelty and Insights: the novelty remains limited and the new insights from the paper remains unclear to me. From my understanding, DPPO and many other prior works have already established most of the foundations for this (divergence, estimators) view, including the TV/KL divergence with MC/Binary/Top-K estimators. Extending from TV/KL to f-divergence is meaningful, but a bit incremental. New insights are also expected beyond the unifed perspective. The zero/non-zero curvature seems plausible, but it's not convincing enough.
- The Reliability of Empical Claims: although I respect the authors' effort, especially for the many algorihtms in evaluation, the reliability of the causal claims remains unclear to me. The concern mainly comes from hyperparameters of . Because Aotmic and Binary estimators share the same with MC, I'm not sure whether the performance gap is due to estimator choice, or hyperparameter choice. For example, for TV divergence, atomic estimation is , while Binary estimation is , they only differ in a constant scale. From Equation 19 and 20, the first term of Binary estimation is exactly the atomic estimation, so there may be a consistent shift between atomic and binary estimations especially when is always positive.
Questions for authors
- The atomic estimation is confusing. For KL, , so the atomic estimation is , which is always positive when and always negative when . Given , does this mean KL with atomic estimation only mask those tokens? What's the exact implementations?
- Line 307 is confusing too. Given MC estimation for KL is , the proximal region should be , which should be . So why we should contrain ?
- As noted in DPPO, the original TRPO bound (Theorem 1) is no longer valid in modern LLM RL due to and group-based advantage estimation. The revised performance bound in DPPO is recommanded for mathematical rigor.
- What's the meaning of notations like {$ r>0: f(r) \leq \delta $} for proximal regions?
- The notations in Equation 10 and 11 are confusing. why does not take as input, but using ?
- To be honest, I don't understand the curvature related claims in Section 3.2. What's the meaining of resistance in this section, and why adaptive resistance is important? I highly recommand rewrite this section to make it easier to follow.
- Evaluations under various models will make the empicial results more solid. Sometimes conclusions may change for different models. So far there is only one evaluated model.
Thank you for the clarifications and additional experiments; they address several concerns and strengthen the empirical evidence. However, my main concern about the mechanistic explanation remains. The revised interpretation clarifies how the clipping rules operate, but it still does not explain why they improve exploration. Since this explanation is central to the paper's contribution, I am keeping my rating at 2.