- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the limitations of fixed clipping in large language model reinforcement learning when the training and rollout policies differ. The authors adapt the existing P3O algorithm to LLM post-training, using the normalized effective sample size (ESS) of token-level importance ratios to determine both the cap on policy-gradient weights and the strength of regularization toward the behavior policy. Experiments examine clipping sensitivity, sampling temperature, mixed-precision rollouts, and asynchronous training. P3O performs comparably to tuned GRPO in some settings and avoids GRPO’s late-stage collapse in the Qwen3-8B FP8-rollout experiment in some configurations.
Critical strengths
The key strength of this paper is replacing fixed clipping thresholds, which require tuning across different training settings, with a batch-adaptive mechanism based on effective sample size (ESS). By using ESS to jointly control importance-weight capping and regularization strength, P3O provides a unified objective for handling varying degrees of policy mismatch without manually selecting a clipping range.
Critical weaknesses
- The method has limited novelty and insufficient insight because the core component (ESS) is inherited from prior works. This paper mainly adapts ESS to LLM training, but lacks a systematic understanding of why it works and when it is beneficial.
- The paper lacks explanation on why the method works and why it might fail. The results support the effectiveness of the objective in some configurations, but do not establish that its batch-adaptive mechanism is responsible for the gains. The inconsistent outcomes across temperatures and models make this explanatory gap particularly important. More effort should be devoted to explaining the observations from the experiments rather than simply displaying the results.
Questions for authors
- Figure 1 shows that the commonly used clipping threshold performs substantially worse than other choices in GRPO and P3O. Could you provide explanation on why this happens, and how different clipping thresholds in standard algorithms affect the stability of training?
- Figure 2 shows that P3O remains stable at higher temperatures but collapses at lower temperatures. What is the reason for this phenomenon and is it a limitation of this method?
- The experiments show that P3O can prevent training collapse, but do not explain why GRPO collapses while P3O remains stable. Could the authors analyze the evolution of policy mismatch, importance-ratio distributions, ESS, and gradient statistics throughout training?
- The FP8 experiments attribute P3O's stability advantage to robustness against precision mismatch, but lack matched BF16-rollout controls and comparisons with GRPO under different clipping thresholds. Moreover, GRPO collapses on Qwen3-8B but not on Qwen2.5-1.5B, without an explanation for this discrepancy. Could you include BF16-versus-FP8 rollout comparisons on the same models and analyze the actual probability mismatch?