- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PNPO, a method that stablizes the cumulative prefix ratio throught prefix-normalized policy weight. The motivation is raw cumulative-weight can reaching the order of during the training. To stablize it, PNPO applies geometric averaging, and use position adaptive clipping to accommodate the scale difference. The experiments show consistent improvement compared with CTPO and GSPO.
Critical strengths
- Identify an important issue in cumulative prefix ratio, the numerical unstability.
- Use a simple and classic geometric averaging to solve the unstability.
Critical weaknesses
- Section 2 takes a lot of space to discuss the deviation of state distribution and the bias of policy gradient, but the proposed method PNPO just adds the bias back to unbiased cumulative prefix ratio, which is confusing.
- All the experiments are conducted on one model and a fixed set of benchmarks. The generality of PNPO is questionable.
Questions for authors
- One of the baseline is CTPO from "Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective". In the Table 1 and 2 of this paper, the result shows CTPO consistently weaker than GRPO, but in the original paper that proposes CTPO, the performance of CTPO is significantly better than GRPO. I wonder which experiment setting causes this discrepancy and what is the key factor for PNPO to surpass other baselines?
- Can PNPO generalize to other settings like shorter response length, larger model, and different tasks?
2 comments
Dear Reviewer:
The rebuttal period is approaching its end. We have provided comprehensive point-by-point responses to all your comments and concerns and sincerely hope you can review them before the window closes. If you have any further questions, we stand ready to provide additional clarification as soon as possible. We appreciate your time and efforts.
The authors
We apologize for not clearly explaining the connection between our exact theoretical derivations and practical approximations. Thank you for your insightful questions and constructive suggestions, which helped us clarify this relationship. We address your concerns below.
Q1. Section 2 devotes substantial space to state-distribution shift and gradient bias, yet PNPO reintroduces bias through normalization. Are these two parts contradictory?
A1. Section 2 establishes exact policy-gradient relations, while Section 3 addresses the bias–variance trade-off in finite-sample training.
Q2. Can PNPO generalize to shorter responses, larger models, and different task settings?
A2. We added short-response experiments with a 7B VLM, using a response limit of 2,048 tokens. We train Qwen2.5-VL-7B-Instruct separately on ViRL39K and MMK12 and evaluate on MathVerse, MathVision, MathVista, and WeMath. Each method selects its best-validation checkpoint within the corresponding training budget. Four-benchmark mean Pass@1 (%) is reported below:
PNPO still outperforms GRPO and GSPO overall with short responses, while its advantages are more pronounced in our main experiments with a 16K length budget (§4.2, Fig. 3; §4.6, Table 4, Fig. 6). We believe this relates to policy differences accumulating along the generated prefix: state-distribution correction may become more beneficial as responses grow longer. Together, the short- and long-response experiments support PNPO’s optimization value and suggest potential for further gains on longer-horizon tasks.
Q3. Why does CTPO perform differently relative to GRPO than in its original paper? What enables PNPO to outperform other baselines?
A3. We consider response length an important factor in CTPO’s relative performance; PNPO’s advantage combines scale control with prefix information.
CTPO is effective only in short-response settings. Its original paper uses an 8,000-token trajectory limit, with mean response lengths of approximately 2,000–3,500 tokens. Our main experiments use a 16K context budget, with CTPO averaging approximately 7,400 tokens. Its clipping rate is not high, but adaptive thresholds do not eliminate the length-dependent scaling risks of raw cumulative weights. In the ViRL39K setting with PPO epochs = 2 and a 2,048-token response limit, CTPO learns effectively: its best-validation checkpoint achieves 56.41% four-benchmark mean external Pass@1, above GRPO’s 55.18% and slightly below PNPO’s 56.79% (App. D.12, Fig. 20).
Scale control supports stable optimization. With four PPO epochs in the main experiments, GRPO, CISPO, and DPPO-KL decline markedly late in training, while PNPO and GSPO, both using geometric normalization, remain stable. This supports the role of cumulative-weight scale control (§4.2, Fig. 3).
Prefix information further improves efficiency and performance. With geometric normalization and fixed clipping thresholds, PNPO and GSPO achieve 52.4% and 50.9% six-benchmark mean Pass@1 (§4.5, Table 3). Extending the text-only comparison (PPO epochs = 2) to 4,800 updates, twice the paper’s budget, PNPO exceeds 53% three-benchmark mean validation Pass@1 at 3,280 updates and reaches 55.20% at 4,320 updates; GSPO peaks at 52.96% within the same budget. The main and extended experiments jointly support the optimization value of position-wise prefix information over shared response-level weights (App. D.10, Fig. 17).
We hope the clarifications and new comparisons address your concerns about PNPO’s formulation, generalization, and gains. We would appreciate your reconsideration.