acceptodds
Under review as a conference paper at ICLR 2027

When Policies Drift, Prefixes Matter: Prefix-Normalized Policy Optimization for LLM Reinforcement Learning

Abstract

In LLM reinforcement learning, the behavior policy can lag behind the current policy through asynchronous generation and updates or repeated rollout reuse; numerical differences between training and inference add further mismatch. Prox- imal methods such as PPO use local likelihood ratios to correct for conditional action-probability mismatch, but not explicitly for differences in prefix visitation induced by earlier decisions. In autoregressive generation, each prefix has a unique path, yielding an exact state–action change-of-measure ratio at each position: the product of local likelihood ratios through that position. However, this ratio can grow or decay rapidly with prefix length, hindering long-sequence optimization. We propose Prefix-Normalized Policy Optimization (PNPO), which geometrically averages local likelihood ratios over each causal prefix and constrains updates with position-adaptive clipping. Under equal optimizer-update budgets, PNPO outperforms GRPO and GSPO in mean Pass@1 across six mathematical reason- ing benchmarks across different policy-lag settings, with a larger advantage under high policy lag. With four PPO epochs, PNPO achieves 52.9% mean Pass@1, compared with 50.9% for GSPO and 50.7% for GRPO. PNPO and GSPO remain stable, whereas GRPO declines late in training. PNPO also matches its one-epoch performance using one quarter as many fresh rollout batches.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, large language models, policy optimization, importance sampling, policy lag, rollout reuse

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jHrROpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PNPO, a method that stablizes the cumulative prefix ratio throught prefix-normalized policy weight. The motivation is raw cumulative-weight can reaching the order of 10610^6 during the training. To stablize it, PNPO applies geometric averaging, and use position adaptive clipping to accommodate the scale difference. The experiments show consistent improvement compared with CTPO and GSPO.

Critical strengths
  1. Identify an important issue in cumulative prefix ratio, the numerical unstability.
  2. Use a simple and classic geometric averaging to solve the unstability.
Critical weaknesses
  1. Section 2 takes a lot of space to discuss the deviation of state distribution and the bias of policy gradient, but the proposed method PNPO just adds the bias back to unbiased cumulative prefix ratio, which is confusing.
  2. All the experiments are conducted on one model and a fixed set of benchmarks. The generality of PNPO is questionable.
Questions for authors
  1. One of the baseline is CTPO from "Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective". In the Table 1 and 2 of this paper, the result shows CTPO consistently weaker than GRPO, but in the original paper that proposes CTPO, the performance of CTPO is significantly better than GRPO. I wonder which experiment setting causes this discrepancy and what is the key factor for PNPO to surpass other baselines?
  2. Can PNPO generalize to other settings like shorter response length, larger model, and different tasks?
2 comments

We apologize for not clearly explaining the connection between our exact theoretical derivations and practical approximations. Thank you for your insightful questions and constructive suggestions, which helped us clarify this relationship. We address your concerns below.

Q1. Section 2 devotes substantial space to state-distribution shift and gradient bias, yet PNPO reintroduces bias through normalization. Are these two parts contradictory?

A1. Section 2 establishes exact policy-gradient relations, while Section 3 addresses the bias–variance trade-off in finite-sample training.

  • Unbiasedness is important, but it is not sufficient for stable optimization or strong final performance. Unbiasedness concerns whether an estimator recovers the true gradient in expectation; it does not guarantee low variance with finite samples. In long responses, products of raw prefix ratios can produce extreme weights, allowing a few samples to dominate updates and cause excessive policy shifts and training instability. PNPO therefore uses normalized cumulative weights, deliberately accepting some bias to mitigate the optimization difficulties caused by high variance. Figure 1(b) illustrates the scale risks of raw products, and Figure 5 shows that training with raw prefix weights does not sustain improvement (§1; §4.4). Thus, establishing exact theoretical forms and obtaining a trainable method through a practical bias–variance trade-off are complementary rather than mutually negating.

Q2. Can PNPO generalize to shorter responses, larger models, and different task settings?

A2. We added short-response experiments with a 7B VLM, using a response limit of 2,048 tokens. We train Qwen2.5-VL-7B-Instruct separately on ViRL39K and MMK12 and evaluate on MathVerse, MathVision, MathVista, and WeMath. Each method selects its best-validation checkpoint within the corresponding training budget. Four-benchmark mean Pass@1 (%) is reported below:

Training datasetBatch/minibatchPPO epochsGRPOGSPOPNPO
ViRL39K256/128157.3756.8557.45
ViRL39K256/64157.3056.3757.50
ViRL39K256/64255.1856.2056.79
ViRL39K256/64454.7355.2855.69
MMK12128/32250.7954.2155.50

PNPO still outperforms GRPO and GSPO overall with short responses, while its advantages are more pronounced in our main experiments with a 16K length budget (§4.2, Fig. 3; §4.6, Table 4, Fig. 6). We believe this relates to policy differences accumulating along the generated prefix: state-distribution correction may become more beneficial as responses grow longer. Together, the short- and long-response experiments support PNPO’s optimization value and suggest potential for further gains on longer-horizon tasks.

Q3. Why does CTPO perform differently relative to GRPO than in its original paper? What enables PNPO to outperform other baselines?

A3. We consider response length an important factor in CTPO’s relative performance; PNPO’s advantage combines scale control with prefix information.

  • CTPO is effective only in short-response settings. Its original paper uses an 8,000-token trajectory limit, with mean response lengths of approximately 2,000–3,500 tokens. Our main experiments use a 16K context budget, with CTPO averaging approximately 7,400 tokens. Its clipping rate is not high, but adaptive thresholds do not eliminate the length-dependent scaling risks of raw cumulative weights. In the ViRL39K setting with PPO epochs = 2 and a 2,048-token response limit, CTPO learns effectively: its best-validation checkpoint achieves 56.41% four-benchmark mean external Pass@1, above GRPO’s 55.18% and slightly below PNPO’s 56.79% (App. D.12, Fig. 20).

  • Scale control supports stable optimization. With four PPO epochs in the main experiments, GRPO, CISPO, and DPPO-KL decline markedly late in training, while PNPO and GSPO, both using geometric normalization, remain stable. This supports the role of cumulative-weight scale control (§4.2, Fig. 3).

  • Prefix information further improves efficiency and performance. With geometric normalization and fixed clipping thresholds, PNPO and GSPO achieve 52.4% and 50.9% six-benchmark mean Pass@1 (§4.5, Table 3). Extending the text-only comparison (PPO epochs = 2) to 4,800 updates, twice the paper’s budget, PNPO exceeds 53% three-benchmark mean validation Pass@1 at 3,280 updates and reaches 55.20% at 4,320 updates; GSPO peaks at 52.96% within the same budget. The main and extended experiments jointly support the optimization value of position-wise prefix information over shared response-level weights (App. D.10, Fig. 17).

We hope the clarifications and new comparisons address your concerns about PNPO’s formulation, generalization, and gains. We would appreciate your reconsideration.

Dear Reviewer:

The rebuttal period is approaching its end. We have provided comprehensive point-by-point responses to all your comments and concerns and sincerely hope you can review them before the window closes. If you have any further questions, we stand ready to provide additional clarification as soon as possible. We appreciate your time and efforts.

The authors

Reviewer_4c4hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies policy lag in LLM reinforcement learning caused by asynchronous training and rollout reuse. The authors argue that the local importance ratios in PPO/GRPO correct only the conditional action probabilities and ignore the shift in the prefix distribution. They propose PNPO, which uses the geometric mean of the likelihood ratios over each causal prefix as a position-wise weight, together with a position-adaptive gating interval to stabilize training; they also prove the equivalence of two exact off-policy policy-gradient representations. On mathematical reasoning with a 1.5B model, PNPO outperforms baselines such as GRPO and GSPO under high policy lag and trains more stably.

Critical strengths
  • Important problem with clear motivation. Policy lag is a practical pain point when reusing rollouts or training asynchronously. The paper clearly identifies the prefix-distribution shift that local ratios ignore, and Figure 1 conveys it intuitively.
  • Easy to implement. PNPO only requires a prefix cumulative sum of token log-ratios and a position-dependent gate, so it can directly replace the weighting and clipping in GRPO/GSPO with negligible overhead.
  • Consistent gains and stable training under high policy lag. With four PPO epochs, PNPO improves mean Pass@1 by 1.2–3.0 points over seven baselines, and the advantage holds in a three-seed comparison; it also remains stable while several baselines collapse late in training.
Critical weaknesses
  • Limited novelty; very close to GSPO/CTPO. PNPO essentially replaces GSPO's response-level geometric-mean ratio with a geometric mean over each prefix. The accompanying idea of scaling the clipping interval with token position was already proposed in CTPO, and when both are expressed on the same scale (the cumulative log-ratio).
  • The "prefix-distribution correction" narrative does not match the mechanism that actually drives the gains. The normalized weights are almost always ≈1 (P99 ≈ 1.0026 in Fig. 1b), so the improvement likely comes from the gating rather than from reweighting, but the paper lacks a gate-only ablation (w ≡ 1) to separate the two.
  • Limited experimental scope; some gaps may be within noise. All experiments use a single 1.5B distilled model on math tasks only, with no validation on larger models, MoE models, or other task types. The one-epoch gap between PNPO and GSPO (~0.6 points) come from single runs and are of the same order as the seed standard deviations in the multi-seed experiment (0.3–0.8 points).
Questions for authors

See weakness.

3 comments

We apologize for not sufficiently clarifying PNPO’s key distinctions from prior methods and its contributions in the original manuscript. Thank you for your careful comparisons with prior work and constructive suggestions. We have clarified these distinctions and added targeted ablations, and address your concerns below.

Q1. PNPO is very close to GSPO/CTPO: it merely replaces GSPO’s response-level geometric mean with a prefix-level geometric mean and adopts position-dependent clipping already proposed in CTPO. Where is the novelty?

A1. PNPO does not merely replace GSPO’s response-level averaging with prefix averaging and add position-dependent clipping; it addresses practical difficulties in state-distribution correction to make it effective for policy optimization. Specifically:

  • Core innovation: prefix normalization preserves information for position-wise state-distribution correction while controlling cumulative-weight scale and fluctuations. In autoregressive generation, each position corresponds to a different prefix state, whose distribution correction requires the cumulative ratio up to that position; GSPO’s shared response-level weight cannot express these position-wise differences. Yet raw cumulative ratios face scale risks that accumulate with length, which CTPO’s adjustment of clipping intervals does not eliminate. PNPO therefore uses prefix geometric normalization to control cumulative-weight scale and fluctuations while preserving causal prefix dependence, rather than heuristically changing GSPO’s averaging scope. In our long-response experiments, CTPO makes limited gains while PNPO continues to improve (§4.2, Fig. 3), demonstrating the practical optimization value of this design.

  • Independent theoretical contribution: we establish and prove the equivalence of two exact policy-gradient representations, specifying the pairing between advantages and score terms. Exact distribution correction does not automatically yield an unbiased policy-gradient estimator. §2.3 (Eq. (5), lines 179–197) and Appendix B.2 (Eq. (41), lines 972–985) provide these results, rather than merely restating the preliminary fact that prefix ratios are computable.

  • Position-dependent clipping is a supporting mechanism, not our claimed core innovation. Prefix geometric normalization scales cumulative log-ratios by 1/t, resulting in different weight fluctuation scales across positions; a shared fixed threshold therefore induces position-dependent clipping tendencies. Position-adaptive thresholds mitigate this imbalance. Even when both methods use geometric normalization and fixed clipping thresholds, PNPO achieves 52.4% six-benchmark mean Pass@1 versus 50.9% for GSPO, only 0.5 percentage points below full PNPO with position-adaptive clipping (52.9%) (§4.5, Table 3).

Continued in Part 2/2.

Continuation of our response (Part 2/2).

Q2. Since normalized weights are nearly 1, do the gains mainly come from gating rather than reweighting? The paper lacks a gate-only ablation that retains the gate and sets w ≡ 1.

A2. We added the requested pure gate-only comparison. In asynchronous training, we set the prefix-normalized policy weight to 1 only in the update; the gate still uses the original weights. Other settings are unchanged. The table shows mean reward over available records in each interval (difference from GRPO). Gate-only achieves higher stage-wise mean reward than GSPO and GRPO, supporting the optimization value of prefix gating itself. However, it performs similarly to PNPO only early on; full PNPO then achieves higher mean reward, with its mean advantage increasing further.

Mean reward (difference from GRPO) (see App. D.9, Fig. 15)

MethodSteps 1–40Steps 41–80Steps 81–130
GRPO−0.8362 (—)−0.5163 (—)−0.3188 (—)
GSPO−0.8344 (+0.0019)−0.5209 (−0.0046)−0.3816 (−0.0627)
Gate-only−0.7984 (+0.0378)−0.4471 (+0.0692)−0.2538 (+0.0650)
PNPO−0.8015 (+0.0348)−0.4309 (+0.0854)−0.2263 (+0.0926)
  • Weights close to 1 do not imply a negligible effect. Geometric normalization compresses the numerical scale while preserving whether each cumulative prefix ratio is above or below 1, so it still amplifies or attenuates update terms accepted by the gate. A small reweighting magnitude per update does not imply an equally small effect on learning: in online RL, policy updates and subsequent rollouts influence one another. GSPO also uses geometric normalization, illustrating that scale compression and effective optimization are compatible.

  • Existing comparisons and new ablations support the prefix term. Keeping the gate based on original weights but replacing the prefix-normalized policy weight in the update with the current-token importance ratio yields lower late-training reward than full PNPO (§4.4, Fig. 5). This retains action reweighting to test the additional role of state reweighting. Together with the earlier result that fixed-threshold PNPO outperforms GSPO, it supports the optimization value of the prefix term.

Q3. Generalization beyond the original model and tasks?

A3. We added Qwen2.5-VL-7B-Instruct experiments on two multimodal datasets. We report mean Pass@1 (%) on MathVerse, MathVision, MathVista, and WeMath at the best-validation checkpoint within each training budget:

Training datasetBatch/minibatchPPO epochsGRPOGSPOPNPO
ViRL39K256/128157.3756.8557.45
ViRL39K256/64157.3056.3757.50
ViRL39K256/64255.1856.2056.79
ViRL39K256/64454.7355.2855.69
MMK12128/32250.7954.2155.50

PNPO leads all five settings and generally has higher training reward (§4.6, Table 4, Fig. 6; App. D.11).

Q4. Could the roughly 0.6-point gap with one PPO epoch reflect random variation?

A4. The introduction describes the one-epoch results as comparable (§1, lines 98–101). With four PPO epochs, PNPO and GSPO achieve three-seed scores of 53.12±0.33% and 51.18±0.31%, respectively. Our point is that performance is similar at low lag, while PNPO’s advantage is clearer at higher lag; training dynamics show the same trend (§4.2, Table 2, Fig. 3).

We hope these clarifications and targeted ablations make PNPO’s contributions and the role of prefix weighting clearer. Thank you for considering this evidence when reassessing our work.

As a follow-up to our earlier responses, we would like to share new results from a PNPO–GSPO comparison on Qwen3-VL-30B-A3B. We compared PNPO and GSPO with a 16/16 training batch/minibatch size, four PPO epochs, and a 4,096-token response limit. All other settings followed the main text-only experiments.

For each trained method, we report the checkpoint with the highest mean validation accuracy across the three benchmarks (mean@32).

MethodAIME24AIME25AMC23MeanGain over Base
Base @030.00%28.54%60.31%39.62%—
GSPO47.50%35.73%87.34%56.86%+17.24 pp
PNPO50.00%39.48%90.08%59.85%+20.23 pp

We evaluated every 20 training steps. PNPO outperformed GSPO on all three benchmarks at six of the seven evaluation checkpoints.

If you have a chance before the discussion closes, we would appreciate your feedback on our earlier point-by-point responses and these additional results, especially if any concerns remain.

Reviewer_HF3UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies policy lag in LLM reinforcement learning, where the behavior policy used to generate rollouts differs from the current policy due to repeated rollout reuse or asynchronous training. The authors argue that token-level importance ratios in PPO/GRPO do not explicitly account for changes in prefix visitation distributions. They propose Prefix-Normalized Policy Optimization (PNPO), which replaces each token's importance ratio with the geometric mean of importance ratios over its causal prefix, together with position-adaptive clipping. Experiments on mathematical reasoning benchmarks show improved performance and training stability, especially when reusing rollout batches for multiple PPO epochs.

Critical strengths
  1. The paper studies an important problem. Improving training stability under policy lag is relevant for efficient LLM reinforcement learning, especially when rollout generation is expensive.

  2. The proposed method is simple and easy to implement. Prefix-normalized importance weighting and position-adaptive clipping introduce relatively little additional computational complexity.

  3. The paper includes useful ablation studies on prefix normalization, clipping strategies, and hyperparameter sensitivity.

Critical weaknesses
  1. The source of the performance improvement is not sufficiently understood. Prefix normalization effectively averages log importance ratios over preceding tokens, which may reduce extreme weight fluctuations. However, this does not necessarily imply lower gradient variance or better gradient estimation. The authors do not directly establish that the improvements come from correcting prefix distribution shift rather than simply smoothing importance weights. Comparisons with alternative smoothing strategies, such as sliding-window averaging or other normalized weighting schemes, would help clarify the contribution.

  2. The experimental settings are relatively limited. All main experiments use a single DeepSeek-R1-Distill-Qwen-1.5B model trained on DAPO-Math-17k, with evaluation restricted to mathematical reasoning benchmarks. Given that the paper focuses on training stability and policy drift, experiments on larger and more recent reasoning models, as well as other tasks, would provide stronger evidence for the generalizability of the findings.

  3. The evaluation of policy lag is not sufficiently comprehensive. The paper primarily increases the number of PPO epochs from one to four to introduce policy lag. However, the motivation also emphasizes asynchronous generation, stale trajectories, and training–inference mismatch, none of which are directly evaluated. It would be helpful to test PNPO under more severe policy lag, longer training runs, and realistic asynchronous or replay-based settings to establish whether its stability advantages persist.

Questions for authors

See weakness.

3 comments

We thank the reviewer for the careful review and insightful feedback, and for recognizing the importance of policy lag and the simplicity of PNPO. Your suggestions helped us strengthen the evidence on prefix weighting, generalization, and practical policy lag. We address each concern below.

Q1. Do PNPO’s gains come merely from normalization reducing extreme importance-weight fluctuations, rather than correcting prefix-distribution shift?

A1. Our comparison shows that position-wise prefix design offers additional optimization benefits when both methods use geometric normalization for scale control. We compare PNPO and GSPO with geometric-mean normalization and fixed clipping thresholds: PNPO uses position-wise causal prefix weights, whereas GSPO shares a response-level weight. This comparison retains normalization in both methods and removes position-adaptive thresholds as a difference. Under equal update budgets, PNPO achieves 52.4% six-benchmark mean Pass@1 versus 50.9% for GSPO, providing evidence for the optimization value of the prefix structure motivated by state-distribution correction (§4.5, Table 3).

Q2. Can comparisons with alternative normalization schemes, such as sliding-window averaging, further clarify the contribution of the full-prefix design?

A2. Thank you for suggesting sliding-window averaging. We added comparisons with window lengths of 8, 16, 32, and 64; full-prefix PNPO shows better training performance overall. We approximately align each window method’s initial clipping rate with PNPO’s. Dynamic clipping is used within the first W tokens of each response, followed by fixed thresholds once the window is full, where W is the window length. At window length 1, the sliding-window weight reduces to GRPO’s token-level importance ratio.

The table reports mean reward over available records in each interval, with differences from GRPO in parentheses. Positive differences indicate higher reward; the best value in each column is bold.

Stage-wise mean reward in sliding-window comparisons (see App. D.8, Fig. 14)

MethodSteps 1–40Steps 41–80Steps 81–130
Window-1 (GRPO)−0.8362 (—)−0.5163 (—)−0.3188 (—)
Window-8−0.8365 (−0.0003)−0.4925 (+0.0238)−0.2437 (+0.0751)
Window-16−0.8162 (+0.0200)−0.4556 (+0.0607)−0.2637 (+0.0551)
Window-32−0.8212 (+0.0150)−0.4670 (+0.0493)−0.2818 (+0.0370)
Window-64−0.7929 (+0.0434)−0.4491 (+0.0672)−0.2632 (+0.0556)
PNPO−0.8015 (+0.0348)−0.4309 (+0.0854)−0.2263 (+0.0926)

Full-prefix PNPO achieves the highest mean reward in both the middle and late stages, supporting its optimization advantage over the tested sliding-window alternatives. Early results also broadly show that smaller windows behave more like GRPO, with Window-8 performing similarly to GRPO.

Our response continues in Part 2/2.

Continuation of our response (Part 2/2).

Q3. Do experiments on a single 1.5B model and math tasks establish generalizability?

A3. We extend evaluation to a 7B VLM, multimodal inputs, and two training datasets. We train Qwen2.5-VL-7B-Instruct separately on ViRL39K and MMK12 and evaluate on MathVerse, MathVision, MathVista, and WeMath. Each method selects its best-validation checkpoint within the corresponding training budget. Four-benchmark mean Pass@1 (%) is reported below:

Training datasetBatch/minibatchPPO epochsGRPOGSPOPNPO
ViRL39K256/128157.3756.8557.45
ViRL39K256/64157.3056.3757.50
ViRL39K256/64255.1856.2056.79
ViRL39K256/64454.7355.2855.69
MMK12128/32250.7954.2155.50

PNPO has the highest four-benchmark mean in all five settings and generally higher training reward, providing evidence from both final evaluation and training dynamics. See §4.6, Table 4 and Fig. 6; per-benchmark scores and configurations are in App. D.11, Tables 10–12 and Figs. 18–19 (selection and evaluation protocol: lines 1625–1637).

Q4. Does PNPO retain its advantages under practical policy lag and longer training?

A4. New experiments support PNPO's optimization advantage under practical policy lag, as well as its learning efficiency and observed peak performance over longer training. The original analysis varies minibatch size and PPO epochs and measures policy divergence via K3 KL (§4.3, Fig. 4). We add:

(1) Asynchronous training. Sampling and updates run asynchronously, using trajectories from older policies with staleness threshold 8. Measured lag stabilizes at about 8 policy versions after startup. The table shows mean reward over available records in each interval (difference from GRPO). See App. D.9, Table 9 for configurations.

Stage-wise mean reward (difference from GRPO) (see App. D.9, Fig. 15)

MethodSteps 1–40Steps 41–80Steps 81–130
GRPO−0.8362 (—)−0.5163 (—)−0.3188 (—)
GSPO−0.8344 (+0.0019)−0.5209 (−0.0046)−0.3816 (−0.0627)
PNPO−0.8015 (+0.0348)−0.4309 (+0.0854)−0.2263 (+0.0926)

PNPO achieves the highest mean reward in all three stages, supporting better optimization under asynchronous policy lag.

(2) Longer training. The text-only comparison (PPO epochs = 2) extends 2,400 updates to 4,800, twice the original budget. The table summarizes the updates required to first exceed 53% three-benchmark mean validation Pass@1, the highest score within this budget, and its update count.

Text-only validation at twice the original update budget (PPO epochs = 2) (see App. D.10, Fig. 17)

MethodUpdates to first exceed 53%Peak three-benchmark mean (%)Updates at peak
PNPO3,28055.204,320
GSPONot reached52.964,800

PNPO attains, with fewer updates, a score GSPO never reaches within the same budget, demonstrating advantages in learning efficiency and observed peak performance. Across all 30 common evaluation points beyond the original budget, PNPO has a higher three-benchmark mean, averaging 2.21 percentage points ahead.

(3) Training–inference mismatch. We also enable bypass on top of four PPO epochs, directly using sampling-backend behavior probabilities as the importance-ratio denominator so backend mismatch enters updates. GSPO's training reward collapses late in training, whereas PNPO retains its optimization advantage (App. D.9, Fig. 16).

We hope the additional comparisons and broader evaluations address your concerns about the source of PNPO’s gains and its generalizability. We would appreciate your reconsideration of our work in light of these results.

Dear Reviewer:

The rebuttal period is approaching its end. We have provided comprehensive point-by-point responses to all your comments and concerns and sincerely hope you can review them before the window closes. If you have any further questions, we stand ready to provide additional clarification as soon as possible. We appreciate your time and efforts.

The authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revised Manuscript and Summary of Added Experiments

Dear Reviewers and Area Chair,

Thank you for your valuable feedback. We have uploaded the revised manuscript incorporating additional experiments and evaluation details addressing your comments. For ease of reference, the additions are summarized below:

  1. Asynchronous training and training–inference mismatch: We add asynchronous sampling and updates, alongside a separate bypass comparison, to evaluate PNPO under practical policy lag and backend mismatch (§4.7; Appendix D.9, Table 9 and Figs. 15–16).

  2. Larger-model and short-response generalization: We add five configurations using a 7B vision-language model on ViRL39K and MMK12, including aggregate scores, per-benchmark results, and training dynamics (§4.6, Table 4 and Fig. 6; Appendix D.11, Tables 10–12 and Figs. 18–19).

  3. Sliding-window comparisons: We add a Window-8 experiment and asynchronous comparisons with window lengths of 8, 16, 32, and 64 to examine the optimization value of the full-prefix design relative to alternative smoothing schemes (Appendix D.8, Figs. 13–14).

  4. Pure gate-only ablation: We retain the original prefix gate and set the prefix-normalized policy weight to 1 only in the update, separating the effects of gating and reweighting (Appendix D.9, Fig. 15).

  5. Extended training: We extend the text-only comparison to 4,800 optimizer updates, twice the original budget, reporting three validation benchmarks and their mean (Appendix D.10, Fig. 17).

  6. Short-response CTPO comparison: We add training-reward curves and four-benchmark external evaluation results to address the difference in CTPO’s relative performance from its original paper (Appendix D.12, Fig. 20).

  7. State-reweighting ablation: A control retaining prefix gating and action reweighting has been added to the existing Fig. 5 to test the additional role of state reweighting (§4.4, Fig. 5).

  8. Evaluation procedures and complete prompts: We provide the evaluation procedures following the official benchmark repositories’ criteria, the judge model, and three complete prompts (Appendices D.11 and D.13).

As the discussion period draws to a close, we would be grateful if you could review our point-by-point responses and revised manuscript before the window closes, and let us know whether any concerns remain. If you have further questions, we are ready to provide additional clarification promptly. Thank you again for your time and efforts.

The Authors

Dear Area Chair,

we have posted point-by-point responses to all three reviews and uploaded a revised manuscript. Unfortunately, we have not yet received any follow-up from the reviewers. We value the opportunity to learn from their perspectives and would be grateful if you could encourage them to discuss our responses and new results during the internal discussion. Thank you for your time and for coordinating the review process.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.