acceptodds
Under review as a conference paper at ICLR 2027

Policy Lag in GRPO: Group-Estimator Bias, Optimizer Memory, and Delayed Feedback

Abstract

Policy lag makes a learner optimize older-policy responses while rollout reward evaluates that older policy. We diagnose this mismatch in group relative policy optimization (GRPO) through its group-gradient estimator, optimizer memory, and feedback channel. In a ten-arm study with 20 paired seed blocks and a 16-update refresh interval, token-level truncated importance sampling (TIS) reduces confirmed terminal rollout failures from 12/20 to 2/20 and raises mean final-learner accuracy from 22.7% to 57.9%. Selective resets identify the first moment as the strongest reset target, and local norm-matched controls favor its update direction. We derive the cross-response bias that survives response-level importance correction and use exact enumeration to separate directional alignment, estimator bias, and finite-batch error. In a disjoint monitoring cohort with 30 development and 50 held-out runs, current-learner evaluation gives 28/35 timely alerts versus 18/35 from rollout reward, with one false alert among 13 no-episode runs for each. These findings distinguish correcting stale-data updates, managing optimizer history, and observing current-policy quality.

Primary area:
reinforcement learning
Keywords:
large language models, off-policy reinforcement learning, group relative policy optimization (GRPO), policy lag, importance sampling, group-estimator bias, optimizer memory, delayed feedback

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EQqWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies policy lag in asynchronous GRPO by separating its effects into stale-data estimator bias, optimizer memory, and delayed feedback. Through controlled policy-lag experiments, it shows that token-level truncated importance sampling and first-moment Adam resets substantially reduce training failures, although combining them provides little additional benefit. The paper further derives the cross-response bias that remains after response-level importance correction because GRPO advantages use shared group normalization, and analyzes the resulting bias–variance trade-off through exact enumeration. Finally, it shows that directly evaluating the current learner detects degradation earlier than rollout reward.

Critical strengths

Systematic diagnosis of policy lag. The paper separates policy lag into estimator bias, optimizer memory, and delayed monitoring, and studies them in a controlled setup with paired runs. This provides a clearer picture of why stale-rollout training fails than treating policy lag as a single issue.

The paper separately ablates Adam’s first moment, second moment, and step counter, and further distinguishes the direction and magnitude of the first moment. This provides stronger evidence that stale momentum direction is a major contributor to training instability, rather than merely showing that optimizer reset helps.

Critical weaknesses

Limited experimental scope. Experiments are conducted on GSM8K or MATH. It is unclear whether the findings generalize to more tasks, or realistic asynchronous training with changing levels of staleness.

Limited novelty and baseline comparison. Importance sampling, stale-rollout filtering, and optimizer reset have all been explored before. The paper’s main contribution is its analysis, but comparisons with recent policy-lag methods are limited.

Questions for authors

How do TIS and first-moment reset compare empirically with recent policy-lag methods such as VCPO and sequence- or prefix-level importance sampling?

TIS and first-moment reset are meant to address different sources of policy lag, but combining them does not help further. Why might this be the case? Could they be fixing the same underlying failure mode?

What is the practical benefit of current-learner monitoring? Can acting on its alerts prevent or recover from training failures, or does it only tell us earlier that the learner has already degraded?

5 comments
AuthorsOpenReview ↗

Gentle Reminder: Follow-up on Our Response to Reviewer EQqW

Dear Reviewer EQqW,

Hope this message finds you well. Thank you for your valuable time reviewing our work. We have submitted our point-by-point rebuttal to address all of your concerns and questions. As the discussion deadline is approaching, we are writing to kindly check whether our response has addressed your questions. Please feel free to let us know if you have further comments.

Thank you again for your efforts. Best regards, The Authors

Thank you for the detailed response and additional experiments. I appreciate the new baseline comparisons, direction–norm ablations, and monitoring-triggered recovery results, which address several of my questions and strengthen the empirical evidence. However, my concerns about the novelty of the contribution and its generalizability beyond the tested mathematical tasks and controlled training settings remain insufficiently resolved.

Thank you for clarifying your remaining concerns. On novelty. Our contribution is to show precisely what an exact response-level importance weight recovers in finite-group GRPO. The weight corrects for the policy that generated the response being scored. Its advantage, however, also depends on other responses to the same prompt, which still come from the older policy. Equation (10) isolates the first-order bias this can leave relative to the mean of the same estimator on fresh, on-policy groups. Section 4.2 shows why that distinction matters. At both endpoints of our four-action example, the weighted mean update points in exactly the on-policy direction, yet its magnitude is wrong. A gradient-alignment check would therefore miss the remaining error in both cases.

Reviewer_Z38dOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper controls behavior-policy refresh intervals within a synchronous veRL training loop. The primary configuration is Qwen2.5-1.5B-Instruct on GSM8K, with K=16K=16, 300 learner updates, and 20 paired seed blocks. Each generated batch supplies one full-batch optimizer update, without response reuse. Ten conditions compare uncorrected training, token-level truncated importance sampling (TIS), an M2PO-style correction rule, selective Adam-state resets, and local direction/norm controls.

TIS reduces confirmed terminal rollout failures from 12/20 to 2/20 and increases mean final-learner accuracy from 22.7% to 57.9%. First-moment reset gives 3/20 failures and 50.8% accuracy. The local bias expansion shows that response-level importance sampling cancels the first-order self-response term while retaining cross-response terms induced by shared group statistics. Exact finite-state calculations distinguish this residual bias from direction and finite-batch MSE. A separate cohort of 30 development and 50 held-out runs compares current-learner evaluation with stale rollout feedback, using monitoring rules frozen before held-out evaluation.

Critical strengths
  1. The decomposition is clear and practically relevant. Behavior-policy refresh, estimator correction, and optimizer reset act on different objects. Rollout reward can accurately characterize the generating policy while failing to describe the current learner. The paper translates these distinctions into separate interventions and measurements.

  2. The primary statistical design is useful. All ten conditions share 20 paired seed blocks; final-accuracy averages include failed runs and unconfirmed terminal tails. Bootstrap resampling uses whole blocks. Relative to uncorrected training, TIS improves final accuracy by 35.2 percentage points, with a pointwise 95% paired bootstrap interval of [18.3, 52.5], and changes confirmed failure incidence by −50 percentage points, with interval [−70, −30] (Tables 1–2).

  3. The selective resets and local direction/norm controls provide more information than a full-reset comparison alone. In the primary configuration, first-moment reset performs better than the other single-state resets. The controls compare candidate updates at a common instantaneous state within each trajectory and retain the downstream feedback caused by the applied rule.

  4. The theoretical reference is carefully defined. uGu_G is the on-policy mean of the same finite-group estimator, preserving its normalization and gradient reduction; it is generally not the expected-reward gradient. Proposition 1, Corollary 1, and Appendix H correctly distinguish the sampling-distribution score, behavior-policy dependence of response weights, and full-group change of measure. Under sequence-first reduction, the contribution includes response-length normalization while the sampling score remains the full-response score.

  5. The conclusions are appropriately qualified. The paper distinguishes terminal failures from recoverable sustained episodes and describes monitoring as passive detection of realized degradation. Additional model/task and lag configurations show reductions in observed confirmed-failure counts under TIS, although their magnitude varies.

Critical weaknesses

1 The practical scope of the single-update regime and its relationship to prior work need clarification

Section 2 and Appendix A.2 specify a detached pre-update learner as the proximal policy, with one full-batch update per generated batch. For a sampled token yy at prefix hh, the ratio is

\rho(\theta_{t-1};y,h)=1,$$ where $\pi_{\mathrm{prox}}=\pi_{\theta_{t-1}}$ is held fixed during differentiation. Proximal clipping therefore does not alter the policy-gradient contribution at this single evaluation point, although the ratio's derivative still supplies the policy score. Global gradient clipping and the M2PO-style mask are separate operations. The detached TIS weight instead uses the behavior-policy mismatch: $$w(y,h)=\min\left\\{\frac{\pi_{\mathrm{prox}}(y\mid h)}{\pi_{\mathrm{beh}}(y\mid h)},\\,2\right\\}.$$ This regime helps isolate correction and optimizer memory. However, the reduction from 60% to 10% confirmed failures is relative to this particular uncorrected baseline. It does not establish the same intervention ranking when behavior-ratio or proximal clipping actively changes the gradient, or when responses are reused for multiple updates. The M2PO-style arm evaluates behavior-ratio weighting combined with adaptive masking, rather than an ordinary active-clipping control. The contribution also needs more direct positioning relative to [Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm](https://proceedings.iclr.cc/paper_files/paper/2026/file/6cad062f78120689e8a678b8af9c6d92-Paper-Conference.pdf) and [Revisiting Group Relative Policy Optimization](https://proceedings.iclr.cc/paper_files/paper/2026/file/b4008025c2182bfe16fcc8566ee14d64-Paper-Conference.pdf), both published at ICLR 2026. These works examine off-policy group-relative objectives, importance weighting, and clipping from different perspectives. Their targets and assumptions differ from the finite-group $u_G$ studied here. Please explain what the present cross-response bias decomposition, optimizer-memory interventions, and controlled lag study add to those results. Explicitly limiting the practical recommendation to the studied regime would largely address the scope concern. If broader claims are intended, a limited experiment with active clipping would strengthen the evidence. Such an experiment is not a prerequisite for my current recommendation. **2 The direction/norm comparison supporting the mechanistic interpretation lacks paired uncertainty estimates** Equation (4) and Appendix B define two useful controls. For nonzero candidate norms, $$u_t^D=\frac{\lVert u_t^C\rVert_2}{\lVert u_t^R\rVert_2}u_t^R,\qquad u_t^N=\frac{\lVert u_t^R\rVert_2}{\lVert u_t^C\rVert_2}u_t^C.$$ Here $u_t^C$ and $u_t^R$ are the carried- and reset-first-moment candidate updates evaluated at the same instantaneous state within a trajectory. The first control uses the reset-derived direction at the carried-update norm; the second uses the carried direction at the reset-update norm. Appendix B specifies the zero-norm fallback. Table 1 reports 3/20 confirmed failures and 50.9% final accuracy for the first control, compared with 9/20 and 26.6% for the second. The first-minus-second differences are −30 percentage points in failure incidence and +24.3 percentage points in mean accuracy. These point estimates favor the reset-derived direction. However, Table 2 and Appendix D do not report the direct paired uncertainty for this comparison, despite its role in the abstract's mechanistic claim. The interval for first-moment reset versus uncorrected training cannot substitute for it, and marginal failure counts alone do not determine paired uncertainty. Please report the paired failure contingency table, failure-rate difference and interval, and final-accuracy difference and interval across the existing 20 blocks. This requires no additional training. The interpretation should retain the authors' existing local qualification: candidate updates share an instantaneous state within each trajectory, but subsequent data, gradients, and optimizer histories evolve under the chosen rule. The results therefore concern local update rules and their downstream feedback, rather than a global decomposition of reset benefits into a pure direction effect. **3 The monitoring cohort's training conditions are insufficiently specified** Appendix C clearly defines the monitoring features, development-time scaling and threshold selection, frozen rules, first-alert procedure, and 1–16-update timely window. Table 6 assigns ten held-out runs to each of five settings: $K=8$, $K=16$, $K=32$, the narrow queue, and the long-age queue. What remains unclear is which correction and reset conditions generated these runs, whether all 80 monitoring runs use the primary model/task configuration, and how the 30 development runs are allocated across settings. This information determines whether the reported 28/35 timely alerts describe uncorrected training, corrected training, or a mixture. A configuration table listing lag setting, model/task, correction rule, reset condition, and development/held-out run counts would resolve the core ambiguity. Per-run onset and alert records would be useful optional reproducibility material. The existing interpretation should be retained. The held-out cohort contains 35 eligible episode runs, 13 no-episode runs, and two unresolved tails excluded from these denominators. Current-learner evaluation gives 28/35 timely alerts versus 18/35 for rollout reward, with one false alert among the 13 no-episode runs for each rule. These observations do not establish equal population false-alert rates. Moreover, 22 of the 28 timely learner alerts occur when measured learner accuracy is already below 0.25. The evidence supports earlier observation of realized degradation relative to stale feedback.
Questions for authors

Q1. Please clarify the scope of the practical recommendation and distinguish the present finite-group target and contributions from the two ICLR 2026 analyses cited above. If the recommendation extends to regimes with active clipping, a limited validation would help; otherwise, explicitly narrowing its scope would largely resolve the concern.

Q2. Please report the direct paired failure and final-accuracy differences, with uncertainty intervals, between the two direction/norm controls using the existing 20 blocks. Include the paired failure contingency table, since the marginal counts do not identify the pairing.

Q3. Please provide the monitoring-cohort configuration table. Which training condition or mixture does the 28/35 timely-alert result summarize? Per-run onset/alert records are optional.

Q4. Optional practical clarification: if already available, please report the additional generated-token or wall-clock cost of evaluating 256 learner-validation prompts every four updates. This would help assess deployment costs and does not require a new system-efficiency experiment.

7 comments
AuthorsOpenReview ↗

Gentle Reminder: Follow-up on Our Response to Reviewer Z38d

Dear Reviewer Z38d,

Hope this message finds you well. Thank you for your valuable time reviewing our work. We have submitted our point-by-point rebuttal to address all of your concerns and questions. As the discussion deadline is approaching, we are writing to kindly check whether our response has addressed your questions. Please feel free to let us know if you have further comments.

Thank you again for your efforts. Best regards, The Authors

The paired analysis in the response addresses my main statistical concern, and I find the evidence favoring the reset direction convincing in the primary setting. I still had difficulty connecting the estimator analysis to the training results: the paper explains why response-level correction leaves a cross-response bias, but it remains unclear how much that bias matters for stability in the actual training runs. The experiments on other models and tasks test the effects of correction and reset, but the direction-versus-step-size comparison is limited to the primary setting. Given those limits on what I can conclude from the study, I’m keeping my score at 2.

Thank you for clarifying your questions about cross-response bias and update direction. To measure how the cross-response bias affects training, we took 640 checkpoints from 40 training runs of a four-action policy. At each checkpoint we copied the learner and its Adam state. Both copies scored responses from the current policy on the same prompts. We changed only the source of the other seven responses used to compute the group advantage. We averaged over the possible responses before each update, isolating the effect of partner choice on the mean gradient.

Reviewer_fzCQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies policy lag in GRPO through three distinct components: the group-normalized gradient estimator, Adam's accumulated state, and delayed observations of learner quality. A controlled refresh protocol compares ten interventions over 20 paired seed blocks. Token-level truncated importance sampling (TIS) reduces confirmed terminal failures from 12/20 to 2/20 and increases mean final-learner accuracy from 22.7% to 57.9%. Selective resets and local direction–norm controls implicate the first moment's update direction. The theoretical analysis identifies cross-response terms that survive response-level importance correction, while exact finite-state examples distinguish directional alignment, estimator bias, and finite-batch error. A separate development/held-out monitoring experiment shows that evaluating the current learner detects realized degradation earlier than stale-rollout reward.

Critical strengths
  1. The primary intervention study is informative and carefully analyzed. Tables 1–2 use paired runs, report all-run accuracy rather than successful-run accuracy, retain unresolved terminal tails in the denominator, and provide paired uncertainty intervals. The +35.2 percentage-point TIS improvement is substantial. Appendix D's threshold/tail sensitivity and explicit caution about degenerate paired intervals strengthen interpretation.

  2. The optimizer controls improve on an undifferentiated reset comparison. Sections 3.2–3.3 and Appendix B separately manipulate the first moment, second moment, and counter, then exchange instantaneous direction and norm. The paper correctly acknowledges that subsequent trajectories diverge, so these are comparisons of local update rules with downstream feedback. The synchronous comparison keeps reset frequency fixed, helping distinguish policy-age effects from reset frequency.

  3. The estimator analysis avoids several common conceptual conflations. Equations (8)–(12) correctly identify why correcting one response does not correct its group partners. Table 3 and Figure 2 make clear that exact distribution correction can have prohibitive variance and that aligned expected direction does not imply unbiasedness. The limitations of the categorical example are stated rather than hidden.

  4. The monitoring result uses a separate held-out cohort and makes a restrained claim. Threshold selection is confined to development runs; Table 6 reports setting-level exceptions; and Section 5 explicitly says that the alerts usually detect already-realized learner degradation. The paper does not conflate earlier observation with successful prevention.

Critical weaknesses
  1. The mathematical diagnosis is not yet closely connected to the training mechanism. Section 4's numerical example has one-token responses, where token and response weighting coincide, and measures error relative to the normalized estimator's own mean. It does not establish that the residual cross-response bias is harmful for reward optimization or quantitatively important in the LLM experiments. Moreover, at the training batch size of 64 groups, untruncated response IS has lower toy-model MSE than cap-two TIS at both endpoints. The LLM study has no untruncated token-IS or alternative-cap comparison. A small such ablation, with ratio and gradient statistics, would help distinguish the benefit of weighting from that of truncation. This is a limitation of explanatory depth, not a contradiction of the stated identities.

  2. The optimizer conclusions are demonstrated in a restricted update regime. Section 2 and Appendix A use one full-batch update per fresh batch, so proximal PPO clipping is inactive throughout. The additional models and lag schedules broaden coverage, but do not test whether the first-moment-reset advantage persists with minibatch reuse or another learning rate. One representative sensitivity experiment would make the practical scope clearer; absent that, recommendations should remain tied to the studied single-update regime.

  3. The monitoring result establishes an observation advantage more clearly than an operational monitoring benefit. Evaluation requires 256 greedy generations every four updates, but the paper reports lead in learner updates without inference overhead or time until results become available. The false-alert comparison has only 13 no-episode runs, so the equal observed count is weak evidence about population false-alert rates, as Appendix D appropriately recognizes. Report evaluation cost/latency and distinguish the offline availability of saved-checkpoint measurements from an online implementation. No prevention experiment is necessary for the present detection claim.

Questions for authors
  1. Can the authors provide an untruncated token-IS comparison or a modest cap sensitivity check, and indicate whether the observed improvements track reduced gradient variance, changed direction, or both?
  2. Which exact model/task and correction/reset condition generated each monitoring cohort? Were the frozen rules evaluated on TIS-stabilized runs, where the episode prevalence may differ substantially?
  3. What is the generation and checkpoint-evaluation overhead of the four-update monitoring cadence, and how much of the reported lead remains when evaluation latency is included?
  4. Can the paired run-level outcomes, frozen training/monitoring configurations, and the verification programs mentioned in the reproducibility statement be made available? The supplied review materials contained the PDF only, so the aggregate experimental summaries could not be independently checked against those artifacts.
1 comment
AuthorsOpenReview ↗

Gentle Reminder: Follow-up on Our Response to Reviewer fzCQ

Dear Reviewer fzCQ,

Hope this message finds you well. Thank you for your valuable time reviewing our work. We have submitted our point-by-point rebuttal to address all of your concerns and questions. As the discussion deadline is approaching, we are writing to kindly check whether our response has addressed your questions. Please feel free to let us know if you have further comments.

Thank you again for your efforts. Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.