- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies policy lag in asynchronous GRPO by separating its effects into stale-data estimator bias, optimizer memory, and delayed feedback. Through controlled policy-lag experiments, it shows that token-level truncated importance sampling and first-moment Adam resets substantially reduce training failures, although combining them provides little additional benefit. The paper further derives the cross-response bias that remains after response-level importance correction because GRPO advantages use shared group normalization, and analyzes the resulting bias–variance trade-off through exact enumeration. Finally, it shows that directly evaluating the current learner detects degradation earlier than rollout reward.
Critical strengths
Systematic diagnosis of policy lag. The paper separates policy lag into estimator bias, optimizer memory, and delayed monitoring, and studies them in a controlled setup with paired runs. This provides a clearer picture of why stale-rollout training fails than treating policy lag as a single issue.
The paper separately ablates Adam’s first moment, second moment, and step counter, and further distinguishes the direction and magnitude of the first moment. This provides stronger evidence that stale momentum direction is a major contributor to training instability, rather than merely showing that optimizer reset helps.
Critical weaknesses
Limited experimental scope. Experiments are conducted on GSM8K or MATH. It is unclear whether the findings generalize to more tasks, or realistic asynchronous training with changing levels of staleness.
Limited novelty and baseline comparison. Importance sampling, stale-rollout filtering, and optimizer reset have all been explored before. The paper’s main contribution is its analysis, but comparisons with recent policy-lag methods are limited.
Questions for authors
How do TIS and first-moment reset compare empirically with recent policy-lag methods such as VCPO and sequence- or prefix-level importance sampling?
TIS and first-moment reset are meant to address different sources of policy lag, but combining them does not help further. Why might this be the case? Could they be fixing the same underlying failure mode?
What is the practical benefit of current-learner monitoring? Can acting on its alerts prevent or recover from training failures, or does it only tell us earlier that the learner has already degraded?
5 comments
Thank you for the detailed response and additional experiments. I appreciate the new baseline comparisons, direction–norm ablations, and monitoring-triggered recovery results, which address several of my questions and strengthen the empirical evidence. However, my concerns about the novelty of the contribution and its generalizability beyond the tested mathematical tasks and controlled training settings remain insufficiently resolved.
Thank you for clarifying your remaining concerns. On novelty. Our contribution is to show precisely what an exact response-level importance weight recovers in finite-group GRPO. The weight corrects for the policy that generated the response being scored. Its advantage, however, also depends on other responses to the same prompt, which still come from the older policy. Equation (10) isolates the first-order bias this can leave relative to the mean of the same estimator on fresh, on-policy groups. Section 4.2 shows why that distinction matters. At both endpoints of our four-action example, the weighted mean update points in exactly the on-policy direction, yet its magnitude is wrong. A gradient-alignment check would therefore miss the remaining error in both cases.
Gentle Reminder: Follow-up on Our Response to Reviewer EQqW
Dear Reviewer EQqW,
Hope this message finds you well. Thank you for your valuable time reviewing our work. We have submitted our point-by-point rebuttal to address all of your concerns and questions. As the discussion deadline is approaching, we are writing to kindly check whether our response has addressed your questions. Please feel free to let us know if you have further comments.
Thank you again for your efforts. Best regards, The Authors