- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how different off-policy correction methods affect gradient direction and realized performance when training language models using stale rollouts. In this paper, the authors introduce a unified framework that expresses the gradients of different correction objectives as weighted sums of token-level score vectors, with coefficients determined by advantages, importance weights, and clipping or masking gates. They introduce Off-policy Gradient Alignment (OGA), measuring the cosine similarity between stale-data gradients and gradients estimated from fresh rollouts at the same model checkpoint. The paper evaluates eight correction rules, including No-IS, CISPO, clipping-based methods, token masking, M2PO, GSPO, and TIC-GRPO, on mathematical reasoning and instruction-following tasks using Qwen3-1.7B models. The results demonstrate substantial variation in gradient alignment across correction rules, including cases where methods with nearly identical coefficient-mass totals produce different gradient directions. Finally, the paper studies whether continued reuse of stale data preserves the improvements achieved during training. It proposes an outcome-calibrated stopping strategy combining a fixed update budget with a KL-based override.
Critical strengths
- It is an important and timely problem setting. Understanding how to learn from stale rollouts is increasingly relevant for asynchronous LLM reinforcement learning and agentic training, where generating new trajectories can be expensive. The paper addresses an important question about how different off-policy corrections affect the useful learning signal extracted from previously generated data.
- There is a systematic comparison of different correction mechanisms. The comparison of eight off-policy correction rules at identical policy checkpoints is a meaningful strength. By fixing the model state, stored rollouts, advantages, and fresh-gradient reference, the authors isolate how individual correction operators change the gradient direction.
- There are some interesting findings on gradient alignment and clipping behavior. The paper demonstrates that correction rules can produce substantially different gradient directions even at relatively low policy drift. The comparison between symmetric clipping and two-sided masking is particularly interesting, as the relative alignment of these methods changes across tasks despite identical ratio bounds.
- I also like that there are detailed analyses and transparent reporting of limitations.
Critical weaknesses
- There is limited experimental scale and generalization. The experimental evaluation is restricted to a single model scale (1.7B) within the Qwen3 family, two tasks, and one training seed. Although the number of evaluated checkpoints and correction rules is relatively large, these are correlated measurements from a limited number of underlying training trajectories rather than independent experiments.
- There is also a very marginal improvement of KL-based stopping over fixed-step stopping. A central practical contribution is the proposed drift-aware stopping strategy. However, the results indicate that most of the improvement comes from stopping training after a fixed number of updates. Consequently, the results strongly support limiting stale-data reuse but provide considerably weaker evidence that KL-based adaptive stopping offers a meaningful improvement over a calibrated fixed budget.
- The gradient alignment is not validated against actual optimizer updates. The paper uses OGA as its primary diagnostic, but this measures the raw gradient rather than the executed parameter update. In practice, AdamW applies coordinatewise preconditioning, momentum, and other transformations that can substantially change the update direction.
- There is insufficient reliability analysis of the fresh-gradient reference. The central OGA measurements depend on a fresh-sample gradient estimate, but the paper does not sufficiently quantify uncertainty in this reference. In particular, repeated fresh-gradient estimates or split-sample agreement would help determine whether the observed directional differences exceed the stochastic variability of the reference itself. This concern is especially relevant for the strongly negative alignment values reported for certain clipping rules in instruction following. Similarly, the reported direction-drift correlations are computed over correlated checkpoints from a single seed.
- The mechanisms behind directional differences are not fully disentangled. The paper highlights that near-equal retained coefficient mass can accompany substantial differences in gradient alignment. However, the prominent comparison between No-IS and Asym-Clip changes both importance weighting and clipping gates. Therefore, the large directional differences cannot be attributed specifically to the deletion of a small fraction of tokens.
Questions for authors
- Can the authors measure the cosine similarity between the actual executed AdamW updates and the fresh-gradient reference, rather than only the raw gradients? Additionally, can they quantify whether OGA predicts subsequent changes in validation reward or one-step task improvement?
- Given that KL+cap improves instruction-following performance by only +0.51 points over fixed-step stopping and provides no additional improvement in mathematics, how robust is this benefit across independent seeds and model sizes?
- Can the authors evaluate the stopping strategy prospectively on new training trajectories, with its thresholds fixed before running those experiments?
- How stable are the estimated fresh-gradient directions across independent rollout samples at the same checkpoint?
- Can the authors separately analyze the directional contributions of importance-ratio reweighting and clipping or masking decisions?
4 comments
Transfer of a preselected reuse budget
A fixed update budget chosen on other trajectories before the new runs gave higher mean reward on both tasks than training to step 256. A standalone KL threshold chosen in advance also limited late losses. In the 12 new 1.7B runs, adding KL rollback at the same step cap tied or lowered reward relative to Fixed under both calibration schemes (Tables G.4–G.6).
We removed the incremental KL+cap gain claim from our contributions. Section 4.2 and the conclusion now highlight transfer of the fixed budget chosen in advance. The prospective stopping test used a new training order at 1.7B.
Replication and checks of the alignment results
The leading alignment ranking replicated under a new training order and at 4B. Among the five rules we retested, No-IS and CISPO were the top two at all 21 low-drift checkpoints across eight reordered 1.7B runs and at step 8 in all six 4B instruction-following runs. Available compute limited the scale of the new experiments, so we prioritized these checks. Section 3.3 reports the result and Appendix F.8 separates the counts of runs and checkpoints.
The negative alignment at instruction-following step 128 persisted across independent fresh samples. The two fresh gradients had cosine 0.9948 there, compared with 0.7772 at step 8. At step 128, the measured Asym-Clip cosine of −0.5102 with the first reference gives an upper bound of about −0.42 with the second. Section 3.1 now presents this check alongside the negative alignment, and Appendix F.7 reports matched-size results. Alignment–drift correlations were negative on 27 of 32 individual trajectories (Table D.5).
Reweighting alone reduced alignment more than gating alone at all three instruction-following checkpoints. Gating had the larger effect at all three math checkpoints (Table D.7). Section 3.2 now isolates deletion with weights held fixed. Tightening Asym-Clip to Sym-Clip removed a median of 0.14% of tokens and raised alignment by 0.017 at the median across 50 low-drift math states (Table D.2).
Thank you for the substantial additional experiments and for revising the stopping-strategy claim. These additions largely address my main concerns, and I am inclined to raise my score after checking the corresponding results in the revised manuscript.
One-step reward prediction from actual updates
Projecting actual AdamW updates onto a fresh reward gradient helped predict which correction rule gained more in one step and the approximate size of larger differences. We measured the cosines of executed updates directly (Table F.6). Across the two full-rate step-64 checkpoints, diagnostic OGA had opposite reward rank correlations, while the executed projection correlated positively at both (Table F.5).
Across the six instruction-following checkpoints, we predicted the observed ordering for all 99 pairs with predicted gaps of at least one point (Figure 4). At 4B, all 12 prespecified mean reward differences had the predicted sign, and 11 paired 95% intervals excluded zero (Appendix F.10, Table F.16).
We added the one-step experiments to our third contribution. Section 2.3 distinguishes gradient alignment from update alignment, and Section 4.1 reports the reward tests.