acceptodds
Under review as a conference paper at ICLR 2027

What Off-Policy Corrections Keep: Gradient Direction and Realized Gains from Stale LLM Rollouts

Abstract

Language-model reinforcement learning increasingly learns from stale rollouts, produced by asynchronous systems or by long agentic interactions that are expensive to repeat, and continued training on them can erode earlier gains. We ask what learning signal different off-policy corrections extract from the same stale responses and when that signal yields realized improvement. We develop a theoretical framework for learning from stale data. It represents each correction through per-token coefficients that factor into advantage, weight, and gate; separates gradient magnitude from off-policy gradient alignment (OGA); and derives a local condition linking alignment and step size to realized improvement. Across mathematical reasoning and instruction following, correction rules produce markedly different directions even at low policy drift. Rules with identical ratio bounds reverse their majority OGA ordering across tasks, and near-equal scalar coefficient-mass totals can accompany large directional differences. OGA generally deteriorates as the policy moves away from the sampling policy, motivating drift as an observable state variable for reuse control. In held-out retrospective evaluation, we show that outcome-calibrated, drift-aware control of reuse retains gains that continued training would otherwise erase, without requiring fresh-gradient measurements.

Primary area:
reinforcement learning
Keywords:
off-policy RL, stale data reuse, LLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hQjBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how different off-policy correction methods affect gradient direction and realized performance when training language models using stale rollouts. In this paper, the authors introduce a unified framework that expresses the gradients of different correction objectives as weighted sums of token-level score vectors, with coefficients determined by advantages, importance weights, and clipping or masking gates. They introduce Off-policy Gradient Alignment (OGA), measuring the cosine similarity between stale-data gradients and gradients estimated from fresh rollouts at the same model checkpoint. The paper evaluates eight correction rules, including No-IS, CISPO, clipping-based methods, token masking, M2PO, GSPO, and TIC-GRPO, on mathematical reasoning and instruction-following tasks using Qwen3-1.7B models. The results demonstrate substantial variation in gradient alignment across correction rules, including cases where methods with nearly identical coefficient-mass totals produce different gradient directions. Finally, the paper studies whether continued reuse of stale data preserves the improvements achieved during training. It proposes an outcome-calibrated stopping strategy combining a fixed update budget with a KL-based override.

Critical strengths
  1. It is an important and timely problem setting. Understanding how to learn from stale rollouts is increasingly relevant for asynchronous LLM reinforcement learning and agentic training, where generating new trajectories can be expensive. The paper addresses an important question about how different off-policy corrections affect the useful learning signal extracted from previously generated data.
  2. There is a systematic comparison of different correction mechanisms. The comparison of eight off-policy correction rules at identical policy checkpoints is a meaningful strength. By fixing the model state, stored rollouts, advantages, and fresh-gradient reference, the authors isolate how individual correction operators change the gradient direction.
  3. There are some interesting findings on gradient alignment and clipping behavior. The paper demonstrates that correction rules can produce substantially different gradient directions even at relatively low policy drift. The comparison between symmetric clipping and two-sided masking is particularly interesting, as the relative alignment of these methods changes across tasks despite identical ratio bounds.
  4. I also like that there are detailed analyses and transparent reporting of limitations.
Critical weaknesses
  1. There is limited experimental scale and generalization. The experimental evaluation is restricted to a single model scale (1.7B) within the Qwen3 family, two tasks, and one training seed. Although the number of evaluated checkpoints and correction rules is relatively large, these are correlated measurements from a limited number of underlying training trajectories rather than independent experiments.
  2. There is also a very marginal improvement of KL-based stopping over fixed-step stopping. A central practical contribution is the proposed drift-aware stopping strategy. However, the results indicate that most of the improvement comes from stopping training after a fixed number of updates. Consequently, the results strongly support limiting stale-data reuse but provide considerably weaker evidence that KL-based adaptive stopping offers a meaningful improvement over a calibrated fixed budget.
  3. The gradient alignment is not validated against actual optimizer updates. The paper uses OGA as its primary diagnostic, but this measures the raw gradient rather than the executed parameter update. In practice, AdamW applies coordinatewise preconditioning, momentum, and other transformations that can substantially change the update direction.
  4. There is insufficient reliability analysis of the fresh-gradient reference. The central OGA measurements depend on a fresh-sample gradient estimate, but the paper does not sufficiently quantify uncertainty in this reference. In particular, repeated fresh-gradient estimates or split-sample agreement would help determine whether the observed directional differences exceed the stochastic variability of the reference itself. This concern is especially relevant for the strongly negative alignment values reported for certain clipping rules in instruction following. Similarly, the reported direction-drift correlations are computed over correlated checkpoints from a single seed.
  5. The mechanisms behind directional differences are not fully disentangled. The paper highlights that near-equal retained coefficient mass can accompany substantial differences in gradient alignment. However, the prominent comparison between No-IS and Asym-Clip changes both importance weighting and clipping gates. Therefore, the large directional differences cannot be attributed specifically to the deletion of a small fraction of tokens.
Questions for authors
  1. Can the authors measure the cosine similarity between the actual executed AdamW updates and the fresh-gradient reference, rather than only the raw gradients? Additionally, can they quantify whether OGA predicts subsequent changes in validation reward or one-step task improvement?
  2. Given that KL+cap improves instruction-following performance by only +0.51 points over fixed-step stopping and provides no additional improvement in mathematics, how robust is this benefit across independent seeds and model sizes?
  3. Can the authors evaluate the stopping strategy prospectively on new training trajectories, with its thresholds fixed before running those experiments?
  4. How stable are the estimated fresh-gradient directions across independent rollout samples at the same checkpoint?
  5. Can the authors separately analyze the directional contributions of importance-ratio reweighting and clipping or masking decisions?
4 comments
AuthorsOpenReview ↗

One-step reward prediction from actual updates

Projecting actual AdamW updates onto a fresh reward gradient helped predict which correction rule gained more in one step and the approximate size of larger differences. We measured the cosines of executed updates directly (Table F.6). Across the two full-rate step-64 checkpoints, diagnostic OGA had opposite reward rank correlations, while the executed projection correlated positively at both (Table F.5).

Across the six instruction-following checkpoints, we predicted the observed ordering for all 99 pairs with predicted gaps of at least one point (Figure 4). At 4B, all 12 prespecified mean reward differences had the predicted sign, and 11 paired 95% intervals excluded zero (Appendix F.10, Table F.16).

We added the one-step experiments to our third contribution. Section 2.3 distinguishes gradient alignment from update alignment, and Section 4.1 reports the reward tests.

AuthorsOpenReview ↗

Transfer of a preselected reuse budget

A fixed update budget chosen on other trajectories before the new runs gave higher mean reward on both tasks than training to step 256. A standalone KL threshold chosen in advance also limited late losses. In the 12 new 1.7B runs, adding KL rollback at the same step cap tied or lowered reward relative to Fixed under both calibration schemes (Tables G.4–G.6).

We removed the incremental KL+cap gain claim from our contributions. Section 4.2 and the conclusion now highlight transfer of the fixed budget chosen in advance. The prospective stopping test used a new training order at 1.7B.

AuthorsOpenReview ↗

Replication and checks of the alignment results

The leading alignment ranking replicated under a new training order and at 4B. Among the five rules we retested, No-IS and CISPO were the top two at all 21 low-drift checkpoints across eight reordered 1.7B runs and at step 8 in all six 4B instruction-following runs. Available compute limited the scale of the new experiments, so we prioritized these checks. Section 3.3 reports the result and Appendix F.8 separates the counts of runs and checkpoints.

The negative alignment at instruction-following step 128 persisted across independent fresh samples. The two fresh gradients had cosine 0.9948 there, compared with 0.7772 at step 8. At step 128, the measured Asym-Clip cosine of −0.5102 with the first reference gives an upper bound of about −0.42 with the second. Section 3.1 now presents this check alongside the negative alignment, and Appendix F.7 reports matched-size results. Alignment–drift correlations were negative on 27 of 32 individual trajectories (Table D.5).

Reweighting alone reduced alignment more than gating alone at all three instruction-following checkpoints. Gating had the larger effect at all three math checkpoints (Table D.7). Section 3.2 now isolates deletion with weights held fixed. Tightening Asym-Clip to Sym-Clip removed a median of 0.14% of tokens and raised alignment by 0.017 at the median across 50 low-drift math states (Table D.2).

Thank you for the substantial additional experiments and for revising the stopping-strategy claim. These additions largely address my main concerns, and I am inclined to raise my score after checking the corresponding results in the revised manuscript.

Reviewer_TjnpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how eight off-policy correction rules affect gradient directions when LLMs learn from stale rollouts. It compares these gradients with gradients from fresh samples and finds clear differences between rules. It also tests whether fixed training limits and KL-based stopping can preserve gains that continued training would erase.

Critical strengths

The comparisons at shared model states are well controlled. Keeping the checkpoint, old samples, advantages, and fresh reference fixed helps isolate how each correction rule changes the gradient, rather than mixing this effect with differences in training history.

Critical weaknesses

The link to actual training updates is not fully tested. In the math setting, the diagnostics and training weight responses differently. The paper also measures gradient alignment before AdamW changes the update direction.

Questions for authors

With the stopping rule fixed in advance, does KL still outperform a fixed step budget chosen separately for each learning rate when tested on new training seeds?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Tjnp

Training objectives and executed updates

Response weighting changed the math reference gradient, and the executed AdamW updates differed from their input gradients in alignment with the same reference. Section 2.3 distinguishes these directions. In Section 4.1, we show that projections of the actual updates onto a fresh reward gradient do help predict one-step reward differences between rules.

At original-order math step 64, we held 6,240 responses, advantages and masks fixed. Giving each token equal weight in the diagnostic or each response equal weight as in training the token-level rules yielded reference gradients with cosine 0.2838 (Appendix F.7). At original-order instruction-following step 8, the input-gradient cosines for No-IS and Asym-Clip were +0.193 and −0.224 against the same fresh reference. The executed-update cosines were +0.0063 and −0.0059 (Table F.6).

At new-order math step 8 we predicted that TSM would gain 0.475 fewer accuracy points than Sym-Clip. We measured 0.841 fewer on independent prompts (Table F.18).

Frozen comparison with a strong fixed budget

We calibrated the budget and threshold separately at each learning rate on other trajectories, then tested one new training order. Across 12 runs, Fixed matched KL+cap ten times and earned higher reward twice (Table G.6). Section 4.2 explains the calibration and same-cap comparison.

We froze the rules before the new runs and kept the behavior-response pools fixed. In both cases where the policies differed, both ran eight updates. KL rolled back to step 4, losing 5.375 points for CISPO and 0.583 for TIC-GRPO relative to Fixed at step 8 (Tables G.5 and G.6).

Under the same per-rate calibration, KL-only gained 1.755 more accuracy points than Fixed on the four math runs, using 168 versus 88 updates on average. Fixed gained 1.244 more reward points on the eight instruction-following runs (Table G.4).

The added analysis of actual updates helps clarify the connection to training, and the stronger fixed-budget comparison is useful. These results ease some of my concerns, though I am still unsure how much KL-based stopping adds beyond a well-chosen budget. On balance, I lean toward keeping my current rating for now.

Thank you for the follow-up. On the new training order, adding KL rollback to a fixed budget calibrated separately for each learning rate produced no additional gain. It returned the same checkpoint on 10 of 12 runs and lowered reward on the other two (Table G.6). As a standalone stopping rule, KL yielded 1.76 percentage points higher mean math accuracy than Fixed, with 168 versus 88 updates on average (Table G.4).

We have refocused our third contribution on how actual optimizer updates translate into reward. In the new one-step experiments, projecting the executed AdamW updates onto a fresh gradient predicted the sign and approximate size of larger reward differences between correction rules (Introduction and Section 4.1). Our practical finding on stopping is that a fixed reuse budget selected on separate trajectories limited late losses on both tasks when applied unchanged to a new training order (Section 4.2).

I appreciate the clarification. The revised focus on actual optimizer updates and reward seems more consistent with the reported results. I also understand that the new experiments do not support an added benefit from KL+cap over the fixed budget. This helps clarify the contribution.

Reviewer_vjk3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper expresses eight off-policy correction rules through per-token coefficients and compares their alignment with a fresh-data gradient at common model states. Experiments on mathematical reasoning and instruction following show that similarly bounded rules can induce different directions and reverse their ordering across tasks. A retrospective study then evaluates calibrated update budgets and KL-based overrides for stopping reuse of a stale rollout pool.

Critical strengths
  1. The common-state comparisons are the strongest contribution. Holding the checkpoint and stale responses fixed separates the correction operator from the trajectory it happened to produce.
  2. The coefficient accounting is precise, especially the role of advantage sign in selecting clipping branches. The paper also includes a calibrated fixed-budget baseline, which reveals that much of the gain retention comes from stopping early. That is a useful practical finding.
Critical weaknesses
  1. The drift statistic is not generally a KL divergence. Section 3.3 averages each response's log-likelihood ratio after dividing by its sampled length. With variable-length responses, this quantity can be negative even in population. The statistic may remain useful as an empirical monitor, but its KL interpretation and the resulting low-drift comparisons need validation.
  2. Declining OGA may reflect a weaker fresh-gradient signal relative to sampling noise, so a matched fresh-versus-fresh control is needed to isolate staleness. The paper also does not test whether higher OGA improves reward under AdamW. Since stopping is calibrated directly on rewards, the link between gradient geometry and realized gains remains unverified.
Questions for authors
  1. Do the low-drift rankings, OGA-drift correlations and stopping results survive using sequence KL, optionally divided by a fixed behavior-policy mean length? Please recalibrate thresholds with the same leave-family-out procedure and report whether the increment over fixed-step stopping persists.
  2. At representative checkpoints, how does stale-versus-fresh alignment compare with fresh-versus-fresh alignment at matched sample sizes? Starting from the same checkpoint and optimizer state, do higher-OGA corrections produce larger immediate reward gains when using the actual training reduction and a matched small update budget, measured by norm or policy KL?
1 comment
AuthorsOpenReview ↗

Response to Reviewer vjk3

KL definition and retained findings

Using sequence KL preserved the leading low-drift rankings, the signs of the alignment–drift correlations, and the original mean instruction-following gain from rollback over a fixed budget.

In Section 3.3 we sum token log-ratios within each stored response and average across responses. After dividing by the fixed mean response length, No-IS and CISPO remained the top two rules at the same cutoff of 0.02 (Table D.1). Comparing Table D.4 in the submitted and revised manuscripts, all 16 pooled correlations and all 16 median within-trajectory correlations kept their signs.

For stopping we recalibrated thresholds on sequence KL itself, using the original method-family holdouts. KL+cap returned the same checkpoints on all 32 original runs, preserving its 0.506-point mean gain over Fixed on instruction following. KL-only changed its choice on 3/16 instruction-following and 5/16 math runs (Table G.1). On the new order, adding rollback at the same pooled cap tied Fixed on eight runs and lowered reward on four (Section 4.2, Table G.6).

Alignment and reward under matched update budgets

In every matched-size comparison at six checkpoints, the two signed clips aligned less closely with a fresh gradient than two fresh gradients aligned with each other. At matched step size, the better-aligned of Sym-Clip and TSM under the training objective earned more reward in all three one-step comparisons.

Table F.10 in Appendix F.7 reports the mean over three splits per checkpoint. At instruction-following step 128, the mean fresh–fresh cosine was 0.988, compared with −0.378 for Asym-Clip against the fresh reference.

We used the training loss reduction on the same next batch from the same checkpoint and AdamW state. We matched the input-gradient norms and kept the executed update norms within 1%. TSM was better aligned and gained 2.00 more reward points at instruction-following step 8. At step 64, Sym-Clip was better aligned and gained 0.57 more. TSM gained 1.17 more accuracy points at original-order math step 8. All three paired 95% intervals excluded zero (Section 4.1, Table D.8).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions and new results

Drift definition. We corrected the measure to sequence KL divided by the fixed mean response length. We sum token log-ratios within each stored response and then average across responses (Section 3.3). The leading low-drift alignment rankings held under the corrected measure (Table D.1). We also replicated the top-two alignment ranking of No-IS and CISPO at low drift under a new training order and at early 4B checkpoints (Appendix F.8).

One-step reward. We added one-step reward prediction experiments to our third contribution. Projecting actual AdamW updates onto a fresh reward gradient helped predict which correction rule gained more and the approximate size of larger differences (Section 4.1).

Stopping. We removed the incremental KL+cap gain claim from our contributions. We now emphasize that a fixed reuse budget chosen in advance on other trajectories transferred to a new training order and limited late losses on both tasks (Section 4.2 and Discussion). Table G.6 reports the same-budget comparison with KL rollback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.