acceptodds
Under review as a conference paper at ICLR 2027

Score Centering Stabilizes Off-policy Reinforcement Learning

12 views

Abstract

Reinforcement learning (RL) of large language models is notoriously sensitive to small differences between training and inference engines, often referred to as the training-inference mismatch (TIM). However, completely eliminating TIM is impractical, as it would come at a major cost to rollout efficiency. In this paper, we show that the instability of RL under TIM is primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step. We derive an additive "score centering" correction term that stabilizes RL under TIM by canceling drift. When training models from 0.6B to 30B parameters, score centering alone matches or outperforms methods based on importance sampling under quantization, with the gap growing as the mismatch becomes more severe. Because the correction is additive, score centering also composes with importance sampling – their composition outperforms pure importance-sampling baselines in our staleness experiments.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, training-inference mismatch

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BXCpOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies training–inference mismatch (TIM) in RL for LLMs. It decomposes the off-policy policy-gradient expectation into a drift term, Eq[R] sˉ\mathbb{E}_q[R]\,\bar s, and a signal term, Covq(R,s)\mathrm{Cov}_q(R, s). The drift term distills the trainer toward the sampler, and the effect compounds because the sampler is periodically synced back to the trainer. Building on this, the paper proposes score centering (SC): at each prefix, it subtracts the sampler’s expected score. This cancels drift in expectation without using importance ratios. The correction is additive and can be combined with TIS/MIS, and a top-k approximation (k=128) keeps its overhead negligible. The experiments compare SC against eight IS-based corrections under a shared REINFORCE objective. On Qwen3-0.6B (Countdown), the settings are synthetic weight noise, an INT8 sampler, and staleness (sync every 64 steps). On Qwen3-30B-A3B-Base (INTELLECT-2 math), the settings are three sampler quantization levels. SC matches or outperforms the IS-based methods, and its advantage grows as the mismatch becomes more severe. Under staleness, SC combined with TIS/MIS performs best.

Critical strengths
  1. A clear mechanistic account of why RL is sensitive to TIM. The drift/signal decomposition unifies several observations. It explains why SFT remains stable even fully offline, why +1/0 rewards are unstable online, and why group centering helps but cannot remove drift. The resulting fix is principled rather than another clipping heuristic. This is the main reason I lean toward acceptance.
  2. A simple, practical method. SC is additive, deterministic given the prefix, adds less than 1% wall-clock overhead, composes with existing IS corrections, and comes with a short reference implementation. It should be easy to adopt in existing RL pipelines.
Critical weaknesses
  1. The evaluation does not reach the regime where the claims matter. (a) There is no held-out downstream benchmark. The 30B results and the quantization/staleness results report smoothed training accuracy, which is the reward of rollouts from the (quantized) sampler, not a measure of the trainer’s capability. (b) Sequences are capped at 512/1024 tokens. In practice, TIM-induced collapse is most severe in long-horizon agentic training, as the paper itself notes in §2.1. The authors argue that short runs with amplified TIM are a proxy for long runs with mild TIM, but this claim is never validated. In the only near-realistic setting (30B, FP8/FP8 KV), uncorrected PG performs on par with every correction method. The practical benefit of SC therefore remains unestablished.
  2. Baselines are run with fixed default hyperparameters. Every method uses its paper or verl defaults with no tuning, together with SGD at lr 1e-2 and one update per batch. The TIS/MIS bounds directly control the bias–variance trade-off under mismatch, and PPO-style clipping was designed for policy movement, not for engine mismatch. The reported gaps could shrink if the baselines were tuned. In addition, the extra 30B seeds were allocated only to the best-performing methods, so most 30B baselines have a single seed.
Questions for authors
  1. For the 30B runs, do the training-reward gaps carry over to held-out benchmarks (e.g., MATH500, AIME) evaluated on the trainer?
  2. Can you show even a single comparison (SC vs. TIS vs. PG) at substantially longer sequences, or in a multi-turn agentic setting, with natural TIM (e.g., a bf16 or FP8 sampler without artificial amplification)? This would directly support the proxy argument.
  3. If the strongest baselines get a small sweep (the TIS cap, the MIS bounds) and are trained with AdamW, does SC keep its advantage? The paper states that SGD performs comparably to AdamW but does not show this.
  4. (Clarification) The conclusion says SC is “the only method that trains stably under severe quantization.” However, TIS reaches 51% at FP8 W/A + FP4 KV. Also, under staleness, SC alone underperforms PPO/DAPO. How do these results fit with the claim that drift is the cause of instability?
1 comment

Thank you for your comprehensive assessment of our paper! We greatly appreciate your summary and questions about the work. Below, we provide answers to each of them:

Q1: Held-out evaluation. We did not manage to report MATH500 or AIME results due to the rebuttal time constraints. However, across all our Qwen3-0.6B runs, we find that held-out accuracy closely tracks training accuracy (Pearson r = 0.98). We also added a new experiment using multi-turn agentic coding that includes both training and held-out accuracy. Please see our general comment for further details.

Q2: Long-horizon runs with natural TIM. Our agentic experiment (general comment and Section 5.5) uses multi-turn terminal tasks with up to 32K tokens of context (about 10K tokens per rollout on average). Even with a bf16 sampler and no artificial amplification, PG collapses around step 130. With an NVFP4 sampler, PG, MIS, and TIS collapse, while SC, TIS + SC, and MIS + SC train stably for more than 400 steps.

Q3: Hyperparameters, AdamW.

We removed the statement "SGD performs comparably to AdamW" from the Appendix and added a tuned comparison in Appendix A.5. In four Countdown settings with Qwen3-0.6B (INT8 sampler, staleness of 64 steps, and weight noise with scales 0.02 and 0.05), we sweep the optimizer learning rates, PG and TIS with caps {1.5, 2, 5, 10}, as well as MIS bounds, with 3 seeds each.

At each method's best learning rate, score centering outperforms the best tuned baseline in all 12 setting-seed pairs with both optimizers, by 6.6 points on average with SGD and by 4.7 points with AdamW. Tuning matters for the baselines: TIS with caps 1.5 and 2 and MIS with bounds [0.5, 2] are stable only at the smallest learning rate, where every method learns slowly, and under staleness every baseline collapses in at least two of three seeds at learning rate 1e-6. Score centering never collapsed with AdamW. Tuned AdamW slightly outperforms tuned SGD (by 2.0 points for score centering and 3.5 points for the baselines), but it does not close the gap.

Regarding seeds, every method in Appendix A.5 uses 3 seeds.

Q4: Severe quantization and staleness.

In the 30B Math setting that limits rollouts to 1024 tokens, TIS was indeed stable under FP8/4 quantization but became unstable under INT8/4 quantization, where only score centering was stable. In the revised version of the paper, we added agentic coding with an NVFP4 sampler and up to 32K token context, where TIS collapses while SC / TIS + SC / MIS + SC remain stable.

The question of why vanilla score centering performs poorly under large staleness is indeed interesting. In Equation 5, we point out that score centering cancels drift but the update still remains computed w.r.t. the sampler's distribution, rather than the trainer's sampling distribution. Under severe quantization error, this is desirable, since we ultimately care about optimizing the sampler rather than the trainer. However, under severe staleness, this becomes problematic, because the mismatch between the sampler and the trainer is systematic and can grow large. For example, if the trainer is 64 steps ahead of the sampler, it might have already unlearned some of the sampler's behavior, in which case we no longer want to train on those samples [1]. Therefore, under staleness, it is better to compute the gradient covariance under the trainer's distribution, which is exactly what IS achieves. We propose using TIS to effectively change the sampling distribution of the gradient, and SC to remove any remaining drift.

[1] Learning Dynamics of LLM Finetuning. Ren and Sutherland, ICLR 2025

Reviewer_rKTpOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies RL instability under training–inference mismatch (TIM). Full-distribution score centering (SC) subtracts the sampler-expected score at each prefix, removing the conditional mean term in the expected update. It provides a top-k approximation and compositions with importance sampling. Qwen3 experiments show substantial gains under severe quantization and benefits from combining SC with TIS/MIS under staleness.

Critical strengths
  1. SC targets a defined conditional mean term without learning a critic. Its scalar loss with detached coefficients and top-k reduction make this implementable, with a direct top-k/full-vocabulary comparison supporting the approximation.
  2. Under the strongest 30B quantization setting, SC reaches approximately 30% training accuracy versus 12% for TIS (Figure 1). The shared-objective comparisons isolate correction rules, and TIS+SC/MIS+SC outperform SC alone under staleness.
Critical weaknesses
  1. The claimed drift mechanism lacks direct quantification and supporting process diagnostics. The abstract identifies accumulating drift as the primary cause, and Section 6 says the experiments establish this cause. SC's successful intervention provides support, but Figure 3 reports accuracy rather than drift magnitude, direction, or temporal accumulation. Entropy, gradient norms, actual output-length distributions, and training–inference KL traces are also absent, limiting interpretation of the pre-collapse dynamics. Section 5.2 fixes one Gaussian weight offset throughout each run without comparing independently resampled offsets, leaving the effect of this temporal structure unresolved.
  2. Practical validation remains limited to short-sequence SGD experiments. Appendix B.1 uses SGD and 512/1024-token caps for the main 0.6B/30B comparisons, and asserts AdamW parity without showing the corresponding results. SGD is a reasonable choice, but SC's behavior under Adam/AdamW and on long reasoning trajectories is not demonstrated. Separately, Sections 5.1 and 6 use severe short runs as a proxy for prolonged milder TIM; Figure 1's mild run remains stable for 800 steps even with PG. Optimizer transfer, longer sequences, and longer training under milder mismatch are distinct validation gaps.
Questions for authors
  1. At frozen checkpoints and fixed prefixes, can drift magnitude, direction, temporal evolution, and uncertainty be quantified using actual advantages and normalization? Please accompany this with entropy, gradient norm, actual output lengths, and training–inference KL for SC and key baselines, distinguishing full-distribution from top-k estimates.
  2. What changes if Section 5.2's offset is independently redrawn before each step's rollout, held fixed within the step, and never accumulated? Please compare drift and stability with the fixed-offset setting at the same noise scale, while measuring realized policy mismatch.
  3. Can existing records substantiate B.1's AdamW parity assertion? A representative SC-versus-strong-baseline comparison under appropriately tuned Adam/AdamW would help establish optimizer applicability.
  4. Can one targeted 32K comparison test SC or its IS composition against a strong baseline on a task producing genuinely long trajectories, reporting actual output lengths, training tokens, and held-out results?
  5. What evidence supports using severe short runs as a proxy for prolonged milder TIM? Existing records or one focused longer run under milder mismatch could help.
1 comment

Thank you for the thorough review and insightful comments! To address your questions, we have conducted a number of additional experiments, which we report in this response. If these results address your concerns, we would appreciate if you could update the review accordingly.

Q4, Q5: Longer runs under natural mismatch. We added a new multi-turn agentic RL experiment specifically to test this, which we summarize in the general comment. In this experiment, PG collapses around step 45 with an NVFP4 sampler and around step 130 with a bf16 sampler -- so a milder mismatch still causes collapse, it just takes more steps, matching the intuition in Section 5.1. Score centering stays stable for >637 steps.

Q1: Pre-collapse diagnostics. We did not manage to run these measurements in the rebuttal window. Our evidence for accumulation is that collapse happens earlier under larger mismatch (Figure 3; bf16 vs. NVFP4 PG in Figure 5), and that removing only the drift term restores stability.

Q2: Resampling the weight offset. We found that a fixed offset is much more harmful, since it accumulates with each step. For example, in Figure 3, relatively small scale fixed noise is sufficient to destabilize every method other than score centering, without meaningfully affecting model performance. In contrast, in Figure 1 we had to make the non-fixed mismatch coming from quantization so severe (Int4 KV) that the base model's accuracy went from 40% to 4% to observe comparable instability.

Q3: AdamW. Thank you for this suggestion! We removed the parity claim from Appendix B.1 and replaced it with a direct comparison of tuned SGD and tuned AdamW in Appendix A.5 (summarized in the general comment). At each method's best learning rate, score centering outperforms the best tuned baseline in all 12 setting-seed pairs with both optimizers, by 6.6 points on average with SGD and 4.7 points with AdamW. Hence, our primary conclusions are not affected by the optimizer choice. We also use AdamW in the agentic experiment in Section 5.5.

Reviewer_LViAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • The paper identifies a drift term in policy-gradient updates under training–inference mismatch and proposes score centering to remove it.
  • The method subtracts the sampler-expected score from each token’s score and can be combined with clipped or masked importance sampling. A top-k approximation avoids storing full sampler distributions.
  • Experiments on Countdown and mathematical reasoning, using models up to 30B parameters, show improved stability under synthetic weight perturbations and severe quantization. Under severe staleness, combining score centering with importance sampling performs better than score centering alone.
Critical strengths
  • The gradient decomposition provides a concrete explanation for an otherwise difficult-to-diagnose failure mode: mismatch can produce nonzero expected updates even under constant rewards. Score centering removes this term without requiring a learned critic or potentially large importance ratios.

  • The experiments test multiple mismatch sources and show where the method needs additional correction. In particular, the stronger results from combining score centering with importance sampling under staleness support the distinction between removing drift and correcting the sampling distribution. Top-k ablations also demonstrate a practical implementation.

Critical weaknesses
  1. The evaluation relies on deliberately amplified mismatch and short sequences of 512–1024 tokens. This does not establish whether the gains persist under natural mismatch in long-context RL. The main 30B results report training accuracy rather than held-out performance, further limiting conclusions about practical training quality.

  2. The top-k implementation does not retain exact drift cancellation unless the sampler’s tail matches the assumed rescaled trainer distribution. The ablations vary k but do not isolate the benefit of tail rescaling over the simpler correction �0��0�, which also vanishes when �1��1�.

Questions for authors
  1. Do you have longer-run results under natural BF16/FP8 mismatch, including held-out evaluation, comparing SC and SC+TIS against TIS? These would clarify whether the gains extend beyond the deliberately severe settings.

  2. In Appendix A.1, have you compared tail rescaling against directly truncating (∑v(qv−pv)∇θlog⁡pv)(\sum_v(q_v-p_v)\nabla_\theta\log p_v) to the sampler’s top-k? How large is the residual drift introduced by the tail approximation, and does rescaling measurably reduce it?

1 comment

Thank you for your detailed review! Please read our response below:

Q1: Longer runs under natural mismatch. We ran a new multi-turn agentic RL experiment specifically to test this, which we summarize in the general comment.

Q2: Tail approximation. We have not compared tail rescaling against direct truncation, but we measured the residual drift of our top-k estimator. In the Qwen3-0.6B INT8 and weight-noise settings, we periodically computed the drift that exact (full-vocabulary) score centering removes at fixed held-out prompts, and compared it to the drift that our top-k estimator with the tail approximation removes. In runs trained with score centering, the relative error stays at about 1.2% with k = 128 (below 1.6% in 95% of the measurements) and about 1.3% with k = 32 throughout training.

Our motivation for introducing the tail approximation was to avoid any systematic bias on tail tokens: without the tail approximation, the "average score" correction term that we subtract from every update would systematically omit tail tokens, resulting in a gradient that increases the probability of tail tokens under a positive advantage and decreases it under a negative advantage.

Our tail approximation is a cheap and low-variance correction term that eliminates this "tail mass" bias, even though it's not a perfectly unbiased estimate of the full score centering correction. One alternative we also considered was to resample the tail, which would both remove the tail bias and provide an unbiased (but higher variance) estimator of full score centering.

With k=128k=128, the head covers >99.9% of the sampler's mass on average in every setting we tested except 30B INT8 + INT4 KV (99.45%), meaning that each of these approximations is very close to full score centering. In all the settings we tested, the tail approximation performed on par with full score centering. We hypothesize that as the training duration is increased, approximations with less bias would be more stable (in particular, the sampling approximation might work well since it is unbiased), but we were unable to test a setting where the tail approximation would not be sufficient.

Comments on the paper

1 comment

We thank all reviewers for the time spent on reading and evaluating our work. We are glad to find that the majority of reviewers (LViA, BXCp) appreciated the principled nature of the signal-drift decomposition and the resulting fix we proposed. In addition, all reviewers (LVia, rKTp, BXCp) highly appreciated the simplicity and ease of practical implementation of score centering. Lastly, each reviewer highlighted composability of score centering with other mismatch corrections as a key strength of the paper.

A common concern was that our experiments relied on deliberately amplified mismatch, short sequence lengths, SGD, and training accuracy.

To address these points, we added a realistic agentic coding experiment with long multi-turn rollouts and held-out evaluation (Section 5.5), and an ablation that tunes the optimizer, the learning rate, and the TIS and MIS hyperparameters (Appendix A.5).

Agentic RL with an NVFP4 sampler. We train Qwen3.5-35B-A3B on multi-turn terminal-use tasks from Endless Terminals (≤20 turns, 32K context, ~10K tokens per rollout on average) using async RL (mean staleness ≈ 2 steps) and AdamW. The trainer runs in bf16, and the sampler uses NVFP4 MoE experts with an FP8 KV cache. Evaluating on 100 held-out tasks:

MethodCollapseFinal validation accuracy
PG (bf16)~step 1300%
PG (NVFP4)~step 450%
MIS (NVFP4)~step 700%
TIS (NVFP4)~step 2551%
SC (NVFP4)none (>637 steps)63%
TIS + SC (NVFP4)none (>417 steps)50%
MIS + SC (NVFP4)none (>578 steps)60%

Under NVFP4 sampling, PG, MIS, and TIS all collapse, while score centering trains stably.

Tuned baselines and AdamW. In Appendix A.5, we tune the baselines in four Countdown settings with Qwen3-0.6B (INT8 sampler, staleness of 64 steps, and weight noise with scales 0.02 and 0.05), with 3 seeds per configuration. With AdamW, we sweep three learning rates for PG, TIS with four caps, MIS with three bounds, and score centering. With SGD, we sweep four learning rates for the two strongest baselines (TIS with cap 10 and MIS with bounds [0.5, 5]) and for score centering. At each method's best learning rate, score centering outperforms the best tuned baseline in all 12 setting-seed pairs across both optimizers: by 6.6 points on average with SGD and by 4.7 points with AdamW.

We hope our updated results provide sufficient further evidence, addressing all key concerns that were raised in reviews.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.