- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies training–inference mismatch (TIM) in RL for LLMs. It decomposes the off-policy policy-gradient expectation into a drift term, , and a signal term, . The drift term distills the trainer toward the sampler, and the effect compounds because the sampler is periodically synced back to the trainer. Building on this, the paper proposes score centering (SC): at each prefix, it subtracts the sampler’s expected score. This cancels drift in expectation without using importance ratios. The correction is additive and can be combined with TIS/MIS, and a top-k approximation (k=128) keeps its overhead negligible. The experiments compare SC against eight IS-based corrections under a shared REINFORCE objective. On Qwen3-0.6B (Countdown), the settings are synthetic weight noise, an INT8 sampler, and staleness (sync every 64 steps). On Qwen3-30B-A3B-Base (INTELLECT-2 math), the settings are three sampler quantization levels. SC matches or outperforms the IS-based methods, and its advantage grows as the mismatch becomes more severe. Under staleness, SC combined with TIS/MIS performs best.
Critical strengths
- A clear mechanistic account of why RL is sensitive to TIM. The drift/signal decomposition unifies several observations. It explains why SFT remains stable even fully offline, why +1/0 rewards are unstable online, and why group centering helps but cannot remove drift. The resulting fix is principled rather than another clipping heuristic. This is the main reason I lean toward acceptance.
- A simple, practical method. SC is additive, deterministic given the prefix, adds less than 1% wall-clock overhead, composes with existing IS corrections, and comes with a short reference implementation. It should be easy to adopt in existing RL pipelines.
Critical weaknesses
- The evaluation does not reach the regime where the claims matter. (a) There is no held-out downstream benchmark. The 30B results and the quantization/staleness results report smoothed training accuracy, which is the reward of rollouts from the (quantized) sampler, not a measure of the trainer’s capability. (b) Sequences are capped at 512/1024 tokens. In practice, TIM-induced collapse is most severe in long-horizon agentic training, as the paper itself notes in §2.1. The authors argue that short runs with amplified TIM are a proxy for long runs with mild TIM, but this claim is never validated. In the only near-realistic setting (30B, FP8/FP8 KV), uncorrected PG performs on par with every correction method. The practical benefit of SC therefore remains unestablished.
- Baselines are run with fixed default hyperparameters. Every method uses its paper or verl defaults with no tuning, together with SGD at lr 1e-2 and one update per batch. The TIS/MIS bounds directly control the bias–variance trade-off under mismatch, and PPO-style clipping was designed for policy movement, not for engine mismatch. The reported gaps could shrink if the baselines were tuned. In addition, the extra 30B seeds were allocated only to the best-performing methods, so most 30B baselines have a single seed.
Questions for authors
- For the 30B runs, do the training-reward gaps carry over to held-out benchmarks (e.g., MATH500, AIME) evaluated on the trainer?
- Can you show even a single comparison (SC vs. TIS vs. PG) at substantially longer sequences, or in a multi-turn agentic setting, with natural TIM (e.g., a bf16 or FP8 sampler without artificial amplification)? This would directly support the proxy argument.
- If the strongest baselines get a small sweep (the TIS cap, the MIS bounds) and are trained with AdamW, does SC keep its advantage? The paper states that SGD performs comparably to AdamW but does not show this.
- (Clarification) The conclusion says SC is “the only method that trains stably under severe quantization.” However, TIS reaches 51% at FP8 W/A + FP4 KV. Also, under staleness, SC alone underperforms PPO/DAPO. How do these results fit with the claim that drift is the cause of instability?
Thank you for your comprehensive assessment of our paper! We greatly appreciate your summary and questions about the work. Below, we provide answers to each of them:
Q1: Held-out evaluation. We did not manage to report MATH500 or AIME results due to the rebuttal time constraints. However, across all our Qwen3-0.6B runs, we find that held-out accuracy closely tracks training accuracy (Pearson r = 0.98). We also added a new experiment using multi-turn agentic coding that includes both training and held-out accuracy. Please see our general comment for further details.
Q2: Long-horizon runs with natural TIM. Our agentic experiment (general comment and Section 5.5) uses multi-turn terminal tasks with up to 32K tokens of context (about 10K tokens per rollout on average). Even with a bf16 sampler and no artificial amplification, PG collapses around step 130. With an NVFP4 sampler, PG, MIS, and TIS collapse, while SC, TIS + SC, and MIS + SC train stably for more than 400 steps.
Q3: Hyperparameters, AdamW.
We removed the statement "SGD performs comparably to AdamW" from the Appendix and added a tuned comparison in Appendix A.5. In four Countdown settings with Qwen3-0.6B (INT8 sampler, staleness of 64 steps, and weight noise with scales 0.02 and 0.05), we sweep the optimizer learning rates, PG and TIS with caps {1.5, 2, 5, 10}, as well as MIS bounds, with 3 seeds each.
At each method's best learning rate, score centering outperforms the best tuned baseline in all 12 setting-seed pairs with both optimizers, by 6.6 points on average with SGD and by 4.7 points with AdamW. Tuning matters for the baselines: TIS with caps 1.5 and 2 and MIS with bounds [0.5, 2] are stable only at the smallest learning rate, where every method learns slowly, and under staleness every baseline collapses in at least two of three seeds at learning rate 1e-6. Score centering never collapsed with AdamW. Tuned AdamW slightly outperforms tuned SGD (by 2.0 points for score centering and 3.5 points for the baselines), but it does not close the gap.
Regarding seeds, every method in Appendix A.5 uses 3 seeds.
Q4: Severe quantization and staleness.
In the 30B Math setting that limits rollouts to 1024 tokens, TIS was indeed stable under FP8/4 quantization but became unstable under INT8/4 quantization, where only score centering was stable. In the revised version of the paper, we added agentic coding with an NVFP4 sampler and up to 32K token context, where TIS collapses while SC / TIS + SC / MIS + SC remain stable.
The question of why vanilla score centering performs poorly under large staleness is indeed interesting. In Equation 5, we point out that score centering cancels drift but the update still remains computed w.r.t. the sampler's distribution, rather than the trainer's sampling distribution. Under severe quantization error, this is desirable, since we ultimately care about optimizing the sampler rather than the trainer. However, under severe staleness, this becomes problematic, because the mismatch between the sampler and the trainer is systematic and can grow large. For example, if the trainer is 64 steps ahead of the sampler, it might have already unlearned some of the sampler's behavior, in which case we no longer want to train on those samples [1]. Therefore, under staleness, it is better to compute the gradient covariance under the trainer's distribution, which is exactly what IS achieves. We propose using TIS to effectively change the sampling distribution of the gradient, and SC to remove any remaining drift.
[1] Learning Dynamics of LLM Finetuning. Ren and Sutherland, ICLR 2025