acceptodds
Under review as a conference paper at ICLR 2027

Demystifying Training-Inference Mismatch in LLM Reinforcement Learning via a Zero-Mismatch Reference

Abstract

Modern LLM RL systems separate rollout generation from policy optimization. These two stages are expected to produce token probabilities that match exactly. However, implementation differences can make them assign different values to the same sequence under the same model weights, inducing Training–Inference Mismatch (TIM). TIM is difficult to inspect because it is entangled with off-policy drift and common stabilization mechanisms. In this work, we isolate TIM in a zero-mismatch diagnostic setting (ExactRL), and show that small token-level numerical disagreements can independently cause training collapse. We further show that TIM changes the effective optimization problem, and identify a set of remedies that could mitigate TIM. Our results suggest that TIM is not benign numerical noise, but a systems-level perturbation that should be treated as a first-order factor in analyzing LLM RL stability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Training–Inference Mismatch, LLM Reinforcement Learning, RL Training Stability, Batch-Invariant Kernels

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_u1xkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies training–inference mismatch (TIM) - the numerical mismatch in token probabilities between the trainer and the inference engine in the context of RL done on LLMs. The authors first eliminate this mismatch in the ExactRL implementation and use it as a baseline to isolate markers corresponding to collapses throughout the paper. The paper makes three contributions: (1) Using ExactRL as a mismatch-free baseline, it shows that TIM alone can destabilize training. (2) It analyzes why both standard ways of getting old-policy probabilities fail under GRPO. (3) Finally, it ablates existing post-hoc corrections and finds that truncated importance sampling combined with sequence-level rejection, based on the gap between the two engines’ probabilities, closely tracks ExactRL.

Critical strengths

The paper dives into a consequential infra issue related to RL training of LLMs which a lot of practitioners do not have (beyond high level behavioral) understanding of, and attempts to do so systematically in a clean setup.

  • The paper systematically shows that TIM can be the primary factor inducing collapses in RL training of LLMs and studies it in a clean manner
  • Through the analysis in Section 4.1, the paper provides an empirical undertstanding of why standard implementations (devoid of algorithmic TIM mitigations) suffer from TIM related issues. The observation of aggregate KL scores not exposing TIM related instabilities is also interesting.
Critical weaknesses

I believe that the exprimental setup of the paper is narrow:

  • All the experimental conclusions, except for Section 3.2 are based of of a single model x task combination.
  • In Section 4.2, the conclusion that K1 and K3 doesn't matter much is based on just one threshold value. The "effective" thresholds can differ for both the estimators. K1 vs K3 should also ideally be compared in a non-TIS setting (i.e., sequence-level rejection only), not just in combination with TIS.
Questions for authors

(This won't affect my score; just generally curious)

What was the runtime of the runs with the ExactRL implementation? Is it scalable, or just feasible to run at the scale of the experiments in the paper?

4 comments

1. Experimental Coverage

We would like to clarify that the primary goal of Section 4.2 is not to systematically compare K1 and K3 or optimize their thresholds, but to demonstrate how ExactRL serves as a zero-mismatch reference for evaluating existing TIM compensation methods. The correction methods and thresholds follow prior work (Qiu et al., 2026; Li et al., 2026), rather than being specifically tuned for our experiments. Accordingly, we have clarified that our observations regarding K1 and K3 are specific to the evaluated configurations. More importantly, our central experimental finding is established in Section 3.2 through extensive experiments across multiple models and settings. Section 4.2 serves a different purpose: demonstrating the diagnostic value of ExactRL under a controlled model–task setting, rather than establishing broadly generalizable conclusions about individual correction methods.

2. System speed

On one H100, our end-to-end rollout inference benchmark on Qwen3-8B/GSM8K achieves 13,976 output tokens/s versus 16,690 for vLLM, equivalent to 19.4% runtime for the same token workload. Across Qwen3-8B and Qwen3-30B-A3B on GSM8K and ShareGPT, runtime overhead is 19–34%; throughput is 0.96–1.23× that of batch-invariant vLLM. Comparisons use matched token workloads, ignored EOS, temperature 1, equal KV-cache caps, and async scheduling for ExactRL.

Thank you again for your constructive feedback. We would appreciate any further thoughts on our clarification of the experimental scope and the additional system performance results, particularly whether these address your concerns. We would be happy to provide further clarification if needed.

I would like to thanks the authors for their response, clarifying the scope of Section 4.2 and providing the throughput numbers.

While I still hold reservations about the breadth of the experimental setup (the number of tasks in particular), this work was an interesting read and provides useful guidance for establishing diagnostic practices in RL post-training pipelines. I would thus increase my score to 3.

I would suggest the authors include the throughput scores in the revised manuscript.

Thank you for taking the time to revisit our submission. We appreciate your recognition and will include the throughput results in the next version of the paper.

Reviewer_TnrjOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies training–inference mismatch in LLM RL: rollout and training stacks assigning different token log-probabilities to the same inputs under the same weights. The authors build ExactRL, which aligns with FSDP through unified model implementations and deterministic, batch-invariant kernels, and use it as a zero-mismatch diagnostic reference. The experiments first use one update per rollout batch in REINFORCE, then compare recomputation, bypass, and existing TIS/SRS corrections in GRPO.

The findings include heterogeneous dense/MoE failure trajectories, limitations of aggregate KL indicators, a same-token contribution probe with two references, and a correction combination that approaches ExactRL within the evaluated window. The paper has genuine engineering and diagnostic value. My main reservations are that the local probes do not yet identify the specific joint-update/transfer mechanism, while the patch comparisons do not separate changes in clipping, selection, and effective updates.

Critical strengths

ExactRL is a working zero-mismatch reference, and the mismatch curves are indeed zero during training. Using single-update REINFORCE for the first comparison avoids interference from PPO mini-step updates and clipping, which is a clean design.

The paper records reward, mismatch, loss, gradient norm and KL estimates together, and shows that failures take different forms; for example, the KL estimates stay almost flat during early degradation. This is useful for diagnosing unstable RL training.

Critical weaknesses

The central conclusion has precedents. That TIM alone causes collapse and that bitwise consistency avoids it has already been shown with direct comparisons: in the Thinking Machines blog [1], the mismatched run collapsed partway through training while the bitwise-consistent run trained smoothly, and the vLLM/TorchTitan blog [2] on Qwen3-1.7B, as well as TBIK [3], which the paper itself cites, report similar comparisons. The paper treats these works only as sources of kernels and does not state what Section 3.2 adds, so this main conclusion offers limited novelty.

[1] Horace He and Thinking Machines Lab. Defeating Nondeterminism in LLM Inference. Thinking Machines Lab: Connectionism, September 2025. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/

[2] vLLM and TorchTitan Teams. No More Train-Inference Mismatch: Bitwise Consistent On-Policy Reinforcement Learning with vLLM and TorchTitan. vLLM Blog, November 2025. https://blog.vllm.ai/2025/11/10/bitwise-consistent-train-inference.html

[3] Ziyang Zhang, Xinheng Ding, Jiayi Yuan, Rixin Liu, Huizi Mao, Jiarong Xing, and Zirui Liu. Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch. arXiv:2511.17826, 2025. https://arxiv.org/abs/2511.17826

The paper claims that ExactRL "can ensure" that every token's log-probability moves in the direction of its advantage sign, which cannot be guaranteed under joint optimization, and in practice training does not require it either. The explanation of why bypass fails in Section 4.1 therefore lacks credible support. Parameters are shared, the change in one token's probability depends on the gradients of all samples in the batch, and correct and incorrect responses in the same group share many identical tokens, so even with no mismatch at all, a joint update almost inevitably moves some tokens against their advantage. The paper's own Figure 5(a) also shows that the zero-centered loss contribution under ExactRL is above zero, meaning that some token probabilities do move against their advantage. The 17.3% figure it is contrasted with is likewise given in a single sentence without the measurement method, for example which engine computes the log-probabilities before and after the update; if different engines are used, their existing numerical gap alone changes the measured direction. The authors should explain exactly how the 17.3% is measured and report the measured rate for ExactRL under the same procedure, rather than only asserting that ExactRL can ensure alignment.

Section 4.2 attributes the effect of TIS combined with sequence-level rejection to token-level truncation, but the actual truncation rate may be very low. To my knowledge, 2 is a very large truncation threshold for train-inference mismatch, and the sequence-level rejection threshold is strict, so accepted sequences are unlikely to contain tokens that need truncation. If truncation rarely happens, the gain can hardly be attributed to mismatch at multiple granularities or to the handling of outlier tokens, and more likely comes from sequence-level rejection making the updates more conservative. The paper does not report the truncation rate of TIS or the number of sequences removed by sequence-level rejection, so the source of the gain cannot be determined.

Questions for authors
  1. Please explain how the 17.3% figure is measured, including which engine computes the log-probabilities before and after the update and how many tokens are counted, and report the rate for ExactRL under the same measurement.

  2. To my knowledge, 2 is a very large truncation threshold for train-inference mismatch, and the truncation rate of TIS may be very low. Please report the actual truncation rate of TIS and the number of sequences rejected by the sequence-level filter.

Update: Thank you for the author's rebuttal. I have updated my score.

2 comments

1. Novelty relative to prior & concurrent work

We would like to clarify our contribution relative to prior and concurrent works. TML blog is the technical foundation of ExactRL, showing that zero-mismatch RL is achievable and beneficial. We have already cited this work in the manuscript. Our contribution is to study the internal mechanism of how TIM collapses training. TBIK is an infrastructure-oriented work, showing that batch-invariant is also achievable under tensor parallelism, not a study for RL training dynamics. We have also cited this work already in the related work section. Similar to TBIK, TorchTitan/vLLM work is also infrastructure-oriented work. We will cite this work as well in the related work section in the revised version.

Our contribution is the in-depth ablation study and analysis on TIM. ExactRL enables controlled isolation of TIM, direct comparison between mismatched and TIM-free training, objective-level failure-mode analysis, and calibration of existing mitigations.

2a. Clarification on the token update direction statement

We would like to clarify our statement on the token probability update direction. The corrected statement is that, with ExactRL, actor/sampler log-probabilities match exactly at both pre-update and post-update weights on all recorded diagnostic tokens. In supplemental Qwen3-1.7B/DAPO experiments, we replay identical tokens before and after the same update using each engine’s own anchors. We show the cross-engine disagreement rate and token update reversal rate for the first 100 updates for both ExactRL and vLLM:

ConfigurationCross-Engine disagreementAgainst advantage
ExactRL + GPG0.00%23.66%
vLLM + GPG21.74%26.57%

ExactRL has zero cross-engine disagreement throughout. Bypass changes the denominator to behavior probabilities, but retains the trainer-side numerator and gradient. It therefore does not enforce execution consistency. In the following section, we further explain why this is related to training collapse.

2b. How token update direction reversal is related to training collapse

Furthermore, we would like to provide more evidence to demonstrate the relationship between token update direction reversal rate and RL training stability. Due to limitations of OpenReview, we show the results as a table here and update the manuscript with more details in Appendix Figure 10. In the following table, we report the training curve of vLLM PPO bypass mode:

  • For the first 125 steps, the training process is effective while the reversal rate is decreasing.
  • For the step from 125 to 225, the training is collapsing while the reversal rate is increasing at the same time.
  • After step 225, the model is fully collapsed (validation accuracy is 0%).
UpdatesReversal rateTrain correctnessAIME2024 accuracy
1–2527.64%8.45%—
26–5027.02%19.53%3.33% (1/30)
51–7526.68%35.12%—
76–10025.19%41.69%20.00% (6/30)
101–12524.39%43.42%—
126–15025.89%41.30%16.67% (5/30)
151–17527.35%40.91%—
176–20029.26%34.44%3.33% (1/30)
201–22529.27%33.92%—
226–40024.37%15.48%0.00% at steps 250, 300, 350, 400

3. TIS threshold

We would like to clarify that the token-level threshold (tok=2) is applied to the correction ratio, rather than directly to the token-level log-probability difference. For every retained token, TIS+SRS applies the weight min⁡(rcorr,2)\min(r_{\mathrm{corr}}, 2), so the loss is still scaled by rcorrr_{\mathrm{corr}} below the cap. We agree that the cap was rarely triggered: over steps 1–800 of the TIS+SRS K3, only 3,571 valid response tokens (0.000202%) exceeded the cap before sequence rejection. Over the same window, TIS+SRS K3 rejected 3.87%, and SRS-only K3 rejected 7.27%. However, this does not imply that importance correction was inactive: TIS still applies the correction ratio to reweight token losses, even when the truncation threshold is not reached.

More importantly, our primary goal in Section 4.2 is not to establish the relative effectiveness of individual correction mechanisms, but to demonstrate how ExactRL serves as a zero-mismatch reference for evaluating existing TIM compensation methods. The correction methods and thresholds (tok=2, seq=0.001) follow prior work (Qiu et al., 2026; Li et al., 2026). Accordingly, we have revised Section 4.2 to remove the attribution to multi-granularity TIM effects and clarify the scope of our conclusions.

Thank you again for your detailed feedback. We would appreciate any further thoughts on our clarification of the contributions relative to prior work and the additional experimental evidence. We would be happy to address any remaining concerns.

Reviewer_MZwCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Understanding how differences between training and rollout engines affect RL stability is valuable. Using ExactRL as a zero-mismatch reference and starting with REINFORCE helps separate numerical mismatch from the effects of PPO clipping and repeated updates.

Critical strengths

Starting with REINFORCE helps separate numerical mismatch from PPO clipping and repeated updates. The experiments on both dense and MoE models provide clear examples of improved stability when the mismatch is removed.

Critical weaknesses

The explanation for bypass failure needs stronger evidence. The paper reports that 17.3% of token log-probability changes disagree with their advantage signs. However, this alone does not establish that mismatch caused those changes: tokens share model parameters, so a joint update need not move every token in its preferred direction, even without mismatch.

Questions for authors

For the same checkpoint, batch, and parameter update, could you directly compare log-probability changes in the training and rollout engines?

3 comments

1. Shared parameters and the guarantee

We agree that joint optimization need not move every token in its advantage direction. We would like to further clarify our statement and update our submission as well. The corrected statement is that, with ExactRL, actor and sampler log-probabilities match exactly at both the pre-update and post-update weights, on all recorded diagnostic tokens, which also helps to bring down the ratio of token probability update that is against its advantage. More details are provided in the following and the updated manuscript as well.

2. Same-checkpoint, same-batch comparison

We performed the requested comparison on Qwen3-1.7B/DAPO: identical tokens and contexts, the same parameter update, and replay after sampler weight synchronization. Let ΔT and ΔR denote each engine’s own post-update minus pre-update log-probability. Cross-engine disagreement is ΔT·ΔR < 0; against-advantage reversal is A·ΔR < 0. The latter excludes A = 0. Unthresholded rates retain zero changes in the denominator. We show the cross-engine disagreement rate and token update reversal rate for the first 100 updates for both ExactRL and vLLM:

ConfigurationCross-Engine DisagreementAgainst-Advantage Reversal
ExactRL + GPG0.00%23.66%
vLLM + GPG21.74%26.57%

ExactRL has zero cross-engine disagreement throughout. In the 2b section of the reply to review Tnrj, we also show the results of how the reversal rate against advantage is related to the training collapse.

Thank you again for your constructive feedback. We would appreciate any further thoughts on our additional same-checkpoint, same-batch experiments, particularly whether they address your concerns. We would be happy to provide further clarification if needed.

The corrected statement and the same-update comparison are helpful. They address the issue with the original claim about token update directions, though the link to training collapse remains less clear to me. I still see value in the problem and in using a zero-mismatch reference to study it. I will keep my current rating, while retaining some reservations about the explanation of the failure mechanism.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all reviewers for their constructive feedback and for recognizing the practical importance of understanding training–inference mismatch (TIM) in LLM reinforcement learning.

We would like to emphasize that our work develops ExactRL, a zero-mismatch reference implementation, and leverages it to systematically investigate how TIM affects RL training dynamics. While prior work has highlighted the importance of numerical consistency in LLM RL, our study goes beyond these observations by isolating TIM from other factors, analyzing training instability and failure behaviors under controlled settings, and evaluating existing mitigation methods against a zero-mismatch reference.

We appreciate the reviewers' suggestions. In response to the reviewers' concerns, we have provided additional evidence and clarified several conclusions:

  • Mechanism analysis: We clarified our statement regarding token update directions and conducted same-checkpoint, same-batch comparisons, observing 0% cross-engine directional disagreement with ExactRL versus 21.74% with vLLM.
  • Correction methods: We reported additional statistics on token-level truncation and sequence-level rejection, and refined the discussion in Section 4.2 to better highlight the diagnostic value of ExactRL in evaluating existing TIM mitigation methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.