- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies training–inference mismatch (TIM) - the numerical mismatch in token probabilities between the trainer and the inference engine in the context of RL done on LLMs. The authors first eliminate this mismatch in the ExactRL implementation and use it as a baseline to isolate markers corresponding to collapses throughout the paper. The paper makes three contributions: (1) Using ExactRL as a mismatch-free baseline, it shows that TIM alone can destabilize training. (2) It analyzes why both standard ways of getting old-policy probabilities fail under GRPO. (3) Finally, it ablates existing post-hoc corrections and finds that truncated importance sampling combined with sequence-level rejection, based on the gap between the two engines’ probabilities, closely tracks ExactRL.
Critical strengths
The paper dives into a consequential infra issue related to RL training of LLMs which a lot of practitioners do not have (beyond high level behavioral) understanding of, and attempts to do so systematically in a clean setup.
- The paper systematically shows that TIM can be the primary factor inducing collapses in RL training of LLMs and studies it in a clean manner
- Through the analysis in Section 4.1, the paper provides an empirical undertstanding of why standard implementations (devoid of algorithmic TIM mitigations) suffer from TIM related issues. The observation of aggregate KL scores not exposing TIM related instabilities is also interesting.
Critical weaknesses
I believe that the exprimental setup of the paper is narrow:
- All the experimental conclusions, except for Section 3.2 are based of of a single model x task combination.
- In Section 4.2, the conclusion that K1 and K3 doesn't matter much is based on just one threshold value. The "effective" thresholds can differ for both the estimators. K1 vs K3 should also ideally be compared in a non-TIS setting (i.e., sequence-level rejection only), not just in combination with TIS.
Questions for authors
(This won't affect my score; just generally curious)
What was the runtime of the runs with the ExactRL implementation? Is it scalable, or just feasible to run at the scale of the experiments in the paper?
4 comments
Thank you again for your constructive feedback. We would appreciate any further thoughts on our clarification of the experimental scope and the additional system performance results, particularly whether these address your concerns. We would be happy to provide further clarification if needed.
I would like to thanks the authors for their response, clarifying the scope of Section 4.2 and providing the throughput numbers.
While I still hold reservations about the breadth of the experimental setup (the number of tasks in particular), this work was an interesting read and provides useful guidance for establishing diagnostic practices in RL post-training pipelines. I would thus increase my score to 3.
I would suggest the authors include the throughput scores in the revised manuscript.
Thank you for taking the time to revisit our submission. We appreciate your recognition and will include the throughput results in the next version of the paper.
1. Experimental Coverage
We would like to clarify that the primary goal of Section 4.2 is not to systematically compare K1 and K3 or optimize their thresholds, but to demonstrate how ExactRL serves as a zero-mismatch reference for evaluating existing TIM compensation methods. The correction methods and thresholds follow prior work (Qiu et al., 2026; Li et al., 2026), rather than being specifically tuned for our experiments. Accordingly, we have clarified that our observations regarding K1 and K3 are specific to the evaluated configurations. More importantly, our central experimental finding is established in Section 3.2 through extensive experiments across multiple models and settings. Section 4.2 serves a different purpose: demonstrating the diagnostic value of ExactRL under a controlled model–task setting, rather than establishing broadly generalizable conclusions about individual correction methods.
2. System speed
On one H100, our end-to-end rollout inference benchmark on Qwen3-8B/GSM8K achieves 13,976 output tokens/s versus 16,690 for vLLM, equivalent to 19.4% runtime for the same token workload. Across Qwen3-8B and Qwen3-30B-A3B on GSM8K and ShareGPT, runtime overhead is 19–34%; throughput is 0.96–1.23× that of batch-invariant vLLM. Comparisons use matched token workloads, ignored EOS, temperature 1, equal KV-cache caps, and async scheduling for ExactRL.