acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Training–Inference Mismatch in LLM Reinforcement Learning: Where It Arises and How to Correct It

11 views

Abstract

We study training–inference mismatch in reinforcement learning with verifiable rewards (RLVR) for large language models, where rollouts are sampled by an inference engine while gradients are computed by a training engine, and the two engines assign different probabilities to the same tokens. To account for this discrepancy in policy updates, we introduce calibrated importance sampling (CIS). CIS is motivated by an empirically supported logit-displacement characterization that expresses the mismatch as an additive displacement in log-odds, determined by the per-logit perturbation before the softmax, whose distribution is approximately invariant to token confidence. This characterization motivates a confidence-aware truncation: large positive displacements are truncated at a single constant threshold, which maps back to an importance-ratio cap that tightens as token confidence increases. Theoretically, we show that CIS replaces the unbounded second moment that governs the error of exact importance sampling with a term bounded by a constant, at the cost of a bias controlled by the truncated excess. In evaluation across three mixture-of-experts models and five mathematical reasoning benchmarks, CIS achieves the highest five-benchmark average on all three models among the evaluated baselines. Diagnostic analyses show that CIS places less truncation bias on low-confidence tokens than truncated importance sampling, while upward clipping of small importance weights reduces held-out accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning with verifiable rewards, training-inference mismatch, importance sampling, large language model post-training, policy optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yHNHOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper rewrites the training-to-inference probability ratio in terms of token confidence and a log-odds displacement. Measurements suggest that the displacement has a much more stable scale across confidence levels than the raw ratio. This motivates CIS, which clips large positive displacements and produces a confidence-dependent importance-weight cap. A small floor handles the high-confidence numerical regime. The authors derive bias-variance bounds and evaluate three MoE models trained on GSM8K, with three seeds and several correction baselines. CIS obtains the highest average across the five math benchmarks for each model.

Critical strengths

The ratio-displacement identity makes the method easy to understand. It explains why the same ratio threshold corresponds to different displacement thresholds at different confidence levels, and the resulting operator is simple. The paper is also clear about the reach of the theory: bounding a second moment motivates truncation, but does not prove that CIS is better than every other bounded cap. The numerical-floor version is covered separately in the appendix.

The experimental details inspire more confidence than an isolated best-score comparison would. There are three seeds, three MoE models, floor and clipping ablations, and a corrected KPop implementation that avoids a numerical masking artifact. The authors also point out where margins are within seed variation. Together with the shared training setup, this makes the reported gains worth taking seriously.

Critical weaknesses

My main reservation is the tuning comparison. As acknowledged in Appendix D.6, CIS's threshold and floor are chosen from a grid on Qwen1.5-MoE, while the baselines use their reference defaults. Matching TIS's truncation rate is a useful diagnostic, but it does not replace tuning its cap. The gains over the strongest baseline are fairly small, so this could affect the conclusion. Keeping CIS's settings fixed on the other two models helps, and is a reason I still view the empirical result positively.

The mechanism is plausible, but the quantitative diagnostics do not directly measure the training gradient. Figure 5(c,d) substitutes a logit-gradient proxy for the advantage-weighted PPO parameter gradient. Also, Appendix A.7 scores step-57 rollouts with step-86 engines when estimating the large moments. That can reveal large ratios, but the sampled tokens do not come from the step-86 behavior distribution assumed in the corresponding variance interpretation. Clarifying or checking these two points would make the explanation of the gains more convincing. They do not undermine the ratio identity or directly contradict the accuracy results.

Questions for authors
  1. Do the gains over TIS and IcePop persist when their thresholds receive a comparable validation budget? Paired seed-level differences would also help interpret the small aggregate margins.

  2. Can you repeat the moment and variance-by-side measurements using rollouts generated by the same checkpoint used for both engine probabilities? Alternatively, how is the step-57/step-86 sampling difference accounted for in Appendix A.7?

  3. Does the bias-variance ordering in Figure 5 survive when using the actual advantage-weighted PPO gradient? A check with gradient projections or a small set of parameter blocks would help connect the diagnostic to the optimizer.

2 comments

We thank the reviewer for recognizing the ratio decomposition, the scope of the theory, and the experimental details, and for the suggestions for further validation.

W1 – My main reservation concerns hyperparameter tuning. Appendix D.6 acknowledges that the CIS threshold and floor are selected through a grid search on Qwen1.5-MoE, while the baselines use reference defaults. Matching the TIS truncation rate is a useful diagnostic, but does not replace tuning its truncation cap. Could this affect the conclusions given the small gains over the strongest baseline? Fixing the CIS parameters on the other two models helps and is one reason I remain positive about the experimental results.

Following this suggestion, we compared three candidate configurations each for CIS, TIS, IcePop, and KPop†, using the same training budget, three training seeds, and selection criterion: average greedy pass@1 across the five benchmarks at the final checkpoint. All parameters are selected on Qwen1.5-MoE and then transferred unchanged to the other two models. For CIS, κ=0.005\kappa=0.005 is fixed beforehand based on numerical precision; performance comparisons across different κ\kappa values are ablations. The CIS parameter search is over λ\lambda.

For TIS, we report both the truncation rate and accuracy at different thresholds.

Qwen1.5-MoE, mean ± standard deviation over three training seeds. Default and truncation-rate-matched results are reused from Tables 1, 2, and 10; C=1.5C=1.5 and C=5C=5 are additional experiments.

Method and settingTruncated tokens (%)Average accuracy across five benchmarks (%)
TIS, C=1.13C=1.134.32 ± 0.648.88 ± 9.24
TIS, C=1.5C=1.51.49 ± 0.0716.32 ± 6.47
TIS, C=2C=2 (default)0.76 ± 0.0731.40 ± 2.47
TIS, C=5C=50.22 ± 0.0631.25 ± 3.02
CIS, λ=2.3, κ=0.005\lambda=2.3,\ \kappa=0.0053.81 ± 0.4234.78 ± 0.25

Among the tested thresholds, TIS reaches a highest mean of 31.40, which is 3.38 percentage points below CIS. Increasing the cap from C=2C=2 to C=5C=5 further reduces truncation but does not improve average accuracy. At C=1.13C=1.13, TIS has a truncation rate close to CIS but much lower accuracy, also showing why matching truncation rates cannot replace threshold tuning.

The best candidates for all three baselines are their reference defaults, so their runs on the other two models in Table 1 already use the configurations selected by this search. Under the same search budget, CIS exceeds the selected TIS, IcePop, and KPop† configurations by 3.38, 0.60, and 1.28 percentage points, respectively.

W2 – The mechanism is plausible, but the quantitative diagnostics do not directly measure training gradients. Figures 5c–d use a logit-gradient proxy in place of advantage-weighted PPO parameter gradients. Appendix A.7 also estimates the large moments by scoring step-57 rollouts with step-86 engines. This can reveal large ratios, but the sampled tokens are not from the step-86 behavior distribution assumed by the variance interpretation. Could these two points be clarified or checked to strengthen the mechanism explanation? They do not affect the ratio identity or directly invalidate the accuracy results.

Thank you for pointing this out. The checkpoint description in Appendix A.7 is incorrect: the original experiment is actually 86/86. We clarify this and add the 57/57 results in Q2. For the gradient-proxy concern, we measured actual PPO parameter gradients, with results reported in Q3.

Q1 – Do the gains remain after giving TIS and IcePop the same validation budget? Paired seed-level differences would also help interpret the relatively small overall gains.

Yes. Under the same search budget, the selected TIS and IcePop configurations are their reference defaults. Based on the mean accuracies in Table 1, CIS outperforms both baselines on all three models.

Differences in average greedy pass@1 across the five benchmarks, in percentage points, calculated from the means in Table 1.

ModelCIS − TISCIS − IcePop
Qwen1.5-MoE-A2.7B3.380.60
DeepSeek-V2-Lite1.140.43
Qwen3-30B-A3B0.740.89

The paired differences against IcePop under the same training seeds are reported below.

Percentage points; each entry is CIS minus IcePop under the same seed.

ModelSeed ASeed BSeed C
Qwen1.5-MoE-A2.7B−0.100.611.31
DeepSeek-V2-Lite0.000.211.08
Qwen3-30B-A3B0.820.841.03

Across the nine pairs, seven differences are positive, one is a tie, and one is negative. The mean gain is 0.64 percentage points, with a paired t-statistic of 3.86 (8 degrees of freedom, p=0.005p=0.005).

Q2 – Could you repeat the moment and side-specific variance measurements using rollouts generated at the same checkpoint used by both engines to compute probabilities? Alternatively, how does Appendix A.7 account for the sampling difference between steps 57 and 86?

Thank you for pointing this out. The checkpoint description in Appendix A.7 is incorrect: the original experiment generated responses at step 86 and scored them with both engines loaded at step 86, i.e., 86/86 rather than the stated 57/86. We will correct this description and add the diagnostic results under the 57/57 setting.

Both experiments use Qwen1.5-MoE with 4,000 trajectories each. Let k=p/qk=p/q and u=(1−p)eεu=(1-p)e^\varepsilon. Inference probabilities in both experiments are obtained through prefill rescoring; displacement moments are computed at positions with finite log-odds.

Metric57/57 (additional)86/86 (Appendix A.7)
Number of tokens578,686510,915
Coverage of finite log-odds77.2%66.3%
E[e2ε]\mathbb E[e^{2\varepsilon}]1.13×1061.13\times10^69.23×1069.23\times10^6
E[u2]\mathbb E[u^2]71.3022.87
E[k2]\mathbb E[k^2]56.016.1
Variance contribution from k>1k>199.99%99.96%
Variance contribution from k<1k<10.01%0.04%
Variance contribution from the top 0.01% of tokens ranked by kk99.8%99.0%
Contribution of the top 10 tokens to E[u2]\mathbb E[u^2]98.7%94.9%

The additional results agree with the original experiment: variance is dominated by a small number of tokens with k>1k>1, supporting the design of truncating large importance ratios on the upper side.

Q3 – Does the bias–variance ranking in Figure 5 hold when using actual advantage-weighted PPO gradients? Gradient projections or measurements on a few parameter blocks would help connect the diagnostics to the optimizer.

Yes. At a fixed step-86 checkpoint and on the same rollouts, we backpropagate through the training PPO loss and compute parameter gradients for the last Transformer block of Qwen1.5-MoE. All methods share the same rollouts and group advantages, changing only the detached mismatch weights. This retains the effects of the advantage and network Jacobian. As in training, each batch is used for one update, so the PPO ratio is 1 at the measurement point and clipping is inactive.

The measurements use 125 independent batches, each containing 8 prompt groups. Bias is the norm of the mean paired gradient difference relative to exact weighting. Additional variance is the increase in the trace of the batch-gradient covariance relative to no correction.

Correction ruleBias normAdditional gradient variance
No correction0.05140
Exact ratio00.200
TIS0.02700.030
CIS0.02070.074

The parameter-gradient results agree with the proxy diagnostics in Figure 5: CIS has lower bias than TIS, while its additional variance is 37% of that under exact weighting. CIS has higher variance than TIS, reflecting their different bias–variance trade-offs.

We thank the reviewer again for the careful review. If these additional experiments and explanations address your concerns, we would appreciate your reconsideration of the rating.

Reviewer_oeSkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies training–inference mismatch in LLM reinforcement learning and proposes Calibrated Importance Sampling (CIS). The method expresses the token-level importance ratio in terms of token confidence and a log-odds displacement, then truncates the displacement to obtain a confidence-dependent ratio cap.

Critical strengths
  1. The paper addresses an important issue in decoupled RL training systems. The measurements of engine disagreement and its association with MoE routing differences provide useful empirical characterization.

  2. Consistent empirical results and useful ablations. The main experiments cover three MoE models, report three training seeds, and evaluate the final checkpoint. CIS achieves the highest five-benchmark average on all three models. The ablations examine fixed caps, lower-side weight modification, and the numerical floor. The training statistics also show substantially smaller gradient spikes than exact importance weighting.

Critical weaknesses
  1. Significant omission of DPPO affects both novelty and methodological positioning.
    The paper does not discuss Rethinking the Trust Region in LLM Reinforcement Learning (Qi et al., ICML 2026), which introduces DPPO. This work directly examines the confidence dependence of probability-ratio clipping and, in Section 5.4, discusses how TIS can disproportionately suppress low-probability tokens and introduce harmful bias. These observations substantially overlap with the present paper’s motivation. Moreover, DPPO’s Section 5.2 investigates the instability of anchoring the trust region to a recomputed training distribution, whereas Equation (1) here retains that decoupled structure. CIS and DPPO are not equivalent, but the manuscript needs to explain its additional insight, address this anchor-related concern, and provide a controlled comparison.

  2. Unequal hyperparameter selection limits the interpretation of the performance gains. Appendix D.6 acknowledges that CIS benefits from parameter selection on Qwen1.5-MoE, while the baselines use their reference implementations’ default settings. This matters because CIS exceeds the strongest evaluated baseline by only 0.60, 0.43, and 0.65 percentage points across the three models. These results establish the highest observed means, but do not yet establish a reliable advantage over well-tuned alternatives.

  3. The key variance diagnostics use a sampling distribution that does not match the importance ratios. According to Appendix A.7, Figures 3 and 5a use tokens generated by the step-57 checkpoint, but compute probabilities with both engines loaded from step 86.

The inference probabilities are also obtained through prefill rescoring. These measurements can reveal engine disagreement on the selected tokens, but they do not directly estimate the second moment governing importance sampling under the step-86 sampling distribution. The quantitative variance claims should be verified using matched-checkpoint rollouts and actual sampling-time inference probabilities.

Questions for authors

What precisely is new relative to DPPO’s analysis of low-probability-token truncation bias, and how does CIS address its findings concerning the trust-region anchor? Can the authors provide a comparison with DPPO using its original rollout-anchored formulation?

7 comments

We thank the reviewer for recognizing the engine-disagreement measurements, MoE routing analysis, and consistency of the experiments, and for the specific questions about DPPO.

W1 – Omitting DPPO substantially affects the novelty and positioning of the method. The paper does not discuss Rethinking the Trust Region in LLM Reinforcement Learning (Qi et al., ICML 2026), which introduces DPPO. That work directly studies the confidence dependence of probability-ratio clipping and discusses in Section 5.4 how TIS can over-suppress low-probability tokens and introduce harmful bias, substantially overlapping with this paper's motivation. In addition, DPPO Section 5.2 studies the instability caused by using a recomputed training distribution as the trust-region anchor, while Eq. (1) retains this decoupled structure.

What does this paper add beyond DPPO's analysis of low-probability-token truncation bias? How does CIS address its findings about the trust-region anchor? Could you compare against DPPO in its original rollout-anchored form?

We have added a citation to DPPO [1] and a discussion of its relation to CIS in the main text. DPPO constrains updates through the divergence between the current training policy and the rollout policy. CIS addresses the mismatch ratio kt=pt/qtk_t=p_t/q_t between the training and inference engines at identical parameters. DPPO Section 5.4 observes that TIS can reduce training stability and attributes this to bias from excessive truncation of low-probability tokens. This observation is directly relevant to our motivation.

Our work adds three findings beyond this observation.

First, the identity kt=pt/qt=pt+(1−pt)eεtk_t=p_t/q_t=p_t+(1-p_t)e^{\varepsilon_t} gives the exact form of the ratio's confidence dependence: the effect of the same log-odds displacement εt\varepsilon_t on the ratio is scaled by 1−pt1-p_t.

Second, our measurements show that the scale of the displacement itself varies little with confidence. Within the region where CIS acts, the MAD of εt\varepsilon_t varies by only a factor of 1.3 across 18 confidence bins, compared with a factor of 115 for log⁡kt\log k_t (Appendix A.6). The heavy tail of εt\varepsilon_t is also associated with MoE routing disagreement (Appendices A.3–A.4).

Third, these findings motivate a common threshold eεt≤1+λe^{\varepsilon_t}\le1+\lambda in displacement coordinates. Mapping it back to ratio space gives the confidence-dependent, one-sided weight ft=min⁡{kt,1+λmax⁡(1−pt,κ)}f_t=\min\{k_t,1+\lambda\max(1-p_t,\kappa)\}. On tokens with pt<0.3p_t<0.3, its truncation bias is less than half that of TIS, and its overall logit-gradient bias is also lower (0.0064 vs. 0.0078, Figures 5b–c).

Regarding the anchor, Eq. (1) retains the structure anchored to recomputed old training probabilities. In our training recipe, however, each batch is used for one parameter update. At gradient evaluation, ρi,t=1\rho_{i,t}=1 and PPO clipping is inactive. Engine mismatch enters the gradient through ki,tk_{i,t} and is controlled by the CIS cap. CIS remains stable in this setting: its peak gradient norm is 3.0 ± 1.1, compared with 69.5 ± 41.4 for exact ratios (Table 10). We will clarify the number of updates per batch in Appendix C.1.

Following the reviewer's suggestion, we added a comparison with the original rollout-anchored DPPO-Binary-KL, retaining its advantage-direction mask and using the official script's default binary-KL threshold of 0.05. Both methods use the same data, optimizer, update budget, training seeds, and final-checkpoint evaluation.

Qwen1.5-MoE-A2.7B, average greedy pass@1 (%) across five benchmarks, mean ± standard deviation over three training seeds. The CIS result is reused from Table 1.

MethodTrust-region anchorFive-benchmark average
DPPO-Binary-KLOriginal rollout policy33.04 ± 0.31
CIS, Eq. (1)Recomputed old training policy34.78 ± 0.25

CIS is 1.74 percentage points higher on average and outperforms DPPO-Binary-KL on all five benchmarks. We will include this comparison in the appendix.

W2 – Different hyperparameter-selection procedures limit the interpretation of the gains. Appendix D.6 acknowledges that CIS is tuned on Qwen1.5-MoE, while the baselines use reference defaults. The margins over the strongest baseline are only 0.60, 0.43, and 0.65 percentage points across the three models. These establish the highest observed means, but do they also establish a reliable advantage over well-tuned alternatives?

Following this suggestion, we compared three candidate configurations each for CIS, TIS, IcePop, and KPop† under the same search budget. All experiments use the same training budget, three training seeds, and selection criterion: average greedy pass@1 across the five benchmarks at the final checkpoint. All parameters are selected on Qwen1.5-MoE-A2.7B; CIS fixes κ=0.005\kappa=0.005 and searches over λ\lambda.

Qwen1.5-MoE-A2.7B, mean ± standard deviation over three training seeds. Bold indicates the selected configuration. Baseline default results are reused from Table 1, the three CIS configurations are from Table 11, and the remaining configurations are additional experiments.

MethodCandidate 1Candidate 2Candidate 3
TISC=1.5C=1.5: 16.32 ± 6.47C=2C=2 (default): 31.40 ± 2.47C=5C=5: 31.25 ± 3.02
IcePop[0.67,1.5][0.67,1.5]: 24.61 ± 3.69[0.5,5][0.5,5] (default): 34.18 ± 0.55[0.2,10][0.2,10]: 33.50 ± 0.58
KPop†ϕ=0.5\phi=0.5: 33.05 ± 1.09ϕ=2\phi=2 (default): 33.50 ± 1.04ϕ=8\phi=8: 31.80 ± 0.62
CISλ=1.6\lambda=1.6: 33.89 ± 0.46λ=2.3\lambda=2.3: 34.78 ± 0.25λ=3.1\lambda=3.1: 33.35 ± 0.66

The best candidates for all three baselines are their reference defaults, so their runs on the other two models in Table 1 already use the configurations selected by this search. Under the same search budget, CIS exceeds the selected TIS, IcePop, and KPop† configurations by 3.38, 0.60, and 1.28 percentage points, respectively.

For consistency across seeds, the nine paired differences against IcePop across the three models contain seven positive differences, one tie, and one negative difference. The mean gain is 0.64 percentage points, with a paired t-statistic of 3.86 (8 degrees of freedom, p=0.005p=0.005). Against TIS, the differences in mean accuracy from Table 1 are 3.38, 1.14, and 0.74 percentage points across the three models. The strongest baseline on Qwen3-30B-A3B is GSPO, over which CIS improves by 0.65 percentage points (69.88 vs. 69.23).

W3 – The key variance diagnostics use a sampling distribution that does not match the importance ratios. Appendix A.7 states that Figures 3 and 5a use tokens generated at step 57, while both engines compute probabilities using step-86 parameters.

Inference probabilities also come from prefill rescoring. These measurements can reveal engine disagreement on the selected tokens, but cannot directly estimate the second moment governing IS variance under the step-86 sampling distribution. Could you validate the quantitative variance claims using matched-checkpoint rollouts and actual sampling-time inference probabilities?

Thank you for pointing this out. The checkpoint description in Appendix A.7 is incorrect: the original experiment generated responses at step 86 and scored them with both engines loaded at step 86, i.e., 86/86 rather than the stated 57/86. We will correct this description and add the diagnostic results under the 57/57 setting.

Both experiments use Qwen1.5-MoE with 4,000 trajectories each. Let k=p/qk=p/q and u=(1−p)eεu=(1-p)e^\varepsilon. Inference probabilities in both experiments are obtained through prefill rescoring; displacement moments are computed at positions with finite log-odds.

Metric57/57 (additional)86/86 (Appendix A.7)
Number of tokens578,686510,915
Coverage of finite log-odds77.2%66.3%
E[e2ε]\mathbb E[e^{2\varepsilon}]1.13×1061.13\times10^69.23×1069.23\times10^6
E[u2]\mathbb E[u^2]71.3022.87
E[k2]\mathbb E[k^2]56.016.1
Variance contribution from k>1k>199.99%99.96%
Variance contribution from k<1k<10.01%0.04%
Variance contribution from the top 0.01% of tokens ranked by kk99.8%99.0%
Contribution of the top 10 tokens to E[u2]\mathbb E[u^2]98.7%94.9%

The additional results agree with the original experiment: variance is dominated by a small number of tokens with k>1k>1. During training, correction uses inference probabilities recorded at sampling time. Across steps 57–86 of the three CIS training runs, the training logs likewise show that 94.8%–97.5% of the variance of kk comes from k>1k>1.

For actual parameter gradients, the additional variance under exact weighting is 2.7 times that under CIS (0.200 vs. 0.074). During training, the peak gradient norm is 69.5 ± 41.4 for exact ratios and 3.0 ± 1.1 for CIS (Table 10). These results support truncating large ratios on the upper side.

References

[1] Penghui Qi et al. Rethinking the Trust Region in LLM Reinforcement Learning. ICML 2026.

We thank the reviewer again for the careful review. If these additional experiments and explanations address your concerns, we would appreciate your reconsideration of the rating.

Reviewer_HMGmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies training–inference mismatch in LLM reinforcement learning and proposes Calibrated Importance Sampling (CIS), which clips importance weights using token confidence. It relates importance weights to logit perturbations and provides estimator error bounds. Experiments on three MoE models show more stable training than untruncated importance sampling and modest accuracy gains over the strongest evaluated baselines.

Critical strengths
  1. The paper is well motivated and generally easy to follow, with a clear progression from analysing training–inference mismatch to developing and evaluating a correction.

  2. The ablations and sensitivity analyses are informative, showing how the confidence floor and one-sided clipping affect performance in the tested setting.

Critical weaknesses
  • The second-moment diagnostic uses a different sampling distribution from the ratio’s denominator. Appendix A.7 (L985–991) uses rollouts generated at step 57 but scores them with both engines at step 86, giving k=p86/q86k=p_{86}/q_{86}. The moments in Figure 3 therefore reflect the old rollout distribution, rather than sampling from q86q_{86}, with no reported reweighting. They do not directly estimate the second-moment term relevant to IS correction at step 86. This diagnostic should be repeated with rollouts generated and scored at the same checkpoint, using inference probabilities recorded during sampling.

  • The claim of lower gradient bias relies on a simplified proxy. Figure 5b measures removed weight, E[k−f]\mathbb{E}[k-f], while Figures 5c–d replace the GRPO parameter gradient with ∇zlog⁡p\nabla_z\log p, omitting advantages, PPO clipping, and the network Jacobian (L406–428). The bias ranking under this proxy need not hold for the actual gradient. Since the theoretical bounds do not establish CIS’s advantage over other bounded clipping rules either (L1713–1716), this claim needs actual gradient bias and variance measurements at a fixed checkpoint and sampling distribution, or verification in a tractable small model. Otherwise, it should be explicitly limited to the proxy.

  • The baseline comparison lacks an equal tuning budget. Appendix D.6 (L1708–1711) states that CIS settings were selected on Qwen1.5-MoE, while baselines use their reference implementations’ defaults. CIS’s average-score margins over the strongest baseline are only 0.60, 0.43, and 0.65 percentage points across the three models (Table 1), so tuning could affect the ranking. Comparable threshold searches for at least TIS, IcePop, and KPop, using the same selection protocol, would help separate the benefit of the correction rule from that of parameter selection.

  • Nits. On p. 22 (L1185–1187), the inequality is missing a << before the final ∞\infty and should end with b2E[kt2]<∞b^2\mathbb{E}[k_t^2]<\infty. On p. 32 (L1690–1692), the reported percentages sum to 16.5%+2.8%+0.45%=19.75%16.5\%+2.8\%+0.45\%=19.75\%, rather than the stated 22%22\%; please correct the figures or explain the discrepancy.

Questions for authors
  • How does the response averaging in Equation (1) match the token average in L209? Equation (1) averages within each response and then across responses, whereas L209 uses G^=1n∑m=1nkmℓm′\widehat{G}=\frac{1}{n}\sum_{m=1}^{n}k_m\ell'_m, with no explicit response-length weights in the definition of ℓm′\ell'_m (L201–204). How is nn defined, and does training average over responses or tokens? Are the length weights absorbed into ℓm′\ell'_m or the sampling procedure? If nn counts all valid batch tokens, these averages generally differ when response lengths vary.

  • How was the numerical floor κ\kappa determined? Appendix C.5 (L1456–1458) says that κ=5×10−3\kappa=5\times10^{-3} was fixed from log-probability storage precision before training, whereas Appendix D.6 (L1708–1709) describes both λ\lambda and κ\kappa as selected from a preregistered grid. Was κ\kappa fixed on numerical grounds or selected through performance comparisons, and how does its value follow from the storage dtype and numerical errors? Removing it reduces average accuracy from 34.78 to 29.02 (Table 12), so a reproducible selection rule matters, particularly when changing precision or inference engines.

4 comments

We thank the reviewer for recognizing the motivation and ablation studies, and for the detailed suggestions on our diagnostic experiments and implementation.

W1 – The sampling distribution used in the second-moment diagnostics does not match the denominator of the importance ratio. Appendix A.7 (L985–991) uses responses generated at step 57, while both engines score them at step 86. The moments in Figure 3 therefore reflect an older rollout distribution, without reported reweighting, and cannot directly estimate the second moment relevant to IS correction at step 86. Could you repeat the diagnostics with generation and scoring at the same checkpoint, using inference probabilities recorded during sampling?

Thank you for pointing this out. The checkpoint description in Appendix A.7 is incorrect: the original experiment generated responses at step 86 and scored them with both engines loaded at step 86, i.e., 86/86 rather than the stated 57/86. We will correct this description and add the diagnostic results under the 57/57 setting.

Both experiments use Qwen1.5-MoE with 4,000 trajectories each. Let k=p/qk=p/q and u=(1−p)eεu=(1-p)e^\varepsilon. Inference probabilities in both experiments are obtained through prefill rescoring; displacement moments are computed at positions with finite log-odds.

Metric57/57 (additional)86/86 (Appendix A.7)
Number of tokens578,686510,915
Coverage of finite log-odds77.2%66.3%
E[e2ε]\mathbb E[e^{2\varepsilon}]1.13×1061.13\times10^69.23×1069.23\times10^6
E[u2]\mathbb E[u^2]71.3022.87
E[k2]\mathbb E[k^2]56.016.1
Variance contribution from k>1k>199.99%99.96%
Variance contribution from k<1k<10.01%0.04%
Variance contribution from the top 0.01% of tokens ranked by kk99.8%99.0%
Contribution of the top 10 tokens to E[u2]\mathbb E[u^2]98.7%94.9%

The additional results agree with the original experiment: variance is dominated by a small number of tokens with k>1k>1, supporting the design of truncating large importance ratios on the upper side.

W2 – The lower-gradient-bias claim relies on simplified proxies. Figure 5b measures the removed weight E[kt−ft]\mathbb E[k_t-f_t], while Figures 5c–d replace the GRPO parameter gradient with ∇zlog⁡pt\nabla_z\log p_t, omitting the advantage, PPO clipping, and network Jacobian (L406–428). The bias ranking under these proxies need not hold for actual gradients, and the theoretical bound does not establish that CIS is superior to other bounded truncation rules (L1713–1716). Could you measure actual gradient bias and variance at a fixed checkpoint and sampling distribution, or verify this on a tractable small model? Otherwise, the claim should be explicitly limited to the proxy diagnostics.

Following this suggestion, we measured actual parameter gradients. At a fixed step-86 checkpoint and on the same rollouts, we backpropagate through the training PPO loss and compute gradients for the last Transformer block of Qwen1.5-MoE. All methods share the same rollouts and group advantages, changing only the detached mismatch weights. This retains the effects of the advantage and network Jacobian. As in training, each batch is used for one update, so the PPO ratio is 1 at the measurement point and clipping is inactive.

The measurements use 125 independent batches, each containing 8 prompt groups. Bias is the norm of the mean paired gradient difference relative to exact weighting. Additional variance is the increase in the trace of the batch-gradient covariance relative to no correction.

Correction ruleBias normAdditional gradient variance
No correction0.05140
Exact ratio00.200
TIS0.02700.030
CIS0.02070.074

The parameter-gradient results agree with the proxy diagnostics in Figure 5: CIS has lower bias than TIS, while its additional variance is 37% of that under exact weighting. CIS has higher variance than TIS, reflecting their different bias–variance trade-offs. These are empirical results under the measured setting; the theoretical bound does not guarantee that CIS outperforms every bounded truncation rule.

W3 – The baseline comparison lacks an equal hyperparameter-tuning budget. Appendix D.6 (L1708–1711) states that CIS is tuned on Qwen1.5-MoE, whereas the baselines use defaults from their reference implementations. In Table 1, CIS improves over the strongest baseline by only 0.60, 0.43, and 0.65 percentage points across the three models, so tuning could affect the ranking. Could you provide threshold searches for at least TIS, IcePop, and KPop under the same selection protocol, to distinguish gains from the correction rule itself from gains due to parameter selection?

We compared three candidate configurations each for CIS, TIS, IcePop, and KPop† under the same search budget. All experiments use the same training budget, three training seeds, and selection criterion: average greedy pass@1 across the five benchmarks at the final checkpoint. All hyperparameters are selected on Qwen1.5-MoE-A2.7B; CIS fixes κ=0.005\kappa=0.005 and searches over λ\lambda.

Qwen1.5-MoE-A2.7B, mean ± standard deviation over three training seeds. Bold indicates the selected configuration. Baseline default results are reused from Table 1, the three CIS configurations are from Table 11, and the remaining configurations are additional experiments.

MethodCandidate 1Candidate 2Candidate 3
TISC=1.5C=1.5: 16.32 ± 6.47C=2C=2 (default): 31.40 ± 2.47C=5C=5: 31.25 ± 3.02
IcePop[0.67,1.5][0.67,1.5]: 24.61 ± 3.69[0.5,5][0.5,5] (default): 34.18 ± 0.55[0.2,10][0.2,10]: 33.50 ± 0.58
KPop†ϕ=0.5\phi=0.5: 33.05 ± 1.09ϕ=2\phi=2 (default): 33.50 ± 1.04ϕ=8\phi=8: 31.80 ± 0.62
CISλ=1.6\lambda=1.6: 33.89 ± 0.46λ=2.3\lambda=2.3: 34.78 ± 0.25λ=3.1\lambda=3.1: 33.35 ± 0.66

Under the same search budget, CIS exceeds the selected TIS, IcePop, and KPop† configurations by 3.38, 0.60, and 1.28 percentage points, respectively. The best candidates for all three baselines are their reference defaults, so their runs on the other two models in Table 1 already use the configurations selected by this search.

W4 – Two minor issues: the inequality on p. 22 (L1185–1187) is missing a << before the final ∞\infty and should end with b2E[kt2]<∞b^2\mathbb E[k_t^2]<\infty. On p. 32 (L1690–1692), the percentages sum to 19.75%, rather than the stated 22%. Could you correct the numbers or explain the discrepancy?

Thank you for the careful reading. The first inequality should be:

\le b^2\mathbb E[k_t^2]<\infty.$$ The three percentages in the second passage do sum to 19.75%, so the statement that they reproduce 22% is incorrect. We will correct both issues in the revision. > **Q1 – How does the response averaging in Eq. (1) agree with the token averaging at L209? Eq. (1) first averages within each response and then across responses, whereas L209 uses $\widehat G=\frac1n\sum_{m=1}^n k_m\ell'_m$, and the definition of $\ell'_m$ (L201–204) contains no explicit response-length weight. How is $n$ defined, and does training average over responses or tokens? Are length weights absorbed into $\ell'_m$ or the sampling procedure? If $n$ is the total number of valid tokens in the batch, these two averages generally differ when response lengths vary.** Thank you for pointing this out. Both the implementation and L209 average over valid tokens in the batch, and $\ell'_m$ contains no response-length weight. The normalization in Eq. (1) was not consistent with this convention. We will change Eq. (1) to use valid-token averaging and clarify at L201 that $n=\sum_iT_i$ is the total number of valid tokens across all responses in the update batch, indexed by $m$. This change does not affect the existing theoretical analysis.

Q2 – How is the numerical floor κ\kappa determined? Appendix C.5 (L1456–1458) says that it is fixed before training based on the storage precision of log-probabilities, while D.6 (L1708–1709) says that both λ\lambda and κ\kappa are selected from a prespecified grid. Is κ\kappa fixed on numerical grounds or selected through performance comparisons? How is its value derived from the dtype and numerical error? Removing it in Table 12 reduces average accuracy from 34.78 to 29.02, so a reproducible selection rule is needed, particularly when changing precision or inference engines.

The value κ=0.005\kappa=0.005 is determined from numerical precision before searching over λ\lambda. Performance comparisons across different κ\kappa values are ablation experiments that examine the role and sensitivity of this numerical safeguard. We will clarify this distinction in the main text.

As pt→1p_t\to1, the original cap 1+λ(1−pt)1+\lambda(1-p_t) approaches 1, so even a small positive numerical error can trigger truncation. The term max⁡(1−pt,κ)\max(1-p_t,\kappa) preserves a minimum truncation tolerance. Similar probability-boundary safeguards appear in existing implementations: verl's DPPO-Binary-KL adds 10−810^{-8} to complementary probabilities [1], while the official Unlikelihood Training implementation directly floors 1−pt1-p_t [2].

Under bf16 precision, we observed discrete spacings of approximately 10−410^{-4}–10−310^{-3} in log⁡kt\log k_t for high-confidence tokens. In an early run, the median ∣log⁡kt∣|\log k_t| among truncated tokens was 0.00070.0007. Based on this observation, we fixed κ=0.005\kappa=0.005 before searching over λ\lambda and transferred it unchanged to the other two models. At the default λ=2.3\lambda=2.3, the minimum truncation tolerance is λκ=0.0115\lambda\kappa=0.0115, giving a cap of at least 1.0115.

The ablations in Table 12 were conducted after the parameters were fixed. Setting κ=0.005\kappa=0.005 and 0.020.02 gives 34.78 and 34.39, respectively; κ=10−3\kappa=10^{-3} reduces accuracy by about 1.4 percentage points, and removing the floor reduces it to 29.02. When changing precision or inference engines, the same procedure can be used: measure the numerical spacing of log⁡kt\log k_t for high-confidence tokens and set the minimum truncation tolerance accordingly.

References

[1] Penghui Qi et al. Rethinking the Trust Region in LLM Reinforcement Learning. ICML 2026. The numerical safeguard cited above is in verl's DPPO-Binary-KL implementation.

[2] Sean Welleck et al. Neural Text Generation with Unlikelihood Training. arXiv:1908.04319, 2019.

We thank the reviewer again for the careful review. If these additional experiments and explanations address your concerns, we would appreciate your reconsideration of the rating.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer HMGm,

Thank you again for your review. Have our responses addressed your concerns? We’d be happy to clarify anything that remains unresolved. If your concerns have been addressed, we would appreciate your reconsideration of the rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewers,

We would appreciate it if you could take a moment to revisit our responses and revised manuscript. Even a brief note on whether your concerns have been addressed would be very helpful. We would be happy to clarify any remaining questions.

Thank you for your time.

Authors

AuthorsOpenReview ↗

Follow-up Before the Rebuttal Deadline

Dear Reviewer,

As the rebuttal period is ending soon, we would appreciate any feedback on whether our responses have addressed your concerns. We would be happy to clarify any remaining questions.

Thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.