- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper rewrites the training-to-inference probability ratio in terms of token confidence and a log-odds displacement. Measurements suggest that the displacement has a much more stable scale across confidence levels than the raw ratio. This motivates CIS, which clips large positive displacements and produces a confidence-dependent importance-weight cap. A small floor handles the high-confidence numerical regime. The authors derive bias-variance bounds and evaluate three MoE models trained on GSM8K, with three seeds and several correction baselines. CIS obtains the highest average across the five math benchmarks for each model.
Critical strengths
The ratio-displacement identity makes the method easy to understand. It explains why the same ratio threshold corresponds to different displacement thresholds at different confidence levels, and the resulting operator is simple. The paper is also clear about the reach of the theory: bounding a second moment motivates truncation, but does not prove that CIS is better than every other bounded cap. The numerical-floor version is covered separately in the appendix.
The experimental details inspire more confidence than an isolated best-score comparison would. There are three seeds, three MoE models, floor and clipping ablations, and a corrected KPop implementation that avoids a numerical masking artifact. The authors also point out where margins are within seed variation. Together with the shared training setup, this makes the reported gains worth taking seriously.
Critical weaknesses
My main reservation is the tuning comparison. As acknowledged in Appendix D.6, CIS's threshold and floor are chosen from a grid on Qwen1.5-MoE, while the baselines use their reference defaults. Matching TIS's truncation rate is a useful diagnostic, but it does not replace tuning its cap. The gains over the strongest baseline are fairly small, so this could affect the conclusion. Keeping CIS's settings fixed on the other two models helps, and is a reason I still view the empirical result positively.
The mechanism is plausible, but the quantitative diagnostics do not directly measure the training gradient. Figure 5(c,d) substitutes a logit-gradient proxy for the advantage-weighted PPO parameter gradient. Also, Appendix A.7 scores step-57 rollouts with step-86 engines when estimating the large moments. That can reveal large ratios, but the sampled tokens do not come from the step-86 behavior distribution assumed in the corresponding variance interpretation. Clarifying or checking these two points would make the explanation of the gains more convincing. They do not undermine the ratio identity or directly contradict the accuracy results.
Questions for authors
-
Do the gains over TIS and IcePop persist when their thresholds receive a comparable validation budget? Paired seed-level differences would also help interpret the small aggregate margins.
-
Can you repeat the moment and variance-by-side measurements using rollouts generated by the same checkpoint used for both engine probabilities? Alternatively, how is the step-57/step-86 sampling difference accounted for in Appendix A.7?
-
Does the bias-variance ordering in Figure 5 survive when using the actual advantage-weighted PPO gradient? A check with gradient projections or a small set of parameter blocks would help connect the diagnostic to the optimizer.
2 comments
Q1 – Do the gains remain after giving TIS and IcePop the same validation budget? Paired seed-level differences would also help interpret the relatively small overall gains.
Yes. Under the same search budget, the selected TIS and IcePop configurations are their reference defaults. Based on the mean accuracies in Table 1, CIS outperforms both baselines on all three models.
Differences in average greedy pass@1 across the five benchmarks, in percentage points, calculated from the means in Table 1.
| Model | CIS − TIS | CIS − IcePop |
|---|---|---|
| Qwen1.5-MoE-A2.7B | 3.38 | 0.60 |
| DeepSeek-V2-Lite | 1.14 | 0.43 |
| Qwen3-30B-A3B | 0.74 | 0.89 |
The paired differences against IcePop under the same training seeds are reported below.
Percentage points; each entry is CIS minus IcePop under the same seed.
| Model | Seed A | Seed B | Seed C |
|---|---|---|---|
| Qwen1.5-MoE-A2.7B | −0.10 | 0.61 | 1.31 |
| DeepSeek-V2-Lite | 0.00 | 0.21 | 1.08 |
| Qwen3-30B-A3B | 0.82 | 0.84 | 1.03 |
Across the nine pairs, seven differences are positive, one is a tie, and one is negative. The mean gain is 0.64 percentage points, with a paired t-statistic of 3.86 (8 degrees of freedom, ).
Q2 – Could you repeat the moment and side-specific variance measurements using rollouts generated at the same checkpoint used by both engines to compute probabilities? Alternatively, how does Appendix A.7 account for the sampling difference between steps 57 and 86?
Thank you for pointing this out. The checkpoint description in Appendix A.7 is incorrect: the original experiment generated responses at step 86 and scored them with both engines loaded at step 86, i.e., 86/86 rather than the stated 57/86. We will correct this description and add the diagnostic results under the 57/57 setting.
Both experiments use Qwen1.5-MoE with 4,000 trajectories each. Let and . Inference probabilities in both experiments are obtained through prefill rescoring; displacement moments are computed at positions with finite log-odds.
| Metric | 57/57 (additional) | 86/86 (Appendix A.7) |
|---|---|---|
| Number of tokens | 578,686 | 510,915 |
| Coverage of finite log-odds | 77.2% | 66.3% |
| 71.30 | 22.87 | |
| 56.0 | 16.1 | |
| Variance contribution from | 99.99% | 99.96% |
| Variance contribution from | 0.01% | 0.04% |
| Variance contribution from the top 0.01% of tokens ranked by | 99.8% | 99.0% |
| Contribution of the top 10 tokens to | 98.7% | 94.9% |
The additional results agree with the original experiment: variance is dominated by a small number of tokens with , supporting the design of truncating large importance ratios on the upper side.
Q3 – Does the bias–variance ranking in Figure 5 hold when using actual advantage-weighted PPO gradients? Gradient projections or measurements on a few parameter blocks would help connect the diagnostics to the optimizer.
Yes. At a fixed step-86 checkpoint and on the same rollouts, we backpropagate through the training PPO loss and compute parameter gradients for the last Transformer block of Qwen1.5-MoE. All methods share the same rollouts and group advantages, changing only the detached mismatch weights. This retains the effects of the advantage and network Jacobian. As in training, each batch is used for one update, so the PPO ratio is 1 at the measurement point and clipping is inactive.
The measurements use 125 independent batches, each containing 8 prompt groups. Bias is the norm of the mean paired gradient difference relative to exact weighting. Additional variance is the increase in the trace of the batch-gradient covariance relative to no correction.
| Correction rule | Bias norm | Additional gradient variance |
|---|---|---|
| No correction | 0.0514 | 0 |
| Exact ratio | 0 | 0.200 |
| TIS | 0.0270 | 0.030 |
| CIS | 0.0207 | 0.074 |
The parameter-gradient results agree with the proxy diagnostics in Figure 5: CIS has lower bias than TIS, while its additional variance is 37% of that under exact weighting. CIS has higher variance than TIS, reflecting their different bias–variance trade-offs.
We thank the reviewer again for the careful review. If these additional experiments and explanations address your concerns, we would appreciate your reconsideration of the rating.
We thank the reviewer for recognizing the ratio decomposition, the scope of the theory, and the experimental details, and for the suggestions for further validation.
Following this suggestion, we compared three candidate configurations each for CIS, TIS, IcePop, and KPop†, using the same training budget, three training seeds, and selection criterion: average greedy pass@1 across the five benchmarks at the final checkpoint. All parameters are selected on Qwen1.5-MoE and then transferred unchanged to the other two models. For CIS, κ=0.005 is fixed beforehand based on numerical precision; performance comparisons across different κ values are ablations. The CIS parameter search is over λ.
For TIS, we report both the truncation rate and accuracy at different thresholds.
Qwen1.5-MoE, mean ± standard deviation over three training seeds. Default and truncation-rate-matched results are reused from Tables 1, 2, and 10; C=1.5 and C=5 are additional experiments.
Among the tested thresholds, TIS reaches a highest mean of 31.40, which is 3.38 percentage points below CIS. Increasing the cap from C=2 to C=5 further reduces truncation but does not improve average accuracy. At C=1.13, TIS has a truncation rate close to CIS but much lower accuracy, also showing why matching truncation rates cannot replace threshold tuning.
The best candidates for all three baselines are their reference defaults, so their runs on the other two models in Table 1 already use the configurations selected by this search. Under the same search budget, CIS exceeds the selected TIS, IcePop, and KPop† configurations by 3.38, 0.60, and 1.28 percentage points, respectively.
Thank you for pointing this out. The checkpoint description in Appendix A.7 is incorrect: the original experiment is actually 86/86. We clarify this and add the 57/57 results in Q2. For the gradient-proxy concern, we measured actual PPO parameter gradients, with results reported in Q3.