acceptodds
Under review as a conference paper at ICLR 2027

Just Reuse the Rollout Log-Probs: Stable Mixture-of-Experts Reinforcement Learning with a Vocabulary-KL Clip

Abstract

In reinforcement learning of large language models, responses are typically generated by a fast inference engine while gradients are computed by a different training backend, and the two disagree on the same tokens because of numerical and, for mixture-of-experts models, expert-routing differences. The PPO/GRPO ratio clip that stabilizes these updates acts on the sampled token's probability ratio: a one-dimensional proxy that neither measures how far the full next-token distribution moved nor, under trainer recompute, reflects which policy generated the data. We find that a double clip stabilizes these updates, and that with it the cheapest correction suffices. VoKL-Clip clips each token by a top- bucket KL between the rollout and current policies (computed for free from the top- log-probs the inference engine returns on request), removing drifted tokens from the gradient by stop-gradient, and keeps a wide ratio clip alongside it. The bucket coarsening yields the sharpest full-vocabulary total-variation bound available from top- information, yet the vocabulary-KL clip provably cannot control importance-weight moments, which the ratio clip bounds; dropping either clip costs 7–11 AIME24 points. With both, VoKL-Clip can just reuse the rollout's own log-probs as the behavior policy: the importance ratio is faithful to the sampler, with no recompute and no routing replay. On Qwen3-30B-A3B trained on DAPO-Math-17k, VoKL-Clip reaches 64.5% AIME24 (ave@32)—versus 58.1% for DAPO and 57.2% for GSPO, each at its overall best rollout↔training mismatch-correction level—leads across the reasoning-benchmark suite, and on AIME24 exceeds every baseline at every correction level at lower correction cost, with stable training dynamics.

Primary area:
reinforcement learning
Keywords:
reinforcement learning for LLMs, mixture-of-experts, training-inference mismatch, KL clip, importance sampling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fSZDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a method for handling the distribution mismatch in RL caused by using different inference and training infrastructure, resulting in numerical differences and/or different experts being used in MOE architectures. The proposed method clips the update for a token using top-k bucket KL divergence and another ratio clip. They show that the KL-clip cannot control importance-weight moments hence propose using the second ratio clip. Experiments show that the method outperforms DAPO and GSPO.

Critical strengths
  • Useful observation that the KL-clip cannot fully control the importance-weight moments.
  • Good performance in experiments.
Critical weaknesses
  • Limited novelty. Clipping by divergence has been studied in prior works, although the observation that it does not fully control the importance-weight moment could be more novel.
  • Weak baselines. Experiments do not compare with recent methods, e.g. DPPO, which according to the authors is the closest to this work.
Questions for authors
  • How would the method compare to more recent methods such as DPPO?
4 comments

Dear Reviewer fSZD,

We appreciate your comments. Below, we respond to your concerns.

W2&Q (DPPO and recent baselines)

We have added DPPO TV and DPPO KL at R0. We align the experimental settings with ours as closely as possible; DPPO KL uses top-16, the same as ours. VoKL-Clip uses different divergence thresholds for positive and negative advantages, whereas DPPO uses a shared threshold, so this aspect cannot be matched exactly. We also ran DPPO KL with R3, following the settings in the original DPPO paper. We will provide the full hyperparameter settings in the appendix. We report VoKL-Clip at its paper checkpoint. For each DPPO variant, we select the checkpoint with the highest AIME24 score, following the AIME24-based selection criterion used in our paper.

MethodAIME24AIME25AIME26AMC23GPQA-DMMLU-Pro
VoKL-Clip (ours, R0)64.543.257.693.262.775.5
DPPO TV (R0)60.640.549.393.054.947.1
DPPO KL (R0)59.640.655.693.760.072.2
DPPO KL (R3)58.142.356.893.660.174.8

VoKL-Clip continues to lead on five of the six benchmarks. These results further demonstrate the effectiveness of our method.

W1 (novelty)

Thank you for highlighting our observation that divergence control alone cannot globally bound the second moment of importance weights. This limitation motivates our study of RL stability, particularly for MoE models. We investigate double clipping at R0 and provide a detailed theoretical analysis. We would like to reiterate our main contributions:

  1. Stabilizing MoE RL at R0. We address rollout–training mismatch caused by numerical and expert-routing differences. VoKL-Clip enables stable training with direct rollout reuse at R0, without an old-policy forward pass or routing replay.
  2. Why double clipping is needed. We show that divergence control alone cannot globally bound the second moment of importance weights (Prop. 3.2; Thm. 3.3). We pair the distributional gate with wide ratio clipping and a dual clip to limit extreme update coefficients.
  3. Component ablations and R0–R3 comparisons. The full method reaches 64.5 on AIME24 at R0, compared with 53.3 without coefficient control, 57.1 without the gate, and 47.9 with a tight clip. It also outperforms every DAPO and GSPO configuration across R0–R3 in our study.

Thanks for your experimental comparison with DPPO. Would you be able to clarify the novelty of your second clip against DPPO's truncated importance sampling?

Dear Reviewer fSZD,

Thank you for your following-up question. We clarify the differences between the second clip and DPPO's truncated importance sampling below.

(1) The second clip treats the two advantage signs differently. TIS uses the same coefficient min⁡(r,C)\min(r,C) for positive- and negative-advantage tokens; DPPO's scaling experiments use CC (DPPO arXiv v2, Eq. 23 and Appendix F). The second clip uses an upper cutoff ρhi\rho_{\mathrm{hi}} for A^>0\hat A>0. For A^<0\hat A<0, it uses a lower cutoff ρlo\rho_{\mathrm{lo}} together with the upper dual-clip cutoff cc (Section 2.2).

The table compares effective importance coefficients for tokens admitted by the divergence masks.

AdvantageDPPO's TISThe second clip
A^>0\hat A>0min⁡(r,C)\min(r,C)rr for r<ρhir<\rho_{\mathrm{hi}}; 0 for r>ρhir>\rho_{\mathrm{hi}}
A^<0\hat A<0min⁡(r,C)\min(r,C)0 for r<ρlor<\rho_{\mathrm{lo}}; rr for ρlo<r<c\rho_{\mathrm{lo}}<r<c; 0 for r>cr>c

(2) The second clip removes the direct gradient on saturated branches. For an unmasked token with r>Cr>C, TIS retains its direct policy-gradient contribution with coefficient CC. On our clipped constant branches, the objective no longer depends on θ\theta, so that token's direct policy-gradient contribution is zero. Thus, the two mechanisms share a coefficient-bounding role but produce different updates.

The second clip removes direct gradient contributions on saturated branches instead of retaining TIS's nonzero weights up to CC. It also stops further direct suppression of negative-advantage tokens below ρlo\rho_{\mathrm{lo}}, where TIS's upper cap is inactive. These safeguards match our design expectations; our theory motivates controlling effective update coefficients separately from distributional drift (Prop. 3.2; Thm. 3.3).

We hope our response has addressed your concerns. We know your time is valuable, and we would be very grateful for any further questions or suggestions.

Reviewer_RHfgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VoKL-Clip - a new style of clipping for GRPO-style RL on LLMs, aimed at the mismatch between the rollout engine and the trainer in mixture-of-experts models. VoKL-Clip drops each token from the gradient when the KL between its rollout and current next-token distributions crosses a threshold. The KL is computed from the top-k log-probs from the inference engine. A wide ratio clip is kept alongside it, so the rollout log-probs can be reused directly, with no recompute and no routing replay. The paper theoretically shows that the top-k KL bounds total variation but cannot control importance-weight moments. The authors evaluate VoKL-Clip on Qwen3-30B-A3B, where it outperforms DAPO and GSPO across six reasoning benchmarks. The authors sweeps the baselines across four mismatch-correction levels and finds VoKL-Clip at the cheapest level beats all of them on AIME24.

Critical strengths
  • The proposed approach shows strong improvements over the baselines (which seem to be fair) in the experimental setup. At R0, VoKL-Clip beats DAPO and GSPO at every correction level on AIME24, including the routing-replay settings, on a 30B MoE model at realistic scale.
  • The authors provide a theoretical explanation of VoKL-Clip. The TV certificate (Theorem 3.1) and the negative result on importance-weight moments explain why neither clip suffices alone. This is substantiated experimentally - dropping either clip costs 7–11 AIME24 points.
Critical weaknesses
  • The experimental setup is rather narrow. It only includes one model, one training dataset. Including atleast one other training dataset, preferably from a domain other than Math would strengthen the work significantly. Additionally, AIME24 is essentially a validation set. However the score on AIME-24 seems to be the highlighted performance.
  • The approach presented in the paper seems to resemble DPPO (Qi et al., 2026) quite closely, as pointed out in the related works, yet the baseline has not been presented.
Questions for authors

Can the authors compare the approach (experimentally) against DPPO?

3 comments

Dear Reviewer RHfg,

Thank you for your review and constructive suggestions.

Q&W2 (DPPO)

We have added DPPO TV and DPPO KL at R0. We align the experimental settings with ours as closely as possible; DPPO KL uses top-16, the same as ours. VoKL-Clip uses different divergence thresholds for positive and negative advantages, whereas DPPO uses a shared threshold, so this aspect cannot be matched exactly. We also ran DPPO KL with R3, following the settings in the original DPPO paper. We will provide the full hyperparameter settings in the appendix. We report VoKL-Clip at its paper checkpoint. For each DPPO variant, we select the checkpoint with the highest AIME24 score, following the AIME24-based selection criterion used in our paper.

MethodAIME24AIME25AIME26AMC23GPQA-DMMLU-Pro
VoKL-Clip (ours, R0)64.543.257.693.262.775.5
DPPO TV (R0)60.640.549.393.054.947.1
DPPO KL (R0)59.640.655.693.760.072.2
DPPO KL (R3)58.142.356.893.660.174.8

VoKL-Clip continues to lead on five of the six benchmarks. These results further demonstrate the effectiveness of our method.

W1 (experimental scope and checkpoint selection)

Thank you for this suggestion. Training MoE models requires substantial time and computational resources, so we prioritized the requested DPPO comparison during the rebuttal. Experiments on agentic and coding tasks are currently underway, and we plan to report them in future work. AIME24 is used for checkpoint selection; the gain over wide-clip DAPO remains at the common step s350 and averages 5.7 points over offline-evaluated checkpoints in s300–360. We will explicitly label AIME24 as the selection metric and add these analyses in the revision.

I would like to thank the reviewers for providing the comparison with DPPO and would suggest updating the submission with new results and analysis. Reporting experiments on agentic and coding tasks are currently underway would materially strengthen this work.

For now, I would like to maintain my positive score.

Thank you for your follow-up and positive assessment. We are glad the DPPO baseline comparison is helpful. We will add those results and analyses to the revision.

Reviewer_1Ri1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a rollout-anchored, top-k bucket-KL mask combined with wide ratio and dual clipping for MoE RL. It reuses rollout log probabilities rather than recomputing the old policy in the training backend. The theoretical analysis gives a tail-aware total-variation envelope and counterexamples showing that KL control does not bound importance-weight variance. Experiments on Qwen3-30B-A3B and DAPO-Math-17k compare DAPO/GSPO across four mismatch-correction levels and ablate bucket size and both clipping components.

Critical strengths
  1. A concrete systems problem is connected to an intelligible algorithm. Reusing the sampler's probabilities gives the appropriate conditional action denominator. The R0-R3 comparison makes implementation choices that are often hidden in RL evaluations visible.
  2. The analysis carefully distinguishes different divergences. The paper does not falsely treat bucket KL as an upper bound on full KL. The total-variation bound includes tail mass, and the importance-weight counterexamples explain why the distribution check alone is insufficient.
  3. The component controls are informative. Removing the coefficient control reduces AIME24 to 53.3; keeping the wide ratio clip without the gate gives 57.1; both together give 64.5. The matched wide-clip control is especially valuable and prevents attributing all improvement to the KL gate.
  4. The experimental account is relatively transparent. The paper discloses single-seed runs, resumed curves, checkpoint selection, correction-level selection, and the DAPO R2 collapse. Training uses temperature 1 and untruncated top-p 1, which avoids a basic inconsistency between sampler probabilities and an untruncated-score interpretation.
Critical weaknesses

W1. The closest algorithmic baseline is missing. Section 5 identifies DPPO as the closest prior work, but experiments compare only DAPO/GSPO and internal variants. DPPO already uses rollout-anchored divergence masking and top-k coarsening; its scaling recipe also includes a finite importance-coefficient cap. A result showing that an unbounded gate-only update can fail does not by itself distinguish VoKL-Clip from that practical recipe. A matched DPPO implementation at R0, with the same coefficient cap, data, and evaluation rule, is the most important comparison. This is a request about a cited close method, not an arbitrary new benchmark. See DPPO, Sections 4.3-4.4 and Appendix F.

W2. The efficiency claim is not measured end to end. Eliminating an old-policy forward pass and routing replay reduces particular costs, but requesting, transferring, and storing top-k probabilities adds work. Figure 3 also shows longer responses for the proposed method, which can materially change rollout compute at a fixed number of updates. The conclusion explicitly acknowledges no wall-clock measurements. Thus “cheapest correction level” is a defensible implementation statement; “lower total training cost” is not yet established. Report time and generated/trained tokens to a fixed performance threshold against the strongest relevant control, including top-k overhead.

W3. The reported gain is vulnerable to training and selection uncertainty. All experiments use one MoE model, one training dataset, and single seeds. AIME24 is evaluated repeatedly and used for checkpoint selection. In addition, the headline baseline correction level is chosen using stability and wins across the evaluation suite, rather than a purely held-out selection criterion. The disclosure is helpful, but these selected test metrics are not independent confirmations. The 64.5 versus 57.1 matched-clamp result is promising; confidence would improve substantially with a focused repeated run and question-level paired uncertainty at a prespecified checkpoint. Multiple samples per question reduce decoding noise, not training-seed or problem-selection uncertainty.

W4. The empirical mechanism is less established than the mathematical non-control result. Theorem 3.1 explicitly limits its guarantee to the evaluated policy, and Proposition 3.2 explicitly bounds the direct logit gradient. Section 2.2 also explains the value-preserving stop-gradient implementation. These are already-present qualifications, not missing explanations. The remaining empirical issue is that sampled-token k1 KL is comparable in the matched clamped runs while entropy separates (Section 4.6 and Figure 3). This is compatible with useful regularization, but does not identify tail-driven importance variance as the dominant failure mode. Gradient-mass and tail-ratio diagnostics would help distinguish that mechanism from other effects of the gate.

Questions for authors
  1. How does VoKL-Clip compare with DPPO at R0 when both use the same top-k information, bounded effective importance coefficients, and training budget?

  2. What are the measured top-k logging/transfer overhead and end-to-end cost to a fixed AIME24 level, accounting for different rollout lengths?

  3. How repeatable is the gain over wide-clip DAPO at a fixed checkpoint? Please separate training-seed variation from evaluation-sampling uncertainty.

  4. What fraction of positive and negative positions are gated, and how much gradient mass is removed? Does the asymmetric threshold help beyond a single shared threshold under the same coefficient bound?

  5. Can tail-ratio and gradient-contribution diagnostics show whether the theoretical importance-variance obstruction is the failure mode encountered in the matched experiments?

3 comments

Dear Reviewer 1Ri1, thank you for your comments. We would like to address each concern below.

Q1&W1

We have added DPPO TV and DPPO KL at R0. We align the experimental settings with ours as closely as possible; DPPO KL uses top-16, the same as ours. VoKL-Clip uses different divergence thresholds for positive and negative advantages, whereas DPPO uses a shared threshold, so this aspect cannot be matched exactly. We also ran DPPO KL with R3, following the settings in the original DPPO paper. We will provide the full hyperparameter settings in the appendix. We report VoKL-Clip at its paper checkpoint. For each DPPO variant, we select the checkpoint with the highest AIME24 score, following the AIME24-based selection criterion used in our paper.

MethodAIME24AIME25AIME26AMC23GPQA-DMMLU-Pro
VoKL-Clip (ours, R0)64.543.257.693.262.775.5
DPPO TV (R0)60.640.549.393.054.947.1
DPPO KL (R0)59.640.655.693.760.072.2
DPPO KL (R3)58.142.356.893.660.174.8

VoKL-Clip continues to lead on five of the six benchmarks. These results further demonstrate the effectiveness of our method.

Q2&W2

We would like to clarify a potential misunderstanding about our efficiency claim. Our cost claim concerns the correction level. R0 saves one old-policy forward pass over all 8,192 responses per step and requires no routing replay. We will qualify "free" in Section 2.2: computing the distance adds no forward pass, but collecting and transferring top-kk log-probs incurs overhead (192 bytes per response position, about 8.1 GB per step at the mean length). Our runs used different GPU types, precluding a controlled wall-clock comparison.

Early in training, longer responses coincide with higher AIME24 scores. At step 40, VoKL-Clip generates 4.9k-token responses (at most 2.9k for the other runs in Fig. 3c) and scores 41.5 on AIME24, compared with at most 38.6 across all baseline correction levels. Together, these observations are consistent with faster learning early in training.

Across the training trajectory, the table below compares progress in training updates: each baseline's final level (mean AIME24 over s310–350), when VoKL-Clip reaches that level, and when the baseline comes within 1 point of it.

BaselineFinal levelVoKL-Clip reaches it atBaseline within 1 point at
wide-clip DAPO (R0)56.3s100s150
DAPO (R3; Table 1)53.8s90s170
GSPO (R1; Table 1)54.8s90s200

Q3&W3

Thank you for the suggestion. Given the high computational cost, reporting one training run per configuration with repeated evaluation sampling (avg@32) is standard practice at this scale, as in DAPO (Yu et al., 2025), GSPO (Zheng et al., 2025), and DPPO (Qi et al., 2026). Full hyperparameters are in Appendix B (Tables 4–6). Since each run takes about 10 days on close to 100 GPUs, we prioritized the requested DPPO comparison during the rebuttal.

Across four hyperparameter configurations, VoKL-Clip has higher mean AIME24 over s310–350 than all listed baselines:

VoKL-Clip k=16k{=}16k=8k{=}8k=32k{=}32radii ×2wide-clip DAPODAPO R3GSPO R1GSPO R3
AIME24, mean s310–35061.459.761.059.856.353.854.854.0

VoKL-Clip also leads wide-clip DAPO on AIME24 at 31/36 evaluated steps. At s350, it leads on four benchmarks, with similar AIME25/AMC23 scores:

s350AIME24AIME25AIME26AMC23GPQA-DMMLU-Pro
VoKL-Clip59.641.656.392.363.475.7
wide-clip DAPO55.341.049.292.860.474.9

Estimated evaluation-sampling variability is ≤1.6 points per checkpoint (2.3 for a difference). A DAPO relaunch with the same configuration differs from the original by ≤2.6 points in five-evaluation window means after s100, below the smallest 3.4-point gap above. Training-seed variability for the matched comparison remains unquantified.

Q4&W4

Asymmetric vs shared threshold. The asymmetric radii were our initial choice, following DAPO's Clip-Higher (ϵhigh=0.28>ϵlow=0.2\epsilon_{\text{high}}=0.28>\epsilon_{\text{low}}=0.2) and SAPO's sign-dependent temperatures, both of which treat negative-advantage updates more conservatively; SAPO's ablation on Qwen3-30B-A3B-Base finds a shared setting less stable (their Fig. 5). Through Thm. 3.1 our radii also mirror DAPO's asymmetry: an admitted token's TV radius is τ/2\sqrt{\tau/2} plus the tail mass, i.e. 0.158 for τ−\tau^- and 0.224 for τ+\tau^+, a ratio of 2≈1.41\sqrt2\approx1.41 against DAPO's 0.28/0.2=1.40.28/0.2=1.4. The one radius variation we tried, doubling both, is worse (AIME24 60.9 vs 64.5; App. C.5).

Gradient mass. The definition of "gradient mass removed" is potentially unclear: it may refer to the sum of gated tokens' gradient norms or the norm of their summed gradient. These differ because token gradients can cancel, and neither directly measures their contribution to the final update after Adam and gradient clipping.

Q5&W4

We clarify that the paper does not attribute the gap in the matched-clamp pair to importance variance. In that pair the sampled-token KL is comparable (2.64×10−32.64\times10^{-3} vs 2.61×10−32.61\times10^{-3}), and what separates the runs is entropy: wide-clip DAPO rises to 1.11 by s360, while VoKL-Clip stays at 0.45. Section 4.6 attributes the gap to the gate holding entropy.

The variance obstruction of Prop. 3.2 is instead the reason we keep a ratio bound. Without it (gate-only), the KL doubles (4.98×10−34.98\times10^{-3}) and spikes to 5.2×10−25.2\times10^{-2}, the median gradient norm is 0.112 vs 0.058 for wide-clip DAPO, and AIME24 falls to 53.3. Reading "tail ratio" as the ratio of sampled tokens outside the rollout's top-16, the case Prop. 3.2 constructs, about 1% of sampled tokens fall there (median rollout tail mass 1.03%). We will make this division of labor explicit in the revision.

Dear Reviewer 1Ri1, As the discussion period is nearing its end, we would like to kindly follow up and check whether our rebuttal has addressed all of your concerns. We hope our clarifications were helpful and would sincerely appreciate it if you could consider adjusting your score accordingly.

Comments on the paper

1 comment

Dear Program Chairs, Senior Area Chairs, and Area Chairs,

Thank you for overseeing the review process. We also appreciate the reviewers' time and constructive feedback. To support your assessment, we highlight two points from our rebuttal:

  1. DPPO comparison. A major concern shared by all three reviewers was the lack of a direct comparison with DPPO. We therefore added DPPO TV and DPPO KL at R0, as well as DPPO KL at R3. VoKL-Clip at R0 leads all three variants on five of the six benchmarks, with only a very small gap on AMC23. These new results are consistent with our original findings and further support the effectiveness of our design.
  2. Additional analyses and clarifications. We provided further evidence that the gains persist beyond the selected checkpoint. We also clarified our theoretical contributions, the differences between the second clip and DPPO's truncated importance sampling, and the scope of our efficiency claim at R0.

These additions strengthen the evidence for our central finding that VoKL-Clip supports stable MoE RL at R0. We will incorporate the new results and clarifications into the revised paper.

We would appreciate your consideration of this additional evidence and our detailed responses when assessing the submission. Thank you again for your time and consideration.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.