acceptodds
Under review as a conference paper at ICLR 2027

When Mismatch Misleads: Stabilizing Low-Precision Reinforcement Learning via Reliable Correction

Abstract

Reinforcement learning (RL) has become a key component of large language model post-training, but its substantial computational cost increasingly motivates end-to-end low-precision training. Despite improving training efficiency, it also introduces numerical perturbations that amplify train-inference mismatch and create a tradeoff between efficiency and stability. Existing approaches primarily stabilize low-precision RL by aligning rollout and actor policies. However, actor-rollout alignment alone does not guarantee improvement of the underlying full-precision objective, since the observed mismatch conflates genuine policy mismatch with numerical noise. We therefore formulate low-precision RL as a three-policy alignment problem, where noisy rollout and actor policies provide observable training signals while the inaccessible full-precision policy defines the learning problem. Based on this formulation, we propose **ERIS (Excess-Risk-guided Importance Sampling)**, an excess-risk-guided method that corrects high-reliability mismatch tokens while retaining identity weighting for noise-dominated tokens. Across multiple Qwen model scales and architectures, ERIS consistently improves FP8 training stability and performance. ERIS improves the average mathematical reasoning score over standard FP8 by 14.29% on Qwen3-8B, while achieving performance comparable to BF16. Mechanism analysis further confirms the advantage of three-policy alignment in preserving consistency with the full-precision policy. Meanwhile, ERIS preserves the efficiency benefits of FP8, achieving up to a 1.17 end-to-end throughput improvement over BF16.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Low Precision Training, Train-Inference Mismatch

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Aqc8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies training instability in end-to-end low-precision (FP8) reinforcement learning for LLMs. The authors argue that the observable log-probability discrepancy zt=log⁡(πθA/πθR)z _ t = \log(\pi _ \theta^{\mathrm{A}} / \pi _ \theta^{\mathrm{R}}) between the low-precision actor πθA\pi _ \theta^{\mathrm{A}} and rollout policy πθR\pi _ \theta^{\mathrm{R}} conflates genuine policy shift zˉt\bar{z} _ t with numerical quantization noise ϵt\epsilon _ t, so indiscriminately reweighting mismatched tokens can degrade rather than improve the underlying full-precision policy πθ\pi _ \theta (Section 3, Figure 1). Framing the update via a three-policy LL-smoothness local improvement certificate (Section 4.1, Eqs. 1–4; Theorem 1), they propose ERIS (Excess-Risk-guided Importance Sampling, Section 4.2). ERIS stratifies tokens by minimum support st=min⁡{πθA,πθR}s _ t = \min\lbrace \pi _ \theta^{\mathrm{A}}, \pi _ \theta^{\mathrm{R}} \rbrace into N=8N=8 surprisal buckets, tracks EMA noise statistics (μ^t,v^t)(\hat{\mu} _ t, \hat{v} _ t) right after policy synchronization, scores tokens by a clipped excess-risk proxy Ht=min⁡((zt−μ^t)2,τ2)−2v^tH _ t = \min((z _ t - \hat{\mu} _ t)^2, \tau^2) - 2\hat{v} _ t (Eq. 5), and applies truncated importance sampling (C=2C=2) only to the top-KK (p=20%p=20\%) tokens with Ht>0H _ t > 0 (Eq. 6).

Evaluated on Qwen3-1.7B, Qwen3-4B, Qwen3-8B, and Qwen2.5-7B trained for one epoch on DAPO-Math-17K (Table 1, Figure 3; SGLang check in Appendix Table 8), ERIS prevents the late-stage reward collapse seen in vanilla FP8 on Qwen3-4B and Qwen2.5-7B, improves math and general-knowledge accuracy over TIS and MIS, and preserves nearly all of FP8's throughput advantage (up to a 1.17×1.17\times end-to-end speedup over BF16, Appendix Figure 10). While the three-policy framing is appealing, there is a significant gap between the population theory (Theorem 1) and the observation-dependent Top-K algorithm actually implemented, and the empirical attribution is clouded by single-run reporting and an unablated high-support cutoff baked into Ht>0H _ t > 0.

Critical strengths

1. Thoughtful three-policy framing of why naive importance correction can backfire in low-precision RL.
Standard importance-sampling fixes (TIS, MIS) implicitly treat any mismatch between πθA\pi _ \theta^{\mathrm{A}} and πθR\pi _ \theta^{\mathrm{R}} as off-policy lag that ought to be corrected toward πθA\pi _ \theta^{\mathrm{A}}. Pointing out that low-precision quantization corrupts low-probability tokens with large log-ratio noise—so reweighting by ρt=πθA/πθR\rho _ t = \pi _ \theta^{\mathrm{A}}/\pi _ \theta^{\mathrm{R}} amplifies quantization artifacts on tail tokens—gives a compelling intuition for why selective, reliability-gated correction outperforms all-token TIS (Random 20% alone already jumps from 32.30% for full TIS to 35.90% on Qwen3-4B Math in Table 2).

2. Broad empirical coverage across four dense models and two rollout engines with minimal throughput cost.
The authors evaluate ERIS across Qwen3-1.7B, 4B, 8B, and Qwen2.5-7B on four math and four general-knowledge benchmarks (Table 1), show that the gains transfer from vLLM to SGLang on Qwen3-4B (Appendix Table 8: 38.65% vs. 34.76% for TIS), and even report a positive effect in BF16 RL (Appendix Table 7). Throughout, ERIS adds only 2%–3% overhead relative to vanilla FP8 while running up to 1.17×1.17\times faster than BF16 (Appendix Figure 10).

Critical weaknesses

1. Disconnect between the 8-bucket excess-risk theory and the actual mechanism (hard st>e−1s _ t > e^{-1} filtering + observation-dependent Top-K).
As detailed under Soundness, Theorem 1 only applies to fixed population coefficients, Assumption 5 (joint conditioning on zˉt\bar{z} _ t and sts _ t) is never empirically tested, and under the default clip τ=log⁡2\tau = \log 2, Eq. (21) and Figure 7 imply that 7 of the 8 surprisal buckets have v^t≥τ2/2\hat{v} _ t \ge \tau^2/2 and can never produce Ht>0H _ t > 0. Furthermore, in Table 3a, removing HtH _ t clipping (w/o H_t clipping)—which is the exact mechanism that blocks the other 7 buckets—actually yields slightly higher math accuracy on Qwen3-4B (37.52% vs. 37.24%). Meanwhile, Table 2 compares ERIS against Random 20% (35.90%) and |z_t| Top-20% (32.78%) over all tokens, without testing a simple support-matched baseline (e.g., selecting |z_t| Top-20% or Random 20% restricted to the high-support bucket −log⁡st∈[0,1)-\log s _ t \in [0, 1)). Consequently, we cannot tell whether the gain over Random 20% (37.24% vs. 35.90%) comes from excess-risk estimation or simply from refusing to apply importance weights to low-probability tokens (st<e−1s _ t < e^{-1}).

2. Single-run point estimates, undefined "divergent" runs, and missing AIS baseline.
All results in Tables 1–3 and 7–8 appear to be single training runs without seed standard deviations or confidence intervals. Several non-divergent margins are fairly small—for instance, on Qwen3-1.7B Math, BF16 (19.35%), ERIS (19.19%), MIS (18.93%), and FP8 (18.73%) sit within 0.62 points of each other, and on Qwen3-8B AIME24, TIS (25.83%) beats ERIS (23.33%). For Qwen3-4B and Qwen2.5-7B (plus p=0.05p=0.05 and C=1.5C=1.5 in Table 3b–c), vanilla FP8 is listed simply as -- (divergent) without stating a formal stopping/divergence rule (especially since Figure 3 shows FP8 on Qwen3-4B completing all 220 steps with late-stage reward oscillations). Finally, while Section 2 and Section 4.3 specifically contrast ERIS with AIS (Zhou et al., 2026), AIS is omitted from all empirical tables.

Questions for authors
  1. Assessment-critical (High-support filtering vs. excess-risk scoring): Appendix D.1 (p. 26) and Figure 7 show that under τ=log⁡2\tau = \log 2, only the first surprisal bucket (−log⁡st∈[0,1)-\log s _ t \in [0, 1), i.e., st>e−1s _ t > e^{-1}) has v^t<τ2/2≈0.24\hat{v} _ t < \tau^2/2 \approx 0.24, so tokens in buckets 2–8 can never be selected (Ht≤0H _ t \le 0). In Table 2 (Qwen3-4B), what happens if you apply |z_t| Top-20% or Random 20% restricted to that same first bucket (−log⁡st∈[0,1)-\log s _ t \in [0, 1))? This control is essential to tell whether the 8-bucket EMA excess-risk machinery adds value beyond simply masking out importance weights on low-probability tokens (st≤e−1s _ t \le e^{-1}).

  2. Clarification (Statistical reliability & divergence criterion): How many training seeds were run for Tables 1, 2, and 3, and can you provide multi-seed mean ±\pm std (or problem-level bootstrap CIs) for the Qwen3-4B main comparisons (TIS, Random 20%, w/o support condition, w/o H_t clipping, and ERIS)? Also, what exact quantitative criterion was used to mark FP8 as -- (divergent) in Tables 1, 3, and 8 given that Figure 3 plots the Qwen3-4B FP8 curve all the way to step 220?

  3. Clarification (Calibration diagnostics & AIS comparison): Can you report per-bucket sample counts and empirical calibration errors E[ϵt−μ^t∣zˉt,st]\mathbb{E}[\epsilon _ t - \hat{\mu} _ t \mid \bar{z} _ t, s _ t] during actual off-policy updates (where zˉt≠0\bar{z} _ t \ne 0, using your BF16 proxy from Section 5.3) to check Assumption 5? And can you include an empirical comparison against AIS (Zhou et al., 2026) on Qwen3-4B?

2 comments
AuthorsOpenReview ↗

Kind reminder on rebuttal

Dear Reviewer Aqc8,

We sincerely appreciate your thoughtful review and valuable feedback. In response to your suggestions, we have provided additional experiments and theoretical clarifications. We have included these additions in our paper and uploaded a new revision.

We hope these responses can address your concerns. If you have any further feedback or remaining concerns, please share with us at your convenience. We would be happy to provide additional clarifications if needed.

Best regards,

Authors of Submission 6969

AuthorsOpenReview ↗

Kind reminder on rebuttal

Dear Reviewer Aqc8,

Thank you again for the time and effort devoted in reviewing our submission and providing constructive feedback. As the discussion period is nearing its end, we would like to follow up on our response. If appropriate, we would appreciate a brief reply before the deadline to let us know whether our response addressed your concerns and if any concerns remain. We would be glad to clarify any remaining points.

Best regards,
Authors of Submission 6969

Reviewer_aHHxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ERIS, a selective importance-correction method for end-to-end FP8 reinforcement learning. It models the observed actor–rollout mismatch as a combination of genuine policy mismatch and numerical noise. ERIS estimates support-conditioned noise statistics after policy synchronization and applies calibrated truncated importance sampling to selected tokens. Other tokens retain unit weights. A theoretical analysis under explicit assumptions motivates the correction rule. Experiments on four Qwen models show improvements in training stability and benchmark performance, with additional results using SGLang.

Critical strengths

1.ERIS addresses a practical problem and improves the average mathematical reasoning score over TIS on all four evaluated models. On Qwen3-8B, it reaches 40.55, compared with 38.44 for TIS and 40.54 for BF16. The gains also hold with SGLang on Qwen3-4B. These improvements retain most of the throughput advantage of FP8 training.

2.The token-selection experiments provide useful insight into the effects of importance correction. Randomly correcting 20% of tokens improves the mathematical reasoning average over full-token TIS, and ERIS raises it further, from 35.90 to 37.24. Selecting tokens solely by mismatch magnitude performs worse. These comparisons and the calibration ablations support accounting for numerical reliability when selecting corrections.

Critical weaknesses

1.The paper does not report the number of independent training runs or variation across runs for the main results and ablations. This makes the smaller gains harder to assess, including the 1.34-point improvement over random 20% correction in Table 2.

2.The selection controls do not fully separate the contribution of the excess-risk score from that of token support. In the calibration statistics shown in Appendix D.1, only the highest-support bucket meets the variance condition for correction. The random and mismatch-only baselines do not match this support distribution. A support-matched comparison would clarify the additional benefit of excess-risk scoring over a simpler probability-based selection rule.

Questions for authors

How many independent training runs underlie the main results and ablations? Please report any available run-to-run variation for a representative comparison, such as ERIS, TIS, and random 20% correction on Qwen3-4B. Were the reported divergence cases reproduced across runs?

4 comments
AuthorsOpenReview ↗

Kind reminder on rebuttal

Dear Reviewer aHHx,

We sincerely appreciate your thoughtful review and valuable feedback. In response to your suggestions, we have provided additional experiments and theoretical clarifications. We have included these additions in our paper and uploaded a new revision.

We hope these responses can address your concerns. If you have any further feedback or remaining concerns, please share with us at your convenience. We would be happy to provide additional clarifications if needed.

Best regards,

Authors of Submission 6969

AuthorsOpenReview ↗

Kind reminder on rebuttal

Dear Reviewer aHHx,

Thank you again for the time and effort devoted in reviewing our submission and providing constructive feedback. As the discussion period is nearing its end, we would like to follow up on our response. If appropriate, we would appreciate a brief reply before the deadline to let us know whether our response addressed your concerns and if any concerns remain. We would be glad to clarify any remaining points.

Best regards,
Authors of Submission 6969

Thank you for the response and new experiments. The additional Qwen3-4B run and support-matched comparisons address my main concerns regarding variance and the benefit of ERIS's selection rule. I maintain my rating of 3 (Weak Accept).

Thank you for taking your time to review our responses! We are glad to know that our responses have resolved your concerns and that you maintain your positive assessment.

Reviewer_Hqe7OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that low-precision actor–rollout mismatch mixes useful correction signals with numerical noise. ERIS estimates support-conditioned noise statistics and applies calibrated importance correction only to tokens with positive estimated excess risk, subject to a correction budget. Experiments across four Qwen models show improved FP8 stability and accuracy approaching BF16, with up to 1.17× throughput improvement.

Critical strengths

Distinguishing mismatch magnitude from correction reliability is consequential. Comparisons with random and mismatch-ranked token selection provide evidence that the selection criterion contributes beyond sparsity alone.

Results span four models and include an alternative rollout backend, component ablations, and general-knowledge evaluation, supporting the method’s practical usefulness.

Critical weaknesses

The theoretical guarantee is conditional on positive reference-gradient alignment and strong calibration assumptions. The optimality result concerns fixed population coefficients, whereas ERIS uses observation-dependent selection; Figure 5 measures mismatch-recovery error rather than actual objective improvement.

Training-seed uncertainty and an AIS comparison are absent. These matter because the method is sensitive enough to diverge at some nearby correction-budget and truncation settings.

Questions for authors

Can you validate the calibration assumptions and show whether selected corrections improve BF16 gradient alignment or measured one-step objective gain?

Do the gains persist across independent training seeds and against AIS?

The supplied launcher defaults to EXCESS_RISK_LOG_CLIP=null, whereas the paper specifies (\tau=\log 2). Which configuration produced the reported results?

3 comments
AuthorsOpenReview ↗

Kind reminder on rebuttal

Dear Reviewer Hqe7,

We sincerely appreciate your thoughtful review and valuable feedback. In response to your suggestions, we have provided additional experiments and theoretical clarifications. We have included these additions in our paper and uploaded a new revision.

We hope these responses can address your concerns. If you have any further feedback or remaining concerns, please share with us at your convenience. We would be happy to provide additional clarifications if needed.

Best regards,

Authors of Submission 6969

Thank you for the detailed rebuttal. The additional training seed, AIS comparison, and support-matched controls strengthen the empirical case. The BF16 gradient-alignment results and clarification of the clipping configuration also address important concerns. Overall, the rebuttal strengthens my support for acceptance.

We sincerely appreciate your time and constructive feedback. We are glad that our responses address your concerns. Thank you again for your support!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.