- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies training instability in end-to-end low-precision (FP8) reinforcement learning for LLMs. The authors argue that the observable log-probability discrepancy between the low-precision actor and rollout policy conflates genuine policy shift with numerical quantization noise , so indiscriminately reweighting mismatched tokens can degrade rather than improve the underlying full-precision policy (Section 3, Figure 1). Framing the update via a three-policy -smoothness local improvement certificate (Section 4.1, Eqs. 1–4; Theorem 1), they propose ERIS (Excess-Risk-guided Importance Sampling, Section 4.2). ERIS stratifies tokens by minimum support into surprisal buckets, tracks EMA noise statistics right after policy synchronization, scores tokens by a clipped excess-risk proxy (Eq. 5), and applies truncated importance sampling () only to the top- () tokens with (Eq. 6).
Evaluated on Qwen3-1.7B, Qwen3-4B, Qwen3-8B, and Qwen2.5-7B trained for one epoch on DAPO-Math-17K (Table 1, Figure 3; SGLang check in Appendix Table 8), ERIS prevents the late-stage reward collapse seen in vanilla FP8 on Qwen3-4B and Qwen2.5-7B, improves math and general-knowledge accuracy over TIS and MIS, and preserves nearly all of FP8's throughput advantage (up to a end-to-end speedup over BF16, Appendix Figure 10). While the three-policy framing is appealing, there is a significant gap between the population theory (Theorem 1) and the observation-dependent Top-K algorithm actually implemented, and the empirical attribution is clouded by single-run reporting and an unablated high-support cutoff baked into .
Critical strengths
1. Thoughtful three-policy framing of why naive importance correction can backfire in low-precision RL.
Standard importance-sampling fixes (TIS, MIS) implicitly treat any mismatch between and as off-policy lag that ought to be corrected toward . Pointing out that low-precision quantization corrupts low-probability tokens with large log-ratio noise—so reweighting by amplifies quantization artifacts on tail tokens—gives a compelling intuition for why selective, reliability-gated correction outperforms all-token TIS (Random 20% alone already jumps from 32.30% for full TIS to 35.90% on Qwen3-4B Math in Table 2).
2. Broad empirical coverage across four dense models and two rollout engines with minimal throughput cost.
The authors evaluate ERIS across Qwen3-1.7B, 4B, 8B, and Qwen2.5-7B on four math and four general-knowledge benchmarks (Table 1), show that the gains transfer from vLLM to SGLang on Qwen3-4B (Appendix Table 8: 38.65% vs. 34.76% for TIS), and even report a positive effect in BF16 RL (Appendix Table 7). Throughout, ERIS adds only 2%–3% overhead relative to vanilla FP8 while running up to faster than BF16 (Appendix Figure 10).
Critical weaknesses
1. Disconnect between the 8-bucket excess-risk theory and the actual mechanism (hard filtering + observation-dependent Top-K).
As detailed under Soundness, Theorem 1 only applies to fixed population coefficients, Assumption 5 (joint conditioning on and ) is never empirically tested, and under the default clip , Eq. (21) and Figure 7 imply that 7 of the 8 surprisal buckets have and can never produce . Furthermore, in Table 3a, removing clipping (w/o H_t clipping)—which is the exact mechanism that blocks the other 7 buckets—actually yields slightly higher math accuracy on Qwen3-4B (37.52% vs. 37.24%). Meanwhile, Table 2 compares ERIS against Random 20% (35.90%) and |z_t| Top-20% (32.78%) over all tokens, without testing a simple support-matched baseline (e.g., selecting |z_t| Top-20% or Random 20% restricted to the high-support bucket ). Consequently, we cannot tell whether the gain over Random 20% (37.24% vs. 35.90%) comes from excess-risk estimation or simply from refusing to apply importance weights to low-probability tokens ().
2. Single-run point estimates, undefined "divergent" runs, and missing AIS baseline.
All results in Tables 1–3 and 7–8 appear to be single training runs without seed standard deviations or confidence intervals. Several non-divergent margins are fairly small—for instance, on Qwen3-1.7B Math, BF16 (19.35%), ERIS (19.19%), MIS (18.93%), and FP8 (18.73%) sit within 0.62 points of each other, and on Qwen3-8B AIME24, TIS (25.83%) beats ERIS (23.33%). For Qwen3-4B and Qwen2.5-7B (plus and in Table 3b–c), vanilla FP8 is listed simply as -- (divergent) without stating a formal stopping/divergence rule (especially since Figure 3 shows FP8 on Qwen3-4B completing all 220 steps with late-stage reward oscillations). Finally, while Section 2 and Section 4.3 specifically contrast ERIS with AIS (Zhou et al., 2026), AIS is omitted from all empirical tables.
Questions for authors
-
Assessment-critical (High-support filtering vs. excess-risk scoring): Appendix D.1 (p. 26) and Figure 7 show that under , only the first surprisal bucket (, i.e., ) has , so tokens in buckets 2–8 can never be selected (). In Table 2 (
Qwen3-4B), what happens if you apply|z_t| Top-20%orRandom 20%restricted to that same first bucket ()? This control is essential to tell whether the 8-bucket EMA excess-risk machinery adds value beyond simply masking out importance weights on low-probability tokens (). -
Clarification (Statistical reliability & divergence criterion): How many training seeds were run for Tables 1, 2, and 3, and can you provide multi-seed mean std (or problem-level bootstrap CIs) for the
Qwen3-4Bmain comparisons (TIS,Random 20%,w/o support condition,w/o H_t clipping, andERIS)? Also, what exact quantitative criterion was used to markFP8as-- (divergent)in Tables 1, 3, and 8 given that Figure 3 plots theQwen3-4BFP8curve all the way to step 220? -
Clarification (Calibration diagnostics &
AIScomparison): Can you report per-bucket sample counts and empirical calibration errors during actual off-policy updates (where , using yourBF16proxy from Section 5.3) to check Assumption 5? And can you include an empirical comparison againstAIS(Zhou et al., 2026) onQwen3-4B?
2 comments
Kind reminder on rebuttal
Dear Reviewer Aqc8,
Thank you again for the time and effort devoted in reviewing our submission and providing constructive feedback. As the discussion period is nearing its end, we would like to follow up on our response. If appropriate, we would appreciate a brief reply before the deadline to let us know whether our response addressed your concerns and if any concerns remain. We would be glad to clarify any remaining points.
Best regards,
Authors of Submission 6969
Kind reminder on rebuttal
Dear Reviewer Aqc8,
We sincerely appreciate your thoughtful review and valuable feedback. In response to your suggestions, we have provided additional experiments and theoretical clarifications. We have included these additions in our paper and uploaded a new revision.
We hope these responses can address your concerns. If you have any further feedback or remaining concerns, please share with us at your convenience. We would be happy to provide additional clarifications if needed.
Best regards,
Authors of Submission 6969