acceptodds
Under review as a conference paper at ICLR 2027

INT8-RL: Training–Inference Co-Optimization for Quantized Rollouts on Ascend

Abstract

Autoregressive rollout generation is a major cost of reinforcement learning (RL) post-training for large language models. INT8 inference can reduce this cost on accelerators with efficient integer kernels, but quantizing the rollout policy changes the distribution from which training trajectories are sampled. Activation outliers further complicate W8A8 execution. We present , a training–inference co-optimization framework for INT8 rollouts on Ascend. The framework combines quantization-aware training with jointly learned smoothing and weight scales, reproducible hash-based rounding, and selective precision fallback. These components adapt the policy to quantized computation while reducing avoidable numerical discrepancies between training and rollout. Experiments on openPangu-7B and Qwen3-30B MoE show substantial reward recovery over the corresponding INT8 baselines. On openPangu-7B with DAPO-Math-17K, the combined W8A8 configuration improves reward from 0.252 to 0.305, compared with 0.309 for BF16. Profiling shows a 19.7% reduction in per-token decode latency. A separate long-sequence ablation that disables training-side fake quantization reduces step time by 18.7%, indicating optimization headroom rather than the measured end-to-end gain of the complete QAT pipeline. These results highlight both the feasibility of INT8 rollouts and the need to control quantization error and training overhead jointly.

Primary area:
reinforcement learning
Keywords:
RL, INT8 Quantization, Quantization-Aware Training, Low-Precision Rollouts, Learnable Smoothing, Training–Inference Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_i3XkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies INT8 rollout generation for reinforcement learning post-training on Ascend hardware. It presents a framework that combines quantization-aware training, jointly learned smoothing factors and weight quantization scales, reproducible hash-based rounding, and selective mixed-precision fallback to mitigate training–rollout numerical mismatch and quantization-induced performance degradation. The framework is evaluated on openPangu-7B and Qwen3-30B MoE using GSM8K and DAPO-Math-17K, with component ablations examining the proposed techniques. The results show reward recovery relative to the INT8 baselines. The paper also measures rollout latency, reporting a 19.7% reduction in per-token decoding latency on openPangu-7B.

Critical strengths

The paper clearly explains the challenges of INT8 rollouts in RL, particularly activation outliers and training–rollout numerical mismatch, and presents a practical combination of techniques to address them.

Critical weaknesses
  1. Adapting established quantization techniques to RL on Ascend is of practical interest. However, the additional methodological or system-level contributions beyond this platform adaptation are not sufficiently demonstrated in the current paper.

  2. Identically labeled configurations have inconsistent results across tables: the openPangu-7B SR-only reward is 0.543 in Table 1 but 0.550 in Table A2. This unresolved difference equals both the WS gain and the combined method’s advantage over BF16 on GSM8K. Without reconciling these results and providing repeated runs with uncertainty estimates, the reliability of these small gains remains unclear.

  3. QAT and learnable quantization parameters introduce training overhead, yet the efficiency evaluation mainly measures rollout latency. The step-time experiment disables training-side fake quantization, so it does not establish whether rollout savings outweigh the complete method’s additional training cost.

Questions for authors
  1. What causes the discrepancies between identically labeled configurations in the main tables and appendix? Could you reconcile these results and provide multi-seed means and standard deviations to assess whether the small reported gains are reliable?

  2. Could you provide a training-time breakdown showing the overhead introduced by each proposed component, particularly fake quantization, learnable smoothing and weight scales, and hash-based rounding, both individually and in combination?

Reviewer_qXuHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets quantized RL rollout on Ascend, integrating QAT, learnable smoothing and weight scaling, deterministic hash rounding, and high-precision fallback for sensitive layers. The DAPO reward of openPangu-7B recovers from 0.252 to 0.305, approaching the BF16 value of 0.309; it also reports a 19.7% reduction in single-token decoding latency.

Critical strengths

The method exports the smoothed weights and quantization status synchronously with strategy updates, which fits dynamic training scenarios better than fixed calibration.

Critical weaknesses
  • The ablation of WS and SS uses different datasets and QAT configurations respectively, which fails to verify the necessity of their joint learning. There is also a lack of comparable experiments with similar quantization RL methods or strong QAT baselines under the same fallback mask.
  • Under the same response length, fallback mask and reward level, what are the respective time costs for training, rollout, and weight refresh in the full QAT pipeline? Is the total time required to achieve the same reward better than BF16?
  • Does hash rounding practically reduce token-level KL, log-probability bias, and the tails of importance weights compared with nearest rounding, independent stochastic rounding, and synchronous stochastic rounding?
Questions for authors

As mentioned in the weaknesses.

Reviewer_4K6VOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

INT8-RL combines quantization-aware training, learned smoothing and weight scales, reproducible hash-based rounding, and selective precision fallback for INT8 RL rollouts on Ascend. Experiments on dense and MoE models demonstrate reward recovery, alongside reduced rollout decoding latency.

Critical strengths

The paper addresses a practical challenge in quantized RL. On DAPO, reward improves from 0.252 to 0.305, approaching the BF16 result of 0.309.

Evaluation on both dense and MoE models provides useful evidence that the approach extends beyond one architecture.

Critical weaknesses

The efficiency benefit of the complete method remains unestablished. The reported 18.7% step-time reduction disables training-side fake quantization; it therefore does not demonstrate end-to-end acceleration with the proposed QAT pipeline.

Component attribution and experimental reliability need stronger evidence. Joint scale learning is not isolated under matched settings, and repeated configurations have unresolved discrepancies across tables. Missing variance estimates further limit interpretation.

Questions for authors

What is the complete QAT pipeline’s end-to-end runtime at matched quality, including weight refresh and synchronization?

Can you provide a matched neither/WS/SS/both ablation and reconcile the repeated-configuration discrepancies?

How are Reward and Pearson computed, and what are the training/evaluation splits and across-seed uncertainties?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.