acceptodds
Under review as a conference paper at ICLR 2027

QCRL: Quantization-Consistent Reinforcement Learning for LLM Reasoning

Abstract

Obtaining a capable W4A4 reasoning model through reinforcement learning with verifiable rewards remains challenging because direct W4A4 optimization suffers from rounding-induced reward instability and rollout-optimization policy mismatch. We propose Quantization-Consistent Reinforcement Learning (QCRL), a dual-view framework that combines two strategies, Quantization-Consistent Rollout and Quantization-Weight Dampening, to improve the final W4A4 policy. Specifically, the rollout strategy collects W4A16 completions from round-to-nearest and stochastic-rounding views of shared latent weights and calibrates their rewards through cross-view comparisons and within-view support to guide W4A4 policy optimization. The weight-dampening strategy stabilizes weights near their quantization centers after parameter updates. On GSM8K with Qwen2.5-3B, QCRL achieves 78.32% pass@1 under W4A4 evaluation, surpassing BF16 reinforcement learning followed by W4A4 post-training quantization by 2.28 percentage points and direct W4A4 reinforcement learning by 4.25 percentage points. These results support reinforcement learning for stronger W4A4 reasoning models while retaining compatibility with W4A16 rollout.

Primary area:
reinforcement learning
Keywords:
Quantization, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KEBQOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper approaches the problem of training quantized LLM policies with RL, particularly in the “W4A4” setup (quantized linear layers use 4-bit weights and 4-bit activations). They proposed a method that consists in two interventions: first, “Quantization-Consistent Rollouts”, which samples GRPO groups into two subgroups based on different quantization methods (“RTN” and “SR”), and apply additional reward heuristics for consistency/calibration. Second, they introduce “Quantization Weights Dampening”, which smooths the quantized weights toward its dequantized values. The paper provides experimental evidence on math reasoning benchmarks to claim that the method surpasses other quantization-based RL approaches.

Critical strengths
  • The approached problem is relevant: quantized RL training is relevant to enable better reasoning models in resource-constrained inference settings.
Critical weaknesses
  • The paper needs to improve substantially in writing clarity, particularly in the Abstract and Introduction. It is full of acronyms and technical terms that are only defined much later in the work. I could only start understanding the method on Section 4 and Figure 3 (right), which is a strong sign that Abstract/Introduction are not communicating the ideas clearly.

    • One example: “the rollout strategy collects W4A16 completions from round-to nearest and stochastic-rounding views of shared latent weights and calibrates their rewards through cross-view comparisons and within-view support to guide W4A4 policy optimization”. It is extremely hard to parse given the amount of technical terms without a proper prior definition, which makes it really hard to communicate the goal of the method.
  • None of the presented results report confidence intervals across different training seeds, so it is hard to judge statistical significance. The improvements might be due to observation noise. This is especially necessary here since RL training has a lot of variance, in particular for quantized settings that are on the edge of instability. Furthermore, some of the presented benchmarks usually contain very few datapoints (e.g., AIME), and reporting a difference of 0.83% to 1.67% in pass@1 must be qualified with confidence intervals.

  • The paper brings an “entropy” analysis, claiming that high entropy in early training means effective exploration, while converging to the BF16 entropy level at the end means stability. The analysis itself is very superficial and does not substantiate the claims: the entropy curve in isolation does not tell anything about exploration and instability (which are a complex byproduct of the whole learning and environment dynamics), so the claimed link needs much stronger evidence.

  • The paper could bring more clarity in the design decisions related to the proposed reward function. It should better explain why these terms were considered and what kind of behaviors in the policy they aim to reinforce.

  • I believe the paper should also bring the limitations/challenges of the proposed method, and contextualize with the aimed practical gains. From my understanding, the method requires keeping a copy of the bf16 weights in memory (for QWD), + 2 quantized versions for rollouts, which is a lot of memory.

Questions for authors

See the weaknesses above, particularly the limitations/challenges point.

4 comments

1. “The paper needs to improve substantially in writing clarity, particularly in the Abstract and Introduction.”

Thank you for your helpful feedback. In future revisions, we will provide clearer explanations and introduce the relevant technical terms earlier to make the paper easier for all readers to understand.

2. “None of the presented results report confidence intervals across different training seeds, so it is hard to judge statistical significance.”

We already report additional training results across multiple random seeds in Table 6 to verify the consistency of these gains. Due to current computational resource constraints, we will include more comprehensive training results across multiple random seeds in the final version.

Method-4 SeedGSM8K Accuracy (mean ± std, %)
W4A4 RL73.94 ± 0.32
QCRL77.24 ± 0.38

3. “the entropy curve in isolation does not tell anything about exploration and instability”

Entropy provides an indirect indication of the model’s degree of exploration and confidence in its answers. We have added experiments that more directly demonstrate the exploration gains from quantization. RTN and SR provide complementary correct responses from the same checkpoint. On 512 GSM8K questions, we compare four RTN responses with two shared RTN responses plus two SR responses at different stages of training. The table below reports the mean number of correct answers unique to SR. SR recovers 17.95%–23.38% of questions missed by all four RTN responses. These results support the complementarity of rounding-based sampling strategies and demonstrate the exploration benefits of intrinsic quantization noise.

CheckpointSR-onlySR rescue among RTN failures
Step 1015.6718.50%
Step 2014.0017.95%
Step 3018.0023.38%

4. “The paper could bring more clarity in the design decisions related to the proposed reward function.”

QCR increases the penalty for an incorrect answer when the other rounding version succeeds, and provides an additional reward for a correct answer when other answers from the same version are also correct. These adjusted rewards directly determine the W4A4 update. In an additional experiment, we performed three training update steps while changing only the rewards and averaged the results over three runs. This modification improves accuracy by 0.39 percentage points and reduces the measured W4A16–W4A4 distribution gap by 7.32%. These results provide empirical evidence that the reward constraints help the quantized model improve consistently.

W4A4 model after updateOriginal rewardQCR rewardDifference
W4A4 Accuracy68.16%68.55%+0.39
Mean RTN W4A16–W4A4 KL0.166630.15443−7.32%

5. “I believe the paper should also bring the limitations/challenges of the proposed method, and contextualize with the aimed practical gains.”

We report the training resource usage for Qwen2.5-3B-Instruct. Both QCRL and direct W4A4 RL use W4A16 rollouts and W4A4 updates. Accounting for both rollout versions and all BF16 training states, QCRL increases training time by 8.81% without incurring additional GPU memory overhead, since the total number of response trajectories generated per prompt remains fixed. The additional runtime comes from the implementation of SR. During training, only a single BF16 copy of the model is maintained, and its parameters are dynamically quantized to 4 bits for computation during inference and updates. The final low-bit model is generated after training and saved in a specific format.

MetricDirect W4A4 RLQCRLDifference
Training-loop time (s)88.8196.63+8.81%
Maximum observed per-GPU memory (GiB)32.95932.9590%

We hope our response addresses your questions regarding QCRL. Before the rebuttal period concludes, we would like to clarify several aspects of the method to support your assessment.

1. How SR complements RTN.

The results of our additional experiments indicate that replacing two of four RTN responses with SR recovers 17.95%–23.38% of the GSM8K questions missed by all four RTN responses. At the same response budget, RTN alone reaches 74.07%, SR alone reaches 71.34%, and mixing them reaches 76.35% with the original rewards. QCR further raises accuracy to 77.41%.

SR serves as a complementary exploration source alongside RTN. Other perturbations can also be useful: our QeRL-style Full baseline reaches 75.21%. But random weight perturbations can produce values outside the quantization grid, introducing additional error when those values are quantized. SR samples directly from the grid, so the sampled weights and their resulting inference behavior can be reproduced exactly using the same quantization grid and inference settings.

2. How W4A16 rollouts are used for W4A4 updates.

In Eq. (2), the W4A4 learner recomputes each token’s probability and divides it by the probability recorded by the W4A16 view that generated that token. We adopted W4A16 rollouts after observing training collapse when using W4A4 for both rollouts and updates. We believe the main issue is that W4A4 rollouts cannot consistently generate correct response trajectories to guide exploration throughout RL training. Both the W4A16 and W4A4 models share the same BF16 weights. Their KL divergence is of the same order of magnitude as that associated with the inherent training–inference mismatch commonly discussed in RL. Clipping extreme outliers with TIS is sufficient to mitigate this mismatch and enable stable updates. We also report the KL divergence values obtained using vLLM and Hugging Face (HF), matching the inference engines used during RL training. These values are computed over 128 responses with fixed prefixes and averaged, yielding numerically close results.

KL, natsMean
vLLM W4A4 → HF W4A40.200787
vLLM W4A16 → HF W4A40.233078

3. Applying QCR to RTN-only responses.

Randomly splitting RTN responses into two groups and applying QCR reaches 74.37%, a gain of about 0.3 percentage points over RTN-only training with the original rewards. With mixed RTN and SR responses, the same reward rule gives a larger gain of 1.06 percentage points, from 76.35% to 77.41%. These results show that QCR provides some regularization for standard quantized RL training and is more effective when applied across different views.

Sample MethodRewardAccuracy
RTN onlyOriginal74.04%
RTN onlyQCR74.37%
RTN-SR mixedQCR77.41%

Dear authors,

While I appreciate the efforts on bringing more evidence during rebuttal, my main concerns related to writing clarity were not addressed. I would note that ICLR allows for revising the draft, so I cannot recommend acceptance in the current form. My second main concern related to statistical significance was addressed partially: I expected all the results from the (main) paper to provide evidence of statistical significance, include the new evidence provided during rebuttal. Finally, the new experiment on entropy shows improvements over training, but does not provide evidence to attribute this to exploration, and the main paper still claims exploration gains based on entropy.

In light to these points, I decide to keep my score.

Thank you for your feedback and for pointing out where the manuscript could be improved. We report averages across multiple random seeds or checkpoints for key metrics in the rebuttal and main results in the manuscript. Given the time constraints, we will provide the corresponding averages for the remaining results in a future revision.

We recognize that entropy provides only indirect evidence of QCRL’s training gains. We therefore added a sampling experiment showing that the dual-view design discovers correct trajectories that RTN alone did not sample. QCR reward calibration gives greater weight in the policy update to correct trajectories supported by other correct responses from the same view.

We have submitted a revised manuscript that addresses the concerns about clarity. The main changes are as follows.

  1. We moved the method overview figure to page 2 so readers can see it earlier. We will also revise the text within the figure in a subsequent version.
  2. We revised the Abstract and Introduction to briefly explain each technical concept before introducing its corresponding term.
  3. We revised all claims based on entropy changes to reflect their role as indirect evidence. We also added the new sampling results to Appendix B.7, where they provide direct evidence of the exploration gains from the dual-view design.
Reviewer_2bkTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes QCRL to improve reasoning models trained and evaluated with W4A4 quantization. It collects W4A16 responses from two rounding views of the same weights and adjusts their training rewards using the success rates within and across views. It also periodically moves weights toward their quantization centers to reduce unstable switching. Experiments show improvements over direct quantized RL and BF16 RL followed by quantization.

Critical strengths
  • The paper addresses a useful problem: preserving reasoning accuracy in the final W4A4 model.
  • The experiments include several model sizes, different math datasets, component ablations, and multiple training seeds on GSM8K.
  • The total response budget is kept fixed. The mixed-view baseline also helps separate the benefit of sampling from the benefit of reward calibration.
Critical weaknesses
  • How does the method address the W4A16–W4A4 mismatch? Both rollout views use W4A16, while updates use W4A4. Why should agreement between these views provide reliable guidance for the W4A4 policy? Appendix A.3 also mentions that fully W4A4 rollout and training collapsed. More details about this failure and the remaining mismatch would help.
  • Is the reward adjustment specific to quantization? Different answers can receive different rewards simply because sampling produces different solutions. Would the same reward rule help if ordinary RTN responses were randomly split into two groups? This control would help separate quantization-specific benefits from general reward shaping.
  • Do the gains hold in actual W4A4 deployment? The accuracy results use fake quantization, and the reported speed measurements concern W4A16 decoding. Please test the final model with a native W4A4 implementation. Full training time and peak memory, including the BF16 training states and both rollout views, would also be useful.
  • What does QWD add beyond existing weight dampening? Earlier work already studies dampening weight oscillations during quantization-aware training. The paper acknowledges this connection. A direct comparison under the same RL setting would help show the benefit of the proposed update and scale protection.
Questions for authors

see weaknesses

4 comments

1. “How does the method address the W4A16–W4A4 mismatch?”

QCRL uses weight-rounding perturbations to improve exploration for the final W4A4 policy. The learner recomputes probabilities under W4A4 quantization and uses each sample’s actual behavior-policy likelihood in Equation (2). We adopted W4A16 rollouts after observing training collapse when using W4A4 for both rollouts and updates. We believe the main issue is that W4A4 rollouts cannot consistently generate correct response trajectories to guide exploration throughout RL training. Both the W4A16 and W4A4 models share the same BF16 weights. Their KL divergence is of the same order of magnitude as that associated with the inherent training–inference mismatch commonly discussed in RL. Clipping extreme outliers with TIS is sufficient to mitigate this mismatch and enable stable updates. We also report the KL divergence values obtained using vLLM and Hugging Face (HF), matching the inference engines used during RL training. These values are computed over 128 responses with fixed prefixes and averaged, yielding numerically close results.

KL, natsMean
vLLM W4A4 → HF W4A40.200787
vLLM W4A16 → HF W4A40.233078

2. “Would the same reward rule help if ordinary RTN responses were randomly split into two groups?”

We have added this control experiment. We randomly split standard RTN responses into two groups and train with QCR constraints, achieving a final accuracy of 74.37%. This is 0.30 percentage points higher than direct training, but still 3.04 percentage points lower than QCR with SR. These results show that QCR provides some regularization for standard quantized RL training and is more effective when applied across different views.

Sample MethodRewardAccuracy
RTN onlyOriginal74.04%
RTN onlyQCR74.37%
RTN-SR mixedQCR77.41%

3. “Do the gains hold in actual W4A4 deployment?”

We have added an evaluation of the trained quantized models on an NVIDIA RTX PRO 5000, using matched quantization formats, scales, and group sizes. QCRL achieves an accuracy of 78.09%, while direct W4A4 RL and BF16 RL + PTQ achieve 74.15% and 75.82%, respectively. The quantized models’ performance and the accuracy gaps between methods on NVFP4-capable hardware are broadly consistent with the results reported in the original manuscript.

4. “Full training time and peak memory, including the BF16 training states and both rollout views, would also be useful.”

We report the training resource usage for Qwen2.5-3B-Instruct. Both QCRL and direct W4A4 RL use W4A16 rollouts and W4A4 updates. Accounting for both rollout versions and all BF16 training states, QCRL increases training time by 8.81% without incurring additional GPU memory overhead, since the total number of response trajectories generated per prompt remains fixed. The additional runtime comes from the implementation of SR. During training, only a single BF16 copy of the model is maintained, and its parameters are dynamically quantized to 4 bits for computation during inference and updates. The final low-bit model is generated after training and saved in a specific format.

MetricDirect W4A4 RLQCRLDifference
Training-loop time (s)88.8196.63+8.81%
Maximum observed per-GPU memory (GiB)32.95932.9590%

5. “What does QWD add beyond existing weight dampening?”

QWD periodically pulls weights toward quantization centers after optimizer updates, without modifying the optimizer state. In a matched comparison with existing damping, our additional scale protection mechanism ensures that QWD remains effective. Compared with the existing damping implementation, QWD improves W4A4 accuracy by 1.06 percentage points.

Sample MethodAccuracy
Base77.41%
Original damping77.26%
QWD78.32%

Thanks for the response. My original questions about the W4A4 deployment test, training time and memory use have been well addressed. I still have two more questions.

The first one is about the W4A4 quantization error in the introduction. When you use SR, the reward disagreement is 24% (Figure 2(b)), but with the default setting RTN it is only about 10%. So in this part, you should state 10% rather than 24%.

My second question is about the function of SR. In Table 1, pure SR results in worse performance than RTN. That means using SR independently is harmful. But according to your experimental results, mixing SR and RTN brings advantages. So I want to know whether other sampling or rounding methods besides SR are also useful, for example, adding Gaussian weight noise to half of the samples, as in QeRL.

Thank you for your feedback.

1.“You should state 10% rather than 24%.”

We acknowledge that our original description was unclear. The default RTN setting gives approximately 0.10 excess reward disagreement, whereas 0.24 corresponds specifically to SR. We will revise the Introduction to foreground the RTN result and identify the SR result separately.

2.“Whether other sampling or rounding methods besides SR are also useful.”

SR serves as a complementary exploration source alongside RTN. Although SR alone underperforms RTN (71.34% vs. 74.07%), mixing them at the same response budget reaches 76.35%, and reward calibration further improves this to 77.41%. Other perturbations can also be useful: our QeRL-style Full baseline reaches 75.21%. But random weight perturbations can produce values outside the quantization grid, introducing additional error when those values are quantized. SR samples directly from the grid, so the sampled weights and their resulting inference behavior can be reproduced exactly using the same quantization grid and inference settings.

Thanks for the response. Now SR's benefit becomes clear for me. Since most of my concerns are addressed, I will increase my rating to WA.

Reviewer_4ok2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

QCRL aims to improve mathematical reasoning in models with 4-bit weights and activations (W4A4). It generates responses using two W4A16 versions of the same model, constructed with nearest rounding and stochastic rounding. It adjusts training rewards using their success rates and periodically pulls weights toward their quantization centers. Policy updates and final evaluation use W4A4 fake quantization. On GSM8K, Qwen2.5-3B achieves 78.32%, compared with 74.07% for direct quantized RL and 76.04% for BF16 RL followed by quantization.

Critical strengths
  1. The paper addresses a meaningful failure of low-precision training. Directly training a quantized model can perform worse than training in BF16 and quantizing afterward. QCRL improves on both baselines, supporting its practical value for preserving reasoning performance under aggressive quantization.
  2. The experiments provide evidence beyond a single final score. Improvements appear across several models and mathematical benchmarks. Component ablations separate sampling, reward adjustment, and weight dampening, while training diagnostics show reduced switching between quantized weight values.
Critical weaknesses
  1. The connection between the reward adjustment and the W4A4 training problem remains unclear. Both sampling versions use W4A16 and differ in weight rounding, whereas the W4A4 learner also experiences activation quantization errors. Higher final accuracy demonstrates effectiveness, but does not establish that the reward adjustment reduces the mismatch between these sampling policies and the W4A4 learner.
  2. Additional memory and compute costs are insufficiently evaluated. Constructing and managing two quantized sampling versions adds quantization and model-state management work, with potential GPU memory overhead. The paper does not report the resulting peak training memory or total training cost. Its efficiency measurements compare W4A16 decoding with BF16 decoding, rather than evaluating the complete QCRL pipeline. Consequently, the practical cost of its accuracy gains remains unclear.
Questions for authors

Could the authors analyze the advantages after group normalization and explain how the reward changes that remain help address the mismatch between W4A16 sampling and W4A4 optimization?

3 comments

1. “The connection between the reward adjustment and the W4A4 training problem remains unclear.”

QCR increases the penalty for an incorrect answer when the other rounding version succeeds, and provides an additional reward for a correct answer when other answers from the same version are also correct. These adjusted rewards directly determine the W4A4 update. In an additional experiment, we performed three training update steps while changing only the rewards and averaged the results over three runs. This modification improves accuracy by 0.39 percentage points and reduces the measured W4A16–W4A4 distribution gap by 7.32%. These results provide empirical evidence that the reward constraints help the quantized model improve consistently.

W4A4 model after updateOriginal rewardQCR rewardDifference
W4A4 Accuracy68.16%68.55%+0.39
Mean RTN W4A16–W4A4 KL0.166630.15443−7.32%

2. “Additional memory and compute costs are insufficiently evaluated.”

We report the training resource usage for Qwen2.5-3B-Instruct. Both QCRL and direct W4A4 RL use W4A16 rollouts and W4A4 updates. Accounting for both rollout versions and all BF16 training states, QCRL increases training time by 8.81% without incurring additional GPU memory overhead, since the total number of response trajectories generated per prompt remains fixed. The additional runtime comes from the implementation of SR. During training, only a single BF16 copy of the model is maintained, and its parameters are dynamically quantized to 4 bits for computation during inference and updates. The final low-bit model is generated after training and saved in a specific format.

MetricDirect W4A4 RLQCRLDifference
Training-loop time (s)88.8196.63+8.81%
Maximum observed per-GPU memory (GiB)32.95932.9590%

3. “Could the authors analyze the advantages after group normalization and explain how the reward changes that remain help address the mismatch between W4A16 sampling and W4A4 optimization?”

Normalization removes shifts and scaling common to the entire group, but does not eliminate newly introduced relative differences between responses. After QCR calibration, these responses no longer receive the same reward, so their normalized learning weights also differ.

Across 24 questions, we analyze 96 fixed response groups, each containing two RTN and two SR responses. Using sample-standard-deviation normalization with ε = 10⁻⁴, reward calibration changes at least one advantage by more than 10⁻³ in 23 of the 64 nonzero-variance groups (35.94%), where nonzero-variance groups contain both correct and incorrect responses.

In our earlier experiments, we also reported gains during actual training. These changes improve W4A4 accuracy by 0.39 percentage points and reduce the measured W4A16–W4A4 distribution gap by 7.32%.

All nonzero-variance groupsGroups with advantage changes > 10⁻³Percentage within category
642335.94%

We hope our response addresses your questions regarding QCRL. Before the rebuttal period concludes, we would like to clarify two aspects of the method to support your assessment. We explain how SR complements RTN and how W4A16 rollouts are used to update the W4A4 policy.

1. How SR complements RTN.

The results of our additional experiments indicate that replacing two of four RTN responses with SR recovers 17.95%–23.38% of the GSM8K questions missed by all four RTN responses. At the same response budget, RTN alone reaches 74.07%, SR alone reaches 71.34%, and mixing them reaches 76.35% with the original rewards. QCR further raises accuracy to 77.41%.

SR serves as a complementary exploration source alongside RTN. Other perturbations can also be useful: our QeRL-style Full baseline reaches 75.21%. But random weight perturbations can produce values outside the quantization grid, introducing additional error when those values are quantized. SR samples directly from the grid, so the sampled weights and their resulting inference behavior can be reproduced exactly using the same quantization grid and inference settings.

2. How W4A16 rollouts are used for W4A4 updates.

In Eq. (2), the W4A4 learner recomputes each token’s probability and divides it by the probability recorded by the W4A16 view that generated that token. We adopted W4A16 rollouts after observing training collapse when using W4A4 for both rollouts and updates. We believe the main issue is that W4A4 rollouts cannot consistently generate correct response trajectories to guide exploration throughout RL training. Both the W4A16 and W4A4 models share the same BF16 weights. Their KL divergence is of the same order of magnitude as that associated with the inherent training–inference mismatch commonly discussed in RL. Clipping extreme outliers with TIS is sufficient to mitigate this mismatch and enable stable updates. We also report the KL divergence values obtained using vLLM and Hugging Face (HF), matching the inference engines used during RL training. These values are computed over 128 responses with fixed prefixes and averaged, yielding numerically close results.

KL, natsMean
vLLM W4A4 → HF W4A40.200787
vLLM W4A16 → HF W4A40.233078

Thank you for the thoughtful response. The additional experiments have addressed my main concerns and strengthened the paper substantially.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.