- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper approaches the problem of training quantized LLM policies with RL, particularly in the “W4A4” setup (quantized linear layers use 4-bit weights and 4-bit activations). They proposed a method that consists in two interventions: first, “Quantization-Consistent Rollouts”, which samples GRPO groups into two subgroups based on different quantization methods (“RTN” and “SR”), and apply additional reward heuristics for consistency/calibration. Second, they introduce “Quantization Weights Dampening”, which smooths the quantized weights toward its dequantized values. The paper provides experimental evidence on math reasoning benchmarks to claim that the method surpasses other quantization-based RL approaches.
Critical strengths
- The approached problem is relevant: quantized RL training is relevant to enable better reasoning models in resource-constrained inference settings.
Critical weaknesses
-
The paper needs to improve substantially in writing clarity, particularly in the Abstract and Introduction. It is full of acronyms and technical terms that are only defined much later in the work. I could only start understanding the method on Section 4 and Figure 3 (right), which is a strong sign that Abstract/Introduction are not communicating the ideas clearly.
- One example: “the rollout strategy collects W4A16 completions from round-to nearest and stochastic-rounding views of shared latent weights and calibrates their rewards through cross-view comparisons and within-view support to guide W4A4 policy optimization”. It is extremely hard to parse given the amount of technical terms without a proper prior definition, which makes it really hard to communicate the goal of the method.
-
None of the presented results report confidence intervals across different training seeds, so it is hard to judge statistical significance. The improvements might be due to observation noise. This is especially necessary here since RL training has a lot of variance, in particular for quantized settings that are on the edge of instability. Furthermore, some of the presented benchmarks usually contain very few datapoints (e.g., AIME), and reporting a difference of 0.83% to 1.67% in pass@1 must be qualified with confidence intervals.
-
The paper brings an “entropy” analysis, claiming that high entropy in early training means effective exploration, while converging to the BF16 entropy level at the end means stability. The analysis itself is very superficial and does not substantiate the claims: the entropy curve in isolation does not tell anything about exploration and instability (which are a complex byproduct of the whole learning and environment dynamics), so the claimed link needs much stronger evidence.
-
The paper could bring more clarity in the design decisions related to the proposed reward function. It should better explain why these terms were considered and what kind of behaviors in the policy they aim to reinforce.
-
I believe the paper should also bring the limitations/challenges of the proposed method, and contextualize with the aimed practical gains. From my understanding, the method requires keeping a copy of the bf16 weights in memory (for QWD), + 2 quantized versions for rollouts, which is a lot of memory.
Questions for authors
See the weaknesses above, particularly the limitations/challenges point.
4 comments
We hope our response addresses your questions regarding QCRL. Before the rebuttal period concludes, we would like to clarify several aspects of the method to support your assessment.
1. How SR complements RTN.
The results of our additional experiments indicate that replacing two of four RTN responses with SR recovers 17.95%–23.38% of the GSM8K questions missed by all four RTN responses. At the same response budget, RTN alone reaches 74.07%, SR alone reaches 71.34%, and mixing them reaches 76.35% with the original rewards. QCR further raises accuracy to 77.41%.
SR serves as a complementary exploration source alongside RTN. Other perturbations can also be useful: our QeRL-style Full baseline reaches 75.21%. But random weight perturbations can produce values outside the quantization grid, introducing additional error when those values are quantized. SR samples directly from the grid, so the sampled weights and their resulting inference behavior can be reproduced exactly using the same quantization grid and inference settings.
2. How W4A16 rollouts are used for W4A4 updates.
In Eq. (2), the W4A4 learner recomputes each token’s probability and divides it by the probability recorded by the W4A16 view that generated that token. We adopted W4A16 rollouts after observing training collapse when using W4A4 for both rollouts and updates. We believe the main issue is that W4A4 rollouts cannot consistently generate correct response trajectories to guide exploration throughout RL training. Both the W4A16 and W4A4 models share the same BF16 weights. Their KL divergence is of the same order of magnitude as that associated with the inherent training–inference mismatch commonly discussed in RL. Clipping extreme outliers with TIS is sufficient to mitigate this mismatch and enable stable updates. We also report the KL divergence values obtained using vLLM and Hugging Face (HF), matching the inference engines used during RL training. These values are computed over 128 responses with fixed prefixes and averaged, yielding numerically close results.
| KL, nats | Mean |
|---|---|
| vLLM W4A4 → HF W4A4 | 0.200787 |
| vLLM W4A16 → HF W4A4 | 0.233078 |
3. Applying QCR to RTN-only responses.
Randomly splitting RTN responses into two groups and applying QCR reaches 74.37%, a gain of about 0.3 percentage points over RTN-only training with the original rewards. With mixed RTN and SR responses, the same reward rule gives a larger gain of 1.06 percentage points, from 76.35% to 77.41%. These results show that QCR provides some regularization for standard quantized RL training and is more effective when applied across different views.
| Sample Method | Reward | Accuracy |
|---|---|---|
| RTN only | Original | 74.04% |
| RTN only | QCR | 74.37% |
| RTN-SR mixed | QCR | 77.41% |
Dear authors,
While I appreciate the efforts on bringing more evidence during rebuttal, my main concerns related to writing clarity were not addressed. I would note that ICLR allows for revising the draft, so I cannot recommend acceptance in the current form. My second main concern related to statistical significance was addressed partially: I expected all the results from the (main) paper to provide evidence of statistical significance, include the new evidence provided during rebuttal. Finally, the new experiment on entropy shows improvements over training, but does not provide evidence to attribute this to exploration, and the main paper still claims exploration gains based on entropy.
In light to these points, I decide to keep my score.
Thank you for your feedback and for pointing out where the manuscript could be improved. We report averages across multiple random seeds or checkpoints for key metrics in the rebuttal and main results in the manuscript. Given the time constraints, we will provide the corresponding averages for the remaining results in a future revision.
We recognize that entropy provides only indirect evidence of QCRL’s training gains. We therefore added a sampling experiment showing that the dual-view design discovers correct trajectories that RTN alone did not sample. QCR reward calibration gives greater weight in the policy update to correct trajectories supported by other correct responses from the same view.
We have submitted a revised manuscript that addresses the concerns about clarity. The main changes are as follows.
- We moved the method overview figure to page 2 so readers can see it earlier. We will also revise the text within the figure in a subsequent version.
- We revised the Abstract and Introduction to briefly explain each technical concept before introducing its corresponding term.
- We revised all claims based on entropy changes to reflect their role as indirect evidence. We also added the new sampling results to Appendix B.7, where they provide direct evidence of the exploration gains from the dual-view design.
1. “The paper needs to improve substantially in writing clarity, particularly in the Abstract and Introduction.”
Thank you for your helpful feedback. In future revisions, we will provide clearer explanations and introduce the relevant technical terms earlier to make the paper easier for all readers to understand.
2. “None of the presented results report confidence intervals across different training seeds, so it is hard to judge statistical significance.”
We already report additional training results across multiple random seeds in Table 6 to verify the consistency of these gains. Due to current computational resource constraints, we will include more comprehensive training results across multiple random seeds in the final version.
3. “the entropy curve in isolation does not tell anything about exploration and instability”
Entropy provides an indirect indication of the model’s degree of exploration and confidence in its answers. We have added experiments that more directly demonstrate the exploration gains from quantization. RTN and SR provide complementary correct responses from the same checkpoint. On 512 GSM8K questions, we compare four RTN responses with two shared RTN responses plus two SR responses at different stages of training. The table below reports the mean number of correct answers unique to SR. SR recovers 17.95%–23.38% of questions missed by all four RTN responses. These results support the complementarity of rounding-based sampling strategies and demonstrate the exploration benefits of intrinsic quantization noise.
4. “The paper could bring more clarity in the design decisions related to the proposed reward function.”
QCR increases the penalty for an incorrect answer when the other rounding version succeeds, and provides an additional reward for a correct answer when other answers from the same version are also correct. These adjusted rewards directly determine the W4A4 update. In an additional experiment, we performed three training update steps while changing only the rewards and averaged the results over three runs. This modification improves accuracy by 0.39 percentage points and reduces the measured W4A16–W4A4 distribution gap by 7.32%. These results provide empirical evidence that the reward constraints help the quantized model improve consistently.
5. “I believe the paper should also bring the limitations/challenges of the proposed method, and contextualize with the aimed practical gains.”
We report the training resource usage for Qwen2.5-3B-Instruct. Both QCRL and direct W4A4 RL use W4A16 rollouts and W4A4 updates. Accounting for both rollout versions and all BF16 training states, QCRL increases training time by 8.81% without incurring additional GPU memory overhead, since the total number of response trajectories generated per prompt remains fixed. The additional runtime comes from the implementation of SR. During training, only a single BF16 copy of the model is maintained, and its parameters are dynamically quantized to 4 bits for computation during inference and updates. The final low-bit model is generated after training and saved in a specific format.