- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
QuantLRM uses weight differences between a reasoning model and an earlier checkpoint to guide weight-only quantization. It assigns higher importance to small and large updates, separately handles zero updates, and incorporates these scores into an AWQ-style scaling search. Experiments across SFT, DPO, and RLVR models report improved reasoning accuracy, particularly at 3-bit precision.
Critical strengths
S1: Consistent gains across models and training stages. The method improves average accuracy over the strongest included baselines across several model families, supporting the practical value of checkpoint differences.
S2: Simple integration with relevant ablations. QuantLRM fits into an existing quantization workflow. Experiments compare different protected regions, mapping functions, and zero-update treatments, including comparisons using AWQ scaling.
Critical weaknesses
W1. The comparison with other delta-aware approaches is limited. DAQ already uses base/post-trained weight differences to guide quantization. Although it is discussed in §2, the experiments do not establish the advantage of QuantLRM’s specific scoring rule over alternative delta-aware objectives. DAQ’s FP8 setting differs from the paper’s W3 focus, so a controlled comparison or a clearer discussion of this distinction would strengthen the contribution.
W2. The interpretation and robustness of zero-update weighting remain unclear. Exact-zero counts can depend on checkpoint precision and rounding, which complicates their interpretation as weight importance. Appendix B also introduces matrix slicing to avoid overflow when computing the scores, but does not clearly establish whether this preserves Equation 5. These details matter because zero-update counts directly multiply the channel scores.
W3. The reported confidence intervals do not reflect the original benchmark sizes. Appendix C uses 50,000 paired draws per benchmark in each bootstrap replicate, substantially exceeding the actual question counts. Without an appropriate adjustment, this produces overly narrow intervals for the reported evaluation sizes. The significance of the smaller gains, particularly the approximately 0.25-point DPO improvement, therefore needs reassessment.
Questions for authors
Q1: Can you compare alternative delta-aware scoring rules or objectives under the same W3 quantizer and search budget, to isolate the benefit of the proposed two-ended score?
Q2: Is the matrix-slicing procedure equivalent to Equation 5? How sensitive are channel rankings and accuracy to the slicing configuration and the precision used to identify zero updates?
Q3: Do the smaller improvements remain statistically supported when paired bootstrap samples match each benchmark’s actual question count? How is stochastic decoding accounted for?