- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how spectral structure of accumulated quantization error relates to predictive degradation. It finds that correcting the dominant error directions provides little benefit, whereas correcting the complementary tail substantially restores NLL and predictive-distribution agreement. Motivated by these findings, the paper proposes Harmful-Tail-Guided Sparse Compensation (HTSC). HTSC constructs an input-conditioned repair teacher and selects modules whose local corrections propagate harmful-tail directions and improve the final predictive distribution. The teacher's predictive distribution is then distilled into low-rank branches at the selected modules of the frozen quantized model.
Critical strengths
- The paper provides substantive empirical evidence that spectral error magnitude can misrepresent predictive damage through the exploration of head/tail interventions, NLL and KL measurements, dimension- and energy-controlled comparisons in Section 3.
- The experimental evaluation demonstrates that sparse compensation (through learning corrections to counteract quantization errors) restores downstream accuracy across model sizes, closely matching their BF16 references (cf. Table 3).
Critical weaknesses
- Insufficient isolation of the harmful-tail guidance: The ablations presented in Table 4 retain the same teacher, leaving its advantage over direct BF16 distillation or next-token training unclear. Module selection is only compared against one random configuration, while other compensation baselines (acknowledged related work [1,2]) are also absent. These missing controls limit the attribution to the HTSC’s distinct components.
- Performance-preservation and inference-overhead claims require stronger support: The empirical evidence provided supports accuracy recovery more strongly than broadly lossless, efficient quantization. While HTSC is able to recover or even exceed the accuracy of the BF16 baselines, see Table 3 and Table 5 (Appendix C), it remains worse on all reported perplexity measurements presented in Table 6 (Appendix D). Moreover, the small average-accuracy gains over the BF16 baseline lack uncertainty estimates, and evaluation covers only one model family (Qwen) and one fake-quantization format (W4A4 NVFP4). Furthermore, although the additional parameter count is small, the claimed low inference overhead is not validated by latency, throughput, or memory measurements; therefore claims such as “with low additional overhead” (line 025) and “with small additional inference cost” (line 080) are hard to empirical verify with the current experimental evaluation.
[1] Zhang et al., LQER: Low-rank quantization error reconstruction for LLMs. ICML 2024
[2] Zhang et al., QERA: an analytical framework for quantization error reconstruction. ICLR 2025
Questions for authors
-
Under matched training conditions, how does the proposed teacher distillation compare with direct BF16 distillation or next-token training? With the teacher and compensation budget fixed, does selecting modules by local output-error magnitude or removing the harmful-tail projection from Equation (3) achieve comparable performance?
-
How does HTSC compare against close compensation baselines such as LQER [1] and QERA [2]?
-
Can you quantify HTSC’s additional inference cost relative to the uncompensated W4A4 NVFP4 model by reporting latency, throughput, and peak memory under matched hardware and workload settings?