acceptodds
Withdrawn from ICLR 2027

The Harmful Tail: Toward Lossless Quantization of Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Low-bit quantization substantially reduces the cost of LLM inference, yet the relationship between error magnitude and predictive degradation remains poorly characterized. We show that this relationship is highly non-uniform across spectral directions. In W4A4-quantized Qwen3 models, dominant high-energy components of the quantization error contribute relatively little to performance loss whereas a low-energy spectral tail explains most of the excess loss. A gold aware analysis further reveals that correcting only four input-conditioned tail directions removes 0.43% of tail-error energy, while increasing mean downstream accuracy beyond the BF16 reference. We attribute this energy function mismatch to loss geometry. Tail directions exhibit systematically higher directional curvature per unit error energy. Motivated by this, we introduce Harmful-Tail-Guided Sparse Compensation, HTSC, which identifies modules whose local corrections propagate beneficially into harmful tail directions and distills this repair into sparse low-rank branches with a frozen backbone. At inference, the reference model and spectral decomposition are not required. Across Qwen3 models, HTSC restores performance to near or slightly above the BF16 reference, with low additional overhead. Code is available at: https://anonymous.4open.science/r/HTSCptq-CE19/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
quantization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8NBfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how spectral structure of accumulated quantization error relates to predictive degradation. It finds that correcting the dominant error directions provides little benefit, whereas correcting the complementary tail substantially restores NLL and predictive-distribution agreement. Motivated by these findings, the paper proposes Harmful-Tail-Guided Sparse Compensation (HTSC). HTSC constructs an input-conditioned repair teacher and selects modules whose local corrections propagate harmful-tail directions and improve the final predictive distribution. The teacher's predictive distribution is then distilled into low-rank branches at the selected modules of the frozen quantized model.

Critical strengths
  • The paper provides substantive empirical evidence that spectral error magnitude can misrepresent predictive damage through the exploration of head/tail interventions, NLL and KL measurements, dimension- and energy-controlled comparisons in Section 3.
  • The experimental evaluation demonstrates that sparse compensation (through learning corrections to counteract quantization errors) restores downstream accuracy across model sizes, closely matching their BF16 references (cf. Table 3).
Critical weaknesses
  • Insufficient isolation of the harmful-tail guidance: The ablations presented in Table 4 retain the same teacher, leaving its advantage over direct BF16 distillation or next-token training unclear. Module selection is only compared against one random configuration, while other compensation baselines (acknowledged related work [1,2]) are also absent. These missing controls limit the attribution to the HTSC’s distinct components.
  • Performance-preservation and inference-overhead claims require stronger support: The empirical evidence provided supports accuracy recovery more strongly than broadly lossless, efficient quantization. While HTSC is able to recover or even exceed the accuracy of the BF16 baselines, see Table 3 and Table 5 (Appendix C), it remains worse on all reported perplexity measurements presented in Table 6 (Appendix D). Moreover, the small average-accuracy gains over the BF16 baseline lack uncertainty estimates, and evaluation covers only one model family (Qwen) and one fake-quantization format (W4A4 NVFP4). Furthermore, although the additional parameter count is small, the claimed low inference overhead is not validated by latency, throughput, or memory measurements; therefore claims such as “with low additional overhead” (line 025) and “with small additional inference cost” (line 080) are hard to empirical verify with the current experimental evaluation.

[1] Zhang et al., LQER: Low-rank quantization error reconstruction for LLMs. ICML 2024

[2] Zhang et al., QERA: an analytical framework for quantization error reconstruction. ICLR 2025

Questions for authors
  1. Under matched training conditions, how does the proposed teacher distillation compare with direct BF16 distillation or next-token training? With the teacher and compensation budget fixed, does selecting modules by local output-error magnitude or removing the harmful-tail projection from Equation (3) achieve comparable performance?

  2. How does HTSC compare against close compensation baselines such as LQER [1] and QERA [2]?

  3. Can you quantify HTSC’s additional inference cost relative to the uncompensated W4A4 NVFP4 model by reporting latency, throughput, and peak memory under matched hardware and workload settings?

Reviewer_azodOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper looks at the error that W4A4 (NVFP4) quantization leaves in the final residual stream of Qwen3 models, and uses an SVD to split it into a small "head" and the remaining "tail". On Qwen3-8B, the top 32 directions (0.78% of the hidden dimension) carry about 36% of the error energy (Fig. 1a). Removing this head barely helps and even raises NLL slightly, while removing the tail recovers almost all of the excess loss and most of the lost accuracy (Fig. 3, Tab. 1). The authors explain this by curvature: per unit of error energy, tail directions have higher curvature under the exact token Hessian (median ratio 2.35 over 192 documents, Fig. 4). An oracle that uses the gold answer shows that correcting just four tail directions lifts accuracy above BF16 (Sec. 3.3).

Building on this, HTSC constructs a teacher that removes the quantization error along up to 32 tail directions per training document, chosen with next-token NLL gradients on WikiText-2. It then distills this teacher into rank-64 branches on at most 14 modules of the frozen quantized model. On Qwen3 0.6B to 30B-A3B and eight downstream tasks, HTSC is compared with NVFP4-RTN, SmoothQuant (Xiao et al., 2023), SpinQuant (Liu et al., 2025), FlatQuant (Sun et al., 2025) and OCGQuant (Yao et al., 2026). It reaches 71.86% mean accuracy on Qwen3-8B, against 71.69% for BF16, with 0.134% extra parameters. The open question for me is whether these gains come from the harmful-tail mechanism or from low-rank distillation on in-domain text.

Critical strengths
  1. A careful diagnostic that questions a common assumption in PTQ, namely that the largest error components matter most. The finding survives an energy-matched control: repairing tail directions that hold the same 37.7% of the energy as the head gives 69.43% accuracy, against 68.23% for the head repair (Tab. 1). The curvature analysis over 192 documents, with bootstrap intervals and a correct second-order decomposition (Fig. 4), gives a plausible explanation. I find this the most valuable part of the paper; on its own it could support a narrower analysis paper.
  2. The method is easy to deploy, and the experiments are reproducible. HTSC keeps the quantized backbone frozen, adds two small low-rank operations per selected module, and leaves attention and the KV cache untouched (Sec. 4.4). It is tested on dense and MoE models, seeds and data splits are fixed, checkpoints are selected on held-out NLL without looking at downstream accuracy or test perplexity (p.14), and code is released. If the gains turn out to come from the harmful-tail teacher, practitioners could adopt the method easily.
Critical weaknesses
  1. The link between the analysis and the method's gains is not established. In practice, HTSC is low-rank distillation on WikiText-2 into a few modules, and all ablations use the same teacher (Tab. 4, p.14). What is missing is the comparison that would isolate the paper's idea: the same modules, rank and data with a plain BF16 teacher, as in standard distillation (Hinton et al., 2015), with a whole-tail teacher, and with plain LoRA (Hu et al., 2022) trained with cross-entropy. The ablations already show that a single module gives +1.30 points and 14 random modules +1.54 over RTN (Tab. 3–4), so generic recovery may explain much of the effect.

    There is also a subtler issue. For each training document, the teacher picks the tail directions whose correction most lowers the NLL of the actual next tokens (Sec. 4.1; "Observed next-token labels are used for teacher construction", p.14). Such a teacher can be better than BF16 on the training text, so "exceeding BF16" may reflect light supervised fine-tuning rather than repaired quantization damage. Running HTSC on the BF16 model itself would show which it is. Finally, all baselines are methods without trained branches (p.8). The closest low-rank compensation methods, LQER (Zhang et al., 2024) and QERA (Zhang et al., 2025), as well as GPTQ (Frantar et al., 2022), appear only in related work, and EoRA (Liu et al., 2024) is not discussed. Without these comparisons I cannot tell whether the harmful-tail teacher, which is the paper's contribution, adds anything.

  2. The headline claims go beyond the paper's own data. The title says "lossless" and p.2 says that HTSC "maintains near-BF16 perplexity". Yet on 4B, 8B and 30B-A3B it closes only about 19–32% of the WikiText-2 perplexity gap and between −9%-9\% and 9%9\% of the C4 gap, and on 8B its C4 perplexity is even slightly worse than RTN (Tab. 6). The abstract calls the tail "low-energy", yet it holds about 62–64% of the error energy. The margins above BF16 are +0.17 (8B) and +0.01 (30B) points, from a single training run without confidence intervals.

    The abstract also reports that the oracle lifts accuracy beyond BF16. This oracle, however, selects directions using the gradient of the gold answer for the very question being evaluated. At k=32k = 32 it reaches 80.48% against 69.85% for BF16, which looks like steering toward the answer rather than repairing damage. The paper does call it an oracle (p.6), but it still concludes that task-harmful error is concentrated in a few tail directions, without a random-direction or head-direction control. Lastly, "low additional overhead" is supported only by parameter counts, with no latency or throughput numbers. Since the title-level claim does not hold on the paper's own tables, I weigh this heavily.

References:

  • Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv:2210.17323, 2022.
  • Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv:1503.02531, 2015.
  • Hu, E. J., et al. LoRA: Low-rank adaptation of large language models. ICLR, 2022.
  • Liu, S.-Y., et al. EoRA: Fine-tuning-free compensation for compressed LLM with eigenspace low-rank approximation. arXiv:2410.21271, 2024.
  • Zhang, C., Cheng, J., Constantinides, G. A., and Zhao, Y. LQER: Low-rank quantization error reconstruction for LLMs. ICML, 2024.
  • Zhang, C., Wong, J. T. H., Xiao, C., Constantinides, G. A., and Zhao, Y. QERA: An analytical framework for quantization error reconstruction. ICLR, 2025.
Questions for authors
  1. Requested experiment: with the same 14 modules, rank 64 and WikiText-2 data, what do a plain BF16 teacher, a whole-tail teacher and LoRA (Hu et al., 2022) with cross-entropy achieve, and what happens if HTSC is applied to the BF16 model itself? I would also like to see LQER (Zhang et al., 2024), QERA (Zhang et al., 2025) and EoRA (Liu et al., 2024) at an equal parameter budget. These are the results most likely to change my assessment.
  2. Clarification: why does downstream accuracy fully recover while WikiText-2 and C4 perplexity barely improve? What are the perplexity and accuracy of the harmful-tail teacher itself, on the training text and on held-out data?
  3. Requested experiment: what does the gold-gradient oracle give when restricted to head directions or to random subspaces, and what does a random perturbation of the BF16 residual with matched energy do?
  4. Requested experiment: could you report results over at least three training seeds (currently a single run with base seed 20260923, p.14), with confidence intervals for Tab. 3–5? Some numbers suggest high evaluation variance: RTN loses 14 HellaSwag points on 30B-A3B (77.66 vs 63.70), SpinQuant (Liu et al., 2025) and FlatQuant (Sun et al., 2025) lose 12.5 and 13.3, and on 4B, OCGQuant (Yao et al., 2026) beats BF16 on GSM8K by 10.46 points (Tab. 5).
  5. Requested experiment: what are the latency and throughput of HTSC relative to the NVFP4 backbone at typical serving batch sizes?
  6. Suggestion: only Qwen3 and W4A4 NVFP4 are tested. Results on a second model family and on other formats (MXFP4, INT4, W4A16) would show whether the harmful-tail effect is general.

References:

  • Hu, E. J., et al. LoRA: Low-rank adaptation of large language models. ICLR, 2022.
  • Liu, S.-Y., et al. EoRA: Fine-tuning-free compensation for compressed LLM with eigenspace low-rank approximation. arXiv:2410.21271, 2024.
  • Liu, Z., et al. SpinQuant: LLM quantization with learned rotations. ICLR, 2025.
  • Sun, Y., et al. FlatQuant: Flatness matters for LLM quantization. ICML, 2025.
  • Yao, Y., et al. OCGQuant: Outlier-companion grouping for NVFP4 quantization. arXiv:2609.00066, 2026.
  • Zhang, C., Cheng, J., Constantinides, G. A., and Zhao, Y. LQER: Low-rank quantization error reconstruction for LLMs. ICML, 2024.
  • Zhang, C., Wong, J. T. H., Xiao, C., Constantinides, G. A., and Zhao, Y. QERA: An analytical framework for quantization error reconstruction. ICLR, 2025.
Reviewer_Q4djOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new and interesting way of understanding the impact of quantization onto LLM outputs. Specifically, it takes the quantized residual state for Qwen 3 (before the last norm + LM head). For each input, the authors split the quantization error into the top 32 singular directions (what they call the "head") and the orthogonal complement (the "tail). The authors first show that subtracting the quant error head from the residual does not improve NLL, but that subtracting the tail recovers the entire gap to BF16. They attribute this to loss curvature. In addition, the authors provide evidence that quantization damage sits in a few directions of the tail. They propose a method called HTSC that turns this into a practical fix, which works by training adapters by distillation to correct outputs towards an unquantized teacher. This works well on small-scale experiments.

Critical strengths

Pros:

  • The paper gives a very interesting (in my view) quant error diagnostic
  • Second-order analysis is well-executed and interesting
  • Covers a good mix of models

Overall, I found the analytical part of the paper interesting. I think it shows well that "Euclidean" energy in residual space is not the right metric to measure quantization quality by. The method half is a KD-trained adapter whose usefulness is a bit confounded. I think there's value in the paper and could raise my score if my questions are well addressed.

Critical weaknesses

Cons:

  • Broadly, my main concern is that alternative explanations for the phenomenona in question are not considered very clearly (see my questions below)
  • The "low-energy tail" framing of the results is misleading since the tail is not low-energy (has 60-something percent of the error)
  • The paper is quite dense so might be inaccessible to a wider audience
Questions for authors
  • Q1: Wouldn't an alternative explanation for the "harmless head" be connected to massive-activation channels? Systematic error along some large fixed channels and/or along the "radial" direction of h could maybe explain some of the stuff you're seeing. Specifically, recall that RMSNorm makes the readout (nearly) invariant to radial error, so the error will have small gradient and small curvature by construction. In this setting, the "energy–function mismatch" would be just about measuring energy before the normalization, not a property of the error spectrum. Can you clarify this? (For instance, you could report the overlap between the head subspace and the massive-activation channels and the per-token direction of h^FP. There's plenty of results that could add clarity here.)

  • Q2: I am quite confused about the low-energy tail claims. For instance, the abstract (and the paper, several times) says stuff like "dominant high-energy components contribute little whereas a low-energy spectral tail explains most of the excess loss" -- but this is per-direction, not in aggregate, right? The aggregate energy of the tail is quite large. The authors seem to confuse themselves on this point: for instance, the conclusion says: "despite its lower energy, the spectral head is less damaging" which seems to have the "inequality" backwards?

  • Q3: Doesn't the "gold-aware" oracle just measure label leakage? (E.g. exceeding BF16.)

  • Q4: Isn't the comparison between HTSC and baselines a bit unfair? Your method trains parameters whereas the baselines are PTQ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.