acceptodds
Withdrawn from ICLR 2027

REGAIN: Restoring Token Distributions in Low-Bit Language Models via Truncated-Support Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Low-bit weight quantization substantially mitigates the memory footprint and latency bottlenecks of large language models, yet aggressive compression regimes incur non-negligible capability degradation. While low-rank quantization error compensation effectively recovers coarse benchmark accuracy with minimal parameter overhead, it fails to restore the nuanced output distributions of the full-precision teacher. Specifically, even when Top-1 predictions appear intact, quantization systematically distorts the probability margins and relative rankings among competing tokens, precipitating severe compounding errors in sequential and structured generation tasks. To resolve this discrepancy, we propose Regain, a parameter-efficient recovery framework based on truncated-support distillation. Instead of optimizing over the entire vocabulary or naively mirroring teacher targets, Regain dynamically constructs a compact token support from the union of the Top- candidates from both the full-precision teacher and the quantized student. This design preserves teacher-favored modes while directly penalizing quantization-induced spurious competitors, using an auxiliary Ghost token to aggregate the residual probability mass for bounded computation. With the low-bit backbone frozen, Regain exclusively trains low-rank adapters under this compact distributional objective. Extensive evaluations across Qwen3 architectures from 8B to 32B, multiple post-training quantization schemes including GPTQ and QuIP#, and aggressive 2 to 4-bit regimes demonstrate that Regain substantially improves distributional fidelity and downstream performance across question answering, tool invocation, and structured reasoning, setting a new Pareto frontier for low-rank recovery of ultra-low-bit language models.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Low-Bit Quantization, Post-Quantization Recovery, Knowledge Distillation, Large Language Models, Parameter-Efficient Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BAz8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes REGAIN, a post-quantization recovery framework that restores token-level output distributions of low-bit LLMs through LoRA-based truncated-support distillation. It constructs a compact supervision set by combining the Top-k tokens from the full-precision teacher and quantized student, with a Ghost token preserving residual probability mass. Experiments on Qwen3 models (8B–32B) under GPTQ and QuIP# quantization demonstrate improved accuracy over RILQ, particularly on reasoning and structured generation tasks, while reducing distillation-cache storage.

Critical strengths

S1. Effective capability recovery. REGAIN achieves notable improvements over RILQ, particularly under aggressive 2-bit quantization. The gains on MATH500, BFCL V3, and ACPBench suggest that output-distribution alignment can benefit low-bit model recovery beyond conventional feature-level compensation.

S2. Efficient distillation supervision. The compact support substantially reduces offline distillation storage (362× compared with Full-KL caching) while maintaining comparable recovery accuracy, demonstrating practical benefits for the recovery training process.

Critical weaknesses

W1. Missing real-world deployment evaluation. The most critical concern is the lack of inference-time efficiency evaluation. REGAIN introduces additional high-precision LoRA computation on top of frozen low-bit backbones, which cannot be trivially merged into existing GPTQ/QuIP# quantized representations. However, the paper reports neither actual inference latency nor total deployment memory overhead, and does not clarify the underlying inference kernels. Since the primary motivation for low-bit quantization is efficient deployment, the absence of such evidence fundamentally weakens the practical significance of the proposed method. Training throughput and distillation-cache savings cannot substitute for real inference measurements.

W2. Limited independent contribution of the proposed support construction. The main methodological novelty lies in the union of teacher and student Top-k candidates. However, Table 4 shows only a marginal improvement over teacher-only Top-k distillation (63.10 vs. 63.03 in zero-shot accuracy), despite the substantial gains over RILQ. This raises concerns about whether the proposed support construction provides meaningful benefits beyond existing output-distribution distillation techniques.

Questions for authors

Q1. Real deployment efficiency. Are the reported experiments conducted using fake quantization or genuinely packed low-bit weights? What are the LoRA precision, rank, and additional memory requirements? Please provide real GPU inference benchmarks comparing the original quantized models, RILQ, and REGAIN, including peak memory usage, prefill/decode latency, and throughput. In particular, how are LoRA computations integrated with GPTQ/QuIP# inference kernels? Convincing evidence that REGAIN preserves the deployment efficiency of low-bit quantization would materially affect my assessment.

Q2. Effectiveness of Union Top-k. Given the marginal gains over teacher-only Top-k distillation in Table 4, can the authors provide stronger evidence that quantization-aware support construction is necessary? In particular, comparisons against teacher-only Top-k + Ghost and Full-KL distillation on challenging generative benchmarks would help establish the independent contribution of the proposed design.

Reviewer_TV4VOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies low-rank quantization error compensation (LQEC), where LoRA adapters are trained to recover the performance of low-bit quantized LLMs. REGAIN uses sparse forward KL distillation over the union of the teacher's and initial quantized model's top-k tokens, together with a Ghost token for the remaining probability mass. The targets can be cached offline, substantially reducing storage compared with dense targets. Experiments cover Qwen3-32B at 2–4 bits and Qwen3-8B/14B at 2–3 bits, using GPTQ and QuIP#, including zero-shot benchmarks, MATH500, BFCL V3, and ACPBench.

Critical strengths
  1. The offline sparse targets are a practical contribution. They remove the need to keep the BF16 teacher in GPU memory during LoRA training and reduce the reported cache size from 89.44 GB to 247 MB. The controlled ablation also shows performance close to full-vocabulary KL.
  2. The evaluation considers more than standard multiple-choice accuracy. MATH500, BFCL V3, ACPBench, and token-distribution metrics offer a useful perspective on recovery quality, and the author illustrates why preserving the selected answer alone may not capture distribution drift.
Critical weaknesses
  1. The contribution of the union support could be made a little clearer. Table 4 offers useful comparisons, although the differences between several sparse-support choices seem relatively modest. A brief discussion of when the union is especially helpful would strengthen the presentation.
  2. The link between improved token distributions and downstream performance could be explained more directly. The paper evaluates both aspects, and some additional interpretation of how they relate would make the results easier to appreciate.
  3. A little more context on the experimental setup would be useful for interpreting the comparisons. In particular, a concise summary of the calibration data and how training budgets were matched across methods would improve clarity.
Questions for authors
  1. Could the authors briefly elaborate on the motivation for using the union support rather than teacher-only top-k support, especially given the relatively small differences in Table 4?
  2. Among the distribution-level metrics, is there one that the authors find particularly informative for understanding the improvements on MATH500, BFCL V3, or ACPBench?
  3. Could the authors briefly clarify how the calibration data and training budgets were selected for the main comparisons? A short summary would be helpful.
Reviewer_1JFaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper the authors study the effect of truncated knowledge distillation for low-rank quantization error recovery techniques. The paper introduces REGAIN, a technique that combines the Top-K teacher tokens with the Top-K student tokens and an additional ghost token that accounts for the remaining probability mass. Their experimental evaluation shows that this technique outperforms other truncated distillation techniques (eg just Top-K from the teacher) and is on par with full vocabulary distillation. They further show that compared to offline Full-KL, truncated Top-K significantly reduce the cache size. Compared to online Full-KL they slightly improve the training throughput.

Critical strengths
  • The paper studies a very interesting and relevant tope, how do token distributions change with quantization, how does it affect downstream tasks beyond top 1 accuracy and how does (truncated) knowledge distillation training interact with it.
  • I find the idea of adding a ghost token/class to account for the remaining missing probability mass interesting and novel. Also the ablation studies clearly show the benefit of this compared to other Top-K approaches.
  • The paper is very well written and easy to follow, compliments to the authors!
Critical weaknesses
  • There are some claims in the paper that are not fully supported by the experimental evaluation:
    • L92: “Full-vocabulary distillation provides dense supervision over token distributions, but capacity-constrained recovery calls for a more targeted and compact objective.” — Based on their own ablation in Table 4, there seems no evidence that full KL hurts compared to truncated KL.
    • L128: “We diagnose that conventional low-rank error compensation methods fail to prevent severe token-level probability distribution distortions under ultra-low bit-widths. We demonstrate that these residual distortions misalign local decision boundaries, substantially degrading performance on decision-critical tasks such as multi-step reasoning and structured generation.” — Unfortunately there is not much diagnose/analysis in the paper, I would have appreciated a much deeper analysis than presented here (e.g. fig 1). Also a lot of what they claim to demonstrate is not supported by the evidence in section 4. There is some partial evidence in figure 3, but that also misses comparing to full KL.
    • L295: “Standard full-vocabulary distillation disperses gradient updates across hundreds of thousands of low-probability tail coordinates, which quickly saturates the low-rank subspace of LoRA.” — similar to first point.
  • The paper could have a broader scope and impact. It is unclear to me why the authors limit their method and investigation specifically to low-rank quantization recovery and do not study other common cases in quantization that use knowledge distillation (e.g. optimize transformations, e2e QAT, post PTQ fine-tuning of remaining FP parameters [gamma, quant scales etc]).
  • The authors should compare more widely to literature. While the comparison to RILQ looks promising (cf table 1&2), REGAIN does not clearly outperform vanilla full-vocabulary knowledge distillation (cf table 4, all metrics are within std). Also additional ablation studies on the effect of number of training tokens, lora rank and other key parameters would strengthen the paper.
  • While the the research question is interesting, the algorithmic novelty is limited and most components are from existing literature (TopK distillation, ghost-token, lora-based error recovery).
Questions for authors
  • Figure 1: why use GPTQ in these examples? This is not using e2e task loss so it is not directly linked to the main claims.
  • How would REGAIN interact with JSD or backward KL?
  • Support budget numbers: for which model is this study? From the regime I assume Qwen3-8B, however I do not see that k=32 result (61.68) matching any of the other tables.
  • Should S_T not be adapted over time as tokens could pop up that are not in the recovery set? What is the effect of keeping it frozen?
  • Can you please elaborate on your exact experimental setting, it seems even not stated in the appendix. The paper only say they do the same as RILQ, but not mention key parameters such as data sets and num tokens etc.
  • How would REGAIN compared to JSD training? This seems to be used more in literature and tend to outperform forward KL only.
  • What is Top-m_t in table 4?

Minor suggestion:

  • 219: typo, should be p_M_T
  • Minor: give S_g u {g} a symbol (eg \hat{S}_g), then eq. 4 can be more precise

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.