- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes REGAIN, a post-quantization recovery framework that restores token-level output distributions of low-bit LLMs through LoRA-based truncated-support distillation. It constructs a compact supervision set by combining the Top-k tokens from the full-precision teacher and quantized student, with a Ghost token preserving residual probability mass. Experiments on Qwen3 models (8B–32B) under GPTQ and QuIP# quantization demonstrate improved accuracy over RILQ, particularly on reasoning and structured generation tasks, while reducing distillation-cache storage.
Critical strengths
S1. Effective capability recovery. REGAIN achieves notable improvements over RILQ, particularly under aggressive 2-bit quantization. The gains on MATH500, BFCL V3, and ACPBench suggest that output-distribution alignment can benefit low-bit model recovery beyond conventional feature-level compensation.
S2. Efficient distillation supervision. The compact support substantially reduces offline distillation storage (362× compared with Full-KL caching) while maintaining comparable recovery accuracy, demonstrating practical benefits for the recovery training process.
Critical weaknesses
W1. Missing real-world deployment evaluation. The most critical concern is the lack of inference-time efficiency evaluation. REGAIN introduces additional high-precision LoRA computation on top of frozen low-bit backbones, which cannot be trivially merged into existing GPTQ/QuIP# quantized representations. However, the paper reports neither actual inference latency nor total deployment memory overhead, and does not clarify the underlying inference kernels. Since the primary motivation for low-bit quantization is efficient deployment, the absence of such evidence fundamentally weakens the practical significance of the proposed method. Training throughput and distillation-cache savings cannot substitute for real inference measurements.
W2. Limited independent contribution of the proposed support construction. The main methodological novelty lies in the union of teacher and student Top-k candidates. However, Table 4 shows only a marginal improvement over teacher-only Top-k distillation (63.10 vs. 63.03 in zero-shot accuracy), despite the substantial gains over RILQ. This raises concerns about whether the proposed support construction provides meaningful benefits beyond existing output-distribution distillation techniques.
Questions for authors
Q1. Real deployment efficiency. Are the reported experiments conducted using fake quantization or genuinely packed low-bit weights? What are the LoRA precision, rank, and additional memory requirements? Please provide real GPU inference benchmarks comparing the original quantized models, RILQ, and REGAIN, including peak memory usage, prefill/decode latency, and throughput. In particular, how are LoRA computations integrated with GPTQ/QuIP# inference kernels? Convincing evidence that REGAIN preserves the deployment efficiency of low-bit quantization would materially affect my assessment.
Q2. Effectiveness of Union Top-k. Given the marginal gains over teacher-only Top-k distillation in Table 4, can the authors provide stronger evidence that quantization-aware support construction is necessary? In particular, comparisons against teacher-only Top-k + Ghost and Full-KL distillation on challenging generative benchmarks would help establish the independent contribution of the proposed design.