- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to optimized a quantized LLM with its quantization metadata and low-bit weight, without dequant it into high-precision form. The paper noticed that a standard zero-order optimization on QPS is enough for low-bit quantization. Experiments on 4-bit checkpoints and related ablation show that the proposed method do not damage the frozen low-bit space while improve model's performance.
Critical strengths
- Performance: The experiments and ablations supports that a quantized checkpoint can be adapted under a fixed forward budget without adding modules or maintaining a fully dequantized trainable copy.
- Theoretical insights: A nonzero aggregate gradient of an updateable untied output head implies a nonzero projected gradient on QPS and a strict local descent direction, which is novel to me.
Critical weaknesses
- Optimizing an already quantized model is not shown to be practical. Quantization is a way to store and serve a model. I cannot come up with some scenarios that a user or system that must adapt the model only after it has been quantized.
Questions for authors
- What learning rate are chosen for mentioned experiments? I'm quite curious about what if a learning rate cannot be represented with a given bit. Will that introduce further noise on gradients?
3 comments
Thank you for the detailed response. The deployment example and explanation on learning rate resolved my questions. I now think that the paper is a complete research work.
I am keeping my rating at 2, but I nevertheless consider acceptance a reasonable outcome. This year’s four-point scale is too coarse to convey that nuance, especially that it is said 2/3 of submissions are rated 2. I would appreciate it if the AC would take this clarification into account when weighing my review.
Official Comment by Authors
Thank you for your thoughtful follow-up. We are glad that our discussion of the practical value and our explanation of learning rate resolved all of your concerns. We especially appreciate your clarifying that you now view the paper as a complete piece of research and consider acceptance a reasonable outcome. Thank you for asking the AC to consider this perspective alongside your numerical rating.
Rebuttal by Authors
W1: Practical scenarios for adapting an already quantized model
Thank you for asking when adapting a model after quantization would be useful in practice.
Adapting an already quantized model is useful when a device needs quantization to fit the model in memory and the data needed for adaptation arrive only after deployment. At that point, the device already holds a compressed checkpoint, and learning from the new data locally must fit the same memory budget.
Consider a phone that ships with a quantized assistant. After release, a user repeatedly corrects which reminders the assistant assigns to their work and personal lists. Those corrections now provide training examples of that user’s preferences. Starting from the full-precision model would require fine-tuning it on the new examples, quantizing the result, and delivering a new checkpoint to the phone. If the corrections must stay on the phone, training must run locally, where only the quantized checkpoint is available and memory is too limited to fine-tune a full-precision copy. Adapting the installed checkpoint instead lets the assistant learn from the corrections while reusing the low-bit weights already on the device.
Section 2.1 formalizes this setting, and Appendix C.1 describes how our experiments start directly from existing quantized checkpoints. QPZO updates only their retained floating-point tensors, keeping the packed weights and quantization metadata fixed. On Qwen-GPTQ/SST-2, this raises accuracy from 90.94% to 94.95% (Table 1) with 9.32 GiB of peak training GPU memory, compared with 17.75 GiB for MeZO on the fully dequantized model (Figure 2). Figure 2 shows lower peak training memory for QPZO across all six model–quantizer settings.
Q1: Learning rates and the backbone bit width
Thank you for raising this question. The specific source of gradient noise you describe is absent in QPZO because the learning rate η is a floating-point scalar and is never quantized to the backbone’s bit width.
Section 2.1 restricts updates to the floating-point tensors retained in QPS and keeps the packed low-bit weights and quantization metadata fixed. In §2.2, Eq. (4) computes the directional estimate Dt=(ℓt+−ℓt−)/(2ε) from two forward losses. The learning rate first appears in Eq. (5), after this estimate has been computed. As shown in Algorithm 1, η scales the update applied directly to the floating-point QPS tensors.
Thus, in all QPZO experiments, including those reported in Table 1, the learning rate remains in floating point regardless of the bit width used to store the backbone weights. More generally, whether η is representable on the backbone’s weight grid has no bearing on the zeroth-order gradient estimate. The table lists the learning rates and perturbation scales for this model and task.