- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CoZO, a memory-efficient ZO fine-tuning method for quantized LLMs. It optimizes fine-grained dequantization scales and zero-points in a continuous space, using alternating optimization and adaptive grouping to balance expressiveness and estimation variance. Theoretical analysis and experiments support its effectiveness.
Critical strengths
The paper is clearly written and well-structured, with its main claims supported by theoretical analysis or experiments. The topic is well-motivated and relevant to the community, as memory-efficient LLM fine-tuning can make LLM adaptation more accessible to users with limited computational resources.
Critical weaknesses
- The novelty is somewhat limited. Compared with QZO, the main difference is the introduction of learnable zero-points, which are common in quantization literature. Alternating optimization and adaptive grouping seem more like add-ons rather than being closely tied to the core motivation.
- The evaluation relies mainly on OPT and LLaMA-2, which are not sufficiently representative of modern LLM architectures. A similar concern applies to the selection of downstream datasets, which are relatively outdated and limited in diversity.
- The paper lacks comparisons with representative PTQ methods, such as OmniQuant, SpinQuant, and FlatQuant. These methods could also serve as meaningful baselines when applied to fine-tuned models and evaluated on downstream tasks. Such comparisons would help better establish the advantages of the proposed method.
Questions for authors
- Have you repeated your main experiments with multiple random seeds? Unlike other research topics, ZO optimization is sensitive to random seeds, leading to noticeable performance variations. Although Table 10 reports results across different seeds, the evaluation is still somewhat limited, especially considering that the improvements from alternating optimization and adaptive grouping are relatively modest.
- From my understanding, CoZO uses the loss difference as the signal for adaptive grouping. However, the theoretical variance bound depends on the gradient norm, which does not necessarily mean that a single loss difference reliably reflects estimation variance. Moreover, different perturbation directions may produce very different loss differences, especially when using only one direction per update as in MeZO. Could you clarify the theoretical justification and reliability of this strategy?
7 comments
W1:The novelty is somewhat limited. Compared with QZO, the main difference is the introduction of learnable zero-points, which are common in quantization literature. Alternating optimization and adaptive grouping seem more like add-ons rather than being closely tied to the core motivation.
We thank the reviewer for the thoughtful comments on the novelty of our method. We agree that learnable zero-points are not a new concept in themselves. However, the core contribution of this work is not simply the introduction of zero-points, but rather a set of interconnected optimization designs in continuous space that address the limitations of the optimization space in zeroth-order fine-tuning of quantized models. We would like to further elaborate on the following aspects.
-
Motivation for Introducing Zero-Points.
Existing zeroth-order fine-tuning methods that optimize only scales are significantly limited in the directions along which they can adjust model weights. As analyzed in Appendix C.9, zero-points provide additional translation directions that cannot be achieved through scale-only optimization. Our introduction of zero-points is therefore not simply intended to increase the number of trainable parameters, but is motivated by an analysis of the optimization structure of the continuous mapping space.
-
Design Motivation for Alternating Optimization and Adaptive Grouping.
However, while fine-grained continuous parameterization enhances expressiveness, it also increases the search dimension of zeroth-order gradient estimation. As the reviewer pointed out in Q1, zeroth-order optimization inherently suffers from estimation instability, which becomes more pronounced as the optimization dimension increases. To address this issue, we introduce alternating optimization to reduce the search dimension of each gradient estimation step, together with adaptive grouping to dynamically balance the expressiveness of the search space and estimation stability throughout training. Therefore, these two designs are not merely additional modules, but interconnected solutions to the dimensionality and stability challenges arising in zeroth-order optimization within fine-grained continuous spaces.
W2:The evaluation relies mainly on OPT and LLaMA-2, which are not sufficiently representative of modern LLM architectures. A similar concern applies to the selection of downstream datasets, which are relatively outdated and limited in diversity.
We agree with the reviewer's suggestion to extend our evaluation to more advanced models and a broader range of downstream tasks. To this end, we conducted additional experiments on Qwen3-8B and introduced the GSM8K mathematical reasoning benchmark, using both Strict EM and Flexible EM metrics to evaluate the applicability of CoZO across newer model architectures and different task types. The corresponding experimental results have been added to Appendix C.7 of the revised manuscript.
Table: Task Evaluation Results on Qwen3-8B
| SST2 | CB | WIC | WSC | GSM8K strict EM | GSM8K flexible EM | |
|---|---|---|---|---|---|---|
| Zero-Shot | 51.49 | 51.79 | 52.04 | 55.17 | 69.98 | 72.71 |
| QZO | 78.32 | 69.64 | 60.66 | 57.69 | 67.85 | 71.27 |
| QuZO | 80.39 | 80.36 | 55.17 | 61.54 | 54.06 | 57.32 |
| CoZO | 81.77 | 83.93 | 60.82 | 63.46 | 74.45 | 75.73 |
We sincerely thank the reviewer for the time and effort devoted to reviewing our paper. Below, we provide point-by-point responses to all the weaknesses and questions raised. We have also revised the original manuscript accordingly, with all changes highlighted in blue.