- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
- The authors argue that when structural pruning is used to reduce model dimensions (such as the number of channels) and is subsequently followed by low-bit quantization—a common practice for real-world deployment—the model’s performance often degrades significantly. This occurs because the structural configuration selected during the pruning phase may be poorly suited to the quantization process, leading to a "misalignment" where channels that appeared optimal during pruning instead become sources of substantial error once quantized. To address this, the authors propose Q-PAT, an extension of the existing PAT (Pruning-Aware Training) framework designed to be more compatible with low-bit deployment. The core ideas behind Q-PAT are: 1) calculating a quantization sensitivity metric for each channel, and 2) incorporating this sensitivity information specifically into the learning process for the structural mask. This approach decouples representation learning from structural selection, using the INT4 sensitivity data as an auxiliary signal to guide the selection of the model structure.
- To validate the effectiveness of their proposed method, the authors conducted experiments using Gemma-2B, Gemma-7B, and Llama2-7B as backbone models, trained on the LaMini dataset. The method was evaluated across 14 benchmarks, and the authors further confirmed that its advantages persist through ablation studies and variations in INT4 group sizes.
Critical strengths
- The analysis and definition of the problem—that channels or structures selected via structural pruning can lead to misalignment issues during subsequent low-bit quantization—are clear and compelling.
- By incorporating an INT4 sensitivity signal as a regularization term in the loss function—using a stop-gradient operation—the method ensures that mask optimization accounts for low-bit sensitivity without interfering with the main task.
- The approach demonstrated superior performance over PAT, based on the average results across 14 tasks.
- This relative advantage was maintained even when quantization settings, such as group size, were varied.
Critical weaknesses
- The sensitivity metric proposed by the authors serves as a surrogate rather than a direct measure of end-to-end performance degradation following deployment; consequently, it may not always align with the actual deterioration in task performance. This issue requires further attention.
- The ablation study results lack statistical significance.
- Although the proposed method performs better on average across 14 tasks, it does not consistently outperform the alternatives; therefore, further analysis of this aspect is necessary.
- A comparison with the latest research on efficient LoRA-based methods is necessary.
Questions for authors
- In cases where the proposed method underperforms compared to PAT, could the authors provide further analysis to determine whether this is due to minimal misalignment—as the authors hypothesized—or if there are other underlying reasons?
- The authors refer to sensitivity as a "surrogate"; could they provide additional explanation or evidence regarding the extent to which this serves as a reliable surrogate?
- Could the authors also provide an assessment of the statistical significance for the ablation study results?
- Could you provide a comparison with the latest relevant research, at least for some of the tasks?
Comment 1
Response. The reviewer asks why Q-PAT can lose on individual tasks or in quantization retention. The local proxy supplies an optimization preference, not a guarantee of improvement for every task; the available results do not identify a causal explanation for each loss.
Evidence. Table 2 shows varying gains in final INT4 average, and the BF16-to-INT4 drop is not uniformly smaller. In the main Gemma-7B run, the remaining-13 INT4 means are nearly equal and the aggregate advantage is concentrated on MMLU. Possible interactions with task importance or model redundancy remain hypotheses rather than demonstrated causes.
Revision place. Sections 4.3–4.4 and Tables 2–3 (pp. 7–8) separate model-level averages, quantization drops, and the MMLU contribution.
Comment 2
Response. The reviewer asks whether Sj predicts downstream degradation. It measures normalized INT4 fake-quantization error of low-rank factors, and the deployed-model analysis shows weak channel-level agreement; we do not infer task-accuracy prediction from it.
Evidence. The Section 4.7 comparison maps training-time Sj from 3,072 channels to the 2,304 physically retained channels of the seed-42 Q-PAT model. The BF16 and GPTQ W4A16 models are evaluated on 128 samples disjoint from the 128 calibration samples. Spearman +0.07 and Pearson +0.06 measure association with channel-wise representation distortion. Figure 3 separately describes association with the learned mask.
Revision place. Section 3.3 (p. 5) defines the proxy; Section 4.7, Figure 3 and Table 5 (pp. 8–9) distinguish learned-mask association from the deployed-model check.
Comment 3
Response. The reviewer requests statistical support for the QSM ablation. We added raw scores for three matched seeds and report paired effects; the final INT4 14-task average gain over w/o QSM is positive in all three.
Evidence. Table 7 reports 1.12 ± 0.18 percentage points (mean ± sample SD). An exploratory two-sided 95% t interval computed from the four-decimal Table 6 entries is approximately 0.67–1.57 points; with n=3, it relies heavily on distributional assumptions and is not presented as a general significance finding. The independent Table 4 single-run difference remains 0.061 points.
Revision place. Section 4.8 and Tables 6–7 (pp. 9–10) add the raw matched data and paired summary; Section 4.5 (p. 8) reports the distinct single-run component control.
Comment 4
Response. The reviewer requests comparison with newer LoRA-based methods. The present experiments compare methods that produce structured pruning decisions under the stated task and precision protocol; they do not establish a matched-budget performance ranking against recent adapter-focused methods.
Evidence. Sections 2.2–2.3 discuss parameter-efficient and low-bit adaptation, while Section 4.2 lists PAT, LLM-Pruner, and SliceGPT as evaluated baselines. Backbone, structural budget, precision, and task-set differences prevent a defensible numerical comparison from published scores alone.
Revision place. The method positioning is stated in Sections 2.2–2.3 (p. 3), and the evaluated baseline scope is explicit in Section 4.2 (p. 6). No new LoRA baseline experiment is claimed in the revised paper.