- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Compression Hypercube, a unified framework for analyzing LLM compression across five dimensions: compression method, compression level, recovery strategy, initialization state, and data distribution. The authors provide theoretical analyses based on compression-induced perturbations and loss-landscape curvature, and conduct experiments across multiple LLM families to investigate compression sensitivity, recovery effectiveness, and configuration selection.
Critical strengths
-
The paper addresses a practically relevant problem by systematically examining the interactions among different compression methods, recovery strategies, and model characteristics.
-
The experiments cover multiple LLM families, compression techniques, and recovery settings, providing a broad empirical comparison of compression behaviors.
-
The paper combines theoretical analysis with empirical observations and explores a lightweight predictor for estimating compression sensitivity without performing trial compression.
Critical weaknesses
-
Limited novel insights. Although the paper addresses an interesting problem, many findings, such as the robustness of 8-bit quantization and the severe degradation caused by aggressive pruning, are already well established. The proposed Compression Hypercube organizes existing techniques into a unified framework, but it remains unclear what fundamentally new or generalizable insights are obtained beyond prior empirical studies.
-
Limited practical guidance from the theoretical analysis and configuration predictor. Theorems 3.1–3.2 mainly rely on standard second-order perturbation analysis, without providing clear guidance for selecting compression methods. More importantly, the proposed configuration predictor does not include the compression method as an input feature. As acknowledged in Appendix F.5, its within-model ranking depends entirely on compression level, since structural features remain constant across configurations. Consequently, it cannot distinguish Wanda, AWQ, and depth pruning at the same nominal compression level. This substantially limits its practical value for compression strategy selection.
-
Insufficient experimental rigor in analyzing recovery strategies. The observed reversal between KD and SFT is potentially interesting: KD outperforms SFT under Wanda-50% pruning (Table 5), whereas SFT outperforms default forward-KL KD under Depth-50% pruning (Table 15). However, these experiments use different recovery objectives and training configurations, making it difficult to attribute the reversal to the compression method itself. A comparison using consistent recovery objectives, training budgets, and hyperparameter tuning protocols is needed to support this conclusion.
-
Incorrect table numbering. The numbering of Tables 3 and 4, as well as Tables 5 and 6, appears to be reversed. The authors should correct the table numbering to ensure consistency with their order of appearance.
Questions for authors
See weaknesses
3 comments
Please review our responses
Dear Reviewer 1ePS,
As we approach the end of the discussion phase, we sincerely request you to review our responses. We have made every effort to carefully address all your concerns and provide detailed clarifications.
If you find our responses satisfactory, we would be grateful if you could consider reassessing your evaluation of our paper. We remain fully committed to addressing any additional questions or concerns you may have.
Thank you for your time, thoughtful feedback, and consideration.
Thanks
Thank you for the detailed response and additional experiments. The matched SFT/KD experiments help address my concerns about experimental consistency.
However, my main concerns remain: (1) the observed compression–recovery interaction is interesting, but the results remain largely descriptive, without explaining why different compression methods favor different recovery objectives or establishing a consistent pattern across evaluation metrics; and (2) the predictor's practical value remains unclear, particularly given the substantial improvement from adding method-specific features.
While the rebuttal improves the empirical rigor, these concerns still limit the strength of the paper's contributions.
Response to Reviewer 1ePS Comments
We thank the reviewer for the careful reading. We address each concern with new experiments.
Limited novel insights
We respectfully disagree that the contribution is organizational. Three contributions are demonstrably new:
Theory and configuration predictor
On the theory: we agree Theorems 3.1–3.2 use standard perturbation techniques. The novelty is not the proof technique but the predictive power it creates: Smin and λmax rank model families by compression robustness before any compression (τ=0.917 across 5 families). No prior work connects pre-compression spectral diagnostics to post-compression quality at this level of cross-family validation.
Smin ordering matches actual degradation. On the predictor: the reviewer is correct that structural features are constant within a model. But our ablation (leave-one-model-out CV, 18 configs, 5 families) shows the predictor's value comes from a different source entirely:
Compression-only (A) has τ=0.000 --- no better than random. It cannot distinguish Wanda, AWQ, and depth pruning at the same compression level, exactly as the reviewer notes. But that is the point: the method × recovery interaction carries 55.4% of signal (R2=0.832), while compression level contributes 0.0%. Structural features (Smin, λmax) serve a different role --- cross-model generalization, predicting how an unseen family responds to compression --- while method × recovery features drive within-model selection.
KD/SFT reversal not isolated
This is a fair concern. We addressed it with a fully matched experiment where every hyperparameter is held constant, only the loss function differs (CE for SFT vs forward-KL for KD). LoRA r=8, α=16, same data subset (10M tokens SlimPajama), cosine LR 2e-4, BF16. No confounds remain.
LLaMA-3.1-8B depth-50% (baseline: PPL 10.83)
Gemma-2-2B all configs (baseline: PPL 14.19)
The pattern is unambiguous: SFT wins depth pruning (22.7% lower PPL on LLaMA, 16.7% on Gemma), KD wins Wanda (ARC +3.1pp, IFEval +13.3pp on Gemma). This replicates across two distinct families under strict controls - the reversal is real and driven by how compression type interacts with the recovery objective.
Table numbering
Thank you for pointing out. We will correct this in the camera-ready version.