- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CACTUS, an ADMM-based solver for curvature-aware post-training compression. Its objective incorporates both input activations and downstream gradients, aiming to preserve model utility and gradient-based influence estimates simultaneously. CACTUS supports pure pruning and sparse-plus-low-rank compression.
Critical strengths
- The paper provides a novel connection between curvature-aware compression and the preservation of gradient-based influence estimates.
- CACTUS offers an efficient and technically well-motivated solver for both pruning and sparse-plus-low-rank constraints, quite impressive.
- The evaluation is extensive, covering several model families and sizes, multiple compression settings, influence metrics, and a downstream data-selection task.
- This method challenges traditional objective with a novel perspective. I think it has contribution to this area and hope for open-source code and model.
Critical weaknesses
- The practical efficiency of using the compressed model as an influence proxy is not fully demonstrated, as end-to-end influence-computation time, memory usage, and sparse backward acceleration are not reported.
- The runtime comparison with LLM-Surgeon uses different hardware and optimization settings.
Questions for authors
No
3 comments
I appreciate authors' effort during rebuttal. However, considering other reviewers' valuable opinion and reported limited improvement, I am not so confident about my score now. Actually, my first glance to this manuscript was quite good, so my opinion will remain positive. Hope for better.
We thank the reviewer for engaging with our rebuttal before the deadline. We are glad that their opinion of our work remains positive. We would like to address the concern about the reported improvement with a few highlights of results from the paper and new results for the Qwen3.8-27B model.
The improvement of CACTUS over the SOTA method ALPS is as large as the one ALPS made over SparseGPT.
- On Llama-3.2-1B under 50% unstructured sparsity, CACTUS results in over 30% reduction of the C4 perplexity gap to dense model relative to the SOTA method ALPS.
- This improvement is as large as the one ALPS made over SparseGPT: in the same setting, ALPS reduces SparseGPT's C4 gap to dense by 35% (26.54 to 22.20) and improves the cosine similarity of the influence vectors from 0.266 to 0.395; CACTUS reduces ALPS's gap by a further 32% (22.20 to 19.60) and improves the cosine similarity by a similar difference (0.395 to 0.524). On Qwen3-4B, CACTUS achieves similar wins against ALPS.
| Llama-3.2-1B, 50% | C4 | WT-2 | |
|---|---|---|---|
| Dense | 14.01 | 9.75 | 1.000 |
| SparseGPT | 26.54 | 19.61 | 0.266 |
| ALPS | 22.20 | 16.44 | 0.395 |
| CACTUS | 19.60 | 14.38 | 0.524 |
- Additionally, for sparse-plus-low-rank compression, CACTUS-SLR results in over 30% reduction of the WT-2 perplexity gap to dense model relative to the SOTA method 3BASiL on Llama-3.2-1B under 50% compression (14.46 to 13.02), and improves the cosine similarity of the influence vectors from 0.475 to 0.585.
The gains remain significant on larger models like Qwen3.8-27B under aggressive compression.
It is a well-documented behavior in model compression that the differences between compression algorithms are most pronounced at more aggressive compression ratios (see ALPS vs. SparseGPT results reported in [1]). We therefore ran the comparison on Qwen3.8-27B at 70% unstructured sparsity (a harder setting than 2:4).
CACTUS reduces the perplexity gap to the dense model compared to ALPS by 22% on C4 and 27% on WT-2, and improves the query Spearman correlation from 0.208 to 0.332.
Perplexity results:
| Qwen3.8-27B | C4 | WT-2 |
|---|---|---|
| Dense | 11.27 | 7.05 |
| ALPS, 70% | 16.93 | 12.92 |
| CACTUS, 70% | 15.66 | 11.32 |
Influence preservation results:
| Qwen3.8-27B | @5% | |||
|---|---|---|---|---|
| ALPS, 70% | 0.334 0.003 | 0.139 0.004 | 0.208 0.012 | 0.113 0.007 |
| CACTUS, 70% | 0.382 0.004 | 0.176 0.006 | 0.332 0.015 | 0.173 0.011 |
We hope that these results and the new experiments address the reviewer's concern about the significance of the improvement of CACTUS over the SOTA method ALPS. We also share the reviewer's view that the K-FAC loss (Equation 1) "challenges traditional objective with a novel perspective": it departs from the layerwise reconstruction error that is the de facto standard for compression algorithms. We hope that future work builds on our findings, for instance with new methods to estimate the gradient covariance , and we would be grateful if the reviewer would take these results into account in their final assessment.
[1] ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models. NeurIPS'24
We thank the reviewer for the positive assessment and for highlighting the strengths of our work: "the novel connection between curvature-aware compression and the preservation of gradient-based influence estimates", the extensive evaluation framework "covering several model families and sizes, multiple compression settings, influence metrics, and a downstream data-selection task", and the K-FAC loss minimization studied in Equation 1 of our paper which "challenges traditional objective with a novel perspective". We especially appreciate that the reviewer finds our solver for both pruning and sparse-plus-low-rank constraints "efficient and technically well-motivated".
We will release the open-source code and a selection of compressed models if the paper is accepted.
W1:
We agree with the reviewer that demonstrating efficiency gains would strengthen the final manuscript. For the rebuttal, following the reviewer's suggestion, we integrated open-source 2:4 sparse kernels into the inference and our influence computation pipelines. For the gradient computation acceleration, we follow [1, 2] and create 2 copies of the weights: one to accelerate the forward pass, and the other to accelerate the backward pass. The TRAK projection and LoGRA projection are applied unchanged for sparse or dense versions.
Qwen3.8-27B with 2:4 sparsity:
On an H100 GPU, the 2:4 sparse version of Qwen3.8-27B runs 1.21$\times$ faster with 31% less peak memory than dense.
Additionally, we run experiments for the LoGRA and LESS pipelines with 2:4 sparsity, as shown in the table below.
Compression is a one-time cost, after which the compressed model can be stored and reused subsequently for every inference query or per-sample gradient computation.
W2:
The reviewer is right that we used higher-end hardware for LLM-Surgeon. The reason is that it required GPUs with more memory than the remaining algorithms (e.g., CACTUS, IProX, SparseGPT). In this sense, LLM-Surgeon has an advantage over all the other procedures, which run on weaker hardware.
[1] Hubara et al., 2021. Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks, NeurIPS 2021.
[2] Hu et al., 2024. Accelerating Transformer Pre-training with 2:4 Sparsity, ICML 2024.