acceptodds
Under review as a conference paper at ICLR 2027

Two for the Price of One: Curvature-Aware Pruning Preserves Influence Functions Too

Abstract

Post-training model pruning enables efficient deployment, but conventional methods do not explicitly preserve internal-gradient alignment between sparse and dense models. We study novel connections between downstream performance of compressed models and the ability to retain gradient information via influence-function preservation. Central to this connection is a curvature-aware objective that accounts for both layer activations and backpropagated gradients. Despite its promise, this objective remains underexplored in pruning due to a lack of efficient solvers. We introduce , a novel ADMM-based optimizer supporting both pruning and sparse-plus-low-rank decompositions of pre-trained weights. Compared to LLM-Surgeon which targets the same objective, achieves over an order-of-magnitude speedup. At % pruning of Llama-3.2-1B, reduces the C4 perplexity gap to the dense model by %, improves compressed-to-dense per-sample gradient cosine similarity on WikiText-2 from to , and runs over faster than LLM-Surgeon on substantially weaker hardware. By preserving influence functions, supports effective data selection. On Llama-3.2-3B, fine-tuning a fresh dense model on the % of data selected by achieves % accuracy on BIG-Bench Hard, compared with % using ALPS and % using the dense-mode selection. Finally, integrating with LoGRA, a recent gradient-projection strategy, on Qwen3-4B with MLP layers pruned using 2:4 sparsity and a rank-64 correction retains a cosine alignment with dense LoGRA-projected C4 gradients.

Primary area:
optimization
Keywords:
Pruning, Influence Functions, Model Compression, KFAC approximation, ADMM, Efficient ML, Data Selection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dXb3OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CACTUS, an ADMM-based solver for curvature-aware post-training compression. Its objective incorporates both input activations and downstream gradients, aiming to preserve model utility and gradient-based influence estimates simultaneously. CACTUS supports pure pruning and sparse-plus-low-rank compression.

Critical strengths
  • The paper provides a novel connection between curvature-aware compression and the preservation of gradient-based influence estimates.
  • CACTUS offers an efficient and technically well-motivated solver for both pruning and sparse-plus-low-rank constraints, quite impressive.
  • The evaluation is extensive, covering several model families and sizes, multiple compression settings, influence metrics, and a downstream data-selection task.
  • This method challenges traditional objective with a novel perspective. I think it has contribution to this area and hope for open-source code and model.
Critical weaknesses
  • The practical efficiency of using the compressed model as an influence proxy is not fully demonstrated, as end-to-end influence-computation time, memory usage, and sparse backward acceleration are not reported.
  • The runtime comparison with LLM-Surgeon uses different hardware and optimization settings.
Questions for authors

No

3 comments

We thank the reviewer for the positive assessment and for highlighting the strengths of our work: "the novel connection between curvature-aware compression and the preservation of gradient-based influence estimates", the extensive evaluation framework "covering several model families and sizes, multiple compression settings, influence metrics, and a downstream data-selection task", and the K-FAC loss minimization studied in Equation 1 of our paper which "challenges traditional objective with a novel perspective". We especially appreciate that the reviewer finds our solver for both pruning and sparse-plus-low-rank constraints "efficient and technically well-motivated".

We will release the open-source code and a selection of compressed models if the paper is accepted.

W1:

We agree with the reviewer that demonstrating efficiency gains would strengthen the final manuscript. For the rebuttal, following the reviewer's suggestion, we integrated open-source 2:4 sparse kernels into the inference and our influence computation pipelines. For the gradient computation acceleration, we follow [1, 2] and create 2 copies of the weights: one to accelerate the forward pass, and the other to accelerate the backward pass. The TRAK projection and LoGRA projection are applied unchanged for sparse or dense versions.

Qwen3.8-27B with 2:4 sparsity:

GPUModelInference time / batchThroughput (tok/s)Peak memory
H100 (80 GB)Dense0.470 s8,71963.4 GiB
H100 (80 GB)CACTUS 2:40.387 s10,57943.5 GiB

On an H100 GPU, the 2:4 sparse version of Qwen3.8-27B runs 1.21$\times$ faster with 31% less peak memory than dense.

Additionally, we run experiments for the LoGRA and LESS pipelines with 2:4 sparsity, as shown in the table below.

PipelineDenseCACTUS 2:4SpeedupPeak memory (dense →\to 2:4)
Inference, Qwen3-4B (MLP 2:4)21,921 tok/s28,044 tok/s1.28$\times$9.47 →\to 7.23 GiB
LoGRA per-sample gradients, Qwen3-4B (MLP 2:4)57.5 ms47.4 ms1.21$\times$24.13 →\to 24.71 GiB
LESS per-sample gradients, Llama-3.2-3B (all linears 2:4)72 ms61 ms1.18$\times$19.67 →\to 20.33 GiB

Compression is a one-time cost, after which the compressed model can be stored and reused subsequently for every inference query or per-sample gradient computation.

W2:

The reviewer is right that we used higher-end hardware for LLM-Surgeon. The reason is that it required GPUs with more memory than the remaining algorithms (e.g., CACTUS, IProX, SparseGPT). In this sense, LLM-Surgeon has an advantage over all the other procedures, which run on weaker hardware.

[1] Hubara et al., 2021. Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks, NeurIPS 2021.

[2] Hu et al., 2024. Accelerating Transformer Pre-training with 2:4 Sparsity, ICML 2024.

I appreciate authors' effort during rebuttal. However, considering other reviewers' valuable opinion and reported limited improvement, I am not so confident about my score now. Actually, my first glance to this manuscript was quite good, so my opinion will remain positive. Hope for better.

We thank the reviewer for engaging with our rebuttal before the deadline. We are glad that their opinion of our work remains positive. We would like to address the concern about the reported improvement with a few highlights of results from the paper and new results for the Qwen3.8-27B model.

The improvement of CACTUS over the SOTA method ALPS is as large as the one ALPS made over SparseGPT.

  • On Llama-3.2-1B under 50% unstructured sparsity, CACTUS results in over 30% reduction of the C4 perplexity gap to dense model relative to the SOTA method ALPS.
    • This improvement is as large as the one ALPS made over SparseGPT: in the same setting, ALPS reduces SparseGPT's C4 gap to dense by 35% (26.54 to 22.20) and improves the cosine similarity of the influence vectors from 0.266 to 0.395; CACTUS reduces ALPS's gap by a further 32% (22.20 to 19.60) and improves the cosine similarity by a similar difference (0.395 to 0.524). On Qwen3-4B, CACTUS achieves similar wins against ALPS.
Llama-3.2-1B, 50%C4WT-2cos⁡tr\cos_{\text{tr}}
Dense14.019.751.000
SparseGPT26.5419.610.266
ALPS22.2016.440.395
CACTUS19.6014.380.524
  • Additionally, for sparse-plus-low-rank compression, CACTUS-SLR results in over 30% reduction of the WT-2 perplexity gap to dense model relative to the SOTA method 3BASiL on Llama-3.2-1B under 50% compression (14.46 to 13.02), and improves the cosine similarity of the influence vectors from 0.475 to 0.585.

The gains remain significant on larger models like Qwen3.8-27B under aggressive compression.

It is a well-documented behavior in model compression that the differences between compression algorithms are most pronounced at more aggressive compression ratios (see ALPS vs. SparseGPT results reported in [1]). We therefore ran the comparison on Qwen3.8-27B at 70% unstructured sparsity (a harder setting than 2:4).

CACTUS reduces the perplexity gap to the dense model compared to ALPS by 22% on C4 and 27% on WT-2, and improves the query Spearman correlation from 0.208 to 0.332.

Perplexity results:

Qwen3.8-27BC4WT-2
Dense11.277.05
ALPS, 70%16.9312.92
CACTUS, 70%15.6611.32

Influence preservation results:

Qwen3.8-27Bcos⁡tr\cos_{\text{tr}}cos⁡te\cos_{\text{te}}ρqry\rho_{\text{qry}}RqryR_{\text{qry}}@5%
ALPS, 70%0.334 ±\pm 0.0030.139 ±\pm 0.0040.208 ±\pm 0.0120.113 ±\pm 0.007
CACTUS, 70%0.382 ±\pm 0.0040.176 ±\pm 0.0060.332 ±\pm 0.0150.173 ±\pm 0.011

We hope that these results and the new experiments address the reviewer's concern about the significance of the improvement of CACTUS over the SOTA method ALPS. We also share the reviewer's view that the K-FAC loss (Equation 1) "challenges traditional objective with a novel perspective": it departs from the layerwise reconstruction error that is the de facto standard for compression algorithms. We hope that future work builds on our findings, for instance with new methods to estimate the gradient covariance G\mathbf{G}, and we would be grateful if the reviewer would take these results into account in their final assessment.

[1] ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models. NeurIPS'24

Reviewer_PuzfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces a curvature-aware LLM pruning method for post-training compression. While classical approaches try to solve an input-to-output local reconstruction problem min⁡W^∣∣X(W^−W)∣∣F2\min_{\widehat W} ||X(\widehat W - W)||_F^2 (where W^\widehat W is the sparse weight matrix), the proposed method minimizes a curvature-aware objective Lkfac(W^)=∣∣X(W^−W)A⊤∣∣F2\mathcal L_\mathrm{kfac}(\widehat W) = || X(\widehat W - W)A^\top ||_F^2, where AA is the downstream gradient matrix of the input XX. This objective was established in LLM-Surgeon and IProX. The main contributions are: 1) efficient curvature-aware pruning via cached eigendecompositions and an efficient Sylvester-equation solution; and 2) extension to sparse-plus-low-rank compression.

Critical strengths
  • The solver is a useful algorithmic contribution. Caching the eigendecompositions makes the two-sided curvature-aware update practical, and the same framework handles both pure pruning and sparse-plus-low-rank compression.

  • The compression results are convincing within the evaluated one-shot setting. For example, at 50% compression on Llama-3.2-1B, CACTUS-prune improves C4 perplexity from 21.39 to 19.60 over five-shot LLM-Surgeon, while reducing compression time from over five hours to about sixteen minutes under the reported hardware configurations.

  • The paper evaluates gradient preservation beyond a single cosine-similarity metric. The query-ranking results, KFAC-weighted evaluation, LESS data-selection experiment, and LoGRA integration provide useful evidence that the compressed models retain information relevant to influence estimation.

Critical weaknesses
  • The conceptual novelty is limited. The curvature-aware objective and its connection to influence preservation were already established in LLM-Surgeon and IProX. The ADMM formulation also closely follows ALPS/3BASiL, and the low-rank update is IPSVD. The main advance is an efficient solver for the existing objective. I find this useful, but the paper's framing makes the connection between compression and influence preservation sound newer than it is.

  • The practical efficiency benefit for influence estimation is not established. The reported timings mainly measure compression, rather than the full cost of building the proxy, computing candidate gradients, and selecting data. The inference speedup in Figure 3 is taken from prior work, rather than measured for this pipeline. Please report end-to-end time and peak memory against dense LESS/LoGRA, including the compression overhead and the workload size needed to recover that cost. This matters because pruning preserves the matrix dimensions, so savings depend on the actual sparse implementation.

  • The downstream data-selection evidence is mixed. In Table 2, CACTUS improves selection-score agreement with the dense model, but on Llama-2-7B its BBH accuracy is slightly below SparseGPT (41.57 versus 41.67). On Llama-3.2-3B, it is close to dense selection, but random selection already obtains 48.89 compared with 49.44 for CACTUS. With 1,080 test examples and no variation across selection/fine-tuning seeds reported, these results do not yet establish a reliable downstream advantage. Repeated runs would help distinguish better agreement with dense scores from better data selection.

  • The gains over strong activation-only baselines can be modest in the larger structured settings. For Llama-3-8B with 2:42{:}4 sparsity and a rank-256 correction, C4 perplexity improves from 12.27 to 12.21 over 3BASiL, while compression time increases from 2h 42m to 4h 07m (Table 10). On the 27B model, Table 13 compares only against SparseGPT, with a substantial increase in compression cost. A clearer cost-quality comparison against ALPS/3BASiL at this scale would help establish when the additional curvature information is worth the expense.

Questions for authors

Please clarify and address the concerns raised in the Weaknesses section

6 comments

We thank the reviewer for their careful reading and constructive suggestions. We address the specific concerns below.

W1:

Connection between compression and influence preservation.

Thank you for noting the usefulness of our framework. As far as we can tell, the connection and the simultaneous study of both compression (sparse-plus-low-rank) and influence preservation have not been done before. For instance, we designed multiple frameworks to measure the fidelity of "gradient preservation beyond a single cosine-similarity metric" as the reviewer has mentioned. We additionally study both under "query-ranking results, KFAC-weighted evaluation, LESS data-selection experiment, and LoGRA integration". This can establish synergies between the two communities, which we find as an interesting contribution.

In fact, the studied objective has been derived in very different prior contexts: LLM-Surgeon for sparsity in weights and IProX for influence preservation. One key difference between the two methods is that LLM-Surgeon is designed only for pruning (sparse weights) and doesn't scale to a Qwen3-4B model on a single A100 GPU (80 GB). On the other hand, IProX scales to large models using low-rank weights, at the cost of severe utility degradation. CACTUS presents a new framework that ensures (i) both influence function preservation and model quality, (ii) is more scalable: scales to a Llama-3-8B on a single L40 GPU (48GB), and (iii) yields a more general family of sparse-plus-low-rank weights with better utility than pure low-rank weights (IProX). In particular, CACTUS handles the special cases of pure sparse weights (LLM-Surgeon) while being more scalable. On Llama-3.2-1B under 50% compression, IProX achieves 102102 WikiText2 perplexity, whereas CACTUS-prune and CACTUS-SLR achieve 14.38 and 13.02, respectively, on WikiText2 perplexity. Under similar compression budgets, LLM-Surgeon only achieves 15.04, while running 15×15\times slower than CACTUS.

ADMM formulation.

ADMM is a general framework which can be applied in different settings as mentioned by the reviewer (ALPS for sparsity and 3BASiL for sparse-plus-low-rank, both under layerwise-reconstruction error). Here, we study a different loss function (K-FAC in Equation (1)) under sparse-plus-low-rank setting. In this case, the ADMM updates, which require solving non-convex problems, are different than standard updates used by ADMM algorithms tailored for the layerwise-reconstruction error. In particular, the Smooth update in our setting requires solving a generalized Sylvester equation. In contrast, 3BASiL involves a simple least-squares update. Notice that naively solving the Sylvester equation (5) via a vectorized Kronecker formulation is extremely expensive and doesn't scale to LLM sized matrices. While the ADMM formulation might look similar, the technical details are different and solving the associated subroutines requires novel tools. Additionally, we present a convergence result for our ADMM procedure for the K-FAC loss function in Theorem 1.

W2:

Thanks for raising this efficiency point. We agree with the reviewer that, since pruning preserves the matrix dimensions, the practical savings depend on the actual sparse implementation. This is why we focus on 2:4 sparsity, which is natively supported by the sparse tensor cores of modern NVIDIA GPUs. For the rebuttal, following the reviewer's suggestion, we have used open-source kernels to obtain the following acceleration results:

Qwen3.8-27B with 2:4 sparsity:

GPUModelInference time / batchThroughput (tok/s)Peak GPU memory
H100 (80 GB)Dense0.470 s8,71963.4 GiB
H100 (80 GB)CACTUS 2:40.387 s10,57943.5 GiB

On an H100 GPU, the 2:4 sparse version of Qwen3.8-27B runs 1.21$\times$ faster with 31% less peak memory than dense.

Additionally, we run experiments for the LoGRA and LESS pipelines with 2:4 sparsity, measuring the per-sample gradient computation that is repeated for every candidate example, as shown in the table below.

PipelineDenseCACTUS 2:4SpeedupPeak memory (dense →\to 2:4)
Inference, Qwen3-4B (MLP 2:4)21,921 tok/s28,044 tok/s1.28$\times$9.47 →\to 7.23 GiB
LoGRA per-sample gradients, Qwen3-4B (MLP 2:4)57.5 ms47.4 ms1.21$\times$24.13 →\to 24.71 GiB
LESS per-sample gradients, Llama-3.2-3B (all linears 2:4)72 ms61 ms1.18$\times$19.67 →\to 20.33 GiB

To obtain acceleration under 2:4 sparsity with gradient propagation, we follow [1, 2] and create 2 copies of the weights one to accelerate the forward pass, and the other to accelerate the backward pass. The TRAK projection and LoGRA projection are applied unchanged for sparse or dense versions.

From a practitioner's perspective, compression is a one-time cost, after which the compressed model can be stored and reused subsequently for every inference query or per-sample gradient computation.

[1] Hubara et al., 2021. Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks, NeurIPS 2021.

[2] Hu et al., 2024. Accelerating Transformer Pre-training with 2:4 Sparsity, ICML 2024.

W3:

Following the reviewer's suggestion, we run additional experiments and retrain the fresh model on each selected subset with 3 seeds and now report the mean and the standard error of BBH EM (%) in the table below.

SelectorLlama-3.2-3BLlama-2-7B
Dense LESS49.07 ±\pm 0.4242.62 ±\pm 0.46
CACTUS-prune48.86 ±\pm 0.5041.79 ±\pm 0.31
ALPS48.33 ±\pm 0.6741.74 ±\pm 0.67
SparseGPT47.84 ±\pm 0.9241.92 ±\pm 0.26
Random 5%47.87 ±\pm 0.5239.41 ±\pm 0.55

The results continue to be aligned with the ones reported in the paper, where CACTUS-prune has the best performance on Llama-3.2-3B among the 4 baselines, and ranks 2 with 41.79 just behind the baseline SparseGPT with 41.92 for Llama-2-7B. Most importantly, our "query-ranking results, KFAC-weighted evaluation, LESS data-selection experiment, and LoGRA integration provide useful evidence that the compressed models retain information relevant to influence estimation." as the reviewer has noted. In subsections 3.3, and 3.4, we demonstrate that CACTUS is much more faithful in influence function preservation as well as scoring and selecting the same data as the dense model across multiple model families, sizes, and compression budgets, and across different projection strategies, which remains the main focus of the paper.

Reviewer_h9qtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CACTUS, an ADMM-based solver for curvature-aware pruning and sparse-plus-low-rank compression of pretrained language models. It combines a Sylvester-equation update with sparse projection and weighted low-rank approximation. The experiments compare compression quality, preservation of dense-model gradient-based influence estimates, downstream data selection, and compatibility with LoGRA.

Critical strengths
  • The solver provides a practical way to optimize the two-sided curvature-aware objective under both sparse and sparse-plus-low-rank constraints.

  • The compression experiments cover multiple model families, sizes, and compression budgets, with substantial runtime improvements over the reported LLM-Surgeon configurations.

  • The evaluation includes influence-ranking metrics and downstream data selection, going beyond gradient cosine similarity alone.

Critical weaknesses
  • The conceptual contribution of curvature-aware influence preservation is limited, and its justification remains incomplete. The objective is inherited from LLM-Surgeon and IProX, while Figure 5 motivates it through local loss perturbations with fixed activations and downstream gradients. The paper should explain more directly why controlling this surrogate preserves gradients recomputed after compression and the resulting influence rankings, beyond the contribution of the new solver.

  • The experiments do not sufficiently isolate the contribution of downstream curvature. Comparisons with ALPS and 3BASiL provide indirect evidence, but the damping study in Table 3 only reports perplexity and does not remove curvature information. A controlled comparison using the same solver with the full G versus an identity matrix, under matched compression and optimization settings, should report both perplexity and influence-ranking preservation.

  • The practical efficiency benefit for influence estimation is not established. The reported runtimes measure model compression, while the inference speedup in Figure 3 is taken from prior work. The authors should report gradient-extraction and scoring time, peak memory, and total cost including compression for the LESS or LoGRA pipeline against the dense model.

  • Better agreement with dense-model selection does not consistently improve downstream performance. In Table 2, CACTUS achieves 49.44% versus 48.89% for random selection on Llama-3.2-3B, and 41.57% versus 41.67% for SparseGPT on Llama-2-7B. Repeated selection and fine-tuning runs are needed to establish the reliability of these gains.

  • The LoGRA comparison gives CACTUS-SLR an additional rank-64 correction while its competitors use pure pruning. Including OATS or 3BASiL with the same sparsity and rank budget would clarify whether the improvement comes from the proposed optimization.

Questions for authors

See weaknesses.

6 comments

We thank reviewer h9qt for their careful reading and constructive suggestions. We address the specific concerns below.

W1:

The studied objective has been derived in very different prior contexts: LLM-Surgeon for sparsity in weights and IProX for influence preservation. One key difference between the two methods is that LLM-Surgeon is designed only for pruning (sparse weights) and doesn't scale to a Qwen3-4B model on a single A100 GPU (80 GB). On the other hand, IProX scales to large models using low-rank weights at the cost of severe utility degradation. CACTUS presents a new framework that ensures (i) both influence function preservation and model quality, (ii) is more scalable: scales to a Llama-3-8B on a single L40 GPU (48GB), and (iii) yields a more general family of sparse-plus-low-rank weights with better utility than pure low-rank weights (IProX). In particular, CACTUS handles the special cases of pure sparse weights (LLM-Surgeon) while being more scalable.

On Llama-3.2-1B under 50% compression, IProX achieves 102102 WikiText2 perplexity, whereas CACTUS-prune and CACTUS-SLR achieve 14.38 and 13.02, respectively on WikiText2 perplexity. Under similar compression budgets, LLM-Surgeon only achieves 15.04, while running 15×15\times slower than CACTUS.

In our paper, Figure 5 aims to show the derivations of each paper separately at a high level; but we agree with the reviewer that including the further explanations that follow, summarized from IProX, about the influence preservation would strengthen the motivation for the K-FAC loss in Equation 1 in our paper.

The IProX authors propose to:

(i) Write the influence of each layer through its per-sample gradient. For a linear layer ℓ\ell with input hℓ−1(z)h_{\ell-1}(z) and upstream gradient δℓ(z)\delta_\ell(z) (in our paper we denote these by xx and gg), the per-sample gradient is ∇WℓL(z)=δℓ(z)hℓ−1(z)⊤\nabla_{W_\ell}\mathcal{L}(z)=\delta_\ell(z) h_{\ell-1}(z)^\top, so the TracIn contribution of layer ℓ\ell factorizes as:

IWℓ(z,z′)=⟨δℓ(z),δℓ(z′)⟩⟨hℓ−1(z),hℓ−1(z′)⟩.I_{W_\ell}(z,z')=\langle \delta_\ell(z),\delta_\ell(z')\rangle \langle h_{\ell-1}(z),h_{\ell-1}(z')\rangle .

Notice that a perturbation Wℓ↦W^ℓ=Wℓ+EℓW_\ell \mapsto \hat W_\ell = W_\ell+E_\ell does not change hℓ−1h_{\ell-1}, so it changes the influence only through the upstream gradients δ^ℓ(z)\hat\delta_\ell(z).

(ii) Measure the perturbation by its layer-local directional effect, i.e. the first-order change of the per-sample loss,

eℓ(z)=δℓ(z)⊤Eℓhℓ−1(z),e_\ell(z)=\delta_\ell(z)^\top E_\ell h_{\ell-1}(z),

and use it to bound the change in influence. Under some regularity assumption, their Proposition 4.1 gives that there is a data-dependent constant CκC_\kappa such that

Ez,z′∣IW^ℓ(z,z′)−IWℓ(z,z′)∣≤CκEz[eℓ(z)2].\mathbb E_{z,z'} \lvert I_{\hat W_\ell}(z,z')-I_{W_\ell}(z,z') \rvert \le C_\kappa \sqrt{\mathbb E_z \left[e_\ell(z)^2\right]} .

The left-hand side is the influence computed with the gradients recomputed after compression.

(iii) Make the surrogate of the right-hand side of the previous equation more tractable with the K-FAC approximation E[δδ⊤⊗hh⊤]≈E[δδ⊤]⊗E[hh⊤]\mathbb{E}[\delta\delta^\top\otimes hh^\top]\approx\mathbb{E}[\delta\delta^\top]\otimes\mathbb{E}[hh^\top]:

Ez[eℓ(z)2]≈∥Cδ1/2(Wℓ−W^ℓ)Ch1/2∥F2=Tr(CδEℓChEℓ⊤),\mathbb E_z\big[e_\ell(z)^2\big] \approx \big\Vert C_{\delta}^{1/2} (W_\ell-\hat W_\ell) C_{h}^{1/2}\big\Vert_F^2 = \mathrm{Tr}\big(C_\delta E_\ell C_h E_\ell^\top\big),

with Ch=E[hℓ−1hℓ−1⊤]C_h=\mathbb{E}[h_{\ell-1}h_{\ell-1}^\top] and Cδ=E[δℓδℓ⊤]C_\delta=\mathbb{E}[\delta_\ell\delta_\ell^\top]. With Ch=H/NcalC_h=\mathbf{H}/N_{\text{cal}}, Cδ=G/NcalC_\delta=\mathbf{G}/N_{\text{cal}} and Eℓ=(W^−W)⊤E_\ell=(\hat{\mathbf{W}}-\mathbf{W})^\top in our input-by-output convention, this is exactly our objective in Equation (1).

Minimizing the K-FAC loss in Equation (1) therefore minimizes the right-hand side of the bound in (ii), which controls the expected change of each layer's influence contribution with recomputed gradients.

W2:

This is an excellent point to make as it fully isolates the benefits of the solver from the curvature-aware loss. Following the reviewer's suggestion, we ran CACTUS with G=I\mathbf G=\mathbf I, everything else fixed, on Llama-3.2-1B at 50% compression (both pure pruning and sparse-plus-low-rank with ratio ρ=0.3\rho=0.3), on Qwen3.8-27B at 2:4 sparsity, and on Qwen3-4B with (2:4 + r64) configuration on the MLP layers. We report perplexity for Llama-3.2-1B and Qwen3.8-27B, and influence preservation for Llama-3.2-1B and Qwen3-4B.

Perplexity (C4 / WT2) (Model Quality):

Model, budgetDenseCACTUS, G=I\mathbf G=\mathbf ICACTUS, full G\mathbf G
Llama-3.2-1B, 50% (sparse)14.02 / 9.7522.32 / 16.4219.60 / 14.38
Llama-3.2-1B, 50% (sparse + low-rank)14.02 / 9.7520.17 / 14.2318.27 / 13.02
Qwen3.8-27B, 2:411.27 / 7.0513.69 / 9.1413.42 / 8.88

Influence preservation metrics:

Model, methodcos⁡tr\cos_{\text{tr}}cos⁡te\cos_{\text{te}}ρqry\rho_{\text{qry}}RqryR_{\text{qry}}@5%
Llama-3.2-1B 50%, CACTUS-prune, G=I\mathbf G=\mathbf I0.3940.3200.0840.108
Llama-3.2-1B 50%, CACTUS-prune, full G\mathbf G0.5270.4430.2290.140
Llama-3.2-1B 50% (S+LR), CACTUS-SLR, G=I\mathbf G=\mathbf I0.4640.4120.1890.148
Llama-3.2-1B 50% (S+LR), CACTUS-SLR, full G\mathbf G0.5910.5090.3120.185
Qwen3-4B MLP 2:4 + r64, CACTUS-SLR, G=I\mathbf G=\mathbf I0.7050.6090.7640.554
Qwen3-4B MLP 2:4 + r64, CACTUS-SLR, full G\mathbf G0.7170.6410.8010.640

With G=I\mathbf G=\mathbf I, our solver is on par with the strongest layerwise-reconstruction solvers. Most importantly, using the full curvature G\mathbf G clearly improves perplexity and gradient preservation versus CACTUS with G=I\mathbf G = \mathbf I on all model quality as well as influence preservation metrics for the studied models and compression configuration. For instance, for Llama-3.2-1B 50%, CACTUS with curvature achieves 14.38 Wikitext-2 perplexity and 0.527 cos⁡tr\cos_{\text{tr}} compared to 16.42 Wikitext-2 perplexity and 0.394 cos⁡tr\cos_{\text{tr}} achieved by running our solver using G=I\mathbf G=\mathbf I. In the revised version, we will add a subsection in the Appendix showing results for CACTUS with G=I\mathbf G=\mathbf I; we thank the reviewer for this insightful remark.

W3:

Thanks for raising this efficiency point.

Our reported compression runtime in the paper already includes gradient collection and forming the H\mathbf{H} and G\mathbf{G} matrices using the small calibration data poitns. The core ADMM algorithm (post G\mathbf{G} and H\mathbf{H} collection) runs at comparable runtimes to other layerwise-reconstruction error algorithms even for the largest models. The differences in overall runtime between CACTUS and the other layerwise-reconstruction algorithms mainly comes from forming the G\mathbf{G} matrix.

For the rebuttal, following the reviewer's suggestion, we have used open-source kernels to obtain the following inference gains:

Qwen3.8-27B with 2:4 sparsity:

GPUModelInference time / batchThroughput (tok/s)Peak memory
H100 (80 GB)Dense0.470 s8,71963.4 GiB
H100 (80 GB)CACTUS 2:40.387 s10,57943.5 GiB

On an H100 GPU, the 2:4 sparse version of Qwen3.8-27B runs 1.21$\times$ faster with 31% less peak memory than dense.

Additionally, we run experiments for the LoGRA and LESS pipelines with 2:4 sparsity, as shown in the table below.

PipelineDenseCACTUS 2:4SpeedupPeak memory (dense →\to 2:4)
Inference, Qwen3-4B (MLP 2:4)21,921 tok/s28,044 tok/s1.28$\times$9.47 →\to 7.23 GiB
LoGRA per-sample gradients, Qwen3-4B (MLP 2:4)57.5 ms47.4 ms1.21$\times$24.13 →\to 24.71 GiB
LESS per-sample gradients, Llama-3.2-3B (all linears 2:4)72 ms61 ms1.18$\times$19.67 →\to 20.33 GiB

To obtain acceleration under 2:4 sparsity with gradient propagation, we follow [1, 2] and create 2 copies of the weights one to accelerate the forward pass, and the other to accelerate the backward pass. The TRAK projection and LoGRA projection are applied unchanged for sparse or dense versions.

From a practitioner's perspective, compression is a one-time cost, after which the compressed model can be stored and reused subsequently for every inference query or per-sample gradient computation.

[1] Hubara et al., 2021. Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks, NeurIPS 2021.

[2] Hu et al., 2024. Accelerating Transformer Pre-training with 2:4 Sparsity, ICML 2024.

Comments on the paper

2 comments
AuthorsOpenReview ↗

A highlight of the paper's contributions and significance results

We thank the reviewers for the time and effort spent to submit their careful reviews.

Our method CACTUS yields gains of up to:

Model compression evaluations (reduction of the WT-2 / C4 perplexity gap to the dense model, relative to the SOTA method of each compression family):

Compression configurationGain (WT-2 / C4)Model
50% unstructured sparsity31% / 32% vs. ALPSLlama-3.2-1B
70% unstructured sparsity42% / 30% vs. ALPSLlama-3.2-1B
50% sparse-plus-low-rank31% / 29% vs. 3BASiLLlama-3.2-1B
70% sparse-plus-low-rank45% / 38% vs. 3BASiLLlama-3.2-1B
70% unstructured sparsity20% / 11% vs. ALPSLlama-3.2-3B
70% unstructured sparsity17% / 28% vs. ALPSQwen3-4B
70% unstructured sparsity27% / 22% vs. ALPSQwen3.8-27B

Influence preservation evaluations (train cosine similarity between compressed and dense influence vectors):

Compression configurationProjectionGain (train cosine)Model
50% unstructured sparsityTRAK0.395 → 0.524 (+33%) vs. ALPSLlama-3.2-1B
50% sparse-plus-low-rankTRAK0.475 → 0.585 (+23%) vs. 3BASiLLlama-3.2-1B
70% unstructured sparsityTRAK0.406 → 0.475 (+17%) vs. ALPSQwen3-4B
2:4 + rank 256TRAK0.518 → 0.573 (+11%) vs. 3BASiLLlama-3-8B
70% unstructured sparsityLoGRA0.334 → 0.382 (+14%) vs. ALPSQwen3.8-27B

Reference values for significance.

In our own tables, the gains of the SOTA method ALPS over SparseGPT are of the same order as the gains of CACTUS over ALPS:

SettingALPS over SparseGPTCACTUS over ALPS
Llama-3.2-1B, 50%: WT-2 / C4 gap reduction32% / 35%31% / 32%
Llama-3.2-1B, 50%: train cosine0.266 → 0.395 (+0.129)0.395 → 0.524 (+0.129)
  • ALPS (NeurIPS 2024) reports as its headline result "a 29% reduction in test perplexity on the WikiText dataset" compared to existing methods (LLaMA3-8B, 70% sparsity). By the same measure, CACTUS reduces the WT-2 perplexity of ALPS by 35% on Llama-3.2-1B at 70% sparsity (63.47 → 41.08).
  • 3BASiL (NeurIPS 2025) reports as its headline result that it "reduces the WikiText2 perplexity gap relative to dense LLaMA-8B model by over 30%" compared to prior methods (2:4 + rank 64). CACTUS-SLR reduces the WT-2 gap of 3BASiL by 31% at 50% and 45% at 70% compression on Llama-3.2-1B.

CACTUS achieves SOTA results on a wide range of compression configurations, model families and sizes, and evaluation metrics. CACTUS also shows SOTA results on both model compression and influence preservation front (using two different gradient projection strategies), whereas prior methods focus on one or the other. We believe that this is a strong contribution to the field of model compression and influence preservation.

AuthorsOpenReview ↗

Rebuttal Requests by the reviewers

During the rebuttal, we have also addressed the following points raised by the reviewers:

Efficiency with open-source 2:4 sparse kernels. We integrated open-source 2:4 sparse kernels into inference and into our influence computation pipelines. For per-sample gradients, we create a second 2:4 sparse copy of the transposed weight, so that both the forward and the backward matrix multiplications use 2:4 sparse tensor cores.

PipelineModelGPUDenseCACTUS 2:4SpeedupPeak memory (dense → 2:4)
InferenceQwen3.8-27B (all linears 2:4)H1008,719 tok/s10,579 tok/s1.21×63.4 → 43.5 GiB
InferenceQwen3-4B (MLP 2:4)L40S21,921 tok/s28,044 tok/s1.28×9.47 → 7.23 GiB
LoGRA per-sample gradientsQwen3-4B (MLP 2:4)L40S57.5 ms47.4 ms1.21×24.13 → 24.71 GiB
LESS per-sample gradientsLlama-3.2-3B (all linears 2:4)H10072 ms61 ms1.18×19.67 → 20.33 GiB

Compression is a one-time cost: the compressed model is computed once, then stored and reused for every subsequent inference query and every per-sample gradient computation.

Novelty of the ADMM solver.

We argued that ADMM is a general purpose technique, and adapting CACTUS to the K-FAC objective changes the required subroutines and requires novel tools compared to layerwise reconstruction methods. The smooth update becomes a two-sided generalized Sylvester equation instead of a one-sided least-squares problem: a naive Kronecker solve is intractable for LLM-sized layers, and we solve it exactly at matrix-multiplication cost in the eigenbases of H\mathbf{H} and G\mathbf{G}. The low-rank update becomes a two-sided IPSVD instead of a one-sided weighted SVD, and we prove convergence of the resulting procedure for the K-FAC loss (Theorem 1).

SubroutineALPS (pruning) / 3BASiL (sparse-plus-low-rank)CACTUS
Objectivelayerwise reconstruction Tr(E⊤HE)\mathrm{Tr}(\mathbf{E}^\top\mathbf{H}\mathbf{E})K-FAC loss Tr(HEGE⊤)\mathrm{Tr}(\mathbf{H}\mathbf{E}\mathbf{G}\mathbf{E}^\top)
Smooth updateone-sided least squares: (H+ρ2I)S=B(\mathbf{H} + \frac{\rho}{2}\mathbf{I})\mathbf{S} = \mathbf{B}two-sided generalized Sylvester equation: HSG+ρ2S=B\mathbf{H}\mathbf{S}\mathbf{G} + \frac{\rho}{2}\mathbf{S} = \mathbf{B}
Solving the smooth updateone linear solvenaive Kronecker solve costs O(m3n3)O(m^3 n^3); our eigenbasis solve costs O(mn(m+n))O(mn(m+n))
Low-rank updateone-sided weighted SVD of H1/2R\mathbf{H}^{1/2}\mathbf{R}two-sided IPSVD of H1/2RG1/2\mathbf{H}^{1/2}\mathbf{R}\mathbf{G}^{1/2}

Here E=W−S−L\mathbf{E} = \mathbf{W} - \mathbf{S} - \mathbf{L} is the compression error of an m×nm \times n layer, B\mathbf{B} is the right-hand side of the update, and R=W−S\mathbf{R} = \mathbf{W} - \mathbf{S} is the residual of the sparse component.

Ablation study: full curvature vs. solver.

Followig reviewer h9qt insightful remark, we ran CACTUS with G=I\mathbf{G} = \mathbf{I}, everything else fixed. This fully isolates the benefit of the full curvature from that of the solver. Our solver using the full curvature performs much better than the solver using the identity, showcasing the value of the K-FAC loss and the full curvature to guide pruning.

Llama-3.2-1B, 50%CACTUS, G=I\mathbf{G} = \mathbf{I}CACTUS, full G\mathbf{G}
Pruning: WT-2 / C416.42 / 22.3214.20 / 19.64
Pruning: train cosine0.3940.527
Sparse-plus-low-rank: WT-2 / C414.23 / 20.1713.07 / 18.45
Sparse-plus-low-rank: train cosine0.4640.591

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.