acceptodds
Under review as a conference paper at ICLR 2027

Prune Gradually, Adapt Sparsely: Efficient Pruning of Large Language Models at Extreme Sparsity

17 views

Abstract

Weight pruning can reduce the storage and inference cost of large language models (LLMs), but preserving quality at high sparsity levels remains challenging. Existing methods face a quality-cost trade-off in this regime. One-shot pruning is cheap but degrades sharply as sparsity increases, even when followed by post-pruning recovery. A growing line of work instead optimizes the mask directly from the dense model under the end-to-end objective; among these methods, ELSA lee2026elsa, the recent state of the art at extreme sparsity, reaches high quality but at the cost of model-sized optimizer state and auxiliary variables. To address this gap, we propose ETAPS, which prunes gradually and adapts sparsely: it tightens a global sparsity constraint in progressive stages while training a sparse delta with a small, fixed budget. Our key observation is that gradual weight removal keeps each pruning step small enough for sparse adaptation to compensate, allowing the adapted model to provide end-to-end gradients that guide the next pruning decision without model-sized optimizer state. Across six models from 125M to 13B parameters, ETAPS achieves lower perplexity than ELSA at every evaluated sparsity from 50% to 90%, with perplexity reductions of up to 42%. ETAPS is also far cheaper: at 90% sparsity, it prunes LLaMA-2-7B on a single H200, where ELSA uses four, with smaller total persistent state and 49% fewer GPU-hours. Together, our findings point to a promising direction for achieving high quality in demanding regimes such as extreme sparsity at substantially lower cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Extreme Sparsity, Sparse Fine-Tuning, Gradual Pruning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_x9jtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces ETAPS, a method for pruning LLMs to high sparsity while keeping optimizer memory small. It gradually increases sparsity and adapts a small sparse delta between pruning steps, with the pretrained weights kept frozen. The next pruning mask is selected globally using weight magnitude multiplied by the mean absolute gradient collected during adaptation. The method also assigns longer adaptation intervals to later pruning stages and reallocates delta coordinates among surviving weights. Experiments show consistent perplexity improvements over ELSA across several model families, with lower persistent memory and fewer GPU-hours in the reported 7B comparison. The main contribution is an effective combination of gradual pruning and sparse adaptation, supported by extensive empirical evaluation.

Critical strengths
  • The paper addresses a useful practical problem: obtaining good models at high sparsity without maintaining full-model optimizer and auxiliary state. The results are consistent across model families and sparsity levels. For example, at 90% sparsity on LLaMA-2-7B, ETAPS improves WikiText-2/C4 perplexity from 27.98/23.20 to 20.98/17.39 over ELSA, while using 26.40 rather than 132.29 GiB of persistent state and 1.66 rather than 3.24 GPU-hours.

  • The ablations are extensive and generally well controlled. Direct and progressive pruning use the same adaptation budget, and the paper separately examines adaptation intervals, saliency scores, schedule shape, and delta capacity. The direct-versus-progressive comparison across three models is particularly useful evidence that the pruning trajectory matters at high sparsity.

  • The method is clearly described, including implementation details that matter for reproduction. The paper distinguishes dense backward computation from sparse optimizer state, reports the difference between persistent and peak memory, and explains how its sparse adaptation differs from SEFT. The experiments up to 70B parameters also demonstrate that the procedure can be executed at a substantial scale.

Critical weaknesses
  • The novelty is mainly in combining existing components. Gradual pruning, sparse fine-tuning, gradient-guided support allocation, and magnitude-gradient saliency are all established ideas, as the paper itself discusses. The ablations support the combination, but do not fully establish why sparse-coordinate adaptation is the best choice for the progressive procedure. A progressive baseline using a sparsity-preserving LoRA-style update under a matched adaptation budget would help separate the benefit of the trajectory from that of the chosen parameterization. The current LoRA comparison starts after one-shot pruning.

  • The efficiency evidence is narrower than the quality evaluation. The detailed timing and memory comparison is for LLaMA-2-7B at 90% sparsity, and the reported state excludes activations and transient buffers. ETAPS still performs dense backward computation, and its wall-clock time is longer than the four-GPU ELSA run despite lower aggregate GPU-hours. Peak memory and cost-quality curves across optimization budgets, including ELSA-L where feasible, would make the practical trade-off clearer. End-to-end sparse inference latency is also not evaluated.

  • The experiments use a single run, and large perplexity gains do not always translate into similarly large downstream gains. At 90% sparsity on LLaMA-2-7B, average zero-shot accuracy improves only from 38.41 to 38.84 over ELSA. Results at 95% and 99% sparsity mainly establish better perplexity; they do not show how much useful downstream capability remains. Repeated runs and downstream evaluation at these sparsity levels would strengthen the extreme-sparsity claim.

  • The 67B/70B experiments demonstrate feasibility, but the comparison at that scale is limited to one-shot methods with much smaller data and compute budgets. The Gemma-2-27B comparison against ELSA-L is more informative. At 67B/70B, a trained recovery baseline, or at least a fuller resource breakdown, would help assess whether the favorable cost-quality trade-off carries over.

Questions for authors

Please clarify and address the concerns raised in the Weaknesses section

6 comments
AuthorsOpenReview ↗

Response for Reviewer x9jt (1/4)

We sincerely thank the reviewer for the thoughtful review, and for recognizing that ETAPS addresses a useful practical problem with consistent results across model families and sparsity levels, supported by extensive ablations. We address each point below and would be glad to continue the discussion.


Sparse adaptation but not LoRA (W1)

The ablations support the combination, but do not fully establish why sparse-coordinate adaptation is the best choice for the progressive procedure. A progressive baseline using a sparsity-preserving LoRA-style update under a matched adaptation budget would help separate the benefit of the trajectory from that of the chosen parameterization.

We thank the reviewer for this insightful suggestion. We would like to clarify that we do not claim sparse-coordinate adaptation is the best possible parameterization, but rather a natural and effective choice for efficient adaptation under extreme sparsity. To further substantiate this choice, we replace ETAPS's sparse delta with SPP [1], an established sparsity-preserving LoRA-style adaptation method, while retaining the progressive pruning procedure. SPP adapts surviving weights through shared learnable row and column scaling factors, preserving the pruning mask. Under a similar adaptation budget, we evaluate both methods on LLaMA-2-7B at 90% sparsity with a micro-batch size of 8 on NVIDIA H200 NVL GPUs.

MethodWiki-2 ↓\downarrowC4 ↓\downarrowDownstream avg. ↑\uparrowWall-clock (h) ↓\downarrowPersistent memory (GiB) ↓\downarrowPeak memory (GiB) ↓\downarrow
ETAPS w/ SPP29.1725.4238.022.1325.44130.47
ETAPS20.9817.3938.841.6626.40116.08

The results demonstrate that the adaptation parameterization matters beyond the progressive pruning trajectory. Compared with SPP, ETAPS reduces WikiText-2/C4 perplexity by 28%/32%, while achieving higher downstream accuracy, 22% shorter runtime, and 11% lower peak GPU memory.

AuthorsOpenReview ↗

Response for Reviewer x9jt (2/4)

Efficiency evaluation reporting (W2)

ETAPS still performs dense backward computation, and its wall-clock time is longer than the four-GPU ELSA run despite lower aggregate GPU-hours.

We acknowledge that ETAPS still performs dense backward computation, as already noted in our Limitations section. However, this is also required by ELSA and ELSA-L; ETAPS instead reduces optimizer and auxiliary-state overhead, achieving substantially lower persistent memory.

Regarding wall-clock time, the reported comparison involves different GPU configurations: ETAPS uses a single GPU (1.66 h), whereas ELSA uses four GPUs (0.81 h). In the single-GPU comparison with ELSA-L, ETAPS is nearly 2$\times$ faster (1.66 vs. 3.27 h) while achieving better perplexity and downstream accuracy. Moreover, ETAPS requires approximately half the aggregate GPU-hours of both ELSA variants.

Peak memory and cost-quality curves across optimization budgets, including ELSA-L where feasible, would make the practical trade-off clearer.

... and the reported state excludes activations and transient buffers.

We have added peak memory to the comparison on LLaMA-2-7B at 90% sparsity, together with ELSA-L. Since peak memory depends on the micro-batch, we report ETAPS with the standard micro-batch of 8 used in Table 4 and additionally ETAPS$^\dagger$ with a micro-batch of 1, matching ELSA-L. Reducing the micro-batch lowers peak memory at the cost of runtime, while persistent state is unchanged. All runs use NVIDIA H200 NVL GPUs with an effective batch size of 8.

MethodWiki-2 ↓\downarrowC4 ↓\downarrowDownstream avg. ↑\uparrowGPUsMicro-batchWall-clock (h) ↓\downarrowGPU-h ↓\downarrowPersistent memory (GiB) ↓\downarrowPeak memory (GiB) ↓\downarrow
ELSA27.9823.2038.41420.813.24132.29179.92
ELSA-L28.5824.6336.10113.273.2762.66101.00
ETAPS20.9817.3938.84181.661.6626.40116.08
ETAPS$^\dagger$20.9517.3739.30112.082.0826.4038.59

The difference between peak and persistent memory reflects activations and transient buffers, which account for 12.19 GiB for ETAPS$^\dagger$, compared with 38.34 GiB for ELSA-L. Notably, ETAPS$^\dagger$ achieves the best quality while requiring 2.6$\times$ less peak memory, 2.4$\times$ less persistent memory, and 1.6$\times$ fewer GPU-hours than ELSA-L.

Regarding cost-quality curves, we agree that they would be helpful. However, we note that producing them would require running every method at multiple budgets, which exceeds our compute given the number of baselines. Moreover, we believe the reported results already characterize the practical trade-off, as the evaluated methods span a wide range of costs, from one-shot pruning in under 0.01 GPU-hours to trained methods requiring several GPU-hours, each reported in the configuration in which it is typically used.

End-to-end sparse inference latency is also not evaluated.

We thank the reviewer for this important suggestion. Below, we evaluate end-to-end sparse inference of LLaMA-2-7B pruned by ETAPS, using the MACKO sparse-matrix vector multiplication (SpMV) kernel [2] on a single consumer-grade RTX 3090 GPU with batch size 1 and 100 generated tokens. The results are reported relative to the dense model.

SparsityDecode speed ↑\uparrowTotal-latency speed ↑\uparrowGPU allocated-memory reduction ↑\uparrowWeight storage (GB) ↓\downarrow
Dense1.00$\times$1.00$\times$1.00$\times$13.48
50%1.32$\times$1.29$\times$1.53$\times$8.69
60%1.55$\times$1.51$\times$1.87$\times$7.09
70%1.91$\times$1.84$\times$2.40$\times$5.48
80%2.41$\times$2.29$\times$3.36$\times$3.89
90%3.26$\times$3.05$\times$5.46$\times$2.34
95%3.91$\times$3.58$\times$7.87$\times$1.60
99%5.01$\times$4.44$\times$11.91$\times$1.02

As observed, the efficiency gains grow with sparsity. At 90% sparsity, the sparse model decodes 3.26$\times$ faster, achieves a 3.05$\times$ end-to-end speedup, and requires 5.46$\times$ less GPU memory, with weights shrinking from 13.5 GB to 2.3 GB. Moreover, since ETAPS achieves the best quality across 90%–99% sparsity (Table 2), these efficiency gains can be realized with less quality degradation than with existing methods. We will include this evaluation in the revised manuscript.

AuthorsOpenReview ↗

Response for Reviewer x9jt (3/4)

Downstream evaluation reporting (W3)

... large perplexity gains do not always translate into similarly large downstream gains. At 90% sparsity on LLaMA-2-7B, average zero-shot accuracy improves only from 38.41 to 38.84 over ELSA.

The highlighted LLaMA-2-7B result occurs at 90% sparsity, where both methods have already suffered substantial downstream accuracy degradation (38.41% for ELSA and 38.84% for ETAPS, compared with 60.21% for the dense model). Perplexity measures next-token prediction quality continuously, whereas downstream accuracy reflects discrete task success, so improvements need not be proportional. At extreme sparsity, challenging tasks can exhibit very low accuracy, making average downstream accuracy less sensitive to improvements that remain visible in perplexity. For instance, both methods achieve around 20% on ARC-Challenge, while ETAPS shows a clearer gain on ARC-Easy (43.98% vs. 40.53%). Importantly, this isolated case is not representative of the broader results: ETAPS improves average downstream accuracy over ELSA in all six model–sparsity comparisons in Table 1, with five showing gains of 2.46–6.93 percentage points.

The experiments use a single run ... Results at 95% and 99% sparsity mainly establish better perplexity; they do not show how much useful downstream capability remains. Repeated runs and downstream evaluation at these sparsity levels would strengthen the extreme-sparsity claim.

We thank the reviewer for this suggestion. To address both concerns, we repeated ETAPS on LLaMA-2-7B with three different seeds at 90%, 95%, and 99% sparsity, reporting the mean and standard deviation of perplexity and all seven downstream tasks below.

SparsityWiki-2 ↓\downarrowC4 ↓\downarrowARC-C ↑\uparrowARC-E ↑\uparrowBoolQ ↑\uparrowHellaSwag ↑\uparrowOBQA ↑\uparrowRTE ↑\uparrowWinoGrande ↑\uparrowDownstream avg. ↑\uparrow
90%20.90±0.1420.90 \pm 0.1417.40±0.0317.40 \pm 0.0319.91±0.7719.91 \pm 0.7744.40±0.4444.40 \pm 0.4459.86±4.0159.86 \pm 4.0132.96±0.2432.96 \pm 0.2417.00±0.6017.00 \pm 0.6052.83±0.2152.83 \pm 0.2149.83±1.8249.83 \pm 1.8239.54±0.6639.54 \pm 0.66
95%28.16±0.3528.16 \pm 0.3522.04±0.0722.04 \pm 0.0718.49±0.6518.49 \pm 0.6536.84±0.2336.84 \pm 0.2360.98±1.2060.98 \pm 1.2029.32±0.0929.32 \pm 0.0916.73±0.9516.73 \pm 0.9552.83±0.2152.83 \pm 0.2149.70±0.6649.70 \pm 0.6637.84±0.3337.84 \pm 0.33
99%46.14±1.5846.14 \pm 1.5832.82±0.3932.82 \pm 0.3917.12±0.7517.12 \pm 0.7531.20±0.5931.20 \pm 0.5961.86±0.4461.86 \pm 0.4427.32±0.1527.32 \pm 0.1512.13±0.3112.13 \pm 0.3152.71±0.3652.71 \pm 0.3650.67±1.7650.67 \pm 1.7636.14±0.4436.14 \pm 0.44

Overall, the run-to-run variation is small relative to the improvements of ETAPS. At 90% sparsity, ETAPS reaches 20.90 ±\pm 0.14 WikiText-2 perplexity, consistently outperforming ELSA at 27.98 (Table 2). In terms of downstream accuracy, the average of ETAPS at 90% (39.54 ±\pm 0.66) exceeds that of ELSA (38.41, Table 21), and decreases only moderately at 95% and 99% sparsity, with low variance across runs.

Reviewer_DHXtOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ETAPS, which combines gradual pruning with fixed-budget sparse-delta adaptation for extreme-sparsity LLM compression. Gradient-weighted importance guides successive pruning decisions. Experiments demonstrate improved perplexity and reduced persistent optimization memory compared with ELSA, with evaluations extending to 70B models.

Critical strengths
  1. The paper is well-written and easy to understand, with clear figures and tables.
  2. LLMs incur substantial inference costs, and sparsity is an effective approach to reducing GPU memory consumption and accelerating inference. The authors propose an effective sparsification method that mitigates quality degradation at high sparsity.
  3. The proposed method is evaluated across multiple LLMs and datasets, demonstrating lower perplexity than ELSA and improved average downstream accuracy in the main comparisons.
Critical weaknesses
  1. Limited methodological novelty. ETAPS mainly combines existing techniques: the cubic pruning schedule follows Zhu and Gupta [1], weight–gradient importance scoring was used in SNIP [2], and sparse-delta adaptation with gradient-guided support updates was explored in SEFT [3]. Overall, the method appears to be an incremental combination of existing pruning and adaptation ideas.
  2. There is a lack of comparison with highly relevant baseline methods. Although the paper discusses "prune-and-tune" and WIP, it does not include experimental comparisons against these iterative pruning approaches. Without such comparative analysis against these highly relevant methods, I cannot determine whether the proposed framework truly offers a significant advantage over existing iterative pruning workflows.
  3. The experiments include OPT, Gemma-2, LLaMA-2, LLaMA-3.2, and the original DeepSeek LLM, but omit newer model families and MoE architectures. Since the effectiveness of global gradient-based selection and sparse adaptation may depend on the underlying architecture, additional results on models such as Qwen3.8-27B and Qwen3.6-35B-A3B are needed to better establish generalizability.
  4. The evaluation mainly uses WikiText-2/C4 perplexity and seven classification tasks, without code-generation or mathematical-reasoning benchmarks such as HumanEval [4] and MATH-500 [5]. Therefore, the current evidence does not establish whether these compressed models retain useful downstream capabilities.
  5. Improvements on downstream tasks are not always compelling. For instance, at 90% sparsity, ETAPS improves the average accuracy of LLaMA-2-7B by only 0.43 percentage points compared to ELSA (38.84 vs. 38.41). Meanwhile, under the 2:4 sparsity pattern, it performs slightly worse than MaskLLM in terms of both perplexity and accuracy (53.74 vs. 53.98).

[1] Zhu M, Gupta S. To prune, or not to prune: exploring the efficacy of pruning for model compression [J]. arXiv preprint arXiv:1710.01878, 2017.

[2] Lee N, Ajanthan T, Torr P H S. Snip: Single-shot network pruning based on connection sensitivity [C]. ICLR 2019.

[3] Xiao Q, Ansell A, Wu B, et al. Leave it to the specialist: Repair sparse LLMs with sparse fine-tuning via sparsity evolution [J]. arXiv preprint arXiv:2505.24037, 2025.

[4] HumanEval. URL: https://github.com/openai/human-eval.

[5] MATH-500. URL: https://huggingface.co/datasets/HuggingFaceH4/MATH-500.

Questions for authors
  1. How does ETAPS differ from the approach that combines existing techniques such as progressive pruning, gradient-based scoring, and sparse adaptation?
  2. Could the authors provide further comparisons with related methods, updated architectures, and results on both mathematical and coding tasks?
  3. Could the authors explain why the performance improvements are so marginal?
6 comments
AuthorsOpenReview ↗

Response for Reviewer DHXt (1/3)

We sincerely thank the reviewer for the constructive feedback. We address each point below and would be glad to continue the discussion.


Incremental combination of methods (W1, Q1)

Overall, the method appears to be an incremental combination of existing pruning and adaptation ideas.

How does ETAPS differ from the approach that combines existing techniques such as progressive pruning, gradient-based scoring, and sparse adaptation?

We agree that cubic pruning schedules [1], gradient-weighted importance [2], and sparse adaptation [3] have individual precedents. However, we respectfully disagree that ETAPS is merely an incremental combination of these techniques. Our contribution is a coupled pruning–adaptation framework that enables end-to-end gradient-guided pruning at extreme sparsity without model-sized optimizer state.

First, the combination addresses a fundamental quality–cost trade-off. Existing inexpensive methods struggle at extreme sparsity, while strong end-to-end optimization methods such as ELSA [4] require substantial model-sized state. ETAPS breaks this trade-off through a complementary interaction: gradual pruning makes a small adaptation budget effective, while sparse adaptation supplies the updated weights and gradients that guide subsequent pruning decisions. Our ablations demonstrate the importance of both components. At 90% sparsity on LLaMA-2-7B, removing adaptation increases WikiText-2 perplexity from 20.98 to 463.2 (Table 19), while direct pruning with the same adaptation budget increases it to 34.17 (Tables 5 and 13).

Second, realizing this interaction requires more than directly applying the existing techniques:

  • Zhu & Gupta [1] provide a cubic pruning schedule but do not address allocating a fixed, limited sparse-adaptation budget across stages. ETAPS introduces increasing adaptation intervals to prioritize the more challenging high-sparsity stages, improving perplexity from 23.19 to 20.98 under the same optimization budget (Table 6).
  • SNIP [2] evaluates weight–gradient sensitivity once at initialization. ETAPS instead updates importance throughout pruning using gradients from the adapted model. Notably, it averages absolute micro-batch gradients rather than averaging signed gradients, avoiding cancellation across batches. This distinction reduces perplexity from 33.22 to 20.98 at 90% sparsity (Table 7).
  • SEFT [3] repairs an already-pruned model by evolving its sparse topology. ETAPS instead constructs the pruning mask progressively, requiring delta-support reallocation to follow a strictly shrinking support. Pruned delta coordinates are reassigned only to surviving weights, while the same accumulated gradients guide both pruning and delta reallocation, without additional backward passes. Please see also Section 3.4 of the paper.

Third, this coupling delivers non-trivial practical gains. At 90% sparsity, ETAPS outperforms both ELSA variants in perplexity and downstream accuracy, while requiring 1.6$\times$ fewer GPU-hours, 2.4$\times$ less persistent state, and 2.6$\times$ lower peak GPU memory than even the memory-efficient ELSA-L under the same micro-batch size (see our response to Reviewer ba3F).

Furthermore, to our knowledge, this combination has not been explored in prior work at extreme sparsity. If the reviewer has specific prior work in mind, we would be happy to provide a conceptual comparison.

AuthorsOpenReview ↗

Response for Reviewer DHXt (2/3)

Marginal improvements (W5, Q3)

Could the authors explain why the performance improvements are so marginal?

For instance, at 90% sparsity, ETAPS improves the average accuracy of LLaMA-2-7B by only 0.43 percentage points compared to ELSA (38.84 vs. 38.41). Meanwhile, under the 2:4 sparsity pattern, it performs slightly worse than MaskLLM in terms of both perplexity and accuracy (53.74 vs. 53.98).

We agree that the downstream accuracy gain on LLaMA-2-7B at 90% sparsity is modest. However, this isolated case is not representative of the broader improvements across models and sparsity levels. ETAPS improves average accuracy over ELSA in all six comparisons in Table 1, with five showing gains of 2.46–6.93 percentage points. On LLaMA-2-13B alone, the gains are 5.99, 6.93, and 4.57 points at 70%, 80%, and 90% sparsity. The highlighted 7B case occurs at extreme sparsity, where both methods suffer substantial accuracy degradation relative to the dense model (38.41%/38.84% vs. 60.21%). Even here, ETAPS reduces WikiText-2/C4 perplexity by 25%, demonstrating substantial language-modeling improvements despite the smaller downstream gain.

For 2:4 sparsity, ETAPS is slightly behind MaskLLM [5] (0.24 points in accuracy, with similarly small perplexity differences). However, MaskLLM is a specialized N:M pruning method trained using 64 A100 GPUs and 1,280 GPU-hours, whereas ETAPS achieves comparable quality on a single H200 with approximately 30$\times$ fewer training tokens. Thus, ETAPS remains competitive with a specialized method while supporting broader sparsity patterns at substantially lower optimization cost.

Evaluation on iterative pruning baselines (W2, Q2)

Although the paper discusses "prune-and-tune" and WIP, it does not include experimental comparisons against these iterative pruning approaches.

We thank the reviewer for highlighting these relevant baselines. To address this concern, we have extended our evaluation to Prune-and-Tune (P&T) [7] and WIP [6] on OPT-1.3B at 70%–90% sparsity, comparing perplexity, downstream accuracy, runtime, and peak GPU memory on an NVIDIA H200 NVL.

SparsityMethodWiki-2 ↓\downarrowC4 ↓\downarrowARC-C ↑\uparrowARC-E ↑\uparrowBoolQ ↑\uparrowHellaSwag ↑\uparrowOBQA ↑\uparrowRTE ↑\uparrowWinoGrande ↑\uparrowDownstream avg. ↑\uparrowWall-clock (h) ↓\downarrowPeak memory (GiB) ↓\downarrow
0%Dense14.6116.0624.4056.4857.2241.2924.2051.9958.7244.90––
70%P&T25.3522.3920.3946.8453.5534.6217.6053.0757.1440.460.5155.30
70%WIP94.3783.4417.9238.4762.2328.7313.0053.0753.9938.200.025.80
70%ETAPS20.2620.3321.5950.1761.5636.7420.6053.0756.6742.910.3534.40
80%P&T40.5530.0118.8641.9658.7231.3116.0054.1552.2539.030.5455.30
80%WIP1,740.43798.1318.6929.1757.0026.1312.2054.8750.7535.540.025.80
80%ETAPS29.4026.6119.1144.7462.1732.4818.6053.0752.1740.340.3434.40
90%P&T86.7945.4619.4535.5259.7627.2412.2053.4349.9636.790.5655.30
90%WIP9,462.644,958.0420.8225.8839.3025.7313.4052.7150.3632.600.035.80
90%ETAPS54.6442.3018.0938.1762.1428.3115.2052.7151.2237.980.3434.40

ETAPS outperforms both methods in perplexity and average downstream accuracy at every sparsity level. Compared with P&T, it also reduces runtime by 31%–39% and peak GPU memory by 38%. WIP's substantially lower cost comes from its fundamentally different setting: it performs iterative pruning without retraining, but suffers severe quality degradation at high sparsity. We will clarify this distinction in Appendix A, where WIP was previously grouped with iterative pruning methods involving retraining.

AuthorsOpenReview ↗

Response for Reviewer DHXt (3/3)

Evaluation on other architectures such as MoE (W3, Q2)

… but omit newer model families and MoE architectures

… additional results on models such as Qwen3.8-27B and Qwen3.6-35B-A3B are needed to better establish generalizability.

Our work focuses on unstructured and N:M weight pruning of dense Transformer LLMs, consistent with the setting studied by established methods such as SparseGPT [8], Wanda [9], ALPS [14], ELSA [4], SEFT [3], and MaskLLM [5]. Within this setting, ETAPS is evaluated across five model families (OPT, Gemma-2, LLaMA-2, LLaMA-3.2, and DeepSeek), spanning 125M–70B parameters. This provides substantial evidence of generalizability across the model architectures and scales targeted by this line of work.

The suggested Qwen models, however, introduce substantially different architectural considerations. Qwen3.8-27B combines Gated DeltaNet [10] linear attention with conventional attention in a multimodal backbone, while Qwen3.6-35B-A3B additionally introduces sparse MoE routing. These differences present challenges for dense-LLM pruning methods more broadly: hybrid linear-attention models involve recurrent state dynamics, MoE architectures introduce routing-dependent expert activation and specialization [12, 13], and multimodal models require preserving vision-language alignment [11]. Addressing these challenges warrants architecture-specific evaluation protocols and appropriate baselines, representing a distinct research avenue for the broader pruning literature rather than a direct extension of conventional dense Transformer pruning. We will clarify this scope in the revised manuscript.

Evaluation on reasoning downstream tasks (W4, Q2)

The evaluation mainly uses WikiText-2/C4 perplexity and seven classification tasks, without code-generation or mathematical-reasoning benchmarks such as HumanEval and MATH-500. Therefore, the current evidence does not establish whether these compressed models retain useful downstream capabilities.

Similar to the architecture concern addressed above, our evaluation follows the established setting of general-purpose LLM weight pruning, where WikiText-2/C4 perplexity and standard zero-shot downstream benchmarks are the primary evaluation metrics, as in SparseGPT [8], Wanda [9], ALPS [14], SparseLLM [15], and ELSA [4]. Beyond perplexity, we evaluate seven downstream tasks spanning science question answering, reading comprehension, commonsense reasoning, and natural language inference.

While HumanEval and MATH-500 would provide additional insights, we respectfully disagree that their absence makes the current evaluation insufficient. Evaluating these specialized generative capabilities would extend the scope of our study beyond the established benchmark protocol for general-purpose LLM pruning. We will clarify this scope in the revised manuscript.

Given that these benchmarks are not part of the standard evaluation protocols of the closest baselines, we would appreciate it if the reviewer could point us to directly comparable LLM pruning papers reporting HumanEval or MATH-500 results, so that we can compare against them under the corresponding protocols.


References

[1] Zhu and Gupta. To prune, or not to prune: Exploring the efficacy of pruning for model compression. ICLR Workshop 2018.

[2] Lee et al. SNIP: Single-shot network pruning based on connection sensitivity. ICLR 2019.

[3] Xiao et al. Leave it to the specialist: Repair sparse LLMs with sparse fine-tuning via sparsity evolution. arXiv 2025.

[4] Lee et al. The unseen frontier: Pushing the limits of LLM sparsity with surrogate-free ADMM. ICLR 2026.

[5] Fang et al. MaskLLM: Learnable semi-structured sparsity for large language models. NeurIPS 2024.

[6] Do et al. WIP: Iterative post-training pruning with weighted importance estimation for large language models. NLDB 2025.

[7] Syed et al. Prune and tune: Improving efficient pruning techniques for massive language models. ICLR Tiny Papers 2023.

[8] Frantar and Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. ICML 2023.

[9] Sun et al. A simple and effective pruning approach for large language models. ICLR 2024.

[10] Yang et al. Gated delta networks: Improving Mamba2 with delta rule. ICLR 2025.

[11] Huang et al. Structural pruning of large vision language models: A comprehensive study on pruning dynamics, recovery, and data efficiency. IJCV 2026.

[12] Lepikhin et al. GShard: Scaling giant models with conditional computation and automatic sharding. ICLR 2021.

[13] Fedus et al. Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity. JMLR 2022.

[14] Meng et al. ALPS: Improved optimization for highly sparse one-shot pruning for large language models. NeurIPS 2024.

[15] Bai et al. SparseLLM: Towards global pruning of pre-trained language models. NeurIPS 2024.

Reviewer_ba3FOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the author proposes ETAPS, which combines progressive pruning with a fixed-budget sparse trainable delta while keeping the pretrained base weights frozen. It ranks surviving weights using their magnitudes and interval-averaged absolute gradients, reallocates delta coordinates at pruning boundaries, and assigns longer adaptation intervals to later stages. Experiments evaluate 50%–90% sparsity across six models, extend to 95%–99% on LLaMA-2-7B, and demonstrate execution on models up to 70B. The paper reports lower perplexity than the evaluated trained and one-shot baselines, together with reduced persistent optimization state and aggregate GPU-hours relative to standard ELSA.

Critical strengths

S1: The quality–cost improvement is meaningful

S2: The controlled ablations substantiate the central recipe rather than merely its final performance

Critical weaknesses

W1: The headline memory comparison does not include the closest memory-efficient competitor. Table 4 compares against standard ELSA, whereas ELSA-L is evaluated only for 27B model quality in Table 3; this leaves the cost–quality advantage over the low-memory variant unresolved. Moreover, the reported 5× reduction concerns aggregate persistent state, explicitly excluding activations and transient buffers, so it does not establish the reduction in peak VRAM or minimum hardware requirements.

W2: Capability preservation is less well established than perplexity improvement at the most aggressive settings. At 90% sparsity on LLaMA-2-7B, mean zero-shot accuracy improves only from 38.41% to 38.84%, compared with 60.21% for the dense model; the 95%–99% and 27B–70B evaluations primarily report perplexity. Together with the single-run evaluation, this limits confidence that the strongest compression settings preserve useful downstream capabilities, even though the perplexity gains are substantial.

Questions for authors

Q1: Why is ELSA-L absent from Table 4? Do existing runs provide a same-model comparison of perplexity, persistent state, peak VRAM, and GPU-hours against this variant?

Q2: What precisions are used for the base weights, gradients, optimizer states, and auxiliary variables in each Table 4 configuration? How much do activations and transient buffers add to the reported persistent-state numbers?

Q3: Do existing evaluations include downstream results at 95%–99% sparsity or on the 27B–70B models, and are uncertainty estimates available for the small 7B accuracy improvement at 90%? I am asking about available results rather than requesting a new benchmark suite during rebuttal.

Q4: Could you clarify how ETAPS relates to SparseLLM (Bai et al., NeurIPS 2024), which makes global pruning tractable through coordinated subproblems, and LoRAPrune (Zhang et al., Findings of ACL 2024), which combines iterative pruning with parameter-efficient adaptation? A conceptual comparison would suffice; no additional experiments are requested. I also recommend adding those references to the related work of the paper.

4 comments
AuthorsOpenReview ↗

Response for Reviewer ba3F (1/3)

We sincerely thank the reviewer for the constructive review, and for recognizing both the meaningful quality–cost improvement of ETAPS and the controlled ablations supporting its design. We address each point below and would be glad to continue the discussion.


Efficiency evaluation reporting (W1, Q1, Q2)

Table 4 compares against standard ELSA, whereas ELSA-L is evaluated only for 27B model quality in Table 3; this leaves the cost–quality advantage over the low-memory variant unresolved.

Why is ELSA-L absent from Table 4? Do existing runs provide a same-model comparison of perplexity, persistent state, peak VRAM, and GPU-hours against this variant?

We mainly focused on standard ELSA in Table 4 because it achieves better model quality, whereas ELSA-L [1] reduces memory overhead through quantized optimizer and auxiliary states, at some cost to quality. However, we agree that a direct cost–quality comparison with ELSA-L is important and have therefore evaluated it on LLaMA-2-7B at 90% sparsity. For a like-for-like comparison of peak memory, we additionally report ETAPS$^\dagger$ with a reduced micro-batch of 1, matching ELSA-L. All runs use the same effective batch size of 8 on NVIDIA H200 NVL GPUs.

MethodWiki-2 ↓\downarrowC4 ↓\downarrowDownstream avg. ↑\uparrowGPUsMicro-batchWall-clock (h) ↓\downarrowGPU-h ↓\downarrowPersistent memory (GiB) ↓\downarrowPeak memory (GiB) ↓\downarrow
ELSA27.9823.2038.41420.813.24132.29179.92
ELSA-L28.5824.6336.10113.273.2762.66101.00
ETAPS20.9817.3938.84181.661.6626.40116.08
ETAPS$^\dagger$20.9517.3739.30112.082.0826.4038.59

ELSA-L reduces the persistent memory of standard ELSA from 132.29 to 62.66 GiB and its peak memory from 179.92 to 101.00 GiB through quantization, with a modest degradation in perplexity and downstream accuracy. Notably, ETAPS$^\dagger$ outperforms both ELSA variants in perplexity and downstream accuracy, while requiring 1.6$\times$ fewer GPU-hours, 2.4$\times$ less persistent memory, and 2.6$\times$ less peak memory than even ELSA's memory-efficient variant, ELSA-L.

Moreover, the reported 5× reduction concerns aggregate persistent state, explicitly excluding activations and transient buffers, so it does not establish the reduction in peak VRAM or minimum hardware requirements.

We have added peak GPU memory measurements in the table above, accounting for activations and transient buffers in addition to persistent state. Under the same micro-batch size and GPU settings, ETAPS$^\dagger$ peaks at only 38.59 GiB, compared with 101.00 GiB for ELSA-L.

What precisions are used for the base weights, gradients, optimizer states, and auxiliary variables in each Table 4 configuration? How much do activations and transient buffers add to the reported persistent-state numbers?

We provide a breakdown of the persistent memory by component and precision for ETAPS, ELSA, and ELSA-L, together with the additional memory consumed by activations and transient buffers. The latter accounts for the difference between the reported persistent state and peak GPU memory. For all listed methods, gradients are computed under bf16 mixed precision and stored in fp32 buffers, which are counted as transient memory.

ETAPS (1 GPU)PrecisionGiB
Model weights (frozen)bf1612.55
Dense saliency accumulatorbf1612.05
Delta valuesfp320.30
Delta indicesint320.30
Optimizer state (AdamW moments)fp320.60
Per coordinate LR factor / step countersfp32/int320.60
Total persistent26.40
Transient (Activation, other) micro-batch 1 (minimal requirement)12.19
Transient (Activation, other) micro-batch 8 (for throughput)89.68
ELSA (4 GPUs, FSDP full sharding)PrecisionGiB
Model/master weightsfp3225.10
Adam first and second momentsfp3250.21
ADMM auxiliary variable zfp3224.12
ADMM dual variable ufp3224.12
Maskbool6.03
Other live allocated tensor2.71
Total persistent132.29
Transient (Activations and other) micro-batch 247.63
ELSA-L (1 GPU)PrecisionGiB
Model/master weightsfp3225.10
Adam first and second moments (8-bit, one fp32 scale per 256 values)int8 + fp3212.75
ADMM dual variable ubf1612.06
ADMM auxiliary variable z (tensor-wise scale)fp8 (e4m3)6.03
Maskbool6.03
Other live allocated tensor0.69
Total persistent62.66
Transient (Activations and other), micro-batch 138.34
AuthorsOpenReview ↗

Response for Reviewer ba3F (2/3)

Downstream accuracy reporting (W2, Q3)

Do existing evaluations include downstream results at 95%–99% sparsity ..., and are uncertainty estimates available for the small 7B accuracy improvement at 90%?

Below, we report downstream results of ETAPS on LLaMA-2-7B at 95% and 99% sparsity, together with 90% for reference. All values are means and standard deviations over 3 runs with different seeds.

SparsityWiki-2 ↓\downarrowC4 ↓\downarrowARC-C ↑\uparrowARC-E ↑\uparrowBoolQ ↑\uparrowHellaSwag ↑\uparrowOBQA ↑\uparrowRTE ↑\uparrowWinoGrande ↑\uparrowDownstream avg. ↑\uparrow
90%20.90±0.1420.90 \pm 0.1417.40±0.0317.40 \pm 0.0319.91±0.7719.91 \pm 0.7744.40±0.4444.40 \pm 0.4459.86±4.0159.86 \pm 4.0132.96±0.2432.96 \pm 0.2417.00±0.6017.00 \pm 0.6052.83±0.2152.83 \pm 0.2149.83±1.8249.83 \pm 1.8239.54±0.6639.54 \pm 0.66
95%28.16±0.3528.16 \pm 0.3522.04±0.0722.04 \pm 0.0718.49±0.6518.49 \pm 0.6536.84±0.2336.84 \pm 0.2360.98±1.2060.98 \pm 1.2029.32±0.0929.32 \pm 0.0916.73±0.9516.73 \pm 0.9552.83±0.2152.83 \pm 0.2149.70±0.6649.70 \pm 0.6637.84±0.3337.84 \pm 0.33
99%46.14±1.5846.14 \pm 1.5832.82±0.3932.82 \pm 0.3917.12±0.7517.12 \pm 0.7531.20±0.5931.20 \pm 0.5961.86±0.4461.86 \pm 0.4427.32±0.1527.32 \pm 0.1512.13±0.3112.13 \pm 0.3152.71±0.3652.71 \pm 0.3650.67±1.7650.67 \pm 1.7636.14±0.4436.14 \pm 0.44

At 90% sparsity, the average accuracy of ETAPS (39.54 ±\pm 0.66 over 3 runs) exceeds that of ELSA (38.41, Table 21) by 1.1 points. At 95% and 99% sparsity, the average accuracy decreases only moderately with low variance across runs.

Do existing evaluations include downstream results ... on the 27B–70B models?

Due to compute constraints, downstream results are not available for the 67B and 70B models. For Gemma-2-27B, we report the downstream results at 90% sparsity below. At this scale, we compare against ELSA-L, as standard ELSA could not practically be run due to its compute cost.

MethodWiki-2 ↓\downarrowC4 ↓\downarrowARC-C ↑\uparrowARC-E ↑\uparrowBoolQ ↑\uparrowHellaSwag ↑\uparrowOBQA ↑\uparrowRTE ↑\uparrowWinoGrande ↑\uparrowDownstream avg. ↑\uparrow
Dense5.689.2662.5487.9284.8065.2637.6067.8779.6469.38
Magnitude4.56e124.16e1222.0125.5948.2625.5215.8053.7949.8034.40
Wanda15,428.7011,082.3519.0325.2551.0725.7315.8052.3551.6234.41
SparseGPT404.64337.0718.8627.6552.2025.9015.2053.0748.7834.52
ELSA-L28.4729.2720.8049.6062.2035.4018.2050.2053.0041.34
ETAPS$^\dagger$16.8518.4425.6053.9663.9443.7924.0052.7160.7746.40

Overall, ETAPS achieves the highest average zero-shot accuracy (46.40%), compared with 41.34% for ELSA-L and around 34.5% for one-shot methods.

AuthorsOpenReview ↗

Response for Reviewer ba3F (3/3)

Clarifications with adjacent methods (Q4)

Could you clarify how ETAPS relates to SparseLLM (Bai et al., NeurIPS 2024) ... and LoRAPrune (Zhang et al., Findings of ACL 2024) ...

We thank the reviewer for suggesting these related methods. Below, we provide a comparison between ETAPS and these methods, which we will add to our extended literature review (Appendix A):

  • SparseLLM [2] shares with ETAPS the high-level idea of making global pruning tractable through decomposition. However, SparseLLM decomposes the model into layer-wise subproblems solved without gradients, whereas ETAPS decomposes the sparsity trajectory into nested constraints and optimizes with end-to-end gradients.
  • LoRAPrune [3] is similar to ETAPS in the general procedure, as it also interleaves pruning with a parameter-efficient adaptation mechanism and uses gradient information to estimate importance. However, since LoRA adapters break sparsity when merged, LoRAPrune mainly targets structured pruning of channels and heads, while ETAPS targets unstructured and N:M sparsity. Moreover, LoRAPrune estimates the importance of the pretrained weights from LoRA gradients, while ETAPS uses the gradients of the effective weights, accumulated in a single dense (but moment-free) buffer.

References

[1] Lee et al. The unseen frontier: Pushing the limits of LLM sparsity with surrogate-free ADMM. ICLR 2026.

[2] Bai et al. SparseLLM: Towards global pruning of pre-trained language models. NeurIPS 2024.

[3] Zhang et al. LoRAPrune: Structured pruning meets low-rank parameter-efficient fine-tuning. Findings of ACL 2024.

Reviewer_ba3FOpenReview ↗

Response to Author Rebuttal

I appreciate the authors for the comprehensive and detailed response, which addressed most of my previous concerns. I would keep my initial score of 3. Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.