- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces ETAPS, a method for pruning LLMs to high sparsity while keeping optimizer memory small. It gradually increases sparsity and adapts a small sparse delta between pruning steps, with the pretrained weights kept frozen. The next pruning mask is selected globally using weight magnitude multiplied by the mean absolute gradient collected during adaptation. The method also assigns longer adaptation intervals to later pruning stages and reallocates delta coordinates among surviving weights. Experiments show consistent perplexity improvements over ELSA across several model families, with lower persistent memory and fewer GPU-hours in the reported 7B comparison. The main contribution is an effective combination of gradual pruning and sparse adaptation, supported by extensive empirical evaluation.
Critical strengths
-
The paper addresses a useful practical problem: obtaining good models at high sparsity without maintaining full-model optimizer and auxiliary state. The results are consistent across model families and sparsity levels. For example, at 90% sparsity on LLaMA-2-7B, ETAPS improves WikiText-2/C4 perplexity from 27.98/23.20 to 20.98/17.39 over ELSA, while using 26.40 rather than 132.29 GiB of persistent state and 1.66 rather than 3.24 GPU-hours.
-
The ablations are extensive and generally well controlled. Direct and progressive pruning use the same adaptation budget, and the paper separately examines adaptation intervals, saliency scores, schedule shape, and delta capacity. The direct-versus-progressive comparison across three models is particularly useful evidence that the pruning trajectory matters at high sparsity.
-
The method is clearly described, including implementation details that matter for reproduction. The paper distinguishes dense backward computation from sparse optimizer state, reports the difference between persistent and peak memory, and explains how its sparse adaptation differs from SEFT. The experiments up to 70B parameters also demonstrate that the procedure can be executed at a substantial scale.
Critical weaknesses
-
The novelty is mainly in combining existing components. Gradual pruning, sparse fine-tuning, gradient-guided support allocation, and magnitude-gradient saliency are all established ideas, as the paper itself discusses. The ablations support the combination, but do not fully establish why sparse-coordinate adaptation is the best choice for the progressive procedure. A progressive baseline using a sparsity-preserving LoRA-style update under a matched adaptation budget would help separate the benefit of the trajectory from that of the chosen parameterization. The current LoRA comparison starts after one-shot pruning.
-
The efficiency evidence is narrower than the quality evaluation. The detailed timing and memory comparison is for LLaMA-2-7B at 90% sparsity, and the reported state excludes activations and transient buffers. ETAPS still performs dense backward computation, and its wall-clock time is longer than the four-GPU ELSA run despite lower aggregate GPU-hours. Peak memory and cost-quality curves across optimization budgets, including ELSA-L where feasible, would make the practical trade-off clearer. End-to-end sparse inference latency is also not evaluated.
-
The experiments use a single run, and large perplexity gains do not always translate into similarly large downstream gains. At 90% sparsity on LLaMA-2-7B, average zero-shot accuracy improves only from 38.41 to 38.84 over ELSA. Results at 95% and 99% sparsity mainly establish better perplexity; they do not show how much useful downstream capability remains. Repeated runs and downstream evaluation at these sparsity levels would strengthen the extreme-sparsity claim.
-
The 67B/70B experiments demonstrate feasibility, but the comparison at that scale is limited to one-shot methods with much smaller data and compute budgets. The Gemma-2-27B comparison against ELSA-L is more informative. At 67B/70B, a trained recovery baseline, or at least a fuller resource breakdown, would help assess whether the favorable cost-quality trade-off carries over.
Questions for authors
Please clarify and address the concerns raised in the Weaknesses section
6 comments
Response for Reviewer x9jt (2/4)
Efficiency evaluation reporting (W2)
ETAPS still performs dense backward computation, and its wall-clock time is longer than the four-GPU ELSA run despite lower aggregate GPU-hours.
We acknowledge that ETAPS still performs dense backward computation, as already noted in our Limitations section. However, this is also required by ELSA and ELSA-L; ETAPS instead reduces optimizer and auxiliary-state overhead, achieving substantially lower persistent memory.
Regarding wall-clock time, the reported comparison involves different GPU configurations: ETAPS uses a single GPU (1.66 h), whereas ELSA uses four GPUs (0.81 h). In the single-GPU comparison with ELSA-L, ETAPS is nearly 2$\times$ faster (1.66 vs. 3.27 h) while achieving better perplexity and downstream accuracy. Moreover, ETAPS requires approximately half the aggregate GPU-hours of both ELSA variants.
Peak memory and cost-quality curves across optimization budgets, including ELSA-L where feasible, would make the practical trade-off clearer.
... and the reported state excludes activations and transient buffers.
We have added peak memory to the comparison on LLaMA-2-7B at 90% sparsity, together with ELSA-L. Since peak memory depends on the micro-batch, we report ETAPS with the standard micro-batch of 8 used in Table 4 and additionally ETAPS$^\dagger$ with a micro-batch of 1, matching ELSA-L. Reducing the micro-batch lowers peak memory at the cost of runtime, while persistent state is unchanged. All runs use NVIDIA H200 NVL GPUs with an effective batch size of 8.
| Method | Wiki-2 | C4 | Downstream avg. | GPUs | Micro-batch | Wall-clock (h) | GPU-h | Persistent memory (GiB) | Peak memory (GiB) |
|---|---|---|---|---|---|---|---|---|---|
| ELSA | 27.98 | 23.20 | 38.41 | 4 | 2 | 0.81 | 3.24 | 132.29 | 179.92 |
| ELSA-L | 28.58 | 24.63 | 36.10 | 1 | 1 | 3.27 | 3.27 | 62.66 | 101.00 |
| ETAPS | 20.98 | 17.39 | 38.84 | 1 | 8 | 1.66 | 1.66 | 26.40 | 116.08 |
| ETAPS$^\dagger$ | 20.95 | 17.37 | 39.30 | 1 | 1 | 2.08 | 2.08 | 26.40 | 38.59 |
The difference between peak and persistent memory reflects activations and transient buffers, which account for 12.19 GiB for ETAPS$^\dagger$, compared with 38.34 GiB for ELSA-L. Notably, ETAPS$^\dagger$ achieves the best quality while requiring 2.6$\times$ less peak memory, 2.4$\times$ less persistent memory, and 1.6$\times$ fewer GPU-hours than ELSA-L.
Regarding cost-quality curves, we agree that they would be helpful. However, we note that producing them would require running every method at multiple budgets, which exceeds our compute given the number of baselines. Moreover, we believe the reported results already characterize the practical trade-off, as the evaluated methods span a wide range of costs, from one-shot pruning in under 0.01 GPU-hours to trained methods requiring several GPU-hours, each reported in the configuration in which it is typically used.
End-to-end sparse inference latency is also not evaluated.
We thank the reviewer for this important suggestion. Below, we evaluate end-to-end sparse inference of LLaMA-2-7B pruned by ETAPS, using the MACKO sparse-matrix vector multiplication (SpMV) kernel [2] on a single consumer-grade RTX 3090 GPU with batch size 1 and 100 generated tokens. The results are reported relative to the dense model.
| Sparsity | Decode speed | Total-latency speed | GPU allocated-memory reduction | Weight storage (GB) |
|---|---|---|---|---|
| Dense | 1.00$\times$ | 1.00$\times$ | 1.00$\times$ | 13.48 |
| 50% | 1.32$\times$ | 1.29$\times$ | 1.53$\times$ | 8.69 |
| 60% | 1.55$\times$ | 1.51$\times$ | 1.87$\times$ | 7.09 |
| 70% | 1.91$\times$ | 1.84$\times$ | 2.40$\times$ | 5.48 |
| 80% | 2.41$\times$ | 2.29$\times$ | 3.36$\times$ | 3.89 |
| 90% | 3.26$\times$ | 3.05$\times$ | 5.46$\times$ | 2.34 |
| 95% | 3.91$\times$ | 3.58$\times$ | 7.87$\times$ | 1.60 |
| 99% | 5.01$\times$ | 4.44$\times$ | 11.91$\times$ | 1.02 |
As observed, the efficiency gains grow with sparsity. At 90% sparsity, the sparse model decodes 3.26$\times$ faster, achieves a 3.05$\times$ end-to-end speedup, and requires 5.46$\times$ less GPU memory, with weights shrinking from 13.5 GB to 2.3 GB. Moreover, since ETAPS achieves the best quality across 90%–99% sparsity (Table 2), these efficiency gains can be realized with less quality degradation than with existing methods. We will include this evaluation in the revised manuscript.
Response for Reviewer x9jt (3/4)
Downstream evaluation reporting (W3)
... large perplexity gains do not always translate into similarly large downstream gains. At 90% sparsity on LLaMA-2-7B, average zero-shot accuracy improves only from 38.41 to 38.84 over ELSA.
The highlighted LLaMA-2-7B result occurs at 90% sparsity, where both methods have already suffered substantial downstream accuracy degradation (38.41% for ELSA and 38.84% for ETAPS, compared with 60.21% for the dense model). Perplexity measures next-token prediction quality continuously, whereas downstream accuracy reflects discrete task success, so improvements need not be proportional. At extreme sparsity, challenging tasks can exhibit very low accuracy, making average downstream accuracy less sensitive to improvements that remain visible in perplexity. For instance, both methods achieve around 20% on ARC-Challenge, while ETAPS shows a clearer gain on ARC-Easy (43.98% vs. 40.53%). Importantly, this isolated case is not representative of the broader results: ETAPS improves average downstream accuracy over ELSA in all six model–sparsity comparisons in Table 1, with five showing gains of 2.46–6.93 percentage points.
The experiments use a single run ... Results at 95% and 99% sparsity mainly establish better perplexity; they do not show how much useful downstream capability remains. Repeated runs and downstream evaluation at these sparsity levels would strengthen the extreme-sparsity claim.
We thank the reviewer for this suggestion. To address both concerns, we repeated ETAPS on LLaMA-2-7B with three different seeds at 90%, 95%, and 99% sparsity, reporting the mean and standard deviation of perplexity and all seven downstream tasks below.
| Sparsity | Wiki-2 | C4 | ARC-C | ARC-E | BoolQ | HellaSwag | OBQA | RTE | WinoGrande | Downstream avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| 90% | ||||||||||
| 95% | ||||||||||
| 99% |
Overall, the run-to-run variation is small relative to the improvements of ETAPS. At 90% sparsity, ETAPS reaches 20.90 0.14 WikiText-2 perplexity, consistently outperforming ELSA at 27.98 (Table 2). In terms of downstream accuracy, the average of ETAPS at 90% (39.54 0.66) exceeds that of ELSA (38.41, Table 21), and decreases only moderately at 95% and 99% sparsity, with low variance across runs.
Response for Reviewer x9jt (1/4)
We sincerely thank the reviewer for the thoughtful review, and for recognizing that ETAPS addresses a useful practical problem with consistent results across model families and sparsity levels, supported by extensive ablations. We address each point below and would be glad to continue the discussion.
Sparse adaptation but not LoRA (W1)
We thank the reviewer for this insightful suggestion. We would like to clarify that we do not claim sparse-coordinate adaptation is the best possible parameterization, but rather a natural and effective choice for efficient adaptation under extreme sparsity. To further substantiate this choice, we replace ETAPS's sparse delta with SPP [1], an established sparsity-preserving LoRA-style adaptation method, while retaining the progressive pruning procedure. SPP adapts surviving weights through shared learnable row and column scaling factors, preserving the pruning mask. Under a similar adaptation budget, we evaluate both methods on LLaMA-2-7B at 90% sparsity with a micro-batch size of 8 on NVIDIA H200 NVL GPUs.
The results demonstrate that the adaptation parameterization matters beyond the progressive pruning trajectory. Compared with SPP, ETAPS reduces WikiText-2/C4 perplexity by 28%/32%, while achieving higher downstream accuracy, 22% shorter runtime, and 11% lower peak GPU memory.