- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes ASAF, combining QuaRot-based low-bit quantization with sensitivity-aware sparsity allocation. A two-stage dynamic programming method determines consecutive layer groups and their sparsity rates under an accuracy budget. Experiments report modest speedups and additional memory savings over QuaRot.
Critical strengths
Optimizing sparsity allocation on an already quantized model is practically relevant, and the explicit accuracy budget provides a clear optimization objective. The paper also evaluates perplexity, downstream accuracy, inference throughput, and memory consumption across multiple model sizes.
Critical weaknesses
-
Limited gains over QuaRot despite accuracy degradation. The headline 3.63× prefill speedup is measured against FP16, while QuaRot already achieves 3.36× (Figure 4). ASAF adds only about 8% in this single-block benchmark, and just 1.2%–3.5% in end-to-end throughput (Table 3). Meanwhile, Llama-2-7B perplexity increases from 6.10 to 6.14, and average zero-shot accuracy decreases by 0.63 percentage points (from 65.64% to 65.01%). The practical benefit of this trade-off is weak without a matched accuracy–latency–memory Pareto comparison.
-
Consecutive grouping and two-stage optimization lack convincing justification. Non-uniform sparsity allocation has already been explored in BESA (ICLR 2024), DSA (NeurIPS 2024), and related work. ASAF motivates consecutive grouping through a smooth depth-wise sensitivity profile, but this does not justify forcing adjacent layers to share sparsity rates. For a fixed group count smaller than the number of layers, grouping couples the sparsity decisions of adjacent layers and reduces allocation flexibility. The paper does not show that this restriction improves either search efficiency or the accuracy–sparsity trade-off relative to independent per-layer allocation. Moreover, the ablation only compares against a uniform-sparsity heuristic, rather than independent per-layer allocation, single-stage DP, or greedy search. The contribution of the two-stage design therefore remains unclear.
-
Comparisons with existing joint compression methods are incomplete. Relevant works include JSQ (ICML 2024), SLiM (ICML 2025), and Optimal Brain Restoration (ICLR 2026). Although Table 5 includes DSA, SLiM, OWL, and SpinQuant, it mainly compares accuracy without matching effective precision, sparsity pattern, compression ratio, or hardware efficiency. The experiments do not show whether ASAF improves the accuracy–efficiency frontier over existing joint compression methods. Evaluation is also restricted to the Llama family.
-
The optimization formulation and hardware realization are not adequately supported. Appendix A.11.4 describes magnitude-based pruning, while Appendix A.13 attributes acceleration to smaller GEMM dimensions using unchanged QuaRot kernels. The pruning granularity and physical matrix reduction are not specified. Structured channel removal requires consistent coupled projections and compatibility with Hadamard transforms, quantization groups, and INT4 kernel alignment constraints. The DP guarantee is also questionable: Equation (2) constructs an additive budget by summing independently measured perplexity increases, which cannot automatically guarantee that the jointly pruned model satisfies the target accuracy budget. Furthermore, does not track the covered layer position, leaving the validity of consecutive partitions unclear. The claimed optimality is therefore not established by the presented formulation.
Questions for authors
- How does ASAF compare with independent per-layer allocation, single-stage DP, and greedy allocation under matched budgets?
- What is the actual pruning structure, and how is GEMM dimensionality reduced while preserving QuaRot kernel compatibility?
- How well does the additive budget proxy in Equation (2) hold under joint pruning, and how does enforce valid layer partitions?
- Can the authors provide matched comparisons with existing joint compression methods and report the offline search cost?
4 comments
(continued)
Summarized W4, Q2 and Q3: Pruning structure, additive budget and the coarse DP state
A4: We answer the three parts in turn. Pruning structure. The pruning unit is a block of 128 consecutive input channels of each of the seven projections, the width of an INT4 kernel tile and, in the 128G rows, of a quantization group, ranked within a group by L2 norm per weight after GPTQ [8] and removed until the group's rate is reached. This shrinks the reduction dimension of the GEMM, leaves output shapes, the residual stream and the rotation unchanged, and keeps the same INT4 kernels.
Additive budget. The search enforces the sum of tabulated degradations, each defined with one group sparsified and tabulated. Table R3 shows that the joint degradation on the calibration subset stays at or below this sum at all four budgets, for example 0.986% against 0.995% at the 1% budget, and the joint test degradation stays below each budget.
Coarse DP state and optimality. The coarse pass selects and narrows the rate intervals; the fine-grained pass, whose state carries the layer index, then splits the layers into consecutive, disjoint groups and returns the minimum-FLOP allocation within the discretized two-phase search space under the 1% budget (Page 1, Lines 24 to 25; Page 2, Lines 71 to 72 and 84 to 85; Page 3, Lines 154 to 155).
Table R3: Tabulated budget against the degradation of the jointly sparsified Llama-2-7B.
| Allocation | tabulated (%) | Joint PPL, calibration subset (%) | Joint PPL, WikiText-2 test (%) |
|---|---|---|---|
| ASAF (Ours), = 0.5% | 0.494 | 0.483 | 0.33 |
| ASAF (Ours), = 1.0% | 0.995 | 0.986 | 0.82 |
| ASAF (Ours), = 1.5% | 1.485 | 1.403 | 1.31 |
| ASAF (Ours), = 2.0% | 1.985 | 1.857 | 1.97 |
Following your suggestions, we have specified the pruning structure in Section 1 (Lines 72–75) and Appendix A.11.4 (Lines 1697–1745), revised the sensitivity definition in Section 3 (Lines 142–146) and the coarse pass in Section 4 (Lines 204–212), and added these results to Appendix A.23 as Table 27 (Lines 2467–2471).
References
[1] Saleh Ashkboos et al. Quarot: Outlier-free 4-bit inference in rotated llms. NeurIPS, 2024.
[2] Zechun Liu et al. Spinquant: LLM quantization with learned rotations. ICLR, 2025.
[3] Peng Xu et al. BESA: Pruning large language models with blockwise parameter-efficient sparsity allocation. ICLR, 2024.
[4] Lujun Li et al. Discovering sparsity allocation for layer-wise pruning of large language models. NeurIPS, 2024.
[5] Jinyang Guo et al. Compressing large language models by joint sparsification and quantization. ICML, 2024.
[6] Hang Guo et al. Optimal brain restoration for joint quantization and sparsification of llms. ICLR, 2026.
[7] Lu Yin et al. Outlier weighed layerwise sparsity (OWL): A missing secret sauce for pruning LLMs to high sparsity. ICML, 2024.
[8] Elias Frantar et al. Optq: Accurate quantization for generative pre-trained transformers. ICLR, 2023.
Best Wishes,
The Authors of Paper 52594
Thanks for your reply and the new experiments. I have raised my score to 2.
Dear Reviewer 6Z6g,
Thank you for your reply and for raising your score.
Our rebuttal added the comparison with the per-layer, greedy and single-stage allocations under the same 1% budget, ASAF on top of SpinQuant, and the pruning structure with the check of the additive budget against the joint degradation. As suggested, we will keep these experiments in the final version.
Thank you again for your constructive feedback.
Best Wishes,
The Authors of Paper 52594
Response to Reviewer 6Z6g
Dear Reviewer 6Z6g:
Thank you for recognizing the task value and the method design of our work. We have responded point by point to your concerns regarding the accuracy trade-off, the two-stage design, matched comparisons, and the pruning structure. In the marked copy appended to the revised PDF (pages 47–97), the revisions made in response to your comments are tagged with @Reviewer 6Z6g. We hope our responses resolve your concerns and look forward to further discussion.
Summarized W1: Gains over QuaRot against the accuracy cost
A1: The degradation budget δmax sets the trade-off between the gain over QuaRot [1] and the accuracy cost, and the search minimizes FLOPs within it. At the 1% budget used throughout the paper, ASAF raises the single-block prefill speedup on Llama-2-70B from 3.33$\times$ to 3.61$\times$, while on Llama-2-7B the perplexity rises from 6.10 to 6.15 and the zero-shot average falls from 65.64% to 65.08%. Tables 14 and 15 of the revised paper trace this trade-off from 5.2% average sparsity at δmax=0.5% to 10.1% at 2.0% on Llama-2-7B, and Tables R1 and R2 below compare ASAF at the same budget with other allocations and on top of SpinQuant [2].
Following your suggestions, we have revised Appendix A.17 (Lines 2140–2170) to make this clearer.
Summarized W2 & Q1: Consecutive grouping and the two-stage design
A2: The novelty is the formulation: ASAF minimizes FLOPs within a degradation budget relative to the quantized base, while BESA [3] and DSA [4] optimize reconstruction error or accuracy at a given overall sparsity. Consecutive grouping follows the smooth variation of sparsity sensitivity with depth, and it lets the degradation of a group pruned together be tabulated directly. Following your suggestion, we further compare ASAF with three allocations of the same problem (i.e., a per-layer DP, greedy allocation and a single-stage DP) on the same tables and the same 1% budget. Table R1 shows that ASAF reaches lower FLOPs than the per-layer and greedy allocations and nearly the same FLOPs as the single-stage DP, 0.9269 against 0.9261, with 2,942,400 DP transitions against 95,027,200, counts computed from the complexity expressions of Appendix A.13.1 (Table 25). A single-stage DP that does not track the group count needs 2,969,600 transitions, and the saving the paper states is against the group-tracking formulation.
Table R1: Allocations of the same constrained problem on QuaRot W4A4 Llama-2-7B, the same δmax and the same tabulated ξ.
Following your suggestions, we have added these results to Appendix A.23 as Table 25 (Lines 2444–2449), and revised the coarse pass and the grouping rationale in Section 4 (Lines 204–242) and Appendix A.3 (Lines 1007–1044).
Summarized W3 & Q4: Matched comparison with joint compression methods
A3: ASAF decides where sparsity goes in a given quantized model, so the matched comparison applies it on top of a strong quantizer; JSQ [5], SLiM and Optimal Brain Restoration [6] prune and quantize jointly in their own formats. Following your suggestion, we apply ASAF on top of SpinQuant [2] (W4A4KV4 on Llama-2-7B, base values as reported in its paper) under the same 1% budget. Table R2 shows that ASAF keeps the average accuracy within 1% of SpinQuant's while reducing its FLOPs, with the eight-task average falling from 64.0% to 63.5% at 0.9250 of SpinQuant's FLOPs. We have removed the unmatched DSA [4], SLiM and OWL [7] rows of the earlier Table 5.
Table R2: SpinQuant and QuaRot, each without and with ASAF on top.
Following your suggestions, we have revised Table 5 and its paragraph in Section 5.3 (Lines 387–399), added these results to Appendix A.23 as Table 26 (Lines 2456–2461), and given the per-benchmark results in Appendix A.21 (Lines 2326–2342).