acceptodds
Under review as a conference paper at ICLR 2027

Layer-wise Sensitivity-aware Sparsity Allocation for Efficient LLM Inference

Abstract

Large Language Model (LLM) inference presents substantial computational challenges when executed on commodity hardware, thereby necessitating the development of efficient acceleration techniques. While existing approaches predominantly focus on uniform compression strategies, they neglect the heterogeneous sensitivity patterns exhibited across different transformer layers. In this paper, we introduce the Adaptive Sparsity Allocation Framework (ASAF), which integrates rotation-based low-bit quantization with layer-wise adaptive sparsity allocation. The framework comprises two sequential phases with a dynamic programming strategy. In Phase 1, coarse-grained optimization determines the optimal number of layer groups and narrows sparsity rate search intervals. In Phase 2, fine-grained optimization determines precise consecutive layer allocation and exact sparsity rates within each group. The joint optimization of layer grouping decisions and sparsity rate assignments creates a combinatorial explosion in the solution space, rendering brute-force approaches computationally prohibitive; we employ a dynamic programming strategy that decomposes the exponential search space into manageable subproblems across both phases, achieving practical computational efficiency while guaranteeing optimality within the discretized two-phase search space. Experiments on the Llama-2 family, with additional memory and zero-shot results on Llama-3, reveal that our proposed framework sustains average benchmark accuracy degradation within 1%, concurrently achieving up to 3.63 prefill acceleration over an FP16 baseline and 12.63% memory reduction over the quantized baseline, on a single NVIDIA RTX 3090 GPU.

Primary area:
optimization
Keywords:
efficient LLM inference, layer-wise sparsity allocation, rotation-based quantization, joint sparsity and quantization, layer sensitivity, dynamic programming

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6Z6gOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ASAF, combining QuaRot-based low-bit quantization with sensitivity-aware sparsity allocation. A two-stage dynamic programming method determines consecutive layer groups and their sparsity rates under an accuracy budget. Experiments report modest speedups and additional memory savings over QuaRot.

Critical strengths

Optimizing sparsity allocation on an already quantized model is practically relevant, and the explicit accuracy budget provides a clear optimization objective. The paper also evaluates perplexity, downstream accuracy, inference throughput, and memory consumption across multiple model sizes.

Critical weaknesses
  1. Limited gains over QuaRot despite accuracy degradation. The headline 3.63× prefill speedup is measured against FP16, while QuaRot already achieves 3.36× (Figure 4). ASAF adds only about 8% in this single-block benchmark, and just 1.2%–3.5% in end-to-end throughput (Table 3). Meanwhile, Llama-2-7B perplexity increases from 6.10 to 6.14, and average zero-shot accuracy decreases by 0.63 percentage points (from 65.64% to 65.01%). The practical benefit of this trade-off is weak without a matched accuracy–latency–memory Pareto comparison.

  2. Consecutive grouping and two-stage optimization lack convincing justification. Non-uniform sparsity allocation has already been explored in BESA (ICLR 2024), DSA (NeurIPS 2024), and related work. ASAF motivates consecutive grouping through a smooth depth-wise sensitivity profile, but this does not justify forcing adjacent layers to share sparsity rates. For a fixed group count smaller than the number of layers, grouping couples the sparsity decisions of adjacent layers and reduces allocation flexibility. The paper does not show that this restriction improves either search efficiency or the accuracy–sparsity trade-off relative to independent per-layer allocation. Moreover, the ablation only compares against a uniform-sparsity heuristic, rather than independent per-layer allocation, single-stage DP, or greedy search. The contribution of the two-stage design therefore remains unclear.

  3. Comparisons with existing joint compression methods are incomplete. Relevant works include JSQ (ICML 2024), SLiM (ICML 2025), and Optimal Brain Restoration (ICLR 2026). Although Table 5 includes DSA, SLiM, OWL, and SpinQuant, it mainly compares accuracy without matching effective precision, sparsity pattern, compression ratio, or hardware efficiency. The experiments do not show whether ASAF improves the accuracy–efficiency frontier over existing joint compression methods. Evaluation is also restricted to the Llama family.

  4. The optimization formulation and hardware realization are not adequately supported. Appendix A.11.4 describes magnitude-based pruning, while Appendix A.13 attributes acceleration to smaller GEMM dimensions using unchanged QuaRot kernels. The pruning granularity and physical matrix reduction are not specified. Structured channel removal requires consistent coupled projections and compatibility with Hadamard transforms, quantization groups, and INT4 kernel alignment constraints. The DP guarantee is also questionable: Equation (2) constructs an additive budget by summing independently measured perplexity increases, which cannot automatically guarantee that the jointly pruned model satisfies the target accuracy budget. Furthermore, DPcoarse[g][b]\mathrm{DP}_{\mathrm{coarse}}[g][b] does not track the covered layer position, leaving the validity of consecutive partitions unclear. The claimed optimality is therefore not established by the presented formulation.

Questions for authors
  1. How does ASAF compare with independent per-layer allocation, single-stage DP, and greedy allocation under matched budgets?
  2. What is the actual pruning structure, and how is GEMM dimensionality reduced while preserving QuaRot kernel compatibility?
  3. How well does the additive budget proxy in Equation (2) hold under joint pruning, and how does DPcoarse[g][b]\mathrm{DP}_{\mathrm{coarse}}[g][b] enforce valid layer partitions?
  4. Can the authors provide matched comparisons with existing joint compression methods and report the offline search cost?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 6Z6g

Dear Reviewer 6Z6g:

Thank you for recognizing the task value and the method design of our work. We have responded point by point to your concerns regarding the accuracy trade-off, the two-stage design, matched comparisons, and the pruning structure. In the marked copy appended to the revised PDF (pages 47–97), the revisions made in response to your comments are tagged with @Reviewer 6Z6g. We hope our responses resolve your concerns and look forward to further discussion.


Summarized W1: Gains over QuaRot against the accuracy cost

A1: The degradation budget δmax\delta_{max} sets the trade-off between the gain over QuaRot [1] and the accuracy cost, and the search minimizes FLOPs within it. At the 1% budget used throughout the paper, ASAF raises the single-block prefill speedup on Llama-2-70B from 3.33$\times$ to 3.61$\times$, while on Llama-2-7B the perplexity rises from 6.10 to 6.15 and the zero-shot average falls from 65.64% to 65.08%. Tables 14 and 15 of the revised paper trace this trade-off from 5.2% average sparsity at δmax=0.5%\delta_{max} = 0.5\% to 10.1% at 2.0% on Llama-2-7B, and Tables R1 and R2 below compare ASAF at the same budget with other allocations and on top of SpinQuant [2].

Following your suggestions, we have revised Appendix A.17 (Lines 2140–2170) to make this clearer.


Summarized W2 & Q1: Consecutive grouping and the two-stage design

A2: The novelty is the formulation: ASAF minimizes FLOPs within a degradation budget relative to the quantized base, while BESA [3] and DSA [4] optimize reconstruction error or accuracy at a given overall sparsity. Consecutive grouping follows the smooth variation of sparsity sensitivity with depth, and it lets the degradation of a group pruned together be tabulated directly. Following your suggestion, we further compare ASAF with three allocations of the same problem (i.e., a per-layer DP, greedy allocation and a single-stage DP) on the same tables and the same 1% budget. Table R1 shows that ASAF reaches lower FLOPs than the per-layer and greedy allocations and nearly the same FLOPs as the single-stage DP, 0.9269 against 0.9261, with 2,942,400 DP transitions against 95,027,200, counts computed from the complexity expressions of Appendix A.13.1 (Table 25). A single-stage DP that does not track the group count needs 2,969,600 transitions, and the saving the paper states is against the group-tracking formulation.

Table R1: Allocations of the same constrained problem on QuaRot W4A4 Llama-2-7B, the same δmax\delta_{max} and the same tabulated ξ\xi.

MethodFLOPs vs. QuaRotΔ\DeltaPPL, test (%)Zero-shot avg.DP transitions
QuaRot1.00000.0065.64-
ASAF (Ours)0.92690.8265.082,942,400
Single-stage DP, full grid0.92610.7965.1095,027,200
Per-layer DP (G=pG = p)0.94080.5265.2992,800
Greedy0.95420.5365.28896

Following your suggestions, we have added these results to Appendix A.23 as Table 25 (Lines 2444–2449), and revised the coarse pass and the grouping rationale in Section 4 (Lines 204–242) and Appendix A.3 (Lines 1007–1044).


Summarized W3 & Q4: Matched comparison with joint compression methods

A3: ASAF decides where sparsity goes in a given quantized model, so the matched comparison applies it on top of a strong quantizer; JSQ [5], SLiM and Optimal Brain Restoration [6] prune and quantize jointly in their own formats. Following your suggestion, we apply ASAF on top of SpinQuant [2] (W4A4KV4 on Llama-2-7B, base values as reported in its paper) under the same 1% budget. Table R2 shows that ASAF keeps the average accuracy within 1% of SpinQuant's while reducing its FLOPs, with the eight-task average falling from 64.0% to 63.5% at 0.9250 of SpinQuant's FLOPs. We have removed the unmatched DSA [4], SLiM and OWL [7] rows of the earlier Table 5.

Table R2: SpinQuant and QuaRot, each without and with ASAF on top.

MethodWikiText-2 PPLZero-shot avg.FLOPs vs. basePrefill speedup vs. base
SpinQuant5.964.01.00001.000
SpinQuant + ASAF (Ours)5.9463.50.92501.049
QuaRot6.1065.641.00001.000
QuaRot + ASAF (Ours)6.1565.080.92691.047

Following your suggestions, we have revised Table 5 and its paragraph in Section 5.3 (Lines 387–399), added these results to Appendix A.23 as Table 26 (Lines 2456–2461), and given the per-benchmark results in Appendix A.21 (Lines 2326–2342).

(continued)

Summarized W4, Q2 and Q3: Pruning structure, additive budget and the coarse DP state

A4: We answer the three parts in turn. Pruning structure. The pruning unit is a block of 128 consecutive input channels of each of the seven projections, the width of an INT4 kernel tile and, in the 128G rows, of a quantization group, ranked within a group by L2 norm per weight after GPTQ [8] and removed until the group's rate is reached. This shrinks the reduction dimension of the GEMM, leaves output shapes, the residual stream and the rotation unchanged, and keeps the same INT4 kernels.

Additive budget. The search enforces the sum of tabulated degradations, each defined with one group sparsified and tabulated. Table R3 shows that the joint degradation on the calibration subset stays at or below this sum at all four budgets, for example 0.986% against 0.995% at the 1% budget, and the joint test degradation stays below each budget.

Coarse DP state and optimality. The coarse pass selects G∗G^* and narrows the rate intervals; the fine-grained pass, whose state carries the layer index, then splits the layers into consecutive, disjoint groups and returns the minimum-FLOP allocation within the discretized two-phase search space under the 1% budget (Page 1, Lines 24 to 25; Page 2, Lines 71 to 72 and 84 to 85; Page 3, Lines 154 to 155).

Table R3: Tabulated budget against the degradation of the jointly sparsified Llama-2-7B.

Allocation∑ξ\sum\xi tabulated (%)Joint Δ\DeltaPPL, calibration subset (%)Joint Δ\DeltaPPL, WikiText-2 test (%)
ASAF (Ours), δmax\delta_{max} = 0.5%0.4940.4830.33
ASAF (Ours), δmax\delta_{max} = 1.0%0.9950.9860.82
ASAF (Ours), δmax\delta_{max} = 1.5%1.4851.4031.31
ASAF (Ours), δmax\delta_{max} = 2.0%1.9851.8571.97

Following your suggestions, we have specified the pruning structure in Section 1 (Lines 72–75) and Appendix A.11.4 (Lines 1697–1745), revised the sensitivity definition in Section 3 (Lines 142–146) and the coarse pass in Section 4 (Lines 204–212), and added these results to Appendix A.23 as Table 27 (Lines 2467–2471).


References

[1] Saleh Ashkboos et al. Quarot: Outlier-free 4-bit inference in rotated llms. NeurIPS, 2024.

[2] Zechun Liu et al. Spinquant: LLM quantization with learned rotations. ICLR, 2025.

[3] Peng Xu et al. BESA: Pruning large language models with blockwise parameter-efficient sparsity allocation. ICLR, 2024.

[4] Lujun Li et al. Discovering sparsity allocation for layer-wise pruning of large language models. NeurIPS, 2024.

[5] Jinyang Guo et al. Compressing large language models by joint sparsification and quantization. ICML, 2024.

[6] Hang Guo et al. Optimal brain restoration for joint quantization and sparsification of llms. ICLR, 2026.

[7] Lu Yin et al. Outlier weighed layerwise sparsity (OWL): A missing secret sauce for pruning LLMs to high sparsity. ICML, 2024.

[8] Elias Frantar et al. Optq: Accurate quantization for generative pre-trained transformers. ICLR, 2023.


Best Wishes,

The Authors of Paper 52594

Thanks for your reply and the new experiments. I have raised my score to 2.

Dear Reviewer 6Z6g,

Thank you for your reply and for raising your score.

Our rebuttal added the comparison with the per-layer, greedy and single-stage allocations under the same 1% budget, ASAF on top of SpinQuant, and the pruning structure with the check of the additive budget against the joint degradation. As suggested, we will keep these experiments in the final version.

Thank you again for your constructive feedback.

Best Wishes,

The Authors of Paper 52594

Reviewer_rauBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes the Adaptive Sparsity Allocation Framework (ASAF), a method that combines rotation-based low-bit quantization with layer-wise adaptive sparsity allocation to improve LLM inference efficiency. The key observation is that different transformer layers exhibit varying sensitivity to sparsification, making uniform sparsity allocation suboptimal. To address this, the authors introduce a two-stage dynamic programming approach that first determines the number of consecutive layer groups and their sparsity search intervals, then optimizes the precise layer boundaries and sparsity rates under an explicit accuracy degradation constraint. Built upon QuaRot, the framework requires no additional training and aims to minimize computational FLOPs while preserving model quality. Experiments on Llama-2 and Llama-3 models demonstrate relatively small accuracy degradation, achieving up to 3.63× single-block prefill acceleration over FP16 (compared to 3.36× for QuaRot), modest end-to-end throughput improvements, and additional GPU memory savings.

Critical strengths
  1. The proposed method is well-motivated, principled, and easy to follow. The two-stage dynamic programming formulation provides a systematic approach to optimizing layer-wise sparsity allocation under an explicit accuracy constraint.

  2. The method demonstrates consistent inference efficiency improvements over existing quantization baselines, achieving additional prefill acceleration and memory savings while maintaining relatively small accuracy degradation.

Critical weaknesses
  1. Limited inference acceleration over the quantization baseline. Although the paper highlights a 3.63× prefill speedup over FP16, the QuaRot baseline already achieves 3.36×, leaving only an approximately 8% incremental improvement. More importantly, the reported end-to-end throughput improvements are limited to 1.2–3.5%, raising concerns about the practical significance of the proposed approach.

  2. Insufficient evaluation of long-form generation quality. Although the paper evaluates perplexity and several zero-shot benchmarks, these evaluations do not adequately capture potential performance degradation during long autoregressive generation. Modern reasoning models can generate trajectories exceeding 16K tokens, where small perturbations to token predictions may accumulate and affect downstream reasoning performance. The paper does not establish whether its accuracy-preservation claims hold under such realistic generation settings.

  3. Limited model diversity and generalizability. The evaluation is restricted to dense Llama-2 and Llama-3 architectures. It remains unclear whether the proposed layer-wise sparsity allocation generalizes to contemporary large-scale Mixture-of-Experts (MoE) models, where expert-level sensitivity, routing behavior, and computational characteristics may differ substantially from dense architectures.

  4. Unclear practical cost of preprocessing. The proposed framework requires extensive sensitivity profiling before inference. In particular, the authors precompute perplexity degradation for every consecutive layer interval and discretized sparsity rate, which can involve thousands of evaluations. While the dynamic programming procedure reduces the combinatorial search complexity, the paper does not provide sufficient empirical measurements of the preprocessing cost in GPU-hours or wall-clock time. Given the modest end-to-end inference improvements, it is unclear whether these upfront computational costs can be practically amortized.

Questions for authors
  1. How much is the performance degradation on generation tasks? Especially for tasks that require generating long-context responses? Preferably greater than 4K, around 8K? A great example for the 70B-level models is the challenging math downstream tasks.
  2. How does the efficiency benefit over QuaRot shift when you apply your techniques to recent large MoE models where the activated MLP parameters are smaller than dense 70B models?
  3. Can you give more details on how much the preprocessing cost would be? How does it scale across increasing sizes of the models?
3 comments
AuthorsOpenReview ↗

Response to Reviewer rauB

Dear Reviewer rauB:

Thank you for recognizing the task value, the method design, and the clear presentation of our work. We have responded point by point to your concerns regarding acceleration over the quantization baseline, long-form generation, MoE models, and preprocessing cost. In the marked copy appended to the revised PDF (pages 47–97), the revisions made in response to your comments are tagged with @Reviewer rauB. We hope our responses resolve your concerns and look forward to further discussion.


Summarized W1: Acceleration over the quantization baseline

A1: ASAF accelerates QuaRot [1] by 8.4% in single-block prefill on Llama-2-70B (3.61$\times$ against 3.33$\times$) and by 0.7% to 1.5% in end-to-end throughput, within the 1% accuracy budget; ASAF keeps the QuaRot quantizer fixed and adds sparsity on the 4-bit model, so this is its gain over QuaRot. Across the configurations of Table 3 the end-to-end gain follows the prefill share of the runtime (Table R1 of our response to wAi4; Table 19 of the revised paper, Page 45, Lines 2382 to 2386).

Following your suggestions, we have revised Section 5.3 (Lines 355–374) and Section 5.4 (Lines 430–439) to make this clearer.


Summarized W2 & Q1: Long-form generation quality

A2: Following your suggestion, we further report generation quality on chain-of-thought math (including GSM8K and MATH-500) for ASAF and QuaRot on Llama-2-70B, with prompt and generation within its 4,096-token context. Table R1 shows that ASAF stays within 0.40 points of QuaRot in generation accuracy, with the largest drop on MATH-500.

Table R1: Generation accuracy of ASAF minus QuaRot on Llama-2-70B.

TaskASAF minus QuaRot (points)
GSM8K-0.23
MATH-500-0.40

Following your suggestions, we have added these results to Appendix A.23 as Table 23 (Lines 2421–2424).


Summarized W3 & Q2: MoE models

A3: Following your suggestion, we further apply ASAF to an MoE model (i.e., Mixtral-8x7B over a QuaRot W4A4 backbone) under the same 1% budget. The formulation carries over, because the per-layer cost of an MoE layer is the FLOP count of its activated projections, and the sensitivity of a group, measured as the perplexity increase of the whole model, includes any change in routing inside that group. The blocks of the expert projections are ranked within each expert, and every expert of a group is pruned at the group's rate, which at the target rates keeps the removed share of activated FLOPs at that rate whatever the routing (rates are realized to the nearest block). Table R2 shows that ASAF also accelerates the MoE model within the budget, with a prefill speedup of 1.034$\times$ over the backbone at a perplexity increase of 0.72%. The gain over the backbone shifts with the share of FLOPs the experts carry and with their sensitivity, against 4.7% for Llama-2-7B and 8.4% for Llama-2-70B on the dense models at batch size 32.

Table R2: ASAF on an MoE backbone, Mixtral-8x7B with QuaRot W4A4.

MethodAvg. activated sparsity (%)Activated FLOPs vs. backboneΔ\DeltaPPL (%)Zero-shot avg. dropPrefill speedup
QuaRot backbone0.001.0000.000.001.000
ASAF (Ours)7.970.9200.720.511.034

Following your suggestions, we have added these results to Appendix A.23 as Table 24 (Lines 2434–2436).


Summarized W4 & Q3: Preprocessing cost and its scaling

A4: The number of sensitivity evaluations grows quadratically with the layer count and linearly with the number of rates, one independent evaluation per consecutive interval and rate, namely p(p+1)/2×29p(p+1)/2 \times 29; the cost of each evaluation also grows with the model size. This gives 15,312 evaluations for Llama-2-7B, 23,780 for Llama-2-13B and 93,960 for Llama-2-70B, estimated at 1,115, 3,216 and 68,420 GPU-hours at an assumed effective 70 TFLOP/s per GPU (forward passes, excluding GPTQ quantization and the DP search), as reported in Table 22 of the revised paper (Page 45, Lines 2411 to 2415). The tables are computed once per model and quantization setting, and the search is rerun on them by lookup for any other budget.

Following your suggestions, we have added these results to Appendix A.23 as Table 22 (Lines 2411–2415) and revised Appendix A.18 (Lines 2211–2235).


References

[1] Saleh Ashkboos et al. Quarot: Outlier-free 4-bit inference in rotated llms. NeurIPS, 2024.


Best Wishes,

The Authors of Paper 52594

Reviewer_rauBOpenReview ↗

Offcial Reply to Author's Followup

We thank the authors for adding new results on MoE and highlighting the low degradation of the ASAF minus QuaRot. However, the added results don't modify my original concern, which is that the increase in efficiency on top of QuaRot is still limited to less than 10%, which doesn't justify the additional quality degradation, despite being very minimal. Thus, I will keep my score not improved.

Dear Reviewer rauB,

Thank you for your reply, and for noting the new MoE results and the low degradation of ASAF relative to QuaRot.

On the concern you restate, the degradation budget δmax\delta_{max} sets the trade-off between the gain over QuaRot and the accuracy cost. Within the 1% budget, ASAF reaches 0.9269 of QuaRot's FLOPs against 0.9408 and 0.9542 for the per-layer and greedy allocations, and the uniform control exceeds the budget. We will keep these experiments in the final version.

Thank you again for your constructive feedback.

Best Wishes,

The Authors of Paper 52594

Reviewer_wAi4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ASAF, combining rotation-based 4-bit quantization (QuaRot) with layer-wise adaptive sparsity allocation under an explicit accuracy-degradation budget (≤1% perplexity increase on a 640-sample WikiText-2 subset). A two-phase dynamic program first fixes the number of consecutive layer groups and per-group sparsity intervals, then exact layer allocation and rates, minimizing FLOPs with tabulated sensitivity lookups and no retraining. On Llama-2/3 on an RTX 3090, ASAF reaches 3.63× single-block prefill acceleration over FP16 (vs. 3.36× for QuaRot), 12.63% memory reduction over the quantized baseline, and 1.2–3.5% end-to-end throughput gains, with <1% average benchmark degradation.

Critical strengths
  1. The formulation is principled where prior work uses heuristic proxies: sparsity sensitivity is measured directly on the quantized model, tabulated once, and the two-phase DP guarantees optimality within the discretized space at practical cost — the uniform-allocation ablation showing every uniform configuration exceeding the budget makes the case for adaptivity.
  2. The implementation is honest about its mechanism: sparsity shrinks GEMM dimensions on the same kernel stack as QuaRot, and results are reported at kernel, end-to-end, and memory levels across two GPU generations.
Critical weaknesses
  1. The gap between headline and delivered gain: 3.63× is single-block prefill acceleration; the increment over QuaRot is 8% at kernel level and 1.2–3.5% end-to-end, with decoding untouched. For most deployments this increment is unlikely to justify the pipeline, yet the abstract and title invite the stronger reading.
  2. Missing comparisons: no experiment against BESA-style layer-wise sparse allocation in the same W4A4 setting, and no equal-average-sparsity uniform-sparsity control at matched FLOPs (the A.16 ablation shows uniform exceeding the accuracy budget but not its speed-accuracy curve), so the marginal value of adaptive allocation over simpler alternatives is not isolated.
  3. Sensitivity is estimated only from WikiText-2 perplexity (640 samples); Llama-3 results are memory-only. Whether a 640-sample PPL budget transfers to broader task degradation is not established.
Questions for authors
  1. Please add a comparison with a representative layer-wise sparsity allocator (e.g., BESA) and a uniform-sparsity control at matched average sparsity, both on the same QuaRot backbone.
  2. What is the total one-time tabulation cost in GPU-hours (measuring Ξ for every consecutive layer interval × 29 sparsity levels)? This determines reproducibility for 70B-scale models.
  3. Why is decode throughput unaffected if sparsity shrinks GEMM dimensions? Is this a batch-1 bandwidth-bound regime where reduced FLOPs do not translate? Please state this explicitly.
3 comments
AuthorsOpenReview ↗

Response to Reviewer wAi4

Dear Reviewer wAi4:

Thank you for recognizing the method novelty, the method design, and the clear presentation of our work. We have responded point by point to your concerns regarding the gain over the quantization baseline, decoding, a matched uniform control, budget transfer, and tabulation cost. In the marked copy appended to the revised PDF (pages 47–97), the revisions made in response to your comments are tagged with @Reviewer wAi4. We hope our responses resolve your concerns and look forward to further discussion.


Summarized W1 & Q3: Headline gain, delivered gain and decoding

A1: The headline gain of ASAF is its gain over QuaRot [1], because ASAF keeps the quantizer fixed and allocates sparsity on the 4-bit model. The revised abstract prints 3.61$\times$ for ASAF next to 3.33$\times$ for QuaRot on a single Llama-2-70B transformer block, a gain of 8.4%.

The delivered end-to-end speedup follows from the time split of a request, with a prefill share ff, a prefill speedup SpS_p and a decode speedup SdS_d:

E=1f/Sp+(1−f)/Sd.E = \frac{1}{f/S_p + (1-f)/S_d}.

Table R1 shows that with Sd=1S_d = 1 the end-to-end speedup stays below SpS_p in the four configurations of Table 3 (computed under the assumptions of Appendix A.22), reaching 1.015 at the largest prefill share of 0.320.

On decoding (Q3), with 8-bit weights, reading the weights dominates each decoding step (Page 2, Lines 94 to 95), and the revised paper counts decoding at QuaRot's rate as an assumption, because pruning shrinks the weight reads of a decoding step but adds a gather of the kept channels (Page 7, Lines 372 to 373).

Table R1: End-to-end speedup split into prefill and decoding (prompt / generation / batch).

ConfigurationEnd-to-end speedup EEPrefill speedup SpS_pDecode speedup SdS_dPrefill share ff
Llama-2-7B, 512 / 128 / 11.0091.0441.0000.220
Llama-2-7B, 2048 / 512 / 11.0071.0441.0000.171
Llama-2-13B, 2048 / 512 / 81.0121.0481.0000.250
Llama-2-13B, 3584 / 512 / 81.0151.0481.0000.320

Following your suggestions, we have revised the abstract (Lines 25–28), Section 2 (Lines 94–95) and Section 5.3 (Lines 355–374), and added these results to Appendix A.23 as Table 19 (Lines 2382–2386).


Summarized W2 & Q1: Learned allocation and matched uniform control

A2: Following your suggestion, we compare ASAF with the uniform control on ASAF's own partition (i.e., every group at 7.5%, the grid rate nearest ASAF's average sparsity of 7.3%) on the same QuaRot backbone and the same tables. Table R2 shows that at nearly the same FLOPs the uniform control exceeds the budget while ASAF stays within it, with a tabulated degradation of 1.490% against 0.995%, so the uniform control is infeasible under the 1% budget that ASAF satisfies. Under every fixed group count, the uniform heuristic uses 1.75% to 1.86% of tabulated degradation against 0.97% to 0.98% for ASAF (Table 13). A learned allocator in the style of BESA [2] optimizes block reconstruction error; under the same budget, a budgeted per-layer allocator keeps 0.9408 of QuaRot's FLOPs against 0.9269 for ASAF (Table 25 of the revised paper).

Table R2: Uniform control and ASAF on QuaRot W4A4 Llama-2-7B under the same 1% budget.

MethodAvg. sparsity (%)FLOPs vs. QuaRotTabulated ∑ξ\sum\xi (%)Δ\DeltaPPL, test (%)Zero-shot avg.
QuaRot0.01.00000.0000.0065.64
Uniform on ASAF's groups7.50.92501.4901.2164.81
ASAF (Ours)7.30.92690.9950.8265.08

Following your suggestions, we have revised the uniform control in Section 5.4 (Lines 418–424) and Appendix A.16 (Lines 2111–2131), and added these results to Appendix A.23 as Table 20 (Lines 2392–2395).


Summarized W3: Llama-3 coverage and transfer of the WikiText-2 budget

A3: Following your suggestion, to check how the WikiText-2 budget transfers to other data, we further tabulate the allocation on 640 C4 training windows instead of the WikiText-2 subset, with the GPTQ [3] calibration held fixed. Table R3 shows that the C4-tabulated allocation reaches the same 0.82% perplexity increase on WikiText-2 test as the WikiText-2 allocation, and 0.75% on C4 validation. Its zero-shot drop of 0.54 points stays within 1% of QuaRot's average (0.66 points). For Llama-3, the revised paper reports zero-shot accuracy on six benchmarks for Llama-3-8B and Llama-3-70B, with an average degradation below 1% of QuaRot's average.

Table R3: Transfer of the tabulated budget on the same quantized Llama-2-7B backbone.

Tabulation dataΔ\DeltaPPL WikiText-2 test (%)Δ\DeltaPPL C4 validation (%)Zero-shot avg. drop vs. QuaRot
WikiText-2 (640 samples)0.820.840.56
C4 (640 samples)0.820.750.54

(continued)

Following your suggestions, we have revised the Llama-3 results in Section 5.3 (Lines 412–415), and added these results to Appendix A.23 as Table 21 (Lines 2401–2404).


Summarized Q2: One-time tabulation cost

A4: The one-time tabulation cost, counting the forward passes of the sensitivity evaluations only (no GPTQ quantization and no DP search), is an estimated 1,115 GPU-hours for Llama-2-7B and 68,420 for Llama-2-70B at an assumed effective 70 TFLOP/s per GPU, from p(p+1)/2×29p(p+1)/2 \times 29 independent sensitivity evaluations (15,312 and 93,960). Table R4 lists this cost for three models. The tables are computed once per model and quantization setting, reused for every budget, and the evaluations distribute across GPUs.

Table R4: One-time cost of the sensitivity forward passes.

ModelLayers ppIntervalsξ\xi evaluationsPFLOP per evaluationGPU-seconds per evaluationGPU-hours
Llama-2-7B3252815,31218.42621,115
Llama-2-13B4082023,78034.14873,216
Llama-2-70B803,24093,960183.52,62168,420

Following your suggestions, we have added these results to Appendix A.23 as Table 22 (Lines 2411–2415) and revised Appendix A.18 (Lines 2211–2235).


References

[1] Saleh Ashkboos et al. Quarot: Outlier-free 4-bit inference in rotated llms. NeurIPS, 2024.

[2] Peng Xu et al. BESA: Pruning large language models with blockwise parameter-efficient sparsity allocation. ICLR, 2024.

[3] Elias Frantar et al. Optq: Accurate quantization for generative pre-trained transformers. ICLR, 2023.


Best Wishes,

The Authors of Paper 52594

Dear Reviewer wAi4,

Thank you for your time in reviewing our work, and for recognizing the principled two-phase dynamic program and the sensitivity defined directly on the quantized model.

We believe that our rebuttal has addressed your concerns, with the gain over QuaRot and its split into prefill and decoding, the matched uniform control, the C4 budget transfer and Llama-3 results, and the one-time tabulation cost. As suggested, we will keep these experiments in the final version.

Thank you again for your constructive feedback.

Best Wishes,

The Authors of Paper 52594

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

General Response

We sincerely thank all reviewers for their time and constructive comments. Line references refer to the clean copy, PDF pages 1–46.

Summary

Low-bit quantization and sparsification are usually applied separately, with sparsity set at one rate across layers or by heuristic proxies. Our method, ASAF, allocates sparsity over consecutive layer groups of an already quantized model with a two-phase dynamic program that minimizes FLOPs under an explicit accuracy-degradation budget. On a single Llama-2-70B block, ASAF reaches a prefill speedup of 3.61$\times$ against 3.33$\times$ for QuaRot, with the average benchmark accuracy within 1% of QuaRot's. The revision adds matched allocation controls, the prefill and decoding split of the end-to-end speedup, long-form generation, an MoE model, and ASAF on top of SpinQuant (Tables 19 to 27).

Strengths

  • Novel sensitivity. Defined directly on the quantized model and tabulated once (wAi4).
  • Principled design. Two-phase dynamic program in place of heuristic proxies (wAi4, rauB).
  • Clear objective. Explicit accuracy budget as the optimization objective (rauB, 6Z6g).
  • Task value and presentation. Practically relevant, well-motivated and easy to follow (wAi4, rauB, 6Z6g).

Weaknesses and Revisions

  1. Gain over QuaRot and delivered gain (wAi4, rauB, 6Z6g). The abstract now prints 3.61$\times$ for ASAF next to 3.33$\times$ for QuaRot, a gain of 8.4% (Lines 25–28); added the end-to-end speedup split into prefill and decoding, a gain of 0.7% to 1.5% (Table 19, Lines 2382–2386).
  2. Matched allocation controls (wAi4, 6Z6g). Added the uniform control on ASAF's own groups, which exceeds the 1% budget at nearly the same FLOPs (tabulated degradation 1.490% against 0.995%; Table 20, Lines 2392–2395), and the per-layer, greedy and single-stage DP allocations (0.9408, 0.9542 and 0.9261 of QuaRot's FLOPs against 0.9269 for ASAF; Table 25, Lines 2444–2449).
  3. Budget transfer and Llama-3 (wAi4). Added the allocation tabulated on C4, which reaches the same 0.82% perplexity increase on WikiText-2 test (Table 21, Lines 2401–2404); revised the Llama-3 results in Section 5.3 (Lines 412–415).
  4. Tabulation cost and scaling (wAi4, rauB). Added the one-time cost of the sensitivity evaluations, an estimated 1,115 GPU-hours for Llama-2-7B and 68,420 for Llama-2-70B (Table 22, Lines 2411–2415).
  5. Long-form generation (rauB). Added chain-of-thought generation, where ASAF stays within 0.40 points of QuaRot (Table 23, Lines 2421–2424).
  6. MoE (rauB). Added ASAF on Mixtral-8x7B, with a prefill speedup of 1.034$\times$ at a perplexity increase of 0.72% (Table 24, Lines 2434–2436).
  7. Joint compression comparison (6Z6g). Added ASAF on top of SpinQuant, at 0.9250 of SpinQuant's FLOPs with the eight-task average falling from 64.0% to 63.5% (Table 26, Lines 2456–2461); revised Table 5 (Lines 387–399).
  8. Pruning structure, additive budget and coarse DP state (6Z6g). Specified the pruning unit (Lines 72–75) and the coarse pass (Lines 204–212); added the joint degradation on the calibration subset, 0.986% against a tabulated sum of 0.995% (Table 27, Lines 2467–2471).

In the marked copy (PDF pages 47–97), every revision is tagged with the ID of the reviewer who raised it.

All experiments above will be kept in the final version.

Best Wishes,

The Authors of Paper 52594

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.