acceptodds
Under review as a conference paper at ICLR 2027

QK-Wanda: Coupling Queries and Keys for Unstructured Pruning

Abstract

Wanda (Sun et al., 2024) prunes large language models by scoring each weight by its magnitude times its input activation, treating every linear projection independently. Self-attention violates this assumption: queries and keys enter the model only through their product, so the importance of a query weight depends on the keys, and vice versa. We introduce QK-Wanda, an unstructured pruning method that scores query and key weights by how much their removal perturbs the joint query-key product rather than either projection alone. Starting from a reconstruction loss on the causally masked query-key product, we derive a closed-form importance score for deleting a single weight; the result is Wanda's magnitude-activation criterion augmented with a term from the opposite projection. Scores are computed from a small calibration set without gradients or retraining, adding only 31-37% to Wanda's calibration and scoring time. On the Llama 3 family, QK-Wanda consistently reduces query-key reconstruction error and lowers perplexity relative to Wanda in most model-sparsity settings, with the largest gains at high sparsity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, model compression, unstructured pruning, one-shot pruning, sparsity, self-attention, query-key coupling

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_5NpLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper frames QK-WANDA as an evolution over WANDA. WANDA scores query (Q) and key (K) weights independently, multiplying each weight’s magnitude by the corresponding input feature’s activation norm. QK-WANDA recognizes that Q and K mutually impact each other through the calculation of attention scores:

$ A=\operatorname{softmax} \left(\frac{Q^\top K}{\sqrt{d_h}}\right) $

Additionally, QK-WANDA highlights that because decoder transformers use causal attention, Q-K interactions must be studied for allowed pairs: query position t can attend to key position u exactly when u ≤ t.

Further, since most models considered in the paper use Grouped Query Attention (GQA), where keys/values are shared for multiple heads, this makes key weights responsible for several query heads. Deleting a shared key weight affects every query head using that key head, so their reconstruction costs are summed, not averaged.

Critical strengths

The method is sound and represents an improvement over WANDA, in that the reconstruction error is reduced.

The method is tested on a broad range of LLMs, at various sizes.

Evaluations show meaningful improvements in most configurations.

Critical weaknesses

W1: The reconstruction error is not always a good predictor of downstream quality: On Llama-3.1-70B at 80%, QK error improves by ~2% but perplexity on WikiText-2 and C4 worsen.

W2: The calibration settings and budget defaults were chosen using WikiText-2 test and C4 validation scores on Llama-3.1-8B therefore there is a real risk of overfitting.

W3: The strongest results come from pruning a small part of the model. Setting half the Q/K weights to zero affects only about 2 to 8% of all model parameters.

Questions for authors

Q1: Are you able to analyze the Llama-3.1-70B shortcoming (improvement in reconstruction error, degradation in downstream metric) further with an intervention study? Are you able to show that this relates to this specific model/data configuration, but the method is generally applicable.

Q2: Could you use different data splits for calibration and evaluations?

Q3: Can you measure inference latency improvements, memory footprint reductions?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5NpL: Reconstruction Mismatch, Data Splits, and Pruning Scope

Thank you for recognizing the soundness of the construction, the range of evaluated models, and the improvements in many configurations. We address the reconstruction mismatch, data selection, and practical scope below.

W1/Q1: Llama-3.1-70B and intervention evidence. We agree that better local reconstruction is not sufficient. At 80% Q/K sparsity, the reported QK error decreases from 5.10% to 3.03%, while WikiText-2/C4 PPL increases from 4.98/9.89 to 18.98/32.37. The diagnostic evaluates the complete masks on common dense block inputs before RoPE; it does not measure post-softmax attention distributions or accumulated errors through the sparse network.

There are two relevant controls in the submission. First, Table 1 and Appendix M separate scoring from allocation. On this same checkpoint at 80%, changing Wanda from row to matrix budgets raises WikiText-2/C4 PPL to 17.69/24.21 without introducing QK scoring. Second, Appendix C explicitly varies whether RoPE is included in the score. In that separate three-seed study on smaller Llama 3 checkpoints, RoPE-aware scoring gives lower post-RoPE error in four of six configurations but worse mean PPL on both datasets in all six. These observations highlight the need to distinguish scoring from allocation and caution against attributing the failure solely to omitted RoPE.

Neither control is a complete intervention study of the 70B failure. In particular, the evidence does not justify claiming that this is an isolated configuration: all four Llama 3 checkpoints have worse PPL at 80%. The supported conclusion is that downstream benefits depend on the checkpoint and pruning regime. Layerwise attention-probability diagnostics and targeted restoration of selected pruned projections would help distinguish candidate explanations, but those results are not established by the submission.

W2/Q2: calibration versus selection data. Calibration tokens are sampled from C4 training, separately from the PPL evaluation data. Nevertheless, the reported WikiText-2 test and C4 validation PPL on Llama-3.1-8B were also used to choose calibration settings and the shared-budget default. We agree that this creates selection bias and that the train/evaluation separation of calibration tokens does not resolve it. The selected configuration was fixed across the other checkpoints, which tests transfer across models on the same benchmarks, not generalization to untouched evaluation data. Section 5 acknowledges this distinction. A fresh evaluation set with frozen settings—or separate development and final-evaluation data when retuning—is required to address the concern.

W3: pruning only a small fraction of total parameters. The Q/K-only experiments intentionally isolate the proposed component, and Appendix D.4 reports the denominator explicitly: 50% Q/K sparsity zeroes 2.23–8.06% of total parameters. However, the submission is not limited to that setting. Section 4.5 and Table 2 prune all seven block projections: Q/K/V/O and the MLP gate/up/down matrices. The hybrid uses QK-Wanda only on Q/K and Wanda elsewhere, compared with Wanda throughout.

At 50% block-projection sparsity, TinyLlama WikiText-2/C4 PPL improves from 11.52/13.85 to 11.22/13.62. On Llama-2-7B, WikiText-2 PPL changes from 6.89 to 6.85 and mean accuracy from 60.23% to 61.09%. These are modest component-level gains, not the larger gains seen in Q/K-only pruning. Embeddings and the output head remain dense, and neither pipeline preserves useful quality at 80% full-block sparsity.

Q3: latency and memory. We agree that weight-removal counts and scoring-time measurements do not establish inference speedups or memory-footprint reductions. Appendix L concerns calibration and scoring; no end-to-end inference benchmark is reported. A deployment claim would require an implemented sparse representation and execution path, with actual latency and memory measurements. The present empirical claim is about the quality of masks at matched pruning budgets, including their use within a broader pruning pipeline—not measured serving acceleration.

Thank you for your responses, however they don't really address my questions/suggestions (no intervention evidence for LLama3.1 70B, no separation of calibration vs selection data, and no latency measurements).

Reviewer_ABvZOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes QK-Wanda, a one-shot unstructured pruning method that accounts for interactions between attention query and key projections. It derives exact single-weight deletion scores from a causal pre-RoPE QK reconstruction objective and adopts a shared Q/K pruning budget without gradients or weight updates. In addition, it provides a theoretical proof for the causal score in the appendix. Experiments across 15 LLM checkpoints (0.5B–72B) evaluate QK reconstruction error, WikiText-2/C4 perplexity, and seven zero-shot tasks. Compared with Wanda, QK-Wanda substantially reduces reconstruction error, while downstream improvements remain model-dependent, showing notable gains on TinyLlama and Llama 2 but degraded perplexity on Llama 3.

Critical strengths
  1. Principled and efficient attention-aware pruning. QK-Wanda derives exact single-weight deletion costs under a causal pre-RoPE reconstruction objective, explicitly accounting for query–key interactions. Its closed-form scores and shared Q/K budget provide a theoretically grounded extension of Wanda without gradients or weight updates.

  2. Extensive experimental validation with meaningful improvements. Evaluations across 15 LLM checkpoints (0.5B–72B) demonstrate substantial reductions in QK reconstruction error. Matched-allocation controls further distinguish scoring improvements from budget allocation effects. Notable high-sparsity gains on Llama 2 models provide empirical support for the method's effectiveness.

Critical weaknesses
  1. Inconsistent downstream effectiveness. Although QK-Wanda generally reduces reconstruction error, it substantially worsens perplexity on Llama 3 at high sparsity. For Llama-3.1-70B at 80% Q/K sparsity, WikiText-2 PPL increases from 4.98 (Wanda) to 18.98, raising concerns about its reliability across model families.

  2. Limitations in experimental validation. Calibration settings and budget allocation were selected using the reported WikiText-2 test and C4 validation benchmarks, without a separate development split. Combined with single-seed main experiments, this limits confidence in the robustness and generalizability of the reported improvements.

  3. Limited effectiveness in method scope. QK-Wanda is an improved version of WAND by coupling queries and keys for unstructured pruning. The method effectiveness scope is very limited. As a reviewer, there are several questions I can ask. For example, is the idea of query–key interaction also benefical for other pruning methods. Is this method benefical for structured or semi-structured pruning. In my opinion, these are very improtant questions. However, this paper only foucus on the limited discussion.

  4. Syntax errors and typos

  • Line 556: "In arXiv preprint arXiv:1803.05457, 2018." -> " arXiv preprint arXiv:1803.05457, 2018."
  • Line 1851: "Task-level results on TinyLlama and Llama-2-7B" -> "Task-level results on TinyLlama and Llama-2-7B."
Questions for authors
  1. Downstream reliability: Why does QK-Wanda substantially worsen Llama-3.1-70B perplexity at 80% sparsity despite lower QK reconstruction error? Can the authors explain this discrepancy and identify conditions under which the method is reliable?

  2. Evaluation robustness: How sensitive are the conclusions to benchmark-based hyperparameter selection and the single calibration seed? Can the authors provide results using an independent development split and multiple calibration seeds?

  3. Method generality: Is Q–K coupling beneficial beyond Wanda-style unstructured pruning, particularly for other pruning methods or structured/semi-structured sparsity? Can the authors provide experimental evidence or theoretical justification supporting broader applicability?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ABvZ: Downstream Reliability, Evaluation, and Generality

Thank you for recognizing the scalar-deletion construction and matched-allocation controls. We address downstream reliability, evaluation independence, and generality below.

1. Llama-3.1-70B: lower QK error is not sufficient for downstream preservation. The reported deterioration is real: at 80% Q/K sparsity, WikiText-2 PPL increases from 4.98 to 18.98 despite a reduction in relative QK error from 5.10% to 3.03%. This failure is explicitly included in the abstract, Table 1, and Section 5; we do not regard it as an insignificant exception.

Appendix D.3 evaluates complete masks on common dense block inputs, not a sum of individual deletion scores. However, it measures pre-RoPE products rather than attention probabilities and excludes error propagated from earlier pruned blocks. These differences explain why the metrics need not agree, but do not establish the mechanism of this particular failure.

Table 1 and Appendix M (Table 22) report the following PPL on Llama-3.1-70B at 80% Q/K sparsity:

MethodWikiText-2C4
Wanda, row budgets4.989.89
Wanda, matrix budgets17.6924.21
QK-Wanda, separate matrix budgets26.0043.09
QK-Wanda, shared budget18.9832.37

Thus, substantial degradation also occurs when Wanda's scoring rule is retained but its row quotas are relaxed. Conversely, sharing the budget improves QK-Wanda relative to its separate-matrix variant here. These controls show that allocation matters alongside the reconstruction target; they do not establish that allocation explains the entire failure, or that QK-Wanda's score is harmless.

The current evidence identifies successful regimes empirically rather than giving an advance reliability test. On Llama-2-70B at 80%, PPL improves from 4.55/6.91 to 3.63/5.97 and mean accuracy from 65.85% to 71.76%; Qwen2.5-72B also improves on both PPL benchmarks and mean accuracy. We do not claim that these gains generalize to every family or sparsity, nor that QK error alone can select a reliable mask.

2. Calibration data, model selection, and seeds. Calibration activations come from C4 training data, whereas PPL is evaluated on WikiText-2 test and C4 validation data. However, as Section 4.1 explicitly states, those reported PPL measurements on Llama-3.1-8B also informed the calibration settings and shared-budget default. This is a model-selection leakage concern, even though the calibration tokens themselves come from a separate split.

The configuration was then frozen across the other checkpoints. This provides cross-checkpoint evidence without per-checkpoint retuning, but does not make the reused benchmarks untouched evaluation data. An independent development/final-evaluation protocol is needed to address that concern; we do not claim that the existing study already provides it.

The main study uses one calibration seed. Appendix C reports three-seed variation under a different calibration budget and compares two QK variants, not QK-Wanda against Wanda. It therefore cannot establish the robustness or statistical significance of the main comparisons. These measurements are descriptive; small gains should not be interpreted as statistically reliable.

3. Generality beyond the current setting. The proposed scores are scalar reconstruction costs, not a mechanism intrinsically restricted to shared unstructured budgets. They can define an additive surrogate under other mask constraints. For example, imposing 2:4 groups would require local mask selection and remove the cross-row/QK allocation freedom; this is a possible extension, not evidence that the reported gains persist under 2:4 constraints. Similarly, incorporating the QK objective into compensated pruning would require accounting for interactions and survivor updates, rather than simply claiming the current score improves SparseGPT or Thanos.

The submission does establish one broader use: Section 4.5 combines QK-Wanda on Q/K with Wanda on the other five block projections. This shows that the method can serve as a component of broader pruning, although the gains are smaller than in Q/K-only experiments and both pipelines fail at 80% full-block sparsity. It does not establish effectiveness for other pruning algorithms or structured masks.

4. Typographical corrections. Thank you for identifying the reference-formatting error at line 556 and the missing punctuation at line 1851. We agree with both corrections.

The contribution is an exact, efficiently computable scalar criterion, matched-allocation evidence, and characterization of failure regimes within mask-only pruning, not universal superiority across pruning settings.

Reviewer_ABvZOpenReview ↗

Reply to Authors' Comments

Thank you for the detailed rebuttal and clarifications. I appreciate the authors' careful responses and acknowledge the theoretical contribution of the proposed QK-aware pruning criterion. However, after considering the rebuttal, I believe my main concerns remain unresolved.

1. Inconsistent Downstream Effectiveness. The proposed method substantially reduces QK reconstruction error, but these improvements do not consistently translate into better downstream performance. In particular, all four Llama 3 checkpoints exhibit worse perplexity at 80% Q/K sparsity. For Llama-3.1-70B, WikiText-2 perplexity increases from 4.98 (Wanda) to 18.98 (QK-Wanda). Although the authors acknowledge this failure and discuss the influence of budget allocation, they have not identified its underlying cause or provided a reliable mechanism for avoiding such degradation. This remains a significant concern regarding the practical effectiveness and robustness of the proposed importance criterion.

2. Insufficient Experimental Validation. The authors acknowledge that the reported WikiText-2 test and C4 validation benchmarks were used for hyperparameter selection without an independent development split. Furthermore, the main experiments rely on a single calibration seed, while the existing multi-seed analysis uses a different experimental protocol. The rebuttal does not provide additional evidence addressing these limitations. Consequently, the reliability and generalizability of the reported improvements remain insufficiently established.

3. Limited Methodological Generality. The authors clarify that QK-aware scoring could potentially be extended to other pruning settings. However, its effectiveness beyond Wanda-style unstructured pruning, particularly for structured or semi-structured sparsity and other pruning algorithms, remains experimentally unverified. I acknowledge that these extensions are outside the paper's explicitly stated scope, but their absence limits the demonstrated applicability of the proposed approach.

Overall Assessment

While the theoretical derivation and extensive model coverage are valuable, the rebuttal mainly clarifies existing limitations without providing sufficient new evidence to resolve my primary concerns regarding downstream reliability and experimental validity.

Therefore, I will maintain my original rating of 1 (Clear Rejection).

Reviewer_aD4mOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes QK-WANDA, an unstructured pruning method for Q/K projection matrices that considers their interactions in attention. Specifically, it introduces a joint reconstruction objective to measure the change in the Query-Key dot-product matrix caused by pruning each weight, and derives closed-form pruning scores by considering the causal mask in autoregressive attention. It also allows weights across different output rows and Q/K projection matrices to share the pruning budget. Experiments show that QK-WANDA generally reduces QK reconstruction error and improves PPL and downstream task performance on some models at high sparsity levels.

Critical strengths
  1. The motivation is reasonable and well justified. The paper identifies an important limitation of existing WANDA-based pruning methods: Query and Key weights are evaluated independently, despite their interactions through attention dot products. Considering the coupled effects of Q/K pruning is a reasonable direction for improving weight importance estimation in attention layers.
  2. The proposed method is relatively comprehensive in its design. It considers not only the joint importance of Query and Key weights, but also the causal nature of autoregressive attention through causal-aware scoring. In addition, it explores different sparsity budget allocation strategies across rows and between the Q and K matrices. These considerations make the proposed pruning framework well thorough.
Critical weaknesses
  1. The reconstruction objective may not accurately reflect the actual impact of pruning on downstream model performance. The proposed importance scores are derived from the reconstruction error of pre-RoPE QK products. However, the actual attention computation further involves RoPE, softmax, and subsequent layers. Consequently, minimizing the proposed local reconstruction objective does not necessarily preserve the model's overall behavior. This limitation is also reflected in the experiments: although QK-WANDA consistently reduces reconstruction error, all four Llama 3 checkpoints exhibit worse PPL than WANDA at 80% Q/K sparsity.
  2. The paper only evaluates unstructured pruning, which does not directly translate into practical inference speedups on current hardware. It would be valuable to evaluate QK-WANDA under hardware-supported structured sparsity patterns, such as 2:4 sparsity, and report the corresponding inference speedups.
Questions for authors
  1. Could the authors compare the reconstruction errors of the attention maps (after applying RoPE and Softmax) produced by QK-WANDA and the original WANDA? Since the proposed method optimizes the Query-Key products before positional encoding and Softmax, it would be helpful to verify whether the improvement in QK reconstruction error also translates into more accurate attention distributions.
  2. Could the authors evaluate QK-WANDA under hardware-supported semi-structured sparsity patterns, such as 2:4 sparsity? Since the proposed method relies on flexible sparsity allocation across different output rows and the Q/K projection matrices, it is unclear how this allocation strategy can be adapted to the fixed local sparsity constraints of 2:4 patterns. Could the authors clarify how QK-WANDA can be extended to such settings and provide corresponding experimental results, including inference speedups?
1 comment
AuthorsOpenReview ↗

Response to Reviewer aD4m: Reconstruction Objective and 2:4 Sparsity

Thank you for the positive assessment of the motivation and the causal/GQA-aware design. Your questions identify two important distinctions: the reconstruction target versus actual attention behavior, and the scoring rule versus the constraints imposed on the mask.

Reconstruction after RoPE and softmax. We agree that lower pre-RoPE QK error does not guarantee better attention distributions or downstream quality. RoPE, causal attention, and softmax remain enabled in every evaluated model; “pre-RoPE” describes the scoring objective and the main reconstruction diagnostic, not a modification of the forward pass. Proposition A.1 establishes exactness for one deletion in that stated objective, not preservation of the complete attention computation or optimality of the final multi-weight mask.

The submission already includes a controlled RoPE-aware variant, QK-Wanda-R, derived in Appendix B.2 and evaluated in Section 4.6 and Appendix C. Table 3 compares causal pre-RoPE and causal-plus-RoPE scoring on Llama-3.2-1B/3B and Llama-3.1-8B at 50% and 80% sparsity, over three calibration seeds. The RoPE-aware variant reduces post-RoPE reconstruction error in four of six model–sparsity pairs, but the pre-RoPE variant has lower mean PPL on both datasets in all six. For example, at 50% on Llama-3.2-1B, post-RoPE error falls from 0.20% to 0.18%, while WikiText-2 PPL increases from 11.1933 to 12.2233. Thus, making the local objective RoPE-aware does not automatically resolve the downstream mismatch.

These results have a specific scope: they use 256 × 32 calibration tokens and separate matrix budgets, unlike the main experiment. They compare two QK-Wanda variants, not QK-Wanda against Wanda after softmax. Appendix C also reports block-output diagnostics, but neither those diagnostics nor post-RoPE logit error are a substitute for the attention-probability comparison you request. That direct comparison remains missing from the submission; we do not infer that it would favor QK-Wanda.

The Llama 3 failure regime. The worse PPL at 80% on all four Llama 3 checkpoints is a genuine limitation, explicitly reported in Section 4.3 and Section 5. Our evidence supports model-dependent gains, not a universal downstream advantage. In contrast, Table 1 shows that on Llama-2-70B at the same Q/K sparsity, WikiText-2/C4 PPL improves from 4.55/6.91 to 3.63/5.97 and mean task accuracy rises from 65.85% to 71.76%. Both outcomes matter when assessing what the reconstruction criterion captures and what it misses.

Extension to 2:4 sparsity. The scalar scores can be computed independently of the mask constraint. A possible 2:4 extension would partition each projection into hardware-compatible groups of four and delete the two lowest-scoring weights in every group. This minimizes the additive scalar-score surrogate under those constraints; it does not in general minimize the true joint reconstruction loss, whose interaction terms are described in Appendix A.6.

Crucially, this would not retain the submitted method's allocation freedom. Fixed 2:4 groups prevent moving the removal budget across rows and between Q and K. Appendix H.2 shows that allocation freedom contributes materially to the unstructured results, so those gains cannot simply be transferred to a 2:4 setting. This describes a possible extension, not an evaluated result: the submission contains neither 2:4 quality measurements nor corresponding inference speedups.

We therefore agree that semi-structured evaluation would strengthen the practical case, while keeping the present claim specific: exact scalar costs for the stated reconstruction objective, efficient mask-only pruning, and experimentally characterized—but non-universal—downstream improvements.

Reviewer_ognvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes QK-Wanda, an unstructured pruning method that considers query–key interactions when scoring weights for removal. The method is evaluated on 15 pretrained models ranging from 0.5B to 72B parameters.

Critical strengths
  1. The paper provides a closed-form derivation of QK-aware pruning scores without requiring gradients or weight updates.
  2. The experiments cover multiple model families and model sizes.
Critical weaknesses
  1. The paper does not convincingly explain its novelty or distinguish its contributions from existing pruning methods. The experimental comparisons are largely limited to Wanda, despite discussing stronger methods such as SparseGPT and Thanos, which mentioned by the paper itself.
  2. All downstream evaluations use a 2K context, with no long-context experiments to assess whether pruning exacerbates performance degradation.
  3. As a pruning method, the paper provides no end-to-end inference latency, throughput, or memory-saving results to demonstrate practical benefits. The reported scoring-time measurements are insufficient to establish whether the method achieves any actual acceleration.
Questions for authors

Since the proposed method explicitly considers Q/K interactions, can the authors also compare its long-context accuracy and efficiency against attention sparsity methods such as MInference and FlexPrefill?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ognv: Novelty, Baselines, and Practical Scope

Thank you for recognizing the closed-form derivation and the breadth of the evaluation. We address the questions about novelty, comparison scope, and practical relevance below.

Novelty and relationship to existing methods. Our contribution is not the general observation that queries and keys interact: Section 2 and Appendix K explicitly discuss prior QK reconstruction objectives. The contribution is a causal, compensation-free scalar-pruning procedure on the original Q/K weights. The exact single-deletion costs in Eqs. (7), (12), and (13) retain token-specific causal interactions, account for every query head affected by a shared key, and put Q and K deletions on a common loss scale. Prefix/suffix accumulators make these scores computable without constructing a separate token-pair error matrix for every candidate weight. This differs from reconstructing each projection independently, quantizing weights, or deleting matched Q/K channels with compensation.

Why Wanda, and what the controls establish. Wanda is the direct baseline for isolating the scoring change while keeping surviving weights fixed. SparseGPT and Thanos also modify surviving weights; comparisons with them would address the broader quality–cost trade-off of complete pruning pipelines, rather than isolate our proposed score. Such comparisons would be useful, but our submission does not establish superiority over these methods.

Importantly, the comparisons are not limited to original row-wise Wanda. Section 4.2 and Appendix M include matrix-budget Wanda with exactly the same per-projection deletion counts as separate-budget QK-Wanda. On Llama-2-70B at 80% Q/K sparsity, WikiText-2/C4 PPL changes from 4.55/6.91 for row-wise Wanda, to 4.13/6.53 for matrix-budget Wanda, to 3.84/6.14 for QK-Wanda with matched matrix budgets, and to 3.63/5.97 with shared Q/K allocation. Across the three largest checkpoints and nine sparsities, the matched-budget QK score gives lower measured WikiText-2 PPL in 21/27 comparisons and lower C4 PPL in 20/27. These controls support a contribution beyond simply relaxing row quotas, while not implying improvement in every configuration.

Long context and MInference/FlexPrefill. The 2K-context evaluation does not establish long-context robustness, and we agree that this is a limitation. MInference and FlexPrefill sparsify token-to-token attention computation during prefill, whereas QK-Wanda removes scalar projection weights offline and retains the model's original attention computation. Consequently, equal numerical “sparsity” would not represent an equal resource budget, and these methods are not direct controls for the weight-importance criterion. A comparison would require a separate end-to-end study with matched hardware, sequence lengths, and quality–latency budgets. We do not claim long-context accuracy or efficiency advantages over either method.

Inference efficiency versus pruning cost. We agree that scoring time is not evidence of inference acceleration. Appendix L measures calibration and score construction, not inference, and Section 5 explicitly limits the claim. Unstructured zeros in an otherwise dense execution do not by themselves demonstrate latency or memory savings. The established benefit is improved quality for some checkpoints at a fixed weight-removal budget, not a demonstrated faster serving system. Section 4.5 additionally evaluates using QK-Wanda for Q/K inside a broader Wanda pruning pipeline, but this remains a quality evaluation rather than a deployment-speed benchmark.

The intended contribution is therefore a controlled study of coupled scalar weight importance, including both its empirical gains and its failure regimes, rather than a claim of state-of-the-art end-to-end acceleration.

Thank you for your rebuttal. However, none of the reply solve any of my concern. The authers do not add any extra experiment I mentioned in Critical Weaknesses or Questions For Authors sections, which does not convince me anything. Could you please add experiments I mentioned in my reviews?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.