- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper frames QK-WANDA as an evolution over WANDA. WANDA scores query (Q) and key (K) weights independently, multiplying each weight’s magnitude by the corresponding input feature’s activation norm. QK-WANDA recognizes that Q and K mutually impact each other through the calculation of attention scores:
$ A=\operatorname{softmax} \left(\frac{Q^\top K}{\sqrt{d_h}}\right) $
Additionally, QK-WANDA highlights that because decoder transformers use causal attention, Q-K interactions must be studied for allowed pairs: query position t can attend to key position u exactly when u ≤ t.
Further, since most models considered in the paper use Grouped Query Attention (GQA), where keys/values are shared for multiple heads, this makes key weights responsible for several query heads. Deleting a shared key weight affects every query head using that key head, so their reconstruction costs are summed, not averaged.
Critical strengths
The method is sound and represents an improvement over WANDA, in that the reconstruction error is reduced.
The method is tested on a broad range of LLMs, at various sizes.
Evaluations show meaningful improvements in most configurations.
Critical weaknesses
W1: The reconstruction error is not always a good predictor of downstream quality: On Llama-3.1-70B at 80%, QK error improves by ~2% but perplexity on WikiText-2 and C4 worsen.
W2: The calibration settings and budget defaults were chosen using WikiText-2 test and C4 validation scores on Llama-3.1-8B therefore there is a real risk of overfitting.
W3: The strongest results come from pruning a small part of the model. Setting half the Q/K weights to zero affects only about 2 to 8% of all model parameters.
Questions for authors
Q1: Are you able to analyze the Llama-3.1-70B shortcoming (improvement in reconstruction error, degradation in downstream metric) further with an intervention study? Are you able to show that this relates to this specific model/data configuration, but the method is generally applicable.
Q2: Could you use different data splits for calibration and evaluations?
Q3: Can you measure inference latency improvements, memory footprint reductions?
2 comments
Thank you for your responses, however they don't really address my questions/suggestions (no intervention evidence for LLama3.1 70B, no separation of calibration vs selection data, and no latency measurements).
Response to Reviewer 5NpL: Reconstruction Mismatch, Data Splits, and Pruning Scope
Thank you for recognizing the soundness of the construction, the range of evaluated models, and the improvements in many configurations. We address the reconstruction mismatch, data selection, and practical scope below.
W1/Q1: Llama-3.1-70B and intervention evidence. We agree that better local reconstruction is not sufficient. At 80% Q/K sparsity, the reported QK error decreases from 5.10% to 3.03%, while WikiText-2/C4 PPL increases from 4.98/9.89 to 18.98/32.37. The diagnostic evaluates the complete masks on common dense block inputs before RoPE; it does not measure post-softmax attention distributions or accumulated errors through the sparse network.
There are two relevant controls in the submission. First, Table 1 and Appendix M separate scoring from allocation. On this same checkpoint at 80%, changing Wanda from row to matrix budgets raises WikiText-2/C4 PPL to 17.69/24.21 without introducing QK scoring. Second, Appendix C explicitly varies whether RoPE is included in the score. In that separate three-seed study on smaller Llama 3 checkpoints, RoPE-aware scoring gives lower post-RoPE error in four of six configurations but worse mean PPL on both datasets in all six. These observations highlight the need to distinguish scoring from allocation and caution against attributing the failure solely to omitted RoPE.
Neither control is a complete intervention study of the 70B failure. In particular, the evidence does not justify claiming that this is an isolated configuration: all four Llama 3 checkpoints have worse PPL at 80%. The supported conclusion is that downstream benefits depend on the checkpoint and pruning regime. Layerwise attention-probability diagnostics and targeted restoration of selected pruned projections would help distinguish candidate explanations, but those results are not established by the submission.
W2/Q2: calibration versus selection data. Calibration tokens are sampled from C4 training, separately from the PPL evaluation data. Nevertheless, the reported WikiText-2 test and C4 validation PPL on Llama-3.1-8B were also used to choose calibration settings and the shared-budget default. We agree that this creates selection bias and that the train/evaluation separation of calibration tokens does not resolve it. The selected configuration was fixed across the other checkpoints, which tests transfer across models on the same benchmarks, not generalization to untouched evaluation data. Section 5 acknowledges this distinction. A fresh evaluation set with frozen settings—or separate development and final-evaluation data when retuning—is required to address the concern.
W3: pruning only a small fraction of total parameters. The Q/K-only experiments intentionally isolate the proposed component, and Appendix D.4 reports the denominator explicitly: 50% Q/K sparsity zeroes 2.23–8.06% of total parameters. However, the submission is not limited to that setting. Section 4.5 and Table 2 prune all seven block projections: Q/K/V/O and the MLP gate/up/down matrices. The hybrid uses QK-Wanda only on Q/K and Wanda elsewhere, compared with Wanda throughout.
At 50% block-projection sparsity, TinyLlama WikiText-2/C4 PPL improves from 11.52/13.85 to 11.22/13.62. On Llama-2-7B, WikiText-2 PPL changes from 6.89 to 6.85 and mean accuracy from 60.23% to 61.09%. These are modest component-level gains, not the larger gains seen in Q/K-only pruning. Embeddings and the output head remain dense, and neither pipeline preserves useful quality at 80% full-block sparsity.
Q3: latency and memory. We agree that weight-removal counts and scoring-time measurements do not establish inference speedups or memory-footprint reductions. Appendix L concerns calibration and scoring; no end-to-end inference benchmark is reported. A deployment claim would require an implemented sparse representation and execution path, with actual latency and memory measurements. The present empirical claim is about the quality of masks at matched pruning budgets, including their use within a broader pruning pipeline—not measured serving acceleration.