- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the interaction between weight quantization and KV-cache eviction based on the model’s own attention scores. A factorial comparison measures the excess loss beyond the sum of the individual compression losses. By transplanting retained token positions between precision settings, the authors distinguish selection changes from changes in sensitivity to the same eviction. Experiments cover multiple Qwen and Llama models, quantization configurations, cache budgets, synthetic retrieval tasks, and LongBench QA tasks.
Critical strengths
-
A useful diagnostic design. Transplanting retained token positions helps distinguish selection changes from differences in model sensitivity to the same eviction. This provides more insight than reporting the performance degradation of joint compression alone.
-
Some findings have substantial robustness evidence. The paper includes paired statistical analyses, multiple-testing corrections, and held-out replications for selected configurations. The Llama LongBench results with GPTQ W4 show that the phenomenon is not confined to weaker quantizers or synthetic tasks. The authors also report negative results and scope limitations transparently.
Critical weaknesses
-
Limited research novelty. Quantization-induced selection changes and non-additive compression errors have been studied previously. The retained-position transplants provide a useful extension to weight-quantized models, but the interaction statistic is a standard factorial contrast, and the exact decomposition follows algebraically from the transplant construction. The additional generalizable mechanistic insight remains limited.
-
Limited robustness of the “near-lossless alone, harmful together” regime. After a fourfold increase in sample size, only one of the six initially significant configurations still meets the near-lossless-alone criteria (Table 12). The interaction persists, but the stronger claim has a narrower scope than the initial results suggest.
-
Limited validation of generation quality and generality. Teacher-forced loss and synthetic-task exact match do not establish the impact on free-generation quality. The three policies supporting the main findings share a similar scoring mechanism; original step-wise TOVA/H2O and broader selection mechanisms remain untested.
-
Limited predictive support for the mechanism. Noise injection supports selection corruption as a source of damage, but displaced attention mass weakly predicts the corresponding loss at 7B. The scoring-window prediction does not hold across model families, and W_K protection shows no consistent advantage over equal-cost W_V protection. The calibration-noise metric has not demonstrated additional ranking value over perplexity in the tested builds.
Questions for authors
Please address the four concerns outlined under Critical Weaknesses.
3 comments
Thank you for the detailed responses and additional experiments. These have addressed my main concerns, and I am raising my rating from 2 to 3 .
Thank you for reading the responses and for raising your score. We will carry the narrowed claims and the new analyses into the final version.
Response to Reviewer 47KB
Thank you. We take the four concerns in turn.
1. Novelty. The interaction statistic is deliberately the standard factorial contrast, and we claim no novelty for it, nor for the Shapley or mediation algebra. The contribution is what the design makes measurable:
What the transplants then show is that the attribution is model-dependent:
The needle-retention gap (130 of 143 cells, Spearman +0.41 with Δ_int) needs only the two self arms' selections. Its link to the oracle term (Spearman +0.32) needs the transplants.
2. The near-lossless-alone regime. Agreed: at four times the items, only one of the six configurations still meets the near-lossless-alone criteria (Table 12). The revision does not rest on it: it is in neither the abstract nor the contributions, and at large Qwen budgets the generation run finds no significant interaction either. What holds at full power is the shipped-build aggregate in our reply to AyEm: over 90 cells, 26 significantly super-additive and 3 sub-additive (Benjamini–Yekutieli 10 / 0); pooled +0.09 [+0.04, +0.14] nats on the needles and +0.22 [+0.06, +0.39] on LongBench, where quantization alone costs a median −0.002 and +0.065 nats.
3. Generation quality and policy breadth.
4. Mechanism. We agree that:
The paper reports all four. The revision adds an observable association that does hold across conditions, the retention gap above, but it is not an item-level predictor (median within-cell correlation +0.08). The noise intervention, which leaves the model untouched, opens the channel at a level within a factor of two of a quantizer's own score error (52 of 66 conditions). The paper's value is a bounded empirical diagnostic, not a predictive theory or a fix.