acceptodds
Under review as a conference paper at ICLR 2027

The Compression Square: Weight Quantization and KV-Cache Eviction Do Not Compose at the Operating Point

Abstract

A deployed long-context language model is compressed twice: its weights are post-training quantized, and its KV cache is pruned by an eviction policy that ranks tokens with the model's own attention. Each axis is evaluated alone and the two losses are assumed to add, which misses that the scores deciding what to evict come from the quantized model. We measure the coupling with the compression square, a factorial over model precision and selection source whose counterfactual transplant cell splits the interaction exactly into an oracle term (the quantized model kept the wrong tokens for itself) and a fragility term (it is more hurt by the same eviction). Across 499k released per-item records on Qwen2.5-7B, Llama-3.1-8B and Qwen2.5-3B at 4k–32k, the deployed cell is significantly worse than additive in 71 of 143 identifiable SnapKV cells (63 after Benjamini–Hochberg) and sub-additive in 3; on the needle tasks the effect is a tail: most items are unaffected and a few are badly hurt. Its size is set by the quantizer (two noisy RTN builds exceed the three shipped W4-g128 recipes in 32 of 33 matched conditions), but the shipped recipes are not exempt on real documents: on Llama-3.1-8B LongBench, GPTQ W4-g128 is super-additive in 9 of 9 cells over three eviction policies at 288 items (5 at 120), and on Qwen2.5-7B LongBench, where eviction alone costs 0.7% of the baseline loss and RTN W4-g128 +0.07 nats, the two together cost +0.49 [+0.14, +0.92] more than their sum. An intervention supports the channel: i.i.d. noise injected into the ranked scores alone, with the model untouched, opens it at a level within a factor of two of a quantizer's own score error. At these budgets, single-axis evaluation does not predict the deployed system. All code and the 499,289 per-item records will be made public upon acceptance at https://github.com/[anonymized]/[anonymized].

Primary area:
foundation or frontier models, including LLMs
Keywords:
weight quantization, kv-cache eviction, model compression, interaction effects, factorial design, llm inference, compression composition

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_47KBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the interaction between weight quantization and KV-cache eviction based on the model’s own attention scores. A factorial comparison measures the excess loss beyond the sum of the individual compression losses. By transplanting retained token positions between precision settings, the authors distinguish selection changes from changes in sensitivity to the same eviction. Experiments cover multiple Qwen and Llama models, quantization configurations, cache budgets, synthetic retrieval tasks, and LongBench QA tasks.

Critical strengths
  1. A useful diagnostic design. Transplanting retained token positions helps distinguish selection changes from differences in model sensitivity to the same eviction. This provides more insight than reporting the performance degradation of joint compression alone.

  2. Some findings have substantial robustness evidence. The paper includes paired statistical analyses, multiple-testing corrections, and held-out replications for selected configurations. The Llama LongBench results with GPTQ W4 show that the phenomenon is not confined to weaker quantizers or synthetic tasks. The authors also report negative results and scope limitations transparently.

Critical weaknesses
  1. Limited research novelty. Quantization-induced selection changes and non-additive compression errors have been studied previously. The retained-position transplants provide a useful extension to weight-quantized models, but the interaction statistic is a standard factorial contrast, and the exact decomposition follows algebraically from the transplant construction. The additional generalizable mechanistic insight remains limited.

  2. Limited robustness of the “near-lossless alone, harmful together” regime. After a fourfold increase in sample size, only one of the six initially significant configurations still meets the near-lossless-alone criteria (Table 12). The interaction persists, but the stronger claim has a narrower scope than the initial results suggest.

  3. Limited validation of generation quality and generality. Teacher-forced loss and synthetic-task exact match do not establish the impact on free-generation quality. The three policies supporting the main findings share a similar scoring mechanism; original step-wise TOVA/H2O and broader selection mechanisms remain untested.

  4. Limited predictive support for the mechanism. Noise injection supports selection corruption as a source of damage, but displaced attention mass weakly predicts the corresponding loss at 7B. The scoring-window prediction does not hold across model families, and W_K protection shows no consistent advantage over equal-cost W_V protection. The calibration-noise metric has not demonstrated additional ranking value over perplexity in the tested builds.

Questions for authors

Please address the four concerns outlined under Critical Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 47KB

Thank you. We take the four concerns in turn.

1. Novelty. The interaction statistic is deliberately the standard factorial contrast, and we claim no novelty for it, nor for the Shapley or mediation algebra. The contribution is what the design makes measurable:

  • Two executable selection counterfactuals. The quantized model decoding from the clean model's retain set, and the converse. A plain 2×2 cannot provide them, and they turn the channel attribution from an assumption into a measurement. Both were in the submission. The revision reports the split through each, their order-free average and their difference.
  • A matched excess-loss measurement. Eviction had been run on weight-quantized models (H₂O App. C, KVzip, Kim et al. 2026), but the interaction had not been estimated with item-level uncertainty, multiplicity control and held-out replication.

What the transplants then show is that the attribution is model-dependent:

  • On Llama-3.1-8B the selection carries about half of the interaction under either reference.
  • On Qwen2.5-7B the quantized retain set mainly costs a decoder that is itself quantized.

The needle-retention gap (130 of 143 cells, Spearman +0.41 with Δ_int) needs only the two self arms' selections. Its link to the oracle term (Spearman +0.32) needs the transplants.

2. The near-lossless-alone regime. Agreed: at four times the items, only one of the six configurations still meets the near-lossless-alone criteria (Table 12). The revision does not rest on it: it is in neither the abstract nor the contributions, and at large Qwen budgets the generation run finds no significant interaction either. What holds at full power is the shipped-build aggregate in our reply to AyEm: over 90 cells, 26 significantly super-additive and 3 sub-additive (Benjamini–Yekutieli 10 / 0); pooled +0.09 [+0.04, +0.14] nats on the needles and +0.22 [+0.06, +0.39] on LongBench, where quantization alone costs a median −0.002 and +0.065 nats.

3. Generation quality and policy breadth.

  • Generated answers. On the Table 21 LongBench cells, the interaction on 1 − F1 is positive in 8 of 9 cells (4 significant, none significantly negative), and on exact match significantly positive in 5 of 9. Without a chat template the answers often run long, and gold-substring match shows no statistically significant interaction. App. G.11 reports all three metrics (see our reply to vggh, Q3).
  • Tail exact match. It is conditional on the gold first token. It agrees with gold-answer containment under free greedy decoding on 98.3% of item-arms in the Qwen 4k grid.
  • Policy breadth (new App. G.12). Three further policies in the same harness, with transplant grids where a policy reaches the band:
    • chunked per-step TOVA and H₂O, evicting after every 32-token prompt chunk and at every answer step;
    • key-norm eviction (Devoto et al., 2024), which never consults a query;
    • head-averaged TOVA, the variant TOVA's authors recommend and kvpress implements.
  • Results.
    • In a 12-item single-needle calibration, the chunked per-step policies never reach a clean tail exact match above 0.17 at any budget up to 2048 of ≈3.9k tokens: the needle is evicted before the question arrives. Key-norm stays at 0.00 up to B = 1024. No budget qualifies, so their interaction is not identifiable. A strict token-by-token run remains open.
    • Head-averaged TOVA does reach the band. On Qwen2.5-7B at 4k (B ∈ {48, 64, 96}), Δ_int is significantly positive in 9 of 20 identifiable cells and negative in 1. All positives are in the noisy builds; the shipped builds give 0 positive and 1 negative of 10.
    • On Llama-3.1-8B LongBench with GPTQ W4-g128 it is +0.43 [+0.07, +0.82] nats at B = 256 and not significant above.
    • The phenomenon is therefore not specific to SnapKV's scoring window. It remains a finding about attention-based selection.

4. Mechanism. We agree that:

  • the displacement → loss link is within noise at 7B;
  • C3 fails across model families;
  • W_K protection does not beat equal-cost W_V protection, our placebo;
  • the σ screen orders quantizers the same way as the perplexity gap.

The paper reports all four. The revision adds an observable association that does hold across conditions, the retention gap above, but it is not an item-level predictor (median within-cell correlation +0.08). The noise intervention, which leaves the model untouched, opens the channel at a level within a factor of two of a quantizer's own score error (52 of 66 conditions). The paper's value is a bounded empirical diagnostic, not a predictive theory or a fix.

Thank you for the detailed responses and additional experiments. These have addressed my main concerns, and I am raising my rating from 2 to 3 .

Thank you for reading the responses and for raising your score. We will carry the narrowed claims and the new analyses into the final version.

Reviewer_AyEmOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The core question the paper poses is whether losses from 3-4 bit weight PTQ and post prefill KV-cache eviction just add up or interact, this poses and interesting question, since query-aware eviction ranks tokens using the quantized model's own attention. The design is a factorial over weight precision and selection source, plus two extra "transplant" conditions where one model decodes from the retain set chosen by the other. That setup splits the 2x2 interaction into an oracle (selection) term and a fragility remainder. Loss is defined as the teacher-forced NLL summed over the gold answer tokens, averaged over items. They only count cells at budgets where the clean policy still works. The models used in the experimants are Qwen2.5 7B, Llama-3.1 8B, and Qwen2.5 3B. Furthermore, tasks include RULER style retrieval and three LongBench QA subsets. The main finding is that about half of the 143 identifiable SnapKV results are significantly super-additive. Only 3 are sub-additive. The paper flags that, the quantizer sets the magnitude, and most positive results come from two noisy RTN builds.

Critical strengths
  1. This is a strong paper. From my point of view, the main contribution is the transplant condition. A plain 2x2 interaction cannot provide this contrast. Because the retain set is a discrete object that can be saved and replayed, the cross-model condition is measured rather than assumed. This allows the oracle term to hold weights and items fixed, isolating the effect of selection alone (Fig 1, Sec 3). I did not find prior work that computes this interaction or decomposes it in this way (see Q4), and the protocol is cheap enough for others to reuse.

  2. The authors report where their claims fail, in the main text: the free-alone certification mostly does not replicate, one of the theory's predictions is unsupported, and protecting WKW_K performs no better than a same-cost placebo at 3 bits and worse at 4. This is particularly relevant in this case because it marks how far the mechanism story can be pushed.

Critical weaknesses
  1. The strongest claims generalize past the evidence. Most positive results come from two noisy RTN builds that a single-axis perplexity or EM check already flags, yet the title, the first sentence of the conclusion, and contribution (ii) state the result for weight quantization in general. On the W4-g128 version most commonly used, the effect is small and mostly not significant: 8 positive and 3 negative out of 47 results (it needs to be mentioned that Llama LongBench at GPTQ W4-g128 is significant). My assessment is that this sits at budgets where eviction alone already costs 7-49% of the full-cache loss, so it is not the cheap regime usually used. The abstract also points to a near-free case, Qwen at B = 2048, where RTN W4-g128 and SnapKV cost 0.07 and 0.06 nats alone, with the interaction adding 0.49 more than the sum of the two. However, this is the only one of the shipped-build results in the table 22 grid that is significant by the mean. Appendix G11 itself concedes that what generalizes is the design and the single-axis blind spot, not the magnitude. I would qualify the title and add the aggregate W4-g128 figures along with their single-axis costs to the abstract.
Questions for authors
  1. The "because" in the conclusion depends on a reference choice. Referencing the decomposition through the other measured transplant gives an equally exact split whose selection channel, Δoq\Delta_{oq}, is null in aggregate. Δoq\Delta_{oq} in fact tracks Δoracle\Delta_{oracle} at Spearman .7. Could you report that split alongside equation 1, with per-model oracle shares and item-level bootstrap confidence intervals, and say whether the cutoff was fixed before the split was computed? If the selection channel accounts for only a small share under the bf16 reference, should that maybe be softened?

  2. For the W4-g128 versions on the 7 to 8B models, using the largest item set per result and a method that accounts for dependence, how many results come out super or sub-additive per these runs?

  3. The noise probe perturbs the clean model's ranking scores, so it answers a different question from the one in the conclusion. Could you add counts on how often the needle is kept under the quantized versus the clean selection, and report whether that gap tracks the super-additive results?

  4. Is the contribution (ii) "first measurement" framing too strong? Q-Hitter, which you cite, already shows heavy hitters being reclassified under cache quantization. does the section 2 description of it as stopping at set overlap still hold? And has eviction been run on weight-quantized models elsewhere in the KV-compression literature in a way contribution (ii) should be compared against?

4 comments
AuthorsOpenReview ↗

Response to Reviewer AyEm (part 1/2)

Thank you for the thorough engagement. Your reference-choice question changed how we state the result.

Q1 — the split depends on the reference. Correct; the revision makes this explicit.

  • The split. Δ_int telescopes exactly through L_1f (Δ_oracle + Δ_frag) and through L_0q (Δ_oq + the remainder). The two selection terms differ by Δ_oracle − Δ_oq = L11 − L1f − L0q + L01. This difference measures how much the cost of changing the retain set depends on decoder precision.
  • What we report. Both splits, their average (the order-free, Shapley, attribution to the selection), and the difference.
  • Intervals. A block bootstrap draws one weight per prompt, so every cell sharing a prompt, also across models, moves together.
cellsvia L_1fvia L_0qaverageΣ(Δ_oracle − Δ_oq), nats
Qwen2.5-7B7621% [−2, 36]−8% [−44, 13]7% [−20, 23]+12.2 [+5.0, +20.2]
Llama-3.1-8B4558% [44, 70]45% [31, 55]52% [39, 60]+2.9 [+0.1, +5.5]
all12534% [23, 43]12% [−6, 24]23% [10, 32]+15.3 [+7.9, +22.8]
exact-index re-run: Qwen2.5-7B7622% [−0.2, 36]−6% [−42, 13]8% [−19, 23]+12.0 [+4.7, +19.3]
exact-index re-run: Llama-3.1-8B4559% [44, 70]45% [31, 56]52% [40, 61]+2.9 [+0.1, +5.6]

The 0.05-nat cut was not pre-specified, so the revision leads with the uncut figures. The cut ones (41% / 22% / 31% over 87 cells) appear only as a sensitivity check; this is not the 70-cell set behind the submission's 41%.

  • Pooled. You are right that under the bf16 reference the selection share is not distinguishable from zero in aggregate: 12% [−6, 24].
  • Qwen. Neither single-reference share is distinguishable from zero, but their difference is. The quantized retain set costs a quantized decoder more than a clean one; we do not claim it costs a clean decoder nothing.
  • Llama. The selection channel carries about half of the interaction under either reference.
  • These are statements about the evaluated grids, which are not matched cell for cell across the two models.

So yes, we have softened the conclusion. The submission said the two compressions do not compose "because the eviction policy's scores come from the quantized model". The revision calls this one reason and gives the model-dependent attribution above (§6).

Q2 — W4-g128 at the largest item set, dependence-aware.

  • Cells. For every W4-g128 cell on Qwen2.5-7B and Llama-3.1-8B we take the run with the most items; no cell pools runs. The band is re-evaluated on that set.
  • Significance. Per cell, the 95% bootstrap interval at 4000 resamples, as throughout the paper. For the multiplicity corrections, p-values at the resampling floor are refined to 200,000 resamples.
  • Coverage. This set is larger than the 47 results in the submission's tables: each cell is taken at its largest item set, and the LongBench cells are included.
methodneedles (56)LongBench (34)all (90)
per cell, 95% interval11 / 315 / 026 / 3
Benjamini–Hochberg3 / 012 / 016 / 0
Benjamini–Yekutieli1 / 07 / 010 / 0

One test per unit at its smallest identifiable budget gives 10 / 2 of 29 overall (Holm 5 / 0).

pooled interaction (mean), natsΔQ mean / medianΔE mean / median
needles+0.09 [+0.04, +0.14]−0.019 / −0.002+0.43 / +0.14
LongBench+0.22 [+0.06, +0.39]+0.035 / +0.065+1.91 / +1.90
  • By build. GPTQ 21 / 0 of 46; RTN W4-g128 5 / 2 of 39; AWQ 0 / 1 of 5. On the needles 10 of the 11 positives are GPTQ, and the median cell there is −0.002 nats. The needle aggregate is therefore a GPTQ effect in a minority of cells.
  • By model, LongBench. Llama 8 / 0 of 8; Qwen 7 / 0 of 26.

Q4 — "first measurement". Restated. Our §2 description of Q-Hitter did not hold and is corrected (see our reply to vggh, W2). Eviction has also been run on weight-quantized models before:

  • H₂O's App. C reports the four cells of eviction × 4-bit on OPT-30B.
  • KVzip evaluates eviction policies on a W8A8KV4 model.
  • Not All Bits Are Equal (ICLR 2026) traces memory–accuracy frontiers of GPTQ-quantized reasoning models under R-KV.

None of them contrasts the joint cell with the sum of the single-axis costs, attaches uncertainty to that contrast, or splits it by channel. Contribution (ii) now claims a controlled interaction estimate with item-level uncertainty. The related work cites all three, plus Q-Hitter's downstream measurement.

AuthorsOpenReview ↗

Response to Reviewer AyEm (part 2/2)

Weakness — the claims outrun the evidence. We agree. Point by point:

  • Abstract. It now gives the shipped-build aggregate (Q2 above) with both single-axis costs, as you suggested. The Qwen B = 2048 cell is gone from the abstract. In §5.4 it carries your caveat: it is a tail, and the generation run finds no counterpart there (−0.2 F1 points [−2.3, +1.6]).
  • Contribution (ii). It gives counts rather than "most cells": the shipped W4-g128 builds' own count (26 super-additive and 3 sub-additive of 90 at their largest item sets) and their exact-match rates (9.2% against 8.0%) stand beside the all-cell figures, which the noisy builds dominate.
  • Conclusion. It now opens "Near the eviction policy's failure boundary, weight quantization and KV-cache eviction need not compose." It also names the model dependence of the attribution.
  • App. G11. The section you cite is now G.13. Its statement, that the design and the single-axis blind spot generalize and the magnitude does not, stands, and the abstract and conclusion now say the same.
  • Title. See below.

The regime. Agreed. Most counted cells sit where eviction alone is not cheap; that is what the operating point is, the tightest budget a single-axis check still passes. The cheapest end on real documents is Llama-3.1-8B LongBench at B = 1024 (288 documents):

  • SnapKV alone costs 8.8% of the full-cache loss and GPTQ W4-g128 1.7%, yet the joint system pays +0.25 [+0.12, +0.38] nats beyond their sum, 1.4× the parts.
  • For the released AWQ it is +0.32 [+0.18, +0.49], 1.5× the parts.

We make no claim at the generous budget fractions usually reported (§6).

Title. We propose "The Compression Square: When Weight Quantization and KV-Cache Eviction Fail to Compose". The 2027 author guide is inconsistent on title changes during discussion, so we will make the change when the AC confirms it is allowed, or at camera-ready.

Q3 — needle kept under the quantized vs the clean selection. Needle retention is the fraction of needle positions in the retain set, averaged over layers and kv-heads. It depends only on the selection.

  • The gap. Pooled across the 143 identifiable cells, the quantized model's own selection keeps a mean 0.020 [0.019, 0.021] less of the needle (block bootstrap over shared prompts): 0.032 on Llama and 0.013 on Qwen. It keeps less in 130 of the 143 cells.
  • Across cells. The gap tracks Δ_int (Spearman +0.41) and Δ_oracle (+0.32; +0.47 on Llama, +0.23 on Qwen). Its larger size on Llama matches the larger selection share there.
  • Super-additive cells. The median gap is 0.030 in the significantly super-additive cells against 0.007 in the rest.
  • Within cells. The item-level rank correlation with the oracle term has a median of only +0.08. This is an association between conditions, not an item-level predictor.
  • By item. Items that lose any needle positions (79%) have a mean oracle term of +0.25 nats, against −0.17 for the rest. With a post hoc threshold of 0.05, the 11% of items beyond it carry 53% of the summed oracle term.

Released checkpoints. Our GPTQ/AWQ builds are re-implementations, so we also ran released 4-bit checkpoints: Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 and -AWQ, and hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4. They are dequantised to bf16 in our harness, with the normalisation weights into which AWQ folds its scales, and validated by perplexity. This is not an end-to-end low-bit kernel test.

  • Llama-3.1-8B, Table 21 design (LongBench, 288 documents). The released AWQ costs +0.11 nats alone. Its interaction with eviction is positive in all 9 cells and significantly super-additive in 6 of the 8 identifiable ones, none sub-additive (median +0.23 nats).
  • Qwen2.5-7B LongBench. 3 of 20 identifiable cells are super-additive, all from the released AWQ; the released GPTQ is 0 of 10.
  • Qwen2.5-7B needles. 1 of 10 is super-additive and 2 are sub-additive, both on the multi-key needle at B = 160 (GPTQ −0.88 [−1.51, −0.23], AWQ −1.04 [−1.63, −0.47] nats).

So the Llama LongBench result does not depend on our re-implementation, while on Qwen the released recipes interact less than ours (App. G.11).

Thank you for the detailed responses. They address my main concerns. Two small clarifications, for the revision rather than for my assessment: (1) how was the 70-cell set behind the submission’s 41% selected, given that it differs from the set the 0.05-nat cut produces? (2) What did the exact-index re-run correct, and did the earlier indexing affect any other results in the paper? I keep my rating of 4. Great work!

Thank you for keeping your rating, and for the two clarifications. Both are now stated in App. G.11 of the updated revision.

(1) The 70-cell set. The submission's 41% was computed over the main grids only (Qwen2.5-7B at 4k and 16k, Llama-3.1-8B and Qwen2.5-3B at 4k): every identifiable SnapKV needle cell there whose submitted Δ_int exceeds 0.05 nats. The cut row of Table 24 applies the same threshold to a different base, the cells that carry both transplants across all six settings.

  • The two sets share 57 cells.
  • The cut row adds 16 Llama-3.1-8B 16k and 14 Qwen2.5-7B 32k cells.
  • It leaves out 13 protection and group-64 cells at Qwen 4k, which ran without the second transplant.

Both shares round to 41%, but they are different statistics, so we report them separately.

(2) What the exact-index re-run corrected. Only the transplant arms. Each transplant reuses the other model's retain set. In the submission that set was re-derived from scores stored in float16, so ties at the top-B boundary, which SnapKV's max-pooling makes common, could break differently from the self arm. The re-run stores the scores in float32, so each transplant now uses exactly the self arm's retain set.

  • Unaffected. Everything built from the four physical cells: Δ_int and every count, interval and replication, the LongBench results and the shipped-build aggregate. The noise probe is also unaffected, because it selects from in-memory scores.
  • Affected in principle, and re-run. Δ_oracle, Δ_frag, Δ_oq and what derives from them. The re-run covers all 143 counted SnapKV cells and the nine Table 21 cells. The changes are a median 0.004 nats and at most 0.10 (at most 0.012 on Table 21), and no share in Table 24 moves by more than two points.
  • Not re-run. The transplant terms of the other policies' needle cells (§5.3), the language-modelling control, the other LongBench grids and the 0.5B study remain float16-based, and App. G.11 now says so. Given the size of the changes where we did re-run, we do not expect them to move materially, but we have not verified it.
  • Mechanism diagnostics. These recompute selections from the stored scores, so they carry the same tie-level perturbation (a median 0.07% of retained positions in the counted cells). That is two orders of magnitude below the 8–22% of the retain set that quantization itself moves (Table 16).
Reviewer_vgghOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates interactions between weight quantization and attention-based KV-cache eviction. It introduces a factorial interaction statistic and cross-model selection transplants to separate selection-induced degradation from increased sensitivity to a fixed eviction. Experiments primarily on Qwen2.5-7B and Llama-3.1-8B, using synthetic retrieval and LongBench QA, demonstrate super-additive losses at particular quantizer–budget combinations. Score-only noise interventions provide additional evidence for the selection pathway.

Critical strengths
  1. Useful experimental framework for studying compression interactions. The factorial contrast and transplant design (§3.1, Eq. 1) provide a meaningful way to distinguish changed token selection from sensitivity to eviction. The score-only intervention strengthens this analysis: perturbing selection scores while keeping model weights unchanged produces significantly positive loss differences in 52/66 tested conditions (Appendix G.2).

  2. Substantial empirical evidence, including calibrated 4-bit quantization on real documents. The main SnapKV analysis retains 63 significantly positive interactions among 143 eligible cells after Benjamini–Hochberg correction (§5.1). More compellingly, GPTQ W4-g128 on Llama-3.1-8B LongBench shows positive interactions in 9/9 cells at 288 items, with 8/9 surviving 10% trimming and 8/9 significant on the 168 held-out documents (Appendix G.8, Table 21). These results establish that the phenomenon extends beyond noisy RTN configurations and isolated synthetic-task outliers.

Critical weaknesses
  1. The implementation does not fully preserve the fixed-selection interpretation of the decomposition. Appendix C (pp. 16–17) reports that self arms select from float32 scores, whereas transplant arms reconstruct selections from float16-stored scores. This changes 0.07–1.92% of retained positions, with per-item loss changes reaching 0.25 nats in the 12-pair check. Although this leaves the four-cell interaction unchanged, it introduces an additional selection perturbation into the oracle/fragility attribution. Given the importance of rare items and boundary ties, small score differences alone do not establish negligible attribution error. A cell-level check using the exact original retained indices is needed.

  2. The novelty relative to Q-Hitter is overstated in §2. Q-Hitter [1] already evaluates downstream perplexity using quantization-perturbed token selections while preserving the KV embeddings in full precision (§4.2, Table 1). For example, Pythia-160M/WikiText-103 perplexity changes from 31.81 under the original selection to 35.15 under the selection induced by 3-bit KV quantization. Thus, describing that work as stopping at set overlap is inaccurate. The weight-quantization pathway and complete factorial decomposition remain useful distinctions, but the contribution should be positioned accordingly.

[1] Zhang et al. Q-Hitter: A Better Token Oracle for Efficient LLM Inference via Sparse-Quantized KV Cache. MLSys 2024, §4.2 and Table 1.

Questions for authors
  1. Attribution validation: When the exact retained indices from the self arms are reused, how do the main oracle/fragility estimates, and the reported 41% oracle contribution, change? Please provide paired cell-level results rather than only individual-item checks.

  2. Scoring clarification: How is the first answer token evaluated on LongBench? Section 4 says every answer token is scored through the pruned cache, whereas the synthetic protocol predicts the first token from pre-eviction prompt-final logits. Is the final prompt token recomputed, and how is consistency maintained across the experimental cells?

  3. Practical significance: For the GPTQ settings in Table 21, do the interactions also translate into lower generated-answer F1 or exact match? Teacher-forced NLL is a useful diagnostic, but generated-answer measurements would materially strengthen the deployment interpretation.

3 comments
AuthorsOpenReview ↗

Response to Reviewer vggh (part 1/2)

Thank you for the precise reading.

W1 / Q1 — exact retained indices. You are right.

  • The problem. Self arms selected from the in-memory float32 scores; transplant arms re-selected from scores stored in float16.
  • The re-run. run_grid.py --exact also stores the scores in float32, which reproduces each self arm's selection index for index; we assert this on every item. Both versions of every transplant arm are built in the same run, so the comparison is paired cell by cell.
  • Your 41%. On the same 70 cells (main grids, submitted Δ_int > 0.05), the oracle share is 41.0% with float16-stored selections and 41.1% with exact ones.
  • Over all 143 counted cells:
    • Float16 storage moved a median 0.07% of a cell's retained positions, at most 0.11%. This is a set difference averaged over each cell's items, so it is not a worst-item bound. The 0.07–1.92% in App. C is a different count: an entry-by-entry comparison of sorted index tensors over three policies, where one displaced position is counted once for every entry it shifts. App. C now says so.
    • Exact indices change Δ_oracle by a median 0.004 nats and at most 0.10 (Spearman 0.999). Δ_int is the same within the run, so Δ_frag changes by exactly the opposite amount.
    • Cells with Δ_oracle significantly positive / negative: 44 / 4 with float16, 46 / 4 exact (45 / 5 on the submitted records).
    • The aggregate share is 37% under both, and Δ_oq changes by at most 0.11 nats.
  • Both references. Table 24 now has a second block computed from the exact arms. No share moves by more than two points (Llama: 52% → 52% by the order-free average; Qwen: 7% → 8%).
  • Paired cells. The nine Table 21 cells (Llama-3.1-8B LongBench, 288 documents), Δ_oracle in nats with float16 → exact. The four physical arms are unchanged, so Δ_frag moves by the opposite amount.
B = 256B = 512B = 1024
SnapKV0.253 → 0.2550.135 → 0.1400.108 → 0.118
TOVA0.232 → 0.2320.190 → 0.1870.056 → 0.060
PyramidKV0.147 → 0.1540.144 → 0.1320.195 → 0.200
  • Reproduction. The re-run reproduces the submitted Δ_int exactly in the 138 cells built by deterministic quantizers. The 5 AWQ cells differ because the submitted AWQ build drew its activation subsample unseeded (disclosed in App. G.13), and the re-run uses the seeded rebuild. This also explains the one sign change: Spearman 0.982 between re-run and submitted Δ_int, same sign in 142 of 143. Δ_int moves by up to 0.44 nats in those cells, in line with the build-to-build gap of the AWQ recipe that App. G.13 reports (up to 0.43 nats on a cell mean). Within that build the float16-vs-exact comparison is still paired.
  • Unaffected. As you note, the four physical cells — hence Δ_int and every count in §5.1 — do not depend on this storage.
  • All cells. The paired values for all 143 cells (Δ_oracle and Δ_oq under both storages, and Δ_int, from which Δ_frag follows) are in the supplement, results_gpu/rebuttal_gpu.json.

W2 — Q-Hitter. Agreed, our sentence was inaccurate. Q-Hitter §4.2 / Table 1 decodes with full-precision KV from the retain set selected under quantized-cache attention (Pythia-160M / WikiText-103: 31.81 → 35.15 at 3 bits). That is the clean-decoder, perturbed-selection arm — our L_0q — for the cache pathway, and the revised related work says so. What we add:

  • the weight pathway, which perturbs the decoder as well as the scores;
  • the matched four-cell contrast with both transplant arms, which makes the split exact under either reference.
AuthorsOpenReview ↗

Response to Reviewer vggh (part 2/2)

Q2 — first answer token on LongBench.

  • Evicted arms. We remove the cached final prompt token, recompute it at its original position through the retained cache, and use its logits to score answer token 0. The remaining answer tokens are teacher-forced through the same cache.
  • Full-cache arms. These keep the original prefill logits, which are the same full-cache conditional.

Position L−1 is in every policy's forced window, so the retain set keeps |S| = B.

The two task families use different rules on purpose, and each rule is the same for every arm within its family:

  • Needles. The first answer token is decided by the prefill before eviction acts, so the tail metrics exclude it.
  • LongBench. Answers are often one token, so the first token is scored through each cell's own cache as above.

§4 now states both rules and the full-cache case.

Q3 — generated answers. On F1 and exact match, yes; on gold-substring match, no.

For the nine Table 21 cells (Llama-3.1-8B, GPTQ W4-g128, SnapKV / TOVA / PyramidKV × B ∈ {256, 512, 1024}, 288 documents), every arm generates its answer greedily from its own pruned cache. We score the answers with LongBench's token F1, normalised exact match and gold-substring match against all gold answers.

  • F1. The interaction on 1 − F1 is positive in 8 of 9 cells and significant in 4 (1 after Benjamini–Hochberg over the nine). None is significantly negative. The median is +1.3 F1 points, up to +4.0; eviction alone costs a median 2.6.
  • Exact match. Significantly positive in 5 of 9 (5 after Benjamini–Hochberg), negative in none.
  • What "positive" means. The joint answer is worse than the additive prediction, not necessarily worse than bf16. Under TOVA at B = 256 the four arms score 21.8 (bf16), 28.4 (GPTQ), 19.4 (bf16 evicted) and 22.0 (GPTQ evicted), against an additive prediction of 26.0.
  • The caveat. Our prompts omit the chat template, so the bf16 model runs to LongBench's length cap on 91% of documents and the GPTQ model on 67%.
    • Quantization alone therefore raises F1 (21.8 → 28.4) and exact match (1.4% → 8.7%) but lowers substring match (48.3% → 41.7%).
    • On substring match, which ignores length, the interaction is not positive in any cell.
    • The F1 oracle term is not significant in any cell, but its intervals are wide. They do not exclude a selection contribution, so they do not show that fragility alone carries the interaction in generation.
  • Large budgets. On Qwen2.5-7B at the large LongBench budgets (8 cells, 120 documents), no generation interaction is significant in either direction. The RTN W4-g128 SnapKV cell at B = 2048 is −0.2 F1 points [−2.3, +1.6].

The generation-level evidence is therefore model- and budget-dependent. App. G.11 reports all three metrics; §5.4 has one sentence.

Tail exact match. Tail exact match scores the answer after its first token. It measures greedy continuation conditional on the gold first token, not unconditional generation.

  • Agreement with generation. On the Qwen2.5-7B 4k grid (four builds, SnapKV at B ∈ {96, 160}, 64 items per task, both transplants), free greedy decoding contains the gold answer exactly when tail exact match holds in 3,021 of 3,072 item-arms (98.3%). The exceptions are 33 tail-only successes and 18 the other way. This check covers that grid only.
  • All 143 cells. On tail exact match the joint system fails on 16.4% of items where the single-axis failure rates add to 8.6% (+7.8 points [6.1, 9.3]; 30 cells significantly positive, 1 negative). This is mostly the noisy builds.
  • Shipped builds. At their largest item sets: 9.2% against 8.0% (+1.2 [+0.3, +2.1]; 4 positive and 1 negative of 56), mostly GPTQ.

Thanks for providing additional experiments and analysis. My primary concerns have been addressed. I will keep my current rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response (to all reviewers and the AC)

We thank the reviewers for careful and constructive reviews. The revision answers the main questions with new runs and with new analyses of the released records. Every number below is regenerated by scripts in the supplement. In the revised PDF, passages added or changed since the submission are in blue. New appendix sections are G.11 (reference choice, exact match, generation, released checkpoints, shipped builds) and G.12 (further policies); the former G.11 is now G.13.

  1. Exact retained indices (vggh). Every counted cell was re-run with transplant arms built both ways in the same run. On the 70 cells behind the submission's 41% oracle share, the re-run gives 41.0% with float16-stored selections and 41.1% with exact ones. No share in the two-reference table (Table 24) moves by more than two points.
  2. Generated answers and released checkpoints (vggh, AyEm, 47KB).
    • On the Table 21 LongBench cells, answers generated freely from each pruned cache show a positive interaction on 1 − F1 in 8 of 9 cells (4 significant, 1 after Benjamini–Hochberg) and on exact match in 5 of 9. We state the protocol's limits in App. G.11: no chat template, so answers often run to the length cap, and gold-substring match shows no statistically significant interaction. At Qwen's large budgets no generation interaction is significant.
    • The released AWQ checkpoint of Llama-3.1-8B is super-additive in 6 of 8 identifiable Table 21 cells, none sub-additive. On Qwen2.5-7B the released builds interact less than ours.
  3. Shipped W4-g128 builds at full power, with both single-axis costs (AyEm). Over 90 identifiable cells: 26 significantly super-additive and 3 sub-additive (Benjamini–Yekutieli 10 / 0).
    • Needles: pooled +0.09 [+0.04, +0.14] nats, where quantization alone costs a median −0.002 nats and eviction alone +0.14.
    • LongBench: +0.22 [+0.06, +0.39] nats, where quantization alone costs +0.065 and eviction alone +1.90 (26% of the full-cache loss).
    • The abstract now reports these numbers.
  4. Narrower claims.
    • Q-Hitter is described correctly, and "first measurement" is withdrawn.
    • Tail exact match is described as what it is: greedy continuation from the gold first token.
    • The new per-step policies are chunked variants.
    • The conclusion now opens "Near the eviction policy's failure boundary, weight quantization and KV-cache eviction need not compose."

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.