- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
WriteKV is a query-agnostic KV-cache eviction method for hybrid-attention models. It uses the native writes produced by linear-attention layers as a signal for which full-attention KV entries are likely to contain novel information. The method scores write directions with block-wise Ridge statistics, aggregates scores across heads and layers, and selects a global layer-token budget for physical cache compaction. On Qwen3.5 models, WriteKV performs well across LongMemEval, RealTalk, and LoCoMo, especially at low retention rates. At 20% KV retention, it reduces packed attention latency to 26.8% of the full-KV value, with a reported 4.20% cache-preparation overhead on an 800K-token prefix.
Critical strengths
- The cross-memory idea is interesting and well motivated. Using the update dynamics of linear-attention states to guide full-attention KV retention gives the method a genuinely query-agnostic signal, without future queries, context reconstruction, or a separately trained scorer.
- The quality gains are consistent across datasets, context lengths, and model sizes. At 10% retention on Qwen3.5-9B, WriteKV reaches 20.97%, 24.49%, and 31.90% F1 on LongMemEval, RealTalk, and LoCoMo, respectively, outperforming the strongest compression baseline by 8.05–19.96 points.
3.The paper takes the systems side seriously. It evaluates block-size trade-offs, physical cache compaction, preparation time, memory usage, and packed-attention latency. The method also reuses the same compressed cache across multiple queries, which is important for the stated query-agnostic setting.
Critical weaknesses
-
The runtime comparison across baselines is not fully controlled. WriteKV and KVzip use FlashAttention 2, while SnapKV, ProtoKV, and PyramidKV use the eager backend. The paper reports matched-backend overhead for WriteKV, which is useful, but the direct total-time comparison still mixes algorithmic overhead with backend differences.
-
Several scoring choices remain heuristic. The method fixes the spectral-energy threshold at (\tau=0.90), uses a block size of 4096, and relies on percentile normalization plus max aggregation. The paper studies block size and mean-versus-max aggregation, but does not show sensitivity to (\tau), regularization, or the association between linear and full-attention layers. This makes it harder to tell how robust the method is outside the reported configuration.
Questions for authors
-
Could you report full end-to-end decode latency, including linear-attention layers, projections, and the generation loop, for different numbers of queries sharing the same prefix? It would also help to report the break-even point where cache preparation becomes worthwhile.
-
Could you compare all systems under the same attention backend, or separate kernel effects from the algorithmic preparation cost more explicitly?
3.How sensitive are quality and cache-preparation cost to the spectral threshold (\tau), Ridge regularization, block size, and the mapping from linear layers to full-attention layers?
5 comments
W2. Several scoring choices remain heuristic. The method fixes the spectral-energy threshold at , uses a block size of 4096, and relies on percentile normalization plus max aggregation. The paper studies block size and mean-versus-max aggregation, but does not show sensitivity to , regularization, or the association between linear and full-attention layers. This makes it harder to tell how robust the method is outside the reported configuration.
Q3. How sensitive are quality and cache-preparation cost to the spectral threshold (), Ridge regularization, block size, and the mapping from linear layers to full-attention layers?
Response to W2 & Q3. We appreciate the reviewer's focus on hyperparameter robustness for both task quality and preparation cost. Referring to Sec. 3.3, Sec. 4.4, and Appendix A, several of these components are adaptive or architecture-defined rather than manually tuned hyperparameters:
Block size and adaptive Ridge regularization . Block size is quantitatively evaluated for both quality and preparation cost in Fig. 8(a)–(c): task quality across and retrospective No-block is nearly identical, while increasing from 2048 to 8192 slightly reduces 100K preparation time (38.25 s → 36.35 s) at a modest temporary peak memory trade-off (21.48 → 23.13 GiB). Regarding Ridge regularization, is not a preset hyperparameter; Eq. (11) computes it adaptively for each block and head from its spectral-tail energy () with negligible scalar overhead, specifically eliminating per-layer or per-length regularization tuning.
Spectral-energy threshold . In terms of preparation cost, eigendecomposition of small matrices takes the same negligible runtime regardless of , so cache-preparation cost is insensitive to . In terms of task quality, does not truncate the eigenvector space of (Appendix A.4)—the complete directional space is always retained, and only determines the scalar damping . While we have not directly swept , keeping a single fixed default without any per-task tuning yields consistent tight-budget gains across all evaluated datasets, 20K–100K context lengths, and model scales/architectures, demonstrating strong cross-setting transferability.
Layer mapping . Assigning each linear-attention layer to the next full-attention layer in model depth is a zero-search, architecture-defined causal rule (ensuring each full-attention layer observes writes produced by its immediately preceding linear layers) that transfers across different layer counts and hidden dimensions without per-model search. Still, we agree that ablating alternative cross-depth observer spans is valuable; we will clarify the rationale for this default mapping in the revision and leave direct sensitivity sweeps over and observer spans as additional analysis.
Q1. Could you report full end-to-end decode latency, including linear-attention layers, projections, and the generation loop, for different numbers of queries sharing the same prefix? It would also help to report the break-even point where cache preparation becomes worthwhile.
A1. We conducted a full-module end-to-end evaluation on Qwen3.5-9B with an 800K-token shared prefix (), covering all linear-attention layers, projections, cache isolation, and the complete autoregressive generation loop. We prefill and snapshot the 800K prefix once, then sequentially serve 2 independent downstream queries from the same cached prefix (restoring isolated state per query and generating 128 tokens via greedy decoding).
Per-query end-to-end decode latency is highly consistent (Full-KV: 22.282 s and 22.271 s, mean 22.28 s; WriteKV: 6.770 s and 6.743 s, mean 6.76 s), yielding the following stage-wise averages: TTFT drops from 387.34 ms to 226.39 ms (−41.55%), TPOT drops from 172.34 ms to 51.40 ms (−70.18%, 3.35×), cache isolation drops from 8.59 ms to 0.66 ms (−92.32%), and complete 128-token end-to-end decoding achieves a 3.30× speedup (saving 15.52 s per query).
Including the one-time 800K prefix preparation cost, cumulative end-to-end time across different numbers of shared queries is reported below ( are directly measured; since every query starts independently from the same deterministic snapshot under greedy decoding, are projected from the measured per-query decoding mean):
| Shared Queries (128 tokens each) | Full-KV Total | WriteKV Total | Net Time Diff (WriteKV − Full-KV) |
|---|---|---|---|
| (prep & snapshot only, measured) | 744.17 s | 789.82 s | +45.64 s |
| (measured) | 766.45 s | 796.57 s | +30.12 s |
| (measured) | 788.73 s | 803.33 s | +14.60 s |
| (projected from measured mean) | 811.00 s | 810.08 s | −0.92 s (break-even) |
| (projected from measured mean) | 855.55 s | 823.60 s | −31.96 s |
Under this 800K setup, WriteKV breaks even and achieves net total-time savings by the 3rd 128-token query; when serving 5 shared queries, break-even requires generating only 76 tokens per query on average.
We thank the reviewer for taking the time to review our paper and provide constructive feedback. We are pleased that the reviewer finds our cross-memory motivation interesting and well founded, recognizes WriteKV's consistent tight-budget performance across datasets, context lengths, and model scales, and appreciates our systematic evaluation of block-size trade-offs, physical cache compaction, and multi-query reuse. In the revision, we will incorporate the matched-backend controlled comparison, full-module end-to-end latency measurements, and clearer design clarifications to further strengthen the paper.
Response to W1 & Q2. We thank the reviewer for highlighting this important experimental control. The mixed backends in Fig. 9(a) arise directly from algorithmic dependencies: SnapKV, PyramidKV, and ProtoKV rely on explicit query–key attention probability weights and therefore require the
eagerattention path, whereas WriteKV and KVzip do not inspect attention weights and natively support FlashAttention 2 (FA2).To isolate kernel speedup effects from pure algorithmic preparation cost, we benchmarked WriteKV (r=0.5) against matched
eager Full-KVunder the exact sameeagerbackend and model setup (measuring end-to-end preparation from context prefill start until the KV cache is ready for decoding, excluding decoding):eagerFull-KVeagerWriteKV TotalAt 100K tokens, WriteKV's pure algorithmic overhead under the
eagerbackend is +4.41 s (7.42%), closely matching the +4.26 s overhead measured under FA2, which demonstrates that WriteKV's scoring and compaction overhead is independent of the attention backend. Accordingly, the total preparation reduction fromSnapKV (eager)(59.55 s) toWriteKV (FA2)(36.91 s) in Fig. 9(a) reflects low algorithmic overhead combined with FA2 compatibility (which attention-weight baselines cannot exploit). We will include this matched-backend table in the revised paper to separate kernel effects from algorithmic cost explicitly.