acceptodds
Under review as a conference paper at ICLR 2027

WriteKV: Query-Agnostic KV Cache Eviction via Native Writes in Hybrid Attention Models

Abstract

During autoregressive inference, transformer-based large language models (LLMs) cache the key and value (KV) representations of previously processed tokens to avoid recomputing historical representations. In hybrid-attention models, linear-attention layers maintain a fixed-size recurrent state, whereas full-attention layers retain an ever-growing KV cache whose memory and per-step attention costs increase with the history length. This coexistence suggests that the update dynamics of linear-attention memory may provide a query-independent signal for compressing the full-attention KV cache. We observe that the native writes generated when linear-attention layers update their fixed-size states are conditioned on the current memory contents: writes aligned with existing memory directions are more likely to be redundant, whereas writes that introduce distinct directions are more likely to encode new memory content. Thus, the novelty of a write direction relative to the existing memory can serve as a retention signal for full-attention KV entries before future queries arrive. Based on this observation, we propose WriteKV, a query-agnostic KV eviction method that quantifies the novelty of write directions using blockwise Ridge statistics and physically compresses the full-attention KV cache through global layer-token selection. WriteKV requires neither additional context reconstruction nor a trained scoring model, and the resulting cache can be reused by multiple queries over the same history. Across three multi-query long-context benchmarks, WriteKV outperforms all evaluated KV-cache eviction baselines under a 20% KV budget. In system measurements with an 800K-token prefix at the same budget, it reduces decode-time attention latency to 26.8% of the full-KV attention latency, while increasing cache-construction time by only 4.20% relative to directly pre-filling the full KV cache.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Hybrid Attention, KV Cache Eviction, Long-Context Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nD47OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WriteKV is a query-agnostic KV-cache eviction method for hybrid-attention models. It uses the native writes produced by linear-attention layers as a signal for which full-attention KV entries are likely to contain novel information. The method scores write directions with block-wise Ridge statistics, aggregates scores across heads and layers, and selects a global layer-token budget for physical cache compaction. On Qwen3.5 models, WriteKV performs well across LongMemEval, RealTalk, and LoCoMo, especially at low retention rates. At 20% KV retention, it reduces packed attention latency to 26.8% of the full-KV value, with a reported 4.20% cache-preparation overhead on an 800K-token prefix.

Critical strengths
  1. The cross-memory idea is interesting and well motivated. Using the update dynamics of linear-attention states to guide full-attention KV retention gives the method a genuinely query-agnostic signal, without future queries, context reconstruction, or a separately trained scorer.
  2. The quality gains are consistent across datasets, context lengths, and model sizes. At 10% retention on Qwen3.5-9B, WriteKV reaches 20.97%, 24.49%, and 31.90% F1 on LongMemEval, RealTalk, and LoCoMo, respectively, outperforming the strongest compression baseline by 8.05–19.96 points.

3.The paper takes the systems side seriously. It evaluates block-size trade-offs, physical cache compaction, preparation time, memory usage, and packed-attention latency. The method also reuses the same compressed cache across multiple queries, which is important for the stated query-agnostic setting.

Critical weaknesses
  1. The runtime comparison across baselines is not fully controlled. WriteKV and KVzip use FlashAttention 2, while SnapKV, ProtoKV, and PyramidKV use the eager backend. The paper reports matched-backend overhead for WriteKV, which is useful, but the direct total-time comparison still mixes algorithmic overhead with backend differences.

  2. Several scoring choices remain heuristic. The method fixes the spectral-energy threshold at (\tau=0.90), uses a block size of 4096, and relies on percentile normalization plus max aggregation. The paper studies block size and mean-versus-max aggregation, but does not show sensitivity to (\tau), regularization, or the association between linear and full-attention layers. This makes it harder to tell how robust the method is outside the reported configuration.

Questions for authors
  1. Could you report full end-to-end decode latency, including linear-attention layers, projections, and the generation loop, for different numbers of queries sharing the same prefix? It would also help to report the break-even point where cache preparation becomes worthwhile.

  2. Could you compare all systems under the same attention backend, or separate kernel effects from the algorithmic preparation cost more explicitly?

3.How sensitive are quality and cache-preparation cost to the spectral threshold (\tau), Ridge regularization, block size, and the mapping from linear layers to full-attention layers?

5 comments

We thank the reviewer for taking the time to review our paper and provide constructive feedback. We are pleased that the reviewer finds our cross-memory motivation interesting and well founded, recognizes WriteKV's consistent tight-budget performance across datasets, context lengths, and model scales, and appreciates our systematic evaluation of block-size trade-offs, physical cache compaction, and multi-query reuse. In the revision, we will incorporate the matched-backend controlled comparison, full-module end-to-end latency measurements, and clearer design clarifications to further strengthen the paper.


W1. The runtime comparison across baselines is not fully controlled. WriteKV and KVzip use FlashAttention 2, while SnapKV, ProtoKV, and PyramidKV use the eager backend. The paper reports matched-backend overhead for WriteKV, which is useful, but the direct total-time comparison still mixes algorithmic overhead with backend differences.

Q2. Could you compare all systems under the same attention backend, or separate kernel effects from the algorithmic preparation cost more explicitly?

Response to W1 & Q2. We thank the reviewer for highlighting this important experimental control. The mixed backends in Fig. 9(a) arise directly from algorithmic dependencies: SnapKV, PyramidKV, and ProtoKV rely on explicit query–key attention probability weights and therefore require the eager attention path, whereas WriteKV and KVzip do not inspect attention weights and natively support FlashAttention 2 (FA2).

To isolate kernel speedup effects from pure algorithmic preparation cost, we benchmarked WriteKV (r=0.5r=0.5) against matched eager Full-KV under the exact same eager backend and model setup (measuring end-to-end preparation from context prefill start until the KV cache is ready for decoding, excluding decoding):

Context Lengtheager Full-KVeager WriteKV TotalAdditional Prep OverheadRelative Overhead
20K8.00 s9.45 s+1.46 s18.25%
40K17.65 s19.95 s+2.30 s13.05%
60K29.36 s32.30 s+2.93 s9.99%
80K43.31 s46.99 s+3.68 s8.50%
100K59.42 s63.83 s+4.41 s7.42%

At 100K tokens, WriteKV's pure algorithmic overhead under the eager backend is +4.41 s (7.42%), closely matching the +4.26 s overhead measured under FA2, which demonstrates that WriteKV's scoring and compaction overhead is independent of the attention backend. Accordingly, the total preparation reduction from SnapKV (eager) (59.55 s) to WriteKV (FA2) (36.91 s) in Fig. 9(a) reflects low algorithmic overhead combined with FA2 compatibility (which attention-weight baselines cannot exploit). We will include this matched-backend table in the revised paper to separate kernel effects from algorithmic cost explicitly.

W2. Several scoring choices remain heuristic. The method fixes the spectral-energy threshold at τ=0.90\tau=0.90, uses a block size of 4096, and relies on percentile normalization plus max aggregation. The paper studies block size and mean-versus-max aggregation, but does not show sensitivity to τ\tau, regularization, or the association between linear and full-attention layers. This makes it harder to tell how robust the method is outside the reported configuration.

Q3. How sensitive are quality and cache-preparation cost to the spectral threshold (τ\tau), Ridge regularization, block size, and the mapping from linear layers to full-attention layers?

Response to W2 & Q3. We appreciate the reviewer's focus on hyperparameter robustness for both task quality and preparation cost. Referring to Sec. 3.3, Sec. 4.4, and Appendix A, several of these components are adaptive or architecture-defined rather than manually tuned hyperparameters:

Block size BB and adaptive Ridge regularization λℓ,h(b)\lambda_{\ell,h}^{(b)}. Block size BB is quantitatively evaluated for both quality and preparation cost in Fig. 8(a)–(c): task quality across B=2048,4096,8192B=2048, 4096, 8192 and retrospective No-block is nearly identical, while increasing BB from 2048 to 8192 slightly reduces 100K preparation time (38.25 s → 36.35 s) at a modest temporary peak memory trade-off (21.48 → 23.13 GiB). Regarding Ridge regularization, λℓ,h(b)\lambda_{\ell,h}^{(b)} is not a preset hyperparameter; Eq. (11) computes it adaptively for each block and head from its spectral-tail energy (λℓ,h(b)=max⁡(Etail/r,Eϵ64/d)\lambda_{\ell,h}^{(b)} = \max(E_{\text{tail}}/r, E\epsilon_{64}/d)) with negligible scalar overhead, specifically eliminating per-layer or per-length regularization tuning.

Spectral-energy threshold τ=0.90\tau=0.90. In terms of preparation cost, eigendecomposition of small d×dd \times d matrices takes the same negligible runtime regardless of τ\tau, so cache-preparation cost is insensitive to τ\tau. In terms of task quality, τ\tau does not truncate the eigenvector space of G\mathbf{G} (Appendix A.4)—the complete directional space is always retained, and τ\tau only determines the scalar damping λ\lambda. While we have not directly swept τ\tau, keeping a single fixed default τ=0.90\tau=0.90 without any per-task tuning yields consistent tight-budget gains across all evaluated datasets, 20K–100K context lengths, and model scales/architectures, demonstrating strong cross-setting transferability.

Layer mapping S(L)\mathcal{S}(L). Assigning each linear-attention layer to the next full-attention layer in model depth is a zero-search, architecture-defined causal rule (ensuring each full-attention layer observes writes produced by its immediately preceding linear layers) that transfers across different layer counts and hidden dimensions without per-model search. Still, we agree that ablating alternative cross-depth observer spans is valuable; we will clarify the rationale for this default mapping in the revision and leave direct sensitivity sweeps over τ\tau and observer spans as additional analysis.

Q1. Could you report full end-to-end decode latency, including linear-attention layers, projections, and the generation loop, for different numbers of queries sharing the same prefix? It would also help to report the break-even point where cache preparation becomes worthwhile.

A1. We conducted a full-module end-to-end evaluation on Qwen3.5-9B with an 800K-token shared prefix (r=0.2,B=4096r=0.2, B=4096), covering all linear-attention layers, projections, cache isolation, and the complete autoregressive generation loop. We prefill and snapshot the 800K prefix once, then sequentially serve 2 independent downstream queries from the same cached prefix (restoring isolated state per query and generating 128 tokens via greedy decoding).

Per-query end-to-end decode latency is highly consistent (Full-KV: 22.282 s and 22.271 s, mean 22.28 s; WriteKV: 6.770 s and 6.743 s, mean 6.76 s), yielding the following stage-wise averages: TTFT drops from 387.34 ms to 226.39 ms (−41.55%), TPOT drops from 172.34 ms to 51.40 ms (−70.18%, 3.35×), cache isolation drops from 8.59 ms to 0.66 ms (−92.32%), and complete 128-token end-to-end decoding achieves a 3.30× speedup (saving 15.52 s per query).

Including the one-time 800K prefix preparation cost, cumulative end-to-end time across different numbers of shared queries NN is reported below (N=0,1,2N=0, 1, 2 are directly measured; since every query starts independently from the same deterministic snapshot under greedy decoding, N=3,5N=3, 5 are projected from the measured per-query decoding mean):

Shared Queries NN (128 tokens each)Full-KV TotalWriteKV TotalNet Time Diff (WriteKV − Full-KV)
N=0N=0 (prep & snapshot only, measured)744.17 s789.82 s+45.64 s
N=1N=1 (measured)766.45 s796.57 s+30.12 s
N=2N=2 (measured)788.73 s803.33 s+14.60 s
N=3N=3 (projected from measured mean)811.00 s810.08 s−0.92 s (break-even)
N=5N=5 (projected from measured mean)855.55 s823.60 s−31.96 s

Under this 800K setup, WriteKV breaks even and achieves net total-time savings by the 3rd 128-token query; when serving 5 shared queries, break-even requires generating only 76 tokens per query on average.

Reviewer_FvX7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes WRITEKV, a query-agnostic KV-cache compression method for hybrid-attention models. The key idea is to use native writes produced by linear-attention layers during the history forward pass as retention signals for full-attention KV caches: the novelty of a write direction indicates whether a historical position contributes new memory components, enabling a reusable compressed KV cache to be constructed before any future query arrives.

Critical strengths
  1. A novel and natural cross-memory retention signal. The paper converts native write trajectories from linear-attention layers into query-agnostic retention signals for full-attention KV, requiring neither future queries, context reconstruction, nor an independently trained scorer. The idea is simple and fits the hybrid-attention structure well.

  2. Strong task performance under tight budgets with genuine multi-query reuse. Under a 20% KV budget, WRITEKV outperforms all evaluated compression baselines on three long-conversation memory benchmarks. Compression is performed before queries arrive, and the same cache can serve multiple queries sharing the same history.

  3. Physical compaction yields practical system benefits with reasonable engineering compatibility. The method physically gathers selected KV entries and releases the rest. In the 800K microbenchmark, attention latency drops to 26.8% and physical cache occupancy decreases by about 80%, while compatibility with FlashAttention 2 keeps preparation overhead at only 4.20% over full-KV.

Critical weaknesses
  1. Limited coverage of model architectures and task types. Main experiments focus on the Qwen3.5 family and long-conversation memory benchmarks with token-overlap/lexical F1. Other hybrid architectures and tasks such as long-document QA, code, retrieval-augmented generation, and multi-document reasoning are not systematically evaluated, limiting generality.

  2. A single query-agnostic mask may discard query-specific information. Different queries may access different historical positions. A unified retention mask can perform well on average but cannot guarantee that every query retains its critical information. Results exceeding Full-KV at some budgets also need more careful explanation than compression regularization alone.

  3. Block-wise approximation and many hyperparameters, with non-trivial preparation memory. Scoring is block-level causal rather than token-level online; tokens in the same block share statistics. Block size, the 90% energy threshold, percentile ranks, max aggregation, and global budget all affect behavior. At B=4096, preparation peak memory is about 21–23 GiB for 100K contexts, which matters for long-context deployment.

  4. High engineering coupling and system experiments far from real serving. The method reads native writes directly from the linear-attention production path, requiring inference frameworks to expose internal signals. Experiments use a single A100, batch size 1, greedy decoding, and 96 generated tokens; latency measurements exclude linear-attention layers, projections, and the generation loop, so 26.8% does not directly translate to end-to-end serving throughput.

  5. Lack of theoretical guarantees, failure analysis, and validation with representation-compression techniques. No theoretical link or error bound connects native-write novelty to full-attention KV importance. There is little failure analysis by query type, history length, or information distribution. The claimed orthogonality to quantization and low-rank projection is not empirically validated in combination, leaving practical deployment value uncertain.

Questions for authors

Can WRITEKV's native-write novelty score reliably predict the marginal importance of full-attention KV for multiple queries in the future under unknown conditions? Where are its effectiveness limits, and when will it systematically lose key historical information?

6 comments

We thank the reviewer for taking the time to evaluate our paper carefully and for providing comprehensive, constructive feedback. We are glad that the reviewer finds our cross-memory approach novel and natural—converting linear-attention native writes into query-agnostic retention signals without future queries, context reconstruction, or a trained scorer—and recognizes WriteKV's strong tight-budget performance under genuine multi-query reuse, as well as the practical latency and memory gains enabled by physical compaction and FlashAttention 2 compatibility. Below, we address each point with additional cross-architecture, non-dialogue, and full-module end-to-end experiments, along with further clarifications.


W1. Limited coverage of model architectures and task types. Main experiments focus on the Qwen3.5 family and long-conversation memory benchmarks with token-overlap/lexical F1. Other hybrid architectures and tasks such as long-document QA, code, retrieval-augmented generation, and multi-document reasoning are not systematically evaluated, limiting generality.

Response to W1. We thank the reviewer for this valuable suggestion. We expand our evaluation across both hybrid model architectures and non-dialogue tasks/metrics.

Extending beyond the Qwen3.5 family. Under the exact same single-A100 BF16 setup and default hyperparameters, we evaluate Olmo Hybrid 7B (which also uses a 3:1 GDN–full-attention ratio but adopts 30-KV-head MHA instead of GQA) against the query-agnostic baseline KVzip across r=0.1–0.7r=0.1\text{–}0.7 (micro-average F1 across all queries; because this checkpoint is a base model prone to generating continuation dialogue turns at higher retention, LongMemEval-20K uniformly evaluates first-line outputs for both methods, while RealTalk uses standard full outputs):

DatasetMethodr=0.1r=0.1r=0.3r=0.3r=0.5r=0.5r=0.7r=0.7
LongMemEval-20K (50 texts / 177 queries)WriteKV16.69%26.30%29.51%30.97%
LongMemEval-20K (50 texts / 177 queries)KVzip10.11%22.18%26.66%28.20%
RealTalk (10 conversations / 728 queries)WriteKV9.93%16.96%18.43%20.54%
RealTalk (10 conversations / 728 queries)KVzip8.90%15.74%15.42%20.36%

Across different hybrid architectures and attention-head designs, WriteKV outperforms KVzip across all 8 dataset–budget pairs (+2.77 to +6.58 pp on LongMemEval-20K and +0.19 to +3.01 pp on RealTalk).

Extending to non-dialogue multi-query tasks and metrics. To move beyond conversational memory and lexical/token-overlap F1, we evaluate two non-dialogue benchmarks from SCBench under the Multi-Request shared-prefix protocol (compressing the prefix once and starting each downstream query independently):

  • Numerical aggregation (SCBench Math.Find; all 100 contexts / 600 queries, official Exact Match):
Retention rrWriteKVKVzipWriteKV − KVzip
0.122.83%16.33%+6.50 pp
0.332.83%30.83%+2.00 pp
0.538.17%38.33%−0.17 pp
  • Long-document summarization (SCBench Summary; full 70 documents / 350 queries, ROUGE-L F1):
Retention rrWriteKVKVzipWriteKV − KVzip
0.129.84%26.09%+3.75 pp
0.333.73%33.29%+0.44 pp
0.535.75%36.32%−0.58 pp

On both non-dialogue tasks and metrics, WriteKV substantially outperforms KVzip under tight budgets (+6.50 pp and +3.75 pp at r=0.1r=0.1; +2.00 pp and +0.44 pp at r=0.3r=0.3) and closely matches KVzip at r=0.5r=0.5 without incurring KVzip's costly second-pass context reconstruction overhead. For broader domains such as code repositories, long-document QA, and multi-document reasoning, we will more clearly define our current evaluation scope around shared-history multi-query reuse in the revision and leave these extensions for future work.

W2. A single query-agnostic mask may discard query-specific information. Different queries may access different historical positions. A unified retention mask can perform well on average but cannot guarantee that every query retains its critical information. Results exceeding Full-KV at some budgets also need more careful explanation than compression regularization alone.

Response to W2.

Query-agnostic objective and per-query stability. We agree that without access to future queries, no query-agnostic method can mathematically guarantee zero loss for every unforeseen query. Unlike query-aware methods conditioned on the current prompt, the principled objective under a fixed budget before queries arrive is to preserve as much non-redundant historical information as possible, thereby maximizing expected coverage across arbitrary future queries. WriteKV uses directional write novelty for this exact purpose rather than overfitting to any specific query.

Empirical distributions confirm that these gains do not come at the expense of severe tail degradation. At r=0.1r=0.1, importance-agnostic uniform random retention has an expected retention rate of 10.00% for future high-attention (P99–100) units. Against this reference:

  • On LoCoMo (a representative conversation with 199 queries), 198/199 queries exceed the 10.00% random expectation (mean 16.45%, median 14.70%, P5–P95: 11.16%–24.47%, so even the 5th-percentile tail query outperforms random retention), holding consistently across single-hop (14.89%), multi-hop (13.99%), temporal (23.21%), and open-domain (15.85%) queries. Under identical conditions, the query-agnostic baseline KVzip achieves a mean retention of 13.83% (median 13.59%, P5–P95: 11.21%–16.79%, 197/199 above 10.00%), trailing WriteKV by 2.62 pp.
  • On LongMemEval-20K (all 26 additional multi-query histories outside Fig. 3, 108 queries total), 90/108 queries exceed the 10.00% random expectation (mean 12.39%).

Why intermediate budgets (r=0.3–0.9r=0.3\text{–}0.9) can exceed Full-KV. We thank the reviewer for raising this observation. Rather than a generic regularization effect, this gain stems primarily from attention denoising induced by physical KV eviction:

  • Softmax attention noise. As shown in Differential Transformer (Ye et al., 2025), standard Softmax must assign non-negative probabilities summing to 1 across all cached tokens, causing long histories with abundant irrelevant or redundant context to siphon away attention mass ("attention noise") and dilute focus on true evidence.
  • Denoising via physical eviction. Consistent with KVzip (Kim et al., 2025), which attributes similar gains to "reduced attention distractions following KV eviction", removing low-novelty redundant or irrelevant entries prevents Softmax from wasting probability mass on distractors, improving signal-to-noise ratio. This also explains why surpassing Full-KV occurs mainly on LongMemEval-20K at intermediate budgets (r=0.3–0.9r=0.3\text{–}0.9, +1.81 to +4.11 pp): LongMemEval pads multi-session histories with substantial distractor sessions, which intermediate budgets prune while preserving sparse key facts, whereas at extreme pruning (r=0.1r=0.1) essential facts start to be evicted and F1 drops below Full-KV as expected.

W3. Block-wise approximation and many hyperparameters, with non-trivial preparation memory. Scoring is block-level causal rather than token-level online; tokens in the same block share statistics. Block size, the 90% energy threshold, percentile ranks, max aggregation, and global budget all affect behavior. At B=4096, preparation peak memory is about 21–23 GiB for 100K contexts, which matters for long-context deployment.

Response to W3.

Clarifying preparation peak memory (Fig. 8c). The reported 21–23 GiB is not an extra memory footprint introduced solely by WriteKV, but rather the total working-set peak memory of the entire prefill pipeline, including model weights, the Full-KV cache, and activations (Sec. 4.4, lines 416–417). As shown in Fig. 8(c) at 100K context length, Full-KV alone already allocates 20.72 GiB of peak memory without any compression scoring. With default B=4096B=4096, WriteKV reaches 22.03 GiB, adding a net temporary overhead of only +1.31 GiB (~6.3%); under tighter memory constraints, setting B=2048B=2048 lowers this net overhead to +0.76 GiB (21.48 GiB) with virtually identical task accuracy (Fig. 8a). Moreover, WriteKV frees block direction buffers immediately after scoring each block (retaining only the compact matrix G\mathbf{G}), so temporary buffers do not accumulate with context length. Crucially, preparation is a one-time cost, after which physical KV occupancy drops substantially (e.g., by 79.9% at r=0.2r=0.2) across all subsequent queries.

Block-wise approximation and configuration choices. First, block-wise approximation incurs negligible accuracy loss: Fig. 8(a) shows that B=2048,4096,8192B=2048, 4096, 8192 closely match the retrospective No-block curve, confirming that expensive token-by-token matrix inversion is unnecessary. Second, most parameters are not tuned per task: the global retention rate ρ\rho is the user-specified budget, the regularization coefficient λℓ,h(b)\lambda_{\ell,h}^{(b)} is computed adaptively from each block-head spectrum (Eq. 11) rather than hand-tuned, and the aggregation design is ablated in Fig. 10. All remaining defaults (τ=0.90,B=4096\tau=0.90, B=4096) are kept completely fixed across all model sizes (4B/7B/9B/27B), context lengths (20K–100K), and datasets without task-specific tuning.

Reviewer_LNNhOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

The paper studies query-agnostic KV-cache eviction for the full-attention layers of hybrid models (Qwen3.5, Gated DeltaNet). It uses native writes from linear-attention layers as a retention signal. Unlike prior query-agnostic methods, it needs no context reconstruction and no trained scorer. Based on the finding (F) and method (M) below, it outperforms existing baselines at tight budgets (r ≤ 0.2). It also adds much less cache-preparation overhead than KVzip, because it does not need a reconstruction pass.

Finding

  • (F.1) After converting scores to percentile ranks, layer-token units with strong future attention tend to rank higher under the query-independent WriteKV score. This holds for four queries sharing one LongMemEval-20K history, even though each query attends to different positions (Fig. 3).

Method

  • (M.1) Block-causal Ridge scoring, which measures how weakly each write direction is covered by past writes, using cumulative second-order statistics (Eq. 2).
  • (M.2) Aggregation of the scores across heads and linear layers into one score per full-attention layer-token unit, followed by a global Top-K selection across layers (Eqs. 3–4).

Main Results

  • (R.1) WriteKV exceeds the baselines at low retention rates and is comparable at higher rates on three dialogue benchmarks (LongMemEval, RealTalk, LoCoMo). This holds across context lengths on LongMemEval (20K–100K) and across model scales (Qwen3.5-4B/9B/27B).
  • (R.2) Ablations support the native-write signal and Ridge scoring over Native-value/Native-beta/Mean-cosine (Fig. 7), and max aggregation over Mean/Mean (Fig. 10). Quality is insensitive to block size (Fig. 8a).
  • (R.3) At 100K, WriteKV adds 4.26 s of preparation overhead over Full-KV, vs. 49.49 s for KVzip, the reconstruction-based query-agnostic baseline (Fig. 9a). Physical compaction at r = 0.2 reduces decode attention latency by 3.74× and cache memory by 79.9% (Fig. 9b).
Critical strengths
  1. The method is cheap. The signal is a by-product of the history forward pass, so it needs no extra pass or trained scorer. The preparation overhead is less than a tenth of KVzip’s (R.3), and the cache is physically compacted, with measured latency and memory.

  2. The gains at tight budgets are large and consistent across three datasets, five context lengths on LongMemEval, and three model sizes.

  3. Using linear attention module to guide the query-agnostic compression of full attention is a new and natural idea for hybrid models, although the link is not yet well supported (W1, W2).

Critical weaknesses

My main concerns are threefold. (1) The finding is not explained, and the method does not follow from it (W1, W2). (2) Ridge scoring adds little over simpler signals, and the closest baselines are missing (W3). (3) Generality is limited in both architecture and tasks (W4, W5).

W1. F.1 lacks an explanation, and its generality is unclear.

  • The paper does not explain why the novelty of a value-side write direction in a linear layer should align with key-based attention from different queries in a full-attention layer.
  • The evidence is limited: one history, four queries, an effect of about 2× uniform, and no baseline in the same plot.

W2. The method does not follow from the motivation, and the link is not shown empirically.

  • Design: if the goal is that the retained set covers what any query needs, a token should matter less when a similar token is already kept. But ν scores each token against all past writes, not against the retained set. For example, copies of a repeated fact lower each other’s scores, so all copies can be evicted while one would suffice. Block size does not address this (Fig. 8a), since every setting, including No-block, uses all past writes as the reference.
  • Evidence: Fig. 3 shows a score distribution for one history, not how many of each query’s high-attention units are retained. The paper should report, per query, the fraction of high-attention units (e.g., P99–100) kept at r = 0.1, compared with baselines.

W3. Ridge scoring, the core component, adds little, and the closest baselines are missing.

  • On LoCoMo at r = 0.1, Native-beta (a scalar gate) already reaches 28.65 in the WriteKV pipeline, about 17 points above the strongest baseline. Ridge on the native write adds only about 3 more points, and the same write direction scored by Mean-cosine (23.45) is worse than the scalar gate (Fig. 7). So most of the gain over the baselines is already obtained without Ridge novelty scoring. A 3-point gain on one dataset and one model, without variance, does not support Ridge as the key component.
  • The closest training-free query-agnostic methods discussed in Sec. 5, namely CurDKV and CapKV (value-side leverage scores), KeyDiff, and ManifoldKV, are not compared. Three of the four baselines are query-dependent methods.

W4. The novel part applies only to delta-rule hybrids.

  • The stated reason for excluding MoE models matters for scale comparisons, not for comparing methods within one model. Kimi Linear (KDA) could be tested. Olmo Hybrid 7B is a dense GDN hybrid with the same 3:1 layer pattern and fits a single A100. Even with the stated 32K context, it covers LoCoMo, RealTalk, and LongMemEval-20K.
  • In Mamba-2 hybrids (e.g., Nemotron-Nano-9B-v2), the write is not conditioned on the state. The residual that WriteKV relies on does not exist, so Ridge would reduce to static input geometry.

W5. The evaluation covers only dialogue QA.

  • All three benchmarks are long-term dialogue memory QA, where questions can target any session. Sec. 1 motivates RAG and agentic use, but neither is evaluated. Retrieval or aggregation tasks (e.g., RULER, SCBench) are needed to test whether the gain generalizes.
Questions for authors

Q1. In Fig. 3, how is future attention aggregated over heads, question tokens, and generated tokens? Are percentiles computed per layer or over all layer-token units?

Q2. w_t is the residual against the decayed state, so it includes a refill term (1 − α_t)S_{t−1}ᵀk_t along the old value, while G accumulates without decay. How much does this term affect u_t, especially in fast-decay heads?

8 comments

We thank the reviewer for taking the time to evaluate our paper carefully and for providing insightful, constructive feedback. We are glad that the reviewer finds our cross-memory approach natural and low-overhead—incurring less than one-tenth of KVzip's preparation time while achieving physical compaction—and recognizes WriteKV's consistent tight-budget gains across benchmarks, context lengths, and model scales. Below, we address each concern with additional experiments and clarifications.


W1 & W2 (Summary): Why does linear value-side write novelty align with full-attention queries (only 1 history, 4 queries, no baseline)? Why score against all past writes rather than the retained set (repeated-fact counterexample)? Report per-query P99–100 retention at r=0.1r=0.1 vs. baselines.

Response to W1 & W2.

Mechanistic link. Rather than direct geometric equivalence between write directions ut\mathbf{u}_t and full-attention keys, they align functionally via deduplication over the shared residual stream: GDN's native write vtnew\mathbf{v}_t^{\text{new}} captures the prediction residual beyond what the recurrent state already encodes. Retaining novel writes minimizes full-attention redundancy; since the fixed-size, decaying linear state struggles to preserve such distinct details without interference, downstream queries naturally attend to these positions.

Scoring reference & repeated facts. Dynamic retained-set scoring is an appealing alternative we considered, but because global Top-K across F×C\mathcal{F} \times \mathcal{C} runs post-prefill (Eq. 4), it would require keeping all historical ut\mathbf{u}_t in memory and performing sequential re-evaluation—forfeiting our streaming low-memory design (+1.31 GiB peak overhead at 100K). Moreover, mutual eviction of repeated facts does not occur for two reasons: (1) Even under identical write directions, block-causal accumulation (Eq. 2) excludes future blocks from G(b)\mathbf{G}^{(b)}, so an important first mention retains its full novelty score while only later collinear repeats drop (uninformative tokens are expected to be evicted). (2) In practice, state-conditioned writes ut\mathbf{u}_t evolve with context and recurrent memory, so verbatim repetition is not collinear. On a real 3-copy verbatim sample, pairwise mean cosine similarities of ut\mathbf{u}_t are only 0.60 / 0.55 / 0.70, yielding mean priorities 0.700 / 0.754 / 0.796 and retaining 41 / 78 / 99 of 584 layer–token units at r=0.1r=0.1. This also explains why retrospective No-block does not collapse (Fig. 8a): Ridge suppresses collinear local redundancy rather than re-contextualized factual mentions.

Multi-query retention validation. WriteKV optimizes expected coverage across unknown queries, whereas single-query attention probes only local subsets; thus dense-query dialogues (LoCoMo) reflect this gain more clearly than sparse-query histories (LongMemEval-20K), though both show positive enrichment. At r=0.1r=0.1, uniform random retention has a 10.00% expected rate for P99–100 units:

  • LongMemEval-20K (all 26 extra ≥4\ge 4-query histories, 108 queries): mean 12.39% (1.24× random; median 12.10%; 90/108 > 10.00%).
  • LoCoMo (199-query representative conversation): mean 16.45% (1.65× random; median 14.70%; P5–P95 11.16%–24.47%; 198/199 > 10.00%), holding across single-hop (14.89%), multi-hop (13.99%), temporal (23.21%), and open-domain (15.85%).
  • Baseline comparison: On the same 199 queries (r=0.1r=0.1), query-agnostic KVzip achieves 13.83% mean (median 13.59%; P5–P95 11.21%–16.79%; 197/199 > 10.00%), trailing WriteKV by 2.62 pp.

W3. Ridge scoring, the core component, adds little, and the closest baselines are missing. On LoCoMo at r = 0.1, Native-beta (a scalar gate) already reaches 28.65 in the WriteKV pipeline, about 17 points above the strongest baseline. Ridge on the native write adds only about 3 more points, and the same write direction scored by Mean-cosine (23.45) is worse than the scalar gate (Fig. 7). So most of the gain over the baselines is already obtained without Ridge novelty scoring. A 3-point gain on one dataset and one model, without variance, does not support Ridge as the key component. The closest training-free query-agnostic methods discussed in Sec. 5, namely CurDKV and CapKV (value-side leverage scores), KeyDiff, and ManifoldKV, are not compared. Three of the four baselines are query-dependent methods.

Response to W3. We address both the contribution/stability of Ridge scoring and the comparison against training-free query-agnostic baselines below.

Contribution and statistical stability of Ridge scoring. First, we clarify the controlled comparisons in Fig. 7. On the exact same write direction ut\mathbf{u}_t, the first-order alternative Mean-cosine (23.45%) underperforms Native-beta (28.65%) by 5.20 pp, whereas second-order Ridge scoring (31.90%) improves over Mean-cosine by +8.44 pp and remains best across r=0.1–0.9r=0.1\text{–}0.9. Without second-order subspace decorrelation, the write direction alone fails to realize its potential; Ridge is thus essential to unlock its directional signal. Moreover, Native-beta is not an external standalone baseline: it operates within our proposed pipeline using the state-conditioned GDN write gate alongside cross-head/cross-layer percentile calibration and global Top-K selection, on top of which Ridge adds +3.25 pp micro-average F1. Across the 10 individual conversations in LoCoMo (r=0.1r=0.1), WriteKV outperforms Native-beta on 8/10 conversations, achieving a conversation-level mean ΔF1\Delta\text{F1} of +3.36 pp (sample std 3.08 pp, range −1.57 to +7.64 pp), demonstrating consistent cross-conversation improvements rather than outlier-driven gains.

Comparison with training-free query-agnostic baselines. We add direct comparisons against the training-free query-agnostic baseline KeyDiff across all three datasets (Qwen3.5-9B, reusing one compressed cache across all queries):

DatasetMethodr=0.1r=0.1r=0.3r=0.3r=0.5r=0.5
LongMemEval-20KWriteKV / KeyDiff20.97% / 12.87% (+8.10)37.13% / 27.90% (+9.23)36.34% / 32.95% (+3.39)
RealTalkWriteKV / KeyDiff24.49% / 9.72% (+14.77)35.57% / 17.55% (+18.02)38.51% / 26.24% (+12.27)
LoCoMoWriteKV / KeyDiff31.90% / 13.69% (+18.21)51.18% / 29.92% (+21.26)55.60% / 43.52% (+12.08)

WriteKV substantially outperforms KeyDiff across all datasets and budgets (+8.10–18.21 pp at r=0.1r=0.1, +9.23–21.26 pp at r=0.3r=0.3, and +3.39–12.27 pp at r=0.5r=0.5). In addition, regarding value-side leverage scoring (CurDKV, CapKV), Native-value (29.74%) in Fig. 7 already evaluates Ridge leverage scoring on static value vectors vt\mathbf{v}_t within the same pipeline, where native writes still outperform static values by +2.16 pp.

W4. The novel part applies only to delta-rule hybrids. The stated reason for excluding MoE models matters for scale comparisons, not for comparing methods within one model. Kimi Linear (KDA) could be tested. Olmo Hybrid 7B is a dense GDN hybrid with the same 3:1 layer pattern and fits a single A100. Even with the stated 32K context, it covers LoCoMo, RealTalk, and LongMemEval-20K. In Mamba-2 hybrids (e.g., Nemotron-Nano-9B-v2), the write is not conditioned on the state. The residual that WriteKV relies on does not exist, so Ridge would reduce to static input geometry.

Response to W4. We thank the reviewer for these constructive suggestions on architectural generality and scope.

Cross-model evaluation on Olmo Hybrid 7B. Under the exact same single-A100 BF16 setup and default hyperparameters, we evaluate Olmo Hybrid 7B (which shares the 3:1 GDN–full-attention ratio but uses 30-KV-head MHA instead of GQA) against the query-agnostic baseline KVzip across r=0.1–0.7r=0.1\text{–}0.7 (micro-average F1 over all queries; because this checkpoint is a base model prone to continuing simulated dialogue turns after answering at higher retention, LongMemEval-20K uniformly evaluates first-line outputs for both methods, while RealTalk uses standard full outputs):

DatasetMethodr=0.1r=0.1r=0.3r=0.3r=0.5r=0.5r=0.7r=0.7
LongMemEval-20K (50 texts / 177 queries)WriteKV16.69%26.30%29.51%30.97%
LongMemEval-20K (50 texts / 177 queries)KVzip10.11%22.18%26.66%28.20%
RealTalk (10 conversations / 728 queries)WriteKV9.93%16.96%18.43%20.54%
RealTalk (10 conversations / 728 queries)KVzip8.90%15.74%15.42%20.36%

Even under a 30-head MHA full-attention configuration, WriteKV consistently outperforms KVzip across all 8 dataset–budget pairs (+2.77 to +6.58 pp on LongMemEval-20K and +0.19 to +3.01 pp on RealTalk).

Refining scope and Mamba-2 boundary. We fully agree with the reviewer: pure SSM hybrids like Mamba-2 lack a state-conditioned residual write term, so WriteKV does not apply to them. Referring broadly to "linear-attention" was imprecise; in the revision, we will narrow our terminology to delta-rule / state-conditioned residual-write hybrids (e.g., GDN, KDA) and explicitly state this boundary for Mamba-2 models.

Note on Kimi Linear 48B. All accuracy and system comparisons in this paper adhere to a controlled single-A100-80GB BF16 protocol. Because a 48B model requires ~96 GB for BF16 weights alone and multi-GPU serving (where quantization or multi-GPU partitioning would introduce confounding system variables), we prioritized evaluating the dense Olmo Hybrid 7B checkpoint within the same single-GPU protocol. Conceptually, Kimi Linear's KDA layers employ an analogous delta-rule residual update and remain compatible in principle.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General statement [1/2]

We sincerely thank all three reviewers (LNNh, FvX7, nD47) for their thoughtful, constructive feedback. We are encouraged that the reviewers find WriteKV's cross-memory mechanism natural and well motivated, recognize its consistent tight-budget performance gains across benchmarks, context lengths, and model scales, and appreciate its low preparation overhead and physical cache compaction benefits. We will incorporate all new experiments, analyses, and clarifications below into the revised manuscript. Specifically, our major additions and clarifications are summarized in G1–G6:

G1. Expanded multi-query attention-coverage validation and repeated-fact analysis (Addressing LNNh W1/W2, FvX7 W2). To validate Fig. 3 beyond a single history, we evaluated per-query retention of future high-attention (P99–100) layer–token units at r=0.1r=0.1 against the 10.00% importance-agnostic uniform random expectation:

  • LongMemEval-20K (all 26 remaining ≥4\ge 4-query histories outside Fig. 3; 108 queries): WriteKV achieves a mean retention of 12.39% (1.24× random; 95% cluster-bootstrap CI of the mean [11.83%, 12.99%]), with 90/108 queries exceeding 10.00%.
  • LoCoMo (representative 199-query conversation): WriteKV reaches 16.45% mean (1.65× random; median 14.70%; P5–P95 11.16%–24.47%; 198/199 > 10.00%), holding across single-hop (14.89%), multi-hop (13.99%), temporal (23.21%), and open-domain (15.85%) queries, and outperforming KVzip (13.83% mean, median 13.59%, P5–P95 11.21%–16.79%, 197/199 > 10.00%) by +2.62 pp.

We also verified on a real 3-copy verbatim repetition sample that state-conditioned write directions ut\mathbf{u}_t evolve with context (pairwise mean cosine similarity only 0.60 / 0.55 / 0.70; mean priorities 0.700 / 0.754 / 0.796; 41 / 78 / 99 of 584 units retained at r=0.1r=0.1), confirming that repeated facts do not mutually cancel out to zero, while cross-block causality protects initial mentions even under collinear repeats.

G2. Direct comparison with training-free query-agnostic baseline KeyDiff and statistical stability of Ridge scoring (Addressing LNNh W3). We added direct comparisons against KeyDiff on Qwen3.5-9B (reusing a single compressed cache per history across all queries):

DatasetMethodr=0.1r=0.1r=0.3r=0.3r=0.5r=0.5
LongMemEval-20KWriteKV / KeyDiff20.97% / 12.87% (+8.10)37.13% / 27.90% (+9.23)36.34% / 32.95% (+3.39)
RealTalkWriteKV / KeyDiff24.49% / 9.72% (+14.77)35.57% / 17.55% (+18.02)38.51% / 26.24% (+12.27)
LoCoMoWriteKV / KeyDiff31.90% / 13.69% (+18.21)51.18% / 29.92% (+21.26)55.60% / 43.52% (+12.08)

WriteKV consistently outperforms KeyDiff across all datasets and budgets. Moreover, on the exact same write direction ut\mathbf{u}_t in Fig. 7, Ridge scoring (31.90%) improves by +8.44 pp over first-order Mean-cosine (23.45%) and remains best across r=0.1–0.9r=0.1\text{–}0.9; over Native-beta (which already uses our write gate and calibration/Top-K pipeline), Ridge improves 8/10 LoCoMo conversations (r=0.1r=0.1) with a conversation-level mean ΔF1\Delta\text{F1} of +3.36 pp (std 3.08 pp).

AuthorsOpenReview ↗

General statement [2/2]

G3. Cross-architecture validation on Olmo Hybrid 7B and scope refinement (LNNh W4, FvX7 W1). Under our single-A100 BF16 setup with default hyperparameters, we evaluated Olmo Hybrid 7B (3:1 GDN–full-attention with 30-KV-head MHA) against KVzip across LongMemEval-20K (50 texts / 177 queries; first-line micro-F1 for both methods since this base checkpoint continues simulated turns at higher rr) and RealTalk (10 conversations / 728 queries; standard micro-F1):

DatasetMethodr=0.1r=0.1r=0.3r=0.3r=0.5r=0.5r=0.7r=0.7
LongMemEval-20KWriteKV16.69%26.30%29.51%30.97%
LongMemEval-20KKVzip10.11%22.18%26.66%28.20%
RealTalkWriteKV9.93%16.96%18.43%20.54%
RealTalkKVzip8.90%15.74%15.42%20.36%

WriteKV outperforms KVzip across all 8 settings (+2.77–6.58 pp on LongMemEval-20K; +0.19–3.01 pp on RealTalk). We will refine our terminology to delta-rule / state-conditioned residual-write hybrids (e.g., GDN, KDA), noting non-applicability to pure SSM hybrids such as Mamba-2.

G4. Non-dialogue multi-query benchmarks on SCBench (LNNh W5, FvX7 W1). Under the same single-A100 BF16 setup and SCBench Multi-Request shared-prefix protocol, we evaluated SCBench Math.Find (100 contexts / 600 queries, Exact Match; Full-KV: 44.33%) and SCBench Summary (70 documents / 350 queries, ROUGE-L F1):

TaskrrWriteKVKVzipWriteKV − KVzip
Math.Find0.122.83%16.33%+6.50 pp
Math.Find0.332.83%30.83%+2.00 pp
Math.Find0.538.17%38.33%−0.17 pp
Summary0.129.84%26.09%+3.75 pp
Summary0.333.73%33.29%+0.44 pp
Summary0.535.75%36.32%−0.58 pp

WriteKV outperforms KVzip on both tasks under tight budgets (r=0.1,0.3r=0.1, 0.3) and matches it at r=0.5r=0.5 without costly second-pass reconstruction.

G5. Matched-eager backend overhead and 800K end-to-end latency (nD47 W1/Q1/Q2, FvX7 W4).

  • Matched eager prep (r=0.5r=0.5): Under an identical eager backend and model setup, WriteKV's algorithmic overhead is backend-independent (matching FA2's +4.26 s at 100K):
Context20K40K60K80K100K
eager Full-KV / WriteKV8.00 / 9.45 s17.65 / 19.95 s29.36 / 32.30 s43.31 / 46.99 s59.42 / 63.83 s
Additional Overhead+1.46 s (18.25%)+2.30 s (13.05%)+2.93 s (9.99%)+3.68 s (8.50%)+4.41 s (7.42%)
  • Full-module 800K end-to-end latency (Qwen3.5-9B, r=0.2r=0.2, 128 tokens, 2 queries): Including linear layers, projections, cache isolation, and the generation loop, per-query decode time drops 3.30× (22.28 → 6.76 s, saving 15.52 s/query; TTFT −41.55%, TPOT −70.18% / 3.35×, isolation −92.32%). Against one-time prep + snapshot overhead (744.17 → 789.82 s, +45.64 s), WriteKV breaks even at the 3rd 128-token query (810.08 vs. 811.00 s) or 76 tokens/query across 5 shared queries.

G6. High-level clarifications on mechanism, boundaries, footprint, and robustness (LNNh W1/Q1–Q2, FvX7 W2–W5/Q, nD47 W2/Q3).

  • Mechanism & scoring dynamics: Native writes align with future queries via residual-stream deduplication; keeping G\mathbf{G} undecayed prevents fast-decay refills from inflating old directions in persistent full-attention KV while max preserves slow-decay heads (Fig. 3 uses max aggregation with per-layer attention vs. global WriteKV percentiles).
  • Accuracy boundaries: Beating Full-KV on distractor-padded LongMemEval-20K (r=0.3–0.9r=0.3\text{–}0.9) reflects attention denoising via distractor eviction (Ye et al., 2025; Kim et al., 2025), whereas extreme-budget drops stem from token quotas saturating below distinct-fact density rather than scenario-specific failure.
  • Deployment footprint: Fig. 8(c)'s 22.03 GiB is the total prefill working set (Full-KV itself uses 20.72 GiB; WriteKV adds only +1.31 GiB net), and vtnew\mathbf{v}_t^{\text{new}} is consumed locally in prefill without exposing external state or modifying decode kernels.
  • Zero-tuning robustness: λℓ,h(b)\lambda_{\ell,h}^{(b)} is spectrum-adaptive (Eq. 11), τ=0.90\tau=0.90 neither truncates G\mathbf{G} nor affects prep runtime, and S(L)\mathcal{S}(L) follows causal depth; defaults (B=4096,τ=0.90B=4096, \tau=0.90) transfer fixed across all models, lengths, and tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.