acceptodds
Under review as a conference paper at ICLR 2027

Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory

Abstract

Recall presents a difficult choice: transformers have a linearly growing memory that slows each successive token, while linear RNNs typically have fixed costs but limited recall. We present Key-Value Means ("KVM"), a novel block-recurrence for attention that can accommodate either fixed-size or growing state. Equipping a strong transformer baseline with fixed-size KVM attention layers yields a strong chunked RNN, while adding only an insignificant number of new parameters. We train a transformer with a growable KVM cache and show it performs competitively on long-context tests with only subquadratic prefill time and sublinear state growth. KVM is implementable with standard operations and without custom kernels, and supports chunk-wise parallelizable training and prefill. It provides many of the benefits of both traditional transformers (expandable context memory, chunk-wise parallelizable training and prefill) and RNNs in a single unified package. It can be used on every layer, saving KV-cache memory, and allowing a continuous range of choices of prefill time complexity between and . We will release our code and trained models under the Apache 2.0 license.

Primary area:
generative models
Keywords:
attention, sparsity, RNN, LRNN, recurrent, KV, KV-Cache, vector quantization, transformer, TTT, test time training, linear attention, rwkv, gdn, kda, compressive state

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HebVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes Key-Value Means (KVM), a block-recurrent attention architecture that combines block sliding-window attention (BSWA) for recent tokens with a compressed key-value state for longer-range context. The state can either have a fixed capacity or grow according to a predefined schedule. KVM updates the state by assigning incoming tokens to existing slots based on key similarity and merging their key-value representations, with additional mechanisms for key normalization and value magnitude preservation. The paper evaluates KVM on language modeling, long-context benchmarks, and kernel-level performance benchmarks, and provides optimized kernel implementations. The results suggest that KVM offers a promising trade-off between long-context modeling quality and computational efficiency.

Critical strengths
  1. Practical motivation and design. KVM addresses the practical challenge of balancing long-context modeling quality, memory usage, and computational cost. Combining exact local attention with a compressed state provides a flexible design whose memory budget can be adjusted according to different requirements.

  2. Broad evaluation and kernel implementation. The paper evaluates KVM across multiple types of tasks, including long-context benchmarks, language modeling, and speed benchmarks. The authors also provide optimized kernel implementations and benchmark the costs of training, prefill, and decoding. This implementation effort is valuable for assessing the practical potential of the proposed architecture.

Critical weaknesses
  1. The novelty and distinction from prior work are not sufficiently clear. The general idea of combining local attention, such as SWA/BSWA, with a state that summarizes longer-range context has appeared in various previous approaches. Although the paper discusses a broad range of related work, it does not sufficiently clarify the core conceptual differences between KVM and existing methods. My understanding is that the main distinction may lie in its state-update mechanism, particularly the similarity-based merging of incoming tokens into existing state slots. However, while Appendix E provides ablations of some design choices, these experiments are preliminary, conducted only at the 120M scale, and lack sufficient detail and analysis to establish how much each component contributes to the overall performance. This weakens the evidence supporting the paper's central novelty.

  2. The model scale is limited, and the baseline comparisons could be stronger. The experiments cover multiple evaluation settings, but the models are relatively small, at 120M and 350M parameters. For example, many reported accuracy values on NIAH-S3 are quite low, making it difficult to determine how well the method handles more challenging retrieval settings, such as multi-needle retrieval or other long-context tasks. In addition, the comparison lacks stronger hybrid-attention baselines, such as architectures that combine full attention with sliding-window or linear attention in different proportions across layers. The use of GPTAlpha-2 as the main backbone also raises questions about how the results generalize to more commonly used model architectures.

Questions for authors
  1. What is the fundamental distinction between KVM and previous state-based attention methods? Beyond the overall combination of local attention and compressed state, which specific components are responsible for the gains? Could mechanisms such as similarity-based merging and Value renormalization also improve existing state-based architectures when incorporated into their state-update procedures?

  2. Can KVM be evaluated at a larger model scale and against stronger hybrid-attention baselines? In particular, it would be useful to compare KVM with architectures that allocate different proportions of layers to full attention and more efficient attention mechanisms, such as a 3:1 hybrid configuration. Such comparisons could help establish whether KVM provides benefits beyond those obtainable through existing layer-wise hybrid designs.

  3. How does KVM fit into modern hybrid-attention architectures? Many recent architectures use different attention mechanisms at different layers, with mechanisms such as DSA for global context and SWA for local context. In this setting, is there a clear advantage to combining local and global information within each KVM attention layer, rather than handling them separately across layers? Can KVM be integrated into such hybrid architectures, and if so, what benefits would it offer?

6 comments
AuthorsOpenReview ↗

Response to the Review [1/3]

Thank you for your detailed review and recognizing KVM’s promising trade-off between long-context modeling quality and computational efficiency, flexible design, practical implementation effort, and broad evaluation. Your questions helped us strengthen the paper: we add more ablations at 120M and 350M, four 750M models, six 350M hybrids, a Qwen3-style backbone comparison, and a test of local attention within KVM when separate SWA layers are present. Results are in Appendices J and K. We provide point-by-point explanations below and will integrate them into the camera-ready.

W1/Q1. Novelty and distinction from prior work.

  1. Budget-controlled growth of a continuously updated KV state for trainable softmax-attention. KVM supports arbitrary feasible nondecreasing state-size budgets. KVM-sqrt has O(sqrt(N)) amortized computation per token and substantially stronger long-context retrieval than fixed/saturating-state models in our experiments. NHA, Block-Recurrent Transformer (BRT), and TransformerFAM have fixed-size recurrent states. While Memory Caching (concurrent) expands accessible memory through frozen checkpoints of an underlying recurrence, KVM expands the set of KV slots available for subsequent dynamic updates. OVQ (concurrent) schedules dictionary growth, but its linear-growth ablation substantially degrades long-context recall relative to its main saturating schedule, without showing a good non-saturating alternative. KVM shows effective non-saturating sublinear growth, together with the distinct allocation, accumulation, and readout mechanisms below. Our experiments with KVM-256 and KVM-sqrt (and OVQ/SWA) show benefits of sqrt(N) (more generally, unbounded sublinear) growth over fixed/saturating state sizes.

  2. Least-redundant append, followed by merge into the expanded state. OVQ merges before appending, preventing merges into appended tokens. KVM’s append-then-merge ordering uses newly allocated capacity for merging - our new ablation O1 shows its benefits. E1 and E2 show retrieval benefits of least-redundant selection over random and evenly spaced append under the same state budget. B1 additionally compares with bipartite compression.

  3. Gated additive writes to individual persistent KV slots, done in chunks. Each overflow token contributes a learned positive weight times its key and value to a single selected KV slot. Updates are batched, and the resulting rows are the KV entries subsequently read by attention. NHA uses a gated recurrent blend over a fixed set of slots. BRT and TransformerFAM update latent states through attention, with the latter also applying an FFN. Titans and LaCT update neural-network fast weights through an inner optimization step. Relative to OVQ’s count-based centroid averaging, KVM learns the strength of incoming writes. This sparse, trainable accumulation rule operates directly on the KV rows used for retrieval. A4 tests the benefits of gating.

  4. JIT normalization with value radius fixed at slot creation. KVM accumulates key and value sums per slot, normalizes keys immediately before readout, and rescales values to the slot’s original radius, so merging changes the directions of readout values while preserving their norms. OVQ instead averages keys and values and uses a log count bias as an explicit prior for a vector quantization-motivated approximation to attention. KVM’s normalization/scaling avoids norm shrinkage from averaging vectors with different directions. A3 replaces radius preservation with division by the number of tokens assigned to the slot. The resulting reductions in retrieval and RULER scores support KVM’s value-normalization choice.

  5. Partial RoPE with positional information zeroed before write. KVM’s specific contribution is the channel-level treatment of keys as they move from the exact window into merged slots. KVM uses partial RoPE for local attention, then removes positional information when keys enter global slots, allowing them to combine with other tokens regardless of their original positions. OVQ does not use RoPE in its main architecture; NHA combines positional local attention with long-term memory lacking positional encoding. Ablations P1 and P2 show the importance of removing positional information during writes into compressed state.

  6. Regional logit scales inside the shared readout. KVM combines regional attention logit scales with the growing, normalized KV state. Each head learns separate inverse temperatures for the compressed-state and BSWA-window logits, allowing flexible regional scales within the same softmax. OVQ and NHA lack this distinction, though OVQ uses the log token count per dictionary cluster as a logit bias to approximate attention. A2 tests the benefits of regional temperature scales.

These distinctions refine the related-work discussion in Appendix A, the design rationale in Appendix H, and the contribution list in Section 1. We will list them explicitly in the camera-ready.

AuthorsOpenReview ↗

Response to the Review [2/3]

W1. Ablation details and new ablations.

Thank you for pointing out that the set of ablations and their descriptions can be improved upon. We do so below via a) further explanation, b) ablations for KVM-sqrt, c) and new ablations that further clarify the roles of these components. We will include related text revisions for camera ready.

We expand the component and positional-encoding ablations to 120M KVM-sqrt and add controlled append-policy comparisons at 350M (Appendix J, Figures 8 and 9).

a) Explanation of existing ablations

  • The “no sink”/A1 ablation describes the case where we formally set S=0S = 0, i.e., no sinks are protected.
  • The “no head temps”/A2 ablation describes the case where the learned per-head scalar inverse temperatures in the original architecture are replaced by 11.
  • The “no v-len normalization”/A3 ablation describes the case where instead of v-len renormalization, we use a strategy that averages value vectors in each slot using the number of tokens assigned to the slot.
  • The “no merge gate”/A4 ablation describes the case where the gate gtg_t is replaced by 11.

b) Explanation of new ablations

  • O1 - Merge before append: Instead of appending tokens and then merging into the enlarged global state, we first merge tokens (selected for merge) into the old global state and then append the tokens selected for appending.
  • E1 - Random append: Tokens to append are chosen randomly from among the overflow tokens.
  • E2 - Evenly spaced append: Tokens to append are chosen at even spacing from among the overflow tokens.
  • B1 - Bipartite: As tokens leave the local window, B1 uses chunked bipartite soft matching, following CentroidKV.
  • P1 - Local-only RoPE: RoPE is applied to local attention, while attention to the global state uses unrotated queries and keys.
  • P2 - RoPE everywhere: RoPE is applied on both local and state tokens but no RoPE subspaces are zero-ed.

Please see Appendix J of the updated paper for these new results and further details.

c) KVM-sqrt ablations

Please see Appendix J Figure 8 of the updated paper for these new results and further details.

Q1. Benefits in other state-based architectures.

Yes, we believe that these mechanisms can be incorporated into other architectures where they are relevant. Their applicability depends on the state representation and the readout.

Our experiments establish the benefits of these mechanisms for KVM, which is a combination of many different parts into a unified whole designed to work well together. We had strong reasons for each choice, and thus incremental application to other work may or may not lead to as strong improvements as we saw here. Testing transfer of these choices to other related architectures is an experimental question that we leave to future work.

W2. Larger trained models.

To address the question of scale, we trained and benchmarked four new 750M scale models. Due to limited compute, we used the same data as the 350M models.

We find that scale helps improve performance at hard retrieval tasks, and the evidence seen at small scales extends to 750M scale too - KVM-256 is a good low-cost option for recall, while KVM-sqrt, due to its growing state, is a good tradeoff between full attention and fixed-state architectures.

The 750M results also substantially improve performance on NIAH-S3. We also report full RULER at 4K and 8K, which evaluates multi-needle and other long context-tasks too.

Please see Appendix K Figure 10 of the updated paper for these new results and further details.

W2/Q2/Q3. Hybrids.

Thank you for these insightful questions which led us to expand our investigation into how KVM might be used in these scenarios. To address the request for stronger hybrid-attention baselines, we have trained a new set of 350M scale (3:1) ratio hybrids. In each row below, three layers of the first type are interleaved with one layer of the second type (Appendix K).

  • H1: (SWA, full attention)
  • H2: (RWKV-7, full attention)
  • H3: (SWA, KVM-sqrt)
  • H4: (RWKV-7, KVM-sqrt)
  • H5: (KVM-256, full attention)
  • H6: (KVM-sqrt, full attention)

We note that KVM can function well in both types of layers in a hybrid setting - the efficient layer as well as the global layer. KVM-256 and KVM-sqrt are both strong contenders for the efficient layer, and KVM-sqrt, alongside SWA as the efficient layer, provides a good choice for global attention at sublinear decoding cost even in a hybrid setting (standard hybrid settings use full attention, which has linear decoding costs in a hybrid setting, lower only by a constant factor than non-hybrid full attention asymptotically).

Please see Appendix K Figure 11 of the updated paper for these new results and further details.

AuthorsOpenReview ↗

Response to the Review [3/3]

W2. Backbone ablations.

Thank you for pointing out that GPTAlpha-2 should be ablated too. We believe that good architecture research requires strong baselines, and that was the motivation behind our choice of GPTAlpha-2. To address concerns regarding GPTAlpha-2 as the main backbone, we experiment at the 350M scale with a Qwen3-style backbone with a similar parameter count and separate additional hyperparameter tuning.

We observe that the GPTAlpha-2 transformer performs better than the Qwen3 transformer across benchmarks (long context extrapolation, context usage within training context length, and short context evals), and the same is observed for GPTAlpha-2 KVM-sqrt versus Qwen3 KVM-sqrt.

On the Qwen3-style backbone, KVM-sqrt also combines improved retrieval and comparable short-context performance with sublinear memory growth. These results extend the empirical support for its memory-quality tradeoff beyond the GPTAlpha-2 backbone.

Please see Appendix K Figure 12 of the updated paper for these new results and further details.

Q3. Benefit of combining local and global state into a single KVM layer.

Regarding the question about the advantage of combining local and global information within each KVM attention layer, our original results did already contain 1:1 hybrids and show some of the impact of having SWA on a separate layer, but we have run a new ablation H7 where we reduce the BSWA window in KVM to a single chunk and do not attend to it (the chunked design of KVM which is essential to allow KVM to be efficient requires at least one chunk remaining). Please see Appendix K Figure 13 of the updated paper for these results.

We observe that local attention within KVM contributes to performance even when separate SWA layers are present, indicating the importance of combining local and global information within the same KVM layer.

We sincerely appreciate the opportunity to improve the paper and address your concerns fully.

Reviewer_3DLbOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

KVM combines block sliding-window attention with a compressed, optionally growing KV state within one softmax attention layer. Similarity-based merging, budgeted expansion, and readout normalization allow different memory–computation trade-offs. Experiments cover 120M/350M models, retrieval tasks, architectural ablations, and sequence-mixing efficiency.

Critical strengths
  1. The architecture offers a coherent way to combine local exact attention with compressed historical memory. Its components form an implementable design with explicit control over state capacity.
  2. The experiments provide useful evidence for several design choices, particularly normalization, sinks, and gating. The efficiency measurements are also informative: Figure 5 includes routing and state-update costs, making the reported sequence-mixing speedups more meaningful than measurements of the attention operation alone.
Critical weaknesses
  1. The evidence for the specific expansion policy is less conclusive than the evidence for the overall architecture. The ablations primarily examine the fixed-state 120M configuration, while comparisons between fixed and growing states change both capacity and update behavior. These comparisons establish the behavior of the growing-state design, but do not isolate the benefit of choosing the least-redundant tokens. Matching only the final state size would also leave differences in capacity during processing. A more informative control would preserve the expansion schedule and number of added entries while changing the token-selection rule, for example to random or recent eligible tokens.
  2. The efficiency results establish benefits at the sequence-mixing level, and the paper clearly states that scope. What remains unclear is how much those gains affect complete-model inference. Fewer historical KV entries provide a clear rationale for memory savings, but do not quantify peak device-memory usage or end-to-end latency. Measurements using existing checkpoints, with quality and efficiency paired under the same configurations, would make the practical trade-off easier to assess.
Questions for authors
  1. What evidence isolates least-redundant-token selection from the benefit of additional state capacity? Would a simpler selection policy perform similarly with the same expansion schedule, number of appended entries, and remaining update rules?
  2. Can existing checkpoints provide paired quality, end-to-end prefill and decoding latency, and peak-memory measurements under the same model and state configurations?
2 comments
AuthorsOpenReview ↗

Response to the Review

Thank you for the careful review and for recognizing the architecture's coherence, the value of the existing ablations, and the careful benchmarking in our efficiency measurements. We address both of the concerns with new experiments in Appendices I and J (Figures 7 and 9), and we believe that these experiments have strengthened our paper significantly. We have also added many other experiments in new Appendices I, J, and K which you may find of interest.

W1/Q1. Isolating the benefit of least-redundant-token expansion from state capacity.

We have added experiments that try alternative expansion strategies (Appendix J, Figure 9), and observe that least-redundant-token expansion contributes significantly to strong long-context retrieval.

We compare least-redundant append with random and uniformly spaced append in the 350M KVM-sqrt setup, keeping the state-budget schedule and all remaining operations unchanged. We additionally evaluate B1, a bipartite compression variant modeled on CentroidKV that pools overflow tokens with existing state entries and compresses them to the same square-root state budget.

W2/Q2. End-to-end comparison for quality, latency and peak memory.

We provide new, realistic, end-to-end timing and memory tests for the 350M KVM-256, KVM-sqrt and GPTA-2 models (Appendix I, Figure 7). We use a batch size of 32 sequences, with chunked prefill for 2K tokens, and decode timings averaged over 256 consecutive tokens to correctly amortize over KVM’s chunk handling, with this value’s median taken over different trials. End-to-end quality tests for these three models are addressed by the existing experimental regime in Table 2 and Appendix E, and we will highlight them in the paper for easier reference.

We observe that at long contexts, KVM variants have significantly better timing and memory profiles than full attention in the end-to-end tests too. Due to common work across architectures (e.g., the FFN part of the model), their end-to-end speedups/memory improvements are smaller than that for a single-layer, but are nevertheless significant enough to motivate considering their use in practical serving, especially at long contexts, such as in long horizon reasoning and agentic tasks.

We sincerely appreciate the opportunity to improve the paper and address your concerns fully.

AuthorsOpenReview ↗

Follow-up on our response

We would like to thank you again for your time and thoughtful feedback. We hope our responses and revisions address the points you raised, and would welcome any further thoughts or questions as the discussion period draws to a close.

Reviewer_VzjJOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Key-Value Means (KVM), a novel block-recurrence for attention that blends many of the benefits of both transformers and RNNs. It addresses the trade-offs of transformers and of RNNs and introduces a method to apply a pre-selected state growth schedule to interpolate between the two.

This paper motivates and justifies the following architectural design methods:

  • KVM processes a sequence in blocks and keeps the keys and values of the most recent blocks in an uncompressed Block Sliding Window Attention (BSWA).
  • Tokens evicted from the BSWA window are compressed into a state in batched fashion. The least redundant tokens are appended as new state rows, and the rest are merged into the state row with the most similar normalized key.
  • The state’s growth rate is selectable, and can range from a fixed size (O(1)) to a size that grows with length (O(n)).
  • State keys and values are renormalized Just-In-Time before attention, with each value rescaled to the norm it had when its row was created.
  • Rotary positional information is removed from keys as they are merged into the state.

To demonstrate the relative performance of the KVM architecture in various configurations, the authors train full attention, RWKV-7 (LRNN), and KVM models at 120M and 350M parameters for 3B and 7.8B tokens respectively on the Prolong dataset at 8k context length and show the relative performance on short- and long-context tasks. They find that KVM matches full attention on short-context benchmarks. Within the training length (8k context length), the results match full attention on the NIAH-S1 dataset, but KVM-256 falls short on the NIAH-S2/S3 variants, where KVM-N\sqrt{N} is close to or above full attention at 350M. Beyond 8k, pure KVM variants keep near-perfect NIAH-S1 accuracy while full attention degrades, and KVM-N\sqrt{N} at 350M parameters also leads benchmarks on most NIAH-S2/S3 settings. KVM is much stronger than RWKV-7 on retrieval but trails full attention on RULER benchmark.

Critical strengths
  1. This paper introduces novel ideas regarding interpolation between transformers and RNNs. KVM’s state can either stay fixed or grow on a chosen schedule, giving one mechanism a range of memory/recall trade-offs. This is consequential because the growing-state variant substantially narrows the recall gap between RNNs and full attention while keeping sublinear state growth.
  2. KVM models outperform the RWKV-7 baseline on needle-in-a-haystack tasks while achieving increasing advantages over full attention in decode, prefill, and backward passes at increasing context lengths. This shows the architecture design works in practice.
Critical weaknesses
  1. On the RULER benchmark, every KVM variant scores below every full-attention model at both the 120M and 350M parameter scales. This shows that by compressing its memory, KVM gives up some recall compared to full attention, even at context lengths the models were trained on.
Questions for authors

(Clarification) Why were the ablations run only on the fixed-state KVM-256 model and not on KVM-N\sqrt{N}, which has the strongest long-context results? Do the ablation conclusions hold for the growing-state variant?

3 comments
AuthorsOpenReview ↗

Response to the Review

Thank you for recognizing that KVM can interpolate between transformers and RNNs, has better recall than RWKV-7, and that it has an increasing efficiency advantage over full attention at longer contexts. In response to the reviews we have added a large number of new experiments and evidence to the paper in Appendices I, J, and K.

The observation that KVM gives up some recall compared to full attention even at trained context lengths is indeed correct, and is in line with what is also observed with our other non-full-attention baselines such as RWKV-7 and OVQ/SWA. We believe that this is broadly due to full attention attending to the full context, while other methods use a compressed representation of the full context. This gives full attention an advantage at certain tasks that require complex usage of context.

Regarding your mention of RULER, we are also including benchmarks in Appendix K (Figure 11) on some new 350M 3:1 hybrid models. Model H6, which is a 3:1 KVM-sqrt:full-attention hybrid, seems to outperform the equivalent dense GPTAlpha-2 transformer on RULER at 4K context.

We would also like to note that although KVM gives up some recall in its trained context length compared to full attention, it has the interesting property of extrapolating to unseen context lengths better than full attention.

Thank you for raising the important point about the scope of the ablations. Per your request, we have provided an extended set of ablations for KVM-sqrt in Appendix J (Figure 8). These results support the importance of the different components in the growing-state variant too, and we will integrate these into the main text. We believe that these suggested experiments as well as the other new evidence shown in the new appendices have significantly strengthened the paper.

AuthorsOpenReview ↗

Follow-up on our response

We would like to thank you again for your time and thoughtful feedback. We hope our responses and revisions address the points you raised, and would welcome any further thoughts or questions as the discussion period draws to a close.

Thank you for the responses and for the additional experiments in Appendices I, J, and K. The KVM-sqrt ablations in Appendix J answer my questions, although the "no merge gate" ablation seems to have advantages on certain tasks. It would also help to present the KVM 256 and KVM-sqrt ablations in a consistent format for easier direct comparison. My concerns have mostly been addressed, and I maintain my score of 3

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

We thank the reviewers for their time and their constructive reviews and valuable suggestions that helped us improve the paper. We appreciate their recognition of KVM’s flexible memory-recall tradeoffs (VzjJ, HebV), coherent and implementable architecture (3DLb), strong retrieval relative to RWKV-7 (VzjJ), and broad evaluation of quality and systems performance (3DLb, HebV).

Summary of main revisions

In response to the reviewers’ feedback, we have added a large number of new experiments in Appendices I, J and K that we summarize below.

  • End-to-end speed and memory benchmarks (3DLb): At long contexts, we observe that KVM variants have substantially lower end-to-end latency and memory use than full attention.
  • Component/positional-encoding ablations on KVM-sqrt (VzjJ, 3DLb, HebV): We observe that each component/positional-encoding decision in KVM contributes to its performance.
  • Comparison of KVM’s append policy with alternatives with the same state schedule (3DLb, HebV): We observe that appending the least redundant tokens is important for the growing state variant, especially for long-context retrieval extrapolation.
  • Models at the 750M scale (HebV): We observe that our findings extend to the 750M scale - KVM-sqrt maintains its retrieval extrapolation advantage over full attention and its recall advantage over fixed-state models, while maintaining comparable short-context performance.
  • Multiple 3:1 hybrids involving KVM, SWA, RWKV-7 and full attention (HebV, VzjJ): We observe that in addition to vanilla KVM-only models, KVM also works well in hybrids with other attention mechanisms, both as the efficient layer (KVM-fixed, KVM-sqrt) and as the global layer (KVM-sqrt), and when used as the global layer with a cheaper efficient layer in a hybrid, it gives end-to-end sublinear amortized decoding cost per token.
  • Comparisons involving alternative (Qwen3-style) backbones (HebV): We observe that our results also transfer to a Qwen3-style backbone and that GPTA-2 is a stronger backbone than Qwen3-style in our experiments.
  • An ablation of the local attention in KVM layers when the model also has separate SWA layers (HebV): We observe that local attention (BSWA) inside KVM is important even when there are other SWA layers in the model.

We also clarify distinctions from prior work in our response to reviewer HebV below.

Implementation note: In the interest of time and due to compute limitations, we used our latest training kernels, which are both faster and use higher precision math for updates, to do the new experiments unless otherwise noted in the paper. The corresponding baseline is always also run with a matched kernel implementation anywhere these were used.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.