- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes Key-Value Means (KVM), a block-recurrent attention architecture that combines block sliding-window attention (BSWA) for recent tokens with a compressed key-value state for longer-range context. The state can either have a fixed capacity or grow according to a predefined schedule. KVM updates the state by assigning incoming tokens to existing slots based on key similarity and merging their key-value representations, with additional mechanisms for key normalization and value magnitude preservation. The paper evaluates KVM on language modeling, long-context benchmarks, and kernel-level performance benchmarks, and provides optimized kernel implementations. The results suggest that KVM offers a promising trade-off between long-context modeling quality and computational efficiency.
Critical strengths
-
Practical motivation and design. KVM addresses the practical challenge of balancing long-context modeling quality, memory usage, and computational cost. Combining exact local attention with a compressed state provides a flexible design whose memory budget can be adjusted according to different requirements.
-
Broad evaluation and kernel implementation. The paper evaluates KVM across multiple types of tasks, including long-context benchmarks, language modeling, and speed benchmarks. The authors also provide optimized kernel implementations and benchmark the costs of training, prefill, and decoding. This implementation effort is valuable for assessing the practical potential of the proposed architecture.
Critical weaknesses
-
The novelty and distinction from prior work are not sufficiently clear. The general idea of combining local attention, such as SWA/BSWA, with a state that summarizes longer-range context has appeared in various previous approaches. Although the paper discusses a broad range of related work, it does not sufficiently clarify the core conceptual differences between KVM and existing methods. My understanding is that the main distinction may lie in its state-update mechanism, particularly the similarity-based merging of incoming tokens into existing state slots. However, while Appendix E provides ablations of some design choices, these experiments are preliminary, conducted only at the 120M scale, and lack sufficient detail and analysis to establish how much each component contributes to the overall performance. This weakens the evidence supporting the paper's central novelty.
-
The model scale is limited, and the baseline comparisons could be stronger. The experiments cover multiple evaluation settings, but the models are relatively small, at 120M and 350M parameters. For example, many reported accuracy values on NIAH-S3 are quite low, making it difficult to determine how well the method handles more challenging retrieval settings, such as multi-needle retrieval or other long-context tasks. In addition, the comparison lacks stronger hybrid-attention baselines, such as architectures that combine full attention with sliding-window or linear attention in different proportions across layers. The use of GPTAlpha-2 as the main backbone also raises questions about how the results generalize to more commonly used model architectures.
Questions for authors
-
What is the fundamental distinction between KVM and previous state-based attention methods? Beyond the overall combination of local attention and compressed state, which specific components are responsible for the gains? Could mechanisms such as similarity-based merging and Value renormalization also improve existing state-based architectures when incorporated into their state-update procedures?
-
Can KVM be evaluated at a larger model scale and against stronger hybrid-attention baselines? In particular, it would be useful to compare KVM with architectures that allocate different proportions of layers to full attention and more efficient attention mechanisms, such as a 3:1 hybrid configuration. Such comparisons could help establish whether KVM provides benefits beyond those obtainable through existing layer-wise hybrid designs.
-
How does KVM fit into modern hybrid-attention architectures? Many recent architectures use different attention mechanisms at different layers, with mechanisms such as DSA for global context and SWA for local context. In this setting, is there a clear advantage to combining local and global information within each KVM attention layer, rather than handling them separately across layers? Can KVM be integrated into such hybrid architectures, and if so, what benefits would it offer?
6 comments
Response to the Review [2/3]
W1. Ablation details and new ablations.
Thank you for pointing out that the set of ablations and their descriptions can be improved upon. We do so below via a) further explanation, b) ablations for KVM-sqrt, c) and new ablations that further clarify the roles of these components. We will include related text revisions for camera ready.
We expand the component and positional-encoding ablations to 120M KVM-sqrt and add controlled append-policy comparisons at 350M (Appendix J, Figures 8 and 9).
a) Explanation of existing ablations
- The “no sink”/A1 ablation describes the case where we formally set , i.e., no sinks are protected.
- The “no head temps”/A2 ablation describes the case where the learned per-head scalar inverse temperatures in the original architecture are replaced by .
- The “no v-len normalization”/A3 ablation describes the case where instead of v-len renormalization, we use a strategy that averages value vectors in each slot using the number of tokens assigned to the slot.
- The “no merge gate”/A4 ablation describes the case where the gate is replaced by .
b) Explanation of new ablations
- O1 - Merge before append: Instead of appending tokens and then merging into the enlarged global state, we first merge tokens (selected for merge) into the old global state and then append the tokens selected for appending.
- E1 - Random append: Tokens to append are chosen randomly from among the overflow tokens.
- E2 - Evenly spaced append: Tokens to append are chosen at even spacing from among the overflow tokens.
- B1 - Bipartite: As tokens leave the local window, B1 uses chunked bipartite soft matching, following CentroidKV.
- P1 - Local-only RoPE: RoPE is applied to local attention, while attention to the global state uses unrotated queries and keys.
- P2 - RoPE everywhere: RoPE is applied on both local and state tokens but no RoPE subspaces are zero-ed.
Please see Appendix J of the updated paper for these new results and further details.
c) KVM-sqrt ablations
Please see Appendix J Figure 8 of the updated paper for these new results and further details.
Q1. Benefits in other state-based architectures.
Yes, we believe that these mechanisms can be incorporated into other architectures where they are relevant. Their applicability depends on the state representation and the readout.
Our experiments establish the benefits of these mechanisms for KVM, which is a combination of many different parts into a unified whole designed to work well together. We had strong reasons for each choice, and thus incremental application to other work may or may not lead to as strong improvements as we saw here. Testing transfer of these choices to other related architectures is an experimental question that we leave to future work.
W2. Larger trained models.
To address the question of scale, we trained and benchmarked four new 750M scale models. Due to limited compute, we used the same data as the 350M models.
We find that scale helps improve performance at hard retrieval tasks, and the evidence seen at small scales extends to 750M scale too - KVM-256 is a good low-cost option for recall, while KVM-sqrt, due to its growing state, is a good tradeoff between full attention and fixed-state architectures.
The 750M results also substantially improve performance on NIAH-S3. We also report full RULER at 4K and 8K, which evaluates multi-needle and other long context-tasks too.
Please see Appendix K Figure 10 of the updated paper for these new results and further details.
W2/Q2/Q3. Hybrids.
Thank you for these insightful questions which led us to expand our investigation into how KVM might be used in these scenarios. To address the request for stronger hybrid-attention baselines, we have trained a new set of 350M scale (3:1) ratio hybrids. In each row below, three layers of the first type are interleaved with one layer of the second type (Appendix K).
- H1: (SWA, full attention)
- H2: (RWKV-7, full attention)
- H3: (SWA, KVM-sqrt)
- H4: (RWKV-7, KVM-sqrt)
- H5: (KVM-256, full attention)
- H6: (KVM-sqrt, full attention)
We note that KVM can function well in both types of layers in a hybrid setting - the efficient layer as well as the global layer. KVM-256 and KVM-sqrt are both strong contenders for the efficient layer, and KVM-sqrt, alongside SWA as the efficient layer, provides a good choice for global attention at sublinear decoding cost even in a hybrid setting (standard hybrid settings use full attention, which has linear decoding costs in a hybrid setting, lower only by a constant factor than non-hybrid full attention asymptotically).
Please see Appendix K Figure 11 of the updated paper for these new results and further details.
Response to the Review [3/3]
W2. Backbone ablations.
Thank you for pointing out that GPTAlpha-2 should be ablated too. We believe that good architecture research requires strong baselines, and that was the motivation behind our choice of GPTAlpha-2. To address concerns regarding GPTAlpha-2 as the main backbone, we experiment at the 350M scale with a Qwen3-style backbone with a similar parameter count and separate additional hyperparameter tuning.
We observe that the GPTAlpha-2 transformer performs better than the Qwen3 transformer across benchmarks (long context extrapolation, context usage within training context length, and short context evals), and the same is observed for GPTAlpha-2 KVM-sqrt versus Qwen3 KVM-sqrt.
On the Qwen3-style backbone, KVM-sqrt also combines improved retrieval and comparable short-context performance with sublinear memory growth. These results extend the empirical support for its memory-quality tradeoff beyond the GPTAlpha-2 backbone.
Please see Appendix K Figure 12 of the updated paper for these new results and further details.
Q3. Benefit of combining local and global state into a single KVM layer.
Regarding the question about the advantage of combining local and global information within each KVM attention layer, our original results did already contain 1:1 hybrids and show some of the impact of having SWA on a separate layer, but we have run a new ablation H7 where we reduce the BSWA window in KVM to a single chunk and do not attend to it (the chunked design of KVM which is essential to allow KVM to be efficient requires at least one chunk remaining). Please see Appendix K Figure 13 of the updated paper for these results.
We observe that local attention within KVM contributes to performance even when separate SWA layers are present, indicating the importance of combining local and global information within the same KVM layer.
We sincerely appreciate the opportunity to improve the paper and address your concerns fully.
Response to the Review [1/3]
Thank you for your detailed review and recognizing KVM’s promising trade-off between long-context modeling quality and computational efficiency, flexible design, practical implementation effort, and broad evaluation. Your questions helped us strengthen the paper: we add more ablations at 120M and 350M, four 750M models, six 350M hybrids, a Qwen3-style backbone comparison, and a test of local attention within KVM when separate SWA layers are present. Results are in Appendices J and K. We provide point-by-point explanations below and will integrate them into the camera-ready.
W1/Q1. Novelty and distinction from prior work.
Budget-controlled growth of a continuously updated KV state for trainable softmax-attention. KVM supports arbitrary feasible nondecreasing state-size budgets. KVM-sqrt has O(sqrt(N)) amortized computation per token and substantially stronger long-context retrieval than fixed/saturating-state models in our experiments. NHA, Block-Recurrent Transformer (BRT), and TransformerFAM have fixed-size recurrent states. While Memory Caching (concurrent) expands accessible memory through frozen checkpoints of an underlying recurrence, KVM expands the set of KV slots available for subsequent dynamic updates. OVQ (concurrent) schedules dictionary growth, but its linear-growth ablation substantially degrades long-context recall relative to its main saturating schedule, without showing a good non-saturating alternative. KVM shows effective non-saturating sublinear growth, together with the distinct allocation, accumulation, and readout mechanisms below. Our experiments with KVM-256 and KVM-sqrt (and OVQ/SWA) show benefits of sqrt(N) (more generally, unbounded sublinear) growth over fixed/saturating state sizes.
Least-redundant append, followed by merge into the expanded state. OVQ merges before appending, preventing merges into appended tokens. KVM’s append-then-merge ordering uses newly allocated capacity for merging - our new ablation O1 shows its benefits. E1 and E2 show retrieval benefits of least-redundant selection over random and evenly spaced append under the same state budget. B1 additionally compares with bipartite compression.
Gated additive writes to individual persistent KV slots, done in chunks. Each overflow token contributes a learned positive weight times its key and value to a single selected KV slot. Updates are batched, and the resulting rows are the KV entries subsequently read by attention. NHA uses a gated recurrent blend over a fixed set of slots. BRT and TransformerFAM update latent states through attention, with the latter also applying an FFN. Titans and LaCT update neural-network fast weights through an inner optimization step. Relative to OVQ’s count-based centroid averaging, KVM learns the strength of incoming writes. This sparse, trainable accumulation rule operates directly on the KV rows used for retrieval. A4 tests the benefits of gating.
JIT normalization with value radius fixed at slot creation. KVM accumulates key and value sums per slot, normalizes keys immediately before readout, and rescales values to the slot’s original radius, so merging changes the directions of readout values while preserving their norms. OVQ instead averages keys and values and uses a log count bias as an explicit prior for a vector quantization-motivated approximation to attention. KVM’s normalization/scaling avoids norm shrinkage from averaging vectors with different directions. A3 replaces radius preservation with division by the number of tokens assigned to the slot. The resulting reductions in retrieval and RULER scores support KVM’s value-normalization choice.
Partial RoPE with positional information zeroed before write. KVM’s specific contribution is the channel-level treatment of keys as they move from the exact window into merged slots. KVM uses partial RoPE for local attention, then removes positional information when keys enter global slots, allowing them to combine with other tokens regardless of their original positions. OVQ does not use RoPE in its main architecture; NHA combines positional local attention with long-term memory lacking positional encoding. Ablations P1 and P2 show the importance of removing positional information during writes into compressed state.
Regional logit scales inside the shared readout. KVM combines regional attention logit scales with the growing, normalized KV state. Each head learns separate inverse temperatures for the compressed-state and BSWA-window logits, allowing flexible regional scales within the same softmax. OVQ and NHA lack this distinction, though OVQ uses the log token count per dictionary cluster as a logit bias to approximate attention. A2 tests the benefits of regional temperature scales.
These distinctions refine the related-work discussion in Appendix A, the design rationale in Appendix H, and the contribution list in Section 1. We will list them explicitly in the camera-ready.