- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Kernelized Linear Attention Activations (KATA), a framework that studies linear-attention feature maps through the geometry of symmetric cones. It analyzes associative recall using spherical-packing bounds and power signal-to-noise ratio (pSNR), showing that rank-one positive semidefinite (PSD) features can reduce interference through squared inner products and improve idealized retrieval capacity. The authors develop several PSD-based linear-attention variants with additive, gated, and delta-rule updates, alongside Triton implementations. Experiments on MQAR, repeated-key overwrite, and 340M-parameter language models demonstrate substantial improvements in associative recall over existing recurrent architectures, although these gains involve larger recurrent states and trade-offs in language-modeling quality.
Critical strengths
-
Principled geometric analysis of associative capacity. The paper connects symmetric-cone geometry, spherical packing, and interference bounds to explain limitations of linear attention. The PSD construction and pSNR analysis provide a useful theoretical perspective on how nonlinear feature maps improve associative recall.
-
Strong associative-recall results across multiple settings. KATA-M1 achieves 0.985 MQAR accuracy at 16× extrapolation, compared with 0.419 for Gated DeltaNet, while KATA-M2 substantially improves high-entropy UUID retrieval in pretrained language models (0.616 vs. 0.004 at 2K). The repeated-key overwrite experiments further demonstrate the complementary benefits of feature geometry and delta-rule updates.
Critical weaknesses
-
Unclear quality-efficiency advantage over existing architectures. The improved recall relies on substantially expanded PSD feature states: KATA-M1 uses 266K state entries versus 65.5K for the largest Gated DeltaNet baseline, while the linear-state PSD kernels remain slower than Gated DeltaNet. Without state-matched comparisons against competitive architectures or simpler quadratic feature maps, it is difficult to establish whether the proposed geometry offers a superior capacity-efficiency trade-off rather than primarily benefiting from increased state size.
-
Limited validation of practical language-modeling benefits. The pretrained experiments are restricted to 340M models with 2K context, where KATA generally exhibits worse perplexity than Gated DeltaNet and softmax attention. Moreover, the strong MQAR length extrapolation deteriorates sharply with RoPE (Table 5), and the pretrained models also struggle beyond their training horizon, raising concerns about whether the associative-recall improvements translate to practical long-context language modeling.
Questions for authors
-
Can the authors provide state-memory-matched comparisons against Gated DeltaNet and simpler quadratic-kernel baselines, together with inference latency, to establish whether KATA improves the recall-efficiency Pareto frontier?
-
Can the authors clarify whether the RoPE extrapolation failures are intrinsic to the squared-inner-product kernel and whether the improved recall persists under controlled, architecture-matched comparisons at longer context lengths?
3 comments
Thank you for the detailed rebuttal and clarifications.
The state-matched comparison between KATA-$\Sigma$2 and Gated DeltaNet addresses my primary concern in Q1. Achieving substantially better MQAR accuracy (0.958 vs. 0.419 at 4K) with comparable state sizes provides convincing evidence that the improvement is not simply due to a larger recurrent state. I also appreciate the parameter-efficiency argument for PSD feature expansion, although a direct latency comparison would further strengthen the practical efficiency claims.
Regarding Q2, I agree that the RoPE extrapolation failure is not specific to KATA, as the softmax baseline exhibits similar degradation. The distinction between associative recall capacity and general language-modeling quality is also well taken. However, the practical benefits remain somewhat limited: while DeltaKATA improves WikiText perplexity over Gated DeltaNet, its LAMBADA perplexity remains higher, and the results suggest that improved recall alone is insufficient for stronger language modeling.
Overall, the rebuttal resolves much of my concern about the recall-efficiency advantage and clarifies the intended scope of the contribution. Given the theoretical insights and convincing associative-recall improvements, I am inclined to increase my rating to reflect them, while retaining some reservations about the practical language-modeling benefits and end-to-end efficiency.
Comment
We appreciate the reviewer's follow-up and for reconsidering their assessment after addressing their concerns.
Regarding the practical language-modeling benefits of linear attention, it's worth noting that it's a core result of the empirical experiments: expanding the capacity of the linear attention mechanism using our KATA approach significantly improves recall, but not fluency. That's also the reason we introduced the (new) overwrite benchmark (which is not part of the Zoology's test suite). Our diagnosis relates exactly to the point you have raised about RoPE: within linear attention mechanisms (not just KATA-derived), RoPE does not provide a positional signal that is strong enough to encode the sequential order of the keys. The delta-rule allows to solve this by explicitly decaying the values associated with already seen keys, but that might not be a desirable effect if the task favors an older value. Hence the need for learnable forget/keep scores. These are promising directions for future work.
Rebuttal P1
We thank the reviewer for recognizing the originality of our geometric analysis and for raising some important questions related to the recall-efficiency advantage of KATA feature maps.
Before we address the raised questions, we invite you to check our responses to the other reviewers and our clarification about our contributions.
The limitation on the 340M LLMs are not due to recall, our results show that PSD kernels do better at retrieval than first order linear attention. What the LLMs results demonstrate is that the improved recall is not sufficient to improve language modeling tasks, especially when mixing similar concepts. This motivates integrating the delta-rule into the KATA framework to incorporate overwrite as well, but is left for future work.
To address the questions raised by the review:
Q1. Can the authors provide state-memory-matched comparisons against Gated DeltaNet and simpler quadratic kernels, together with inference latency? A1: The advantage of using PSD outer product expansion is that we can get larger state space without adding parameters to the neural network. When k have d dimensions, we can map the key to a D=d(d+1)/2 state space via the outer product kk⊤. First order attention (GLA, GDN) has to map to D directly to get the same capacity. In the Qwen3.5 4B, that means that the model has to add 16B parameters in its GatedDeltaNet layers to achieve similar capacity to the quadratic PSD kernel. The inference in this case we be dominated by the projections WK and WQ, not by the the linear attention state-space update.
Our paper also compares different architectures at various states space sizes. Even in equal state space size, PSD kernels are better at recall.
Based on Table 1 (other expanded results are in the appendix), closely matched feature-value state sizes:
We have deliberately introduced KATA-Σ2 (PSD features) to get smaller feature-value space, and it achieves 0.958 versus 0.419 at 16× the training length, with similar state entries and 5.33× fewer parameters. The improvement therefore persists at substantially smaller state sizes than KATA-M1's 266,240 entries (our response to Reviewer 6Aqr discusses the MQAR selection protocol).
So KATA PSD feature maps are not only closer to recall-efficiency pareto boundary, but also to the parameterization-recall pareto frontier. We have expanded on the later using Qwen3.5-4B as example in our response to Reviewer iocT: how KATA does not require additional model parameters to improve recall. The expanded state space is 10x cheaper than expanding the attention head dimension.
Q2. Are RoPE extrapolation failures intrinsic to the squared kernel, and does recall persist in controlled longer-context comparisons?
A2: The failure is not unique to the squared kernel. Table 5 includes the following MQAR ablation at 4K, after training up to 256 tokens:
The pretrained models share a 340M backbone, while mixer components differ. At 2K, UUID recall is 0.616 for KATA-M2, 0.004 for GDN, and 0.930 for softmax. Replacing KATA-M2's RoPE with a GDN-style short convolution retains 0.530, so its in-window recall benefit does not require RoPE.
KATA's strong result without RoPE shows that the squared kernel can extrapolate on this task. The issue when RoPE is used is two folds:
The motivation behind KATA is maximizing recall at fixed state space, which PSD feature map is optimal under hardware constraints. Whether that translates to better language modeling is a different question: our empirical results show that it's negative. When we incorporated Delta-rule with KATA, we got better perplexity on Wikitext than GDN and reduced it over LAMBADA by 1 nat. Delta-rule in our (novel) overwrite benchmark showed that it resolves the issue of mixing different values for the same key, which we interpreted as better fluency. In a much simpler analogy: if one has better ability at memorizing the information they hear, that does not mean they're better at combining pieces of information with subtle differences. Softmax attention which its large capacity is still better at the mixing. These results show that treating attention as pure associative recall mechanism is not doing it justice and suggests better architecture via a DeltaKATA variant.