- Rating
- 4(+2)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies post-training reduction of the matrix-valued hidden state in linear attention. It links the state's rank to keys/values and low utilization to query-noise amplification at readout. The authors exploit the joint key/query orthogonal invariance of (Gated) DeltaNet to build a structured, axis-aligned channel-pruning framework compatible with causal convolutions and existing kernels; adapt L1, S-Wanda, and gradient criteria; and propose DRRQR, a selector based on Strong Rank-Revealing QR with provable conditioning bounds. Experiments on DeltaNet/GDN (370M–1.3B), Mamba-2 (370M, 7B), and hybrid Qwen3.5-9B show ~50% of key/query channels can be removed with modest degradation after recovery fine-tuning; decode speedups reach ~1.4× at large batch.
Critical strengths
-
Clean, self-contained theory. The rank bounds, the effective-rank lower bound via key conditioning, and the readout-noise amplification result directly motivate the method, and the invariance analysis honestly explains the axis-aligned restriction.
-
Extensive experiments. Four model families (370M–9B), compression ratios 30–75%, pre-/post-RFT comparisons, calibration-robustness and per-head analyses, and kernel-level vs. end-to-end efficiency breakdowns (Tables 17–19).
-
Useful diagnostic. The rank-utilization analysis documents that (Gated) DeltaNet models under-use their recurrent capacity.
Critical weaknesses
W1 (main): Storage cost of linear attention states is not a bottleneck in the hybrid architectures this paper targets, especially in long-context scenarios. For Qwen3.5-9B (24 GDN layers, 0.52M-scalar states each; 8 attention layers, 2,048 scalars/token/layer; native context 262K), one linear layer's full state equals the KV cache of only 256 tokens in one softmax layer; all 24 combined equal a 768-token context (~0.3% of native). At native context, KV (~8 GB/sequence) exceeds total linear state (24–48 MB) by orders of magnitude; halving every state saves <0.3% of activation memory. Qwen3.8-Flash-Next (Aug 2026) confirms: its speed comes from sparsifying its 12 softmax layers (12:1 GQA plus a sparse-attention indexer) while its 36 linear states stay full-size. The paper's own Table 4 shows hybrid recall is carried by the softmax layers (GDN-1.3B: 81.3→61.9 NIAH at 50% reduction; Qwen3.5-9B unchanged) — yet the pipeline compresses the linear side and freezes the softmax side, optimizing a non-bottleneck while leaving the bottleneck in place. No wall-clock or memory numbers are reported for Qwen3.5-9B, and the hybrid long-context evaluation stops at 4K — the only regime where linear-state traffic rivals KV traffic.
W2: The regime where the method's speedups materialize has no industrial footprint. All wall-clock gains are measured on purely recurrent 1.3B models (Tables 17–19), but no maintained production linear-attention LLM is purely recurrent: every deployed system — Qwen3.5 (4B–9B, down to edge scale), Qwen3.8-Flash-Next, Kimi Linear, Nemotron — is a hybrid, and even Mamba-3 (Mar 2026) reports its best results in hybrid configurations (one attention layer per five Mamba-3 layers). The sole purely recurrent lineage still trained at scale, RWKV-7 (community-driven), has minimal industrial adoption.
W3 (secondary): Generality is narrower than the title promises. The key ingredient, joint key/query orthogonal invariance, is specific to the DeltaNet-style rank-1 erase-write recurrence (and Mamba-2 via SSD duality); empirical support beyond the DeltaNet family is a single set of pre-RFT Mamba-2 results where DRRQR no longer beats the gradient baseline (App. C.3), and on Qwen3.5-9B only DRRQR vs. L1 is reported.
Questions for authors
-
Can you report end-to-end memory and latency for Qwen3.5-9B (baseline vs. 50% reduction) at representative contexts (ideally ≥32K), with a linear-state vs. KV-cache breakdown? If supplemented experiments demonstrate an effective end-to-end speedup in the hybrid setting, or otherwise refute the bottleneck analysis above, I would be willing to raise my rating to Accept.
-
Since state capacity is not a memory/speed pressure in hybrids, the more valuable direction is strengthening linear attention from my perspective: how would your theory transfer to state expansion (MoM, arXiv:2502.13685; SSE, arXiv:2507.16577, ICLR 2026)? Could rank utilization guide efficient expansion — where and how much capacity to add, and how to raise hidden-state utilization — and what would it mean for hybrid and purely recurrent models, respectively?
-
Section 5 states Grad is compared on all evaluated models including Qwen3.5-9B, yet Tables 12–13 report only DRRQR and L1 without explanation — despite Grad being the strongest selector on Mamba-Codestral (Table 3). Why the omission? Does DRRQR remain competitive with Grad included?
6 comments
I appreciate the authors for their contributions, including the additional experimental results and the preservation of most of the performance under state reduction, both of which are strengths of the model. However, as the authors mentioned in the rebuttal, when the context length reaches 32k, state reduction yields only a 2.3% GPU memory saving for the entire Qwen hybrid architecture. This suggests that state reduction may be effective only for state sizes within 16k/8k.
Therefore, I am more inclined to view this paper as a systematic analysis of the redundancy phenomenon in linear attention's memory capacity, rather than as a further breakthrough along a major research direction in linear attention.
Based on my experience in linear attention, I believe that the efficient expansion of the memory capacity of linear attention is becoming the mainstream direction, and has been increasingly emphasized by recent frontier works [1][2][3][4]. My reason for raising questions related to state expansion is to see whether the authors’ theory of state reduction can be analogically transferred to the setting of state expansion, and whether some general methods and philosophies can be extracted from this work, and be used to deeply analyze memory utilization efficiency, memory redundancy, and the gap between the mathematically effective capacity expansion and the physical capacity expansion under various forms of state expansion techniques. I think the "well-conditioned keys" mentioned in the authors' initial response is not detailed nor abundant, and suggest the author to take two representative strategies into consideration: One is represented by SSE[1] and MoM[2], which leverages a router to determine the activated memory partition for key-value updates, thus alleviating the collision and interference between different key-value associations. The other is represented by March[3] and HLA[4], which applies direct attention to state checkpoints at boundaries of previous sequential chunks, where the analysis strategy in DRRQR may need some modification, since they directly retrieve the earlier state checkpoints, rather than strike to recover old information from a single matrix memory. I think utilizing DRRQR's strategy to identify the redundancy of storing historical checkpoints and improve the corresponding efficiency is very interesting.
My other concerns have all been addressed. I understand that the rebuttal period is time-constrained, so I did not ask the authors to add analytical experiments related to state expansion. If the authors can provide a direct and in-depth theoretical response to my questions about state expansion, I believe this would make the paper’s phenomenon analysis more complete, and could provide meaningful insights, inspirations and guidance for frontier research directions including state expansion. If this remaining issue can be well addressed, I would be glad to raise my score.
[1] https://arxiv.org/abs/2507.16577
[2] https://arxiv.org/abs/2502.13685
[3] https://arxiv.org/abs/2608.12435
[4] https://arxiv.org/abs/2610.05842
We thank Reviewer SrWP for their thoughtful follow up and for acknowledging that we have addressed almost all of your concerns. We hope to resolve the remaining question about state expansion below, which we agree is an interesting and complementary research direction to state reduction. Consistent with the interpretation of our work of poor rank utilization, state expansion is a way of increasing the absolute available memory at low relative utilization.
SSE We thank the reviewer for pointing out this very interesting connection. In fact, SSE [1] already explicitly connects memory expansion to memory utilization: its Figure 5 reports higher singular value entropy than GLA and describes a "less compressible and more effectively utilized state composition". This closely connects to the question studied in our paper. We will make sure to include this connection in the final version of the submission.
Our framework could shed light on that observation. Our analysis in Section 2.2 suggests studying how routing changes the diversity of representations entering each memory.
SSE's row sparse formulation uses top- selection followed by softmax to build the keys indexing the linear associative memory. We now study the top-1 case. Let and be the number of writes to coordinate up to time T, then . If every coordinate is visited, , so equal usage gives perfectly conditioned keys, supporting better memory utilization.
SSE's state expanded instantiation selects top- sub memories. Its load balancing loss encourages uniform memory usage. We again investigate a special case theoretically. For top-1 routing among routed memories, excluding the always selected shared memory partition, let denote the selected partition. At time , let be the within partition softmax key after applying its write gate and decay. The expanded key places in block and zeros elsewhere. With and , the full expanded state satisfies
\widetilde{S}&=[S^{(1)}\\ \cdots\\ S^{(N)}]=V\widetilde{K}^\top. \end{aligned}$$ Let $d_{\mathrm{exp}}=\min(d_v,Nd_k)$ and $P_{\widetilde{K}}$ project onto the row space of $\widetilde{K}$, and let $G_m=\sum_{t:\\,m_t=m}\bar{k}_t\bar{k}_t^\top$. Applying our bound in Section 2.2 to the full expanded state gives $$u(\widetilde{S})\geq \frac{\operatorname{er}(VP_{\widetilde{K}})}{d_{\mathrm{exp}}} \frac{\min_m\lambda_{\min}(G_m)}{\max_m\lambda_{\max}(G_m)}.$$ The minimum and maximum compare the weakest effective key direction across all partitions with the strongest, potentially in a different partition. The ratio therefore captures imbalance both within and between partitions. The load balancer targets one possible source of the above mentioned imbalance by encouraging similar numbers of writes to each submemory. To quantify its role, let $n_m>0$ denote the number of writes to partition $m$ and $C_m=G_m/n_m$. Suppose there is a matrix $C\succ0$ and $0\leq\varepsilon<1$ such that $$(1-\varepsilon)C\preceq C_m\preceq(1+\varepsilon)C.$$ (Here, $A\preceq B$ means $v^\top Av\leq v^\top Bv$ for every vector $v$.) Then $$u(\widetilde{S})\geq \frac{\operatorname{er}(VP_{\widetilde{K}})}{d_{\mathrm{exp}}} \frac{n_{\min}}{n_{\max}} \frac{1-\varepsilon}{1+\varepsilon} \frac{1}{\kappa(C)}.$$ This bound connects load balancing directly to utilization of the expanded state. In the above limit case, a balanced router improves the ratio $n_{\min}/n_{\max}$ and thus strengthens the memory utilization bound. In summary, increasing state size adds theoretical capacity. Our above analysis identifies how key geometry and balanced writes support the effective utilization of that capacity. **MoM** Similar to the SSE's state expanded formulation, MoM [2] maintains a mixture of states (memories), with a router assigning tokens to memories. We provide another, slightly different perspective on utilization of routed memories. For a routed memory $m$, let $K_m$ contain its assigned keys and $U_m$ the corresponding effective write vectors (either plain linear attention or decayed, delta corrected writes), giving $S_m=U_mK_m^\top$. With $P_{K_m}$ denoting the orthogonal projector onto the row space of $K_m$, our bound in Section 2.2 gives $$u(S_m)\geq \frac{\operatorname{er}(U_mP_{K_m})} {d_m\\,\kappa(K_m)^2}, \qquad d_m=\min(d_v^{(m)},d_k^{(m)}).$$ From the perspective of our paper, this could suggest considering key alignment during routing to promote diverse keys within each memory and support higher rank utilization.
We thank Reviewer SrWP for their thoughtful review. We appreciate their comments highlighting our “Clean, self-contained theory” and “Extensive experiments”, and hope to address the remaining questions below.
We have uploaded a revised version of the paper, with all changes highlighted in blue. All references to pages, sections, figures, tables, and equations use the revised version's numbering. Citations in square brackets refer to this response.
Positioning
We want to start by clarifying the positioning of our paper. One of its central contributions is to shed light on the low-rank structure of pretrained linear recurrences. We explore pruning as one direct application of this analysis as our second contribution. The latter connects to the research communities interest in post-training methods for better efficiency of linear attention hybrids. Concurrent state-quantization work, including DAMP [1], STEPQuant [2], LeapQuant [3], and QReS [4] also targets recurrent states in hybrid models. However, they quantize the state, while we analyze its recurrent capacity from the perspective of effective rank. Our work is therefore complementary to that branch of research.
W1: Long Context
The table below reports Qwen3.5-9B retrieval accuracy (%) at longer contexts under 50% DRRQR state reduction. It shows that, even at long context, the softmax layers seem to maintain a long-range path.
W1/W2/Q1: Efficiency
We acknowledge the reviewer's concerns about the relevance of state reduction in linear attention layers, especially for hybrid models. Before providing explicit data, we again want to highlight the research community's interest in adjacent problems. In particular, QReS [4] (Figure 6) motivates state quantization in hybrids by showing that state- and weight quantization allow for consistent speedups across a range of batch sizes. The benefit of either state- or weight-quantization alone would be smaller for small or large batch-sizes.
We have updated Figure 4 in the revised paper with end-to-end decode measurements for Qwen3.5-9B and Nemotron-H-8B. State reduction yields speedups with both BF16 and FP8 KV caches up to 32K, with the largest gains at larger batch sizes and lower KV-cache precision. The table below reports Qwen's per-sequence cache memory at 50% state reduction with a BF16 KV cache. Memory saved is relative to the combined baseline state and KV cache.
Compressing the KV cache increases the relative memory savings from state reduction. A natural follow-up is the reviewer's Q2, which we address below.
W3/Q3: Generality.
The rotational invariance holds for linear recurrences with scalar or no decay, as well as those with and without delta-correction. We have generalized Proposition 3.1 accordingly in the revised submission. We thank the reviewer for pointing out that Grad was missing in Table 13. The table below provides the Qwen3.5-9B results before RFT.
DRRQR leads on Qwen in the selector comparison above, while Grad leads on Mamba-Codestral (Table 9). We have added the missing results to Table 13.
Q2: State expansion.
Our analysis diagnoses underutilization, and pruning tests whether capacity can be removed. We agree that the opposite approach is also a promising avenue for future work. Both state expansion and increased state utilization are promising directions for optimizing linear recurrences. Beyond MoM and SSE, which the reviewer mentioned, our analysis points to well-conditioned keys for increased memory utilization in linear recurrences. Well-conditioned keys allow for more directed indexing into the linear associative memory.
We again thank the reviewer for their thoughtful review. We hope that positioning our paper within concurrent work on hybrid efficiency, reflecting the research community's interest, together with the additional empirical evidence above, addresses the reviewer's concerns.
References
[1] Anonymous. DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization. ICLR 2027.
[2] Anonymous. STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization. ICLR 2027.
[3] Anonymous. LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization. ICLR 2027.
[4] Anonymous. QReS: Quantizing Recurrent Language Models Is Easier Than You Think. ICLR 2027.