acceptodds
Under review as a conference paper at ICLR 2027

On State Reduction in Linear Attention

14 views

Abstract

Linear attention offers a computationally efficient yet expressive alternative to softmax attention. However, recent empirical results indicate that the hidden state of trained linear attention models often exhibits a low-rank structure, suggesting that these models underexploit their capacity in practice. To understand this phenomenon, we analyze how the keys and values shape the rank of the recurrent state, providing a theoretical perspective on memory utilization in linear attention. In addition to these theoretical insights, we conjecture that the low-rank states can be substantially reduced post-training with only minimal performance degradation. To this end, we propose a hardware-aware approach that structurally prunes key and query matrices, reducing the state size while retaining compatibility with existing kernels. We adapt several existing pruning strategies to fit our framework and, building on our theoretical analysis, propose a robust structured pruning method based on a rank-revealing QR decomposition. Our empirical evaluations across models of varying sizes and on various downstream tasks, reveal that the state of pretrained linear attention models can be halved with only a minor decrease of next token prediction capabilities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language modeling, linear attention, pruning, associative memory

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_SrWPOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies post-training reduction of the matrix-valued hidden state in linear attention. It links the state's rank to keys/values and low utilization to query-noise amplification at readout. The authors exploit the joint key/query orthogonal invariance of (Gated) DeltaNet to build a structured, axis-aligned channel-pruning framework compatible with causal convolutions and existing kernels; adapt L1, S-Wanda, and gradient criteria; and propose DRRQR, a selector based on Strong Rank-Revealing QR with provable conditioning bounds. Experiments on DeltaNet/GDN (370M–1.3B), Mamba-2 (370M, 7B), and hybrid Qwen3.5-9B show ~50% of key/query channels can be removed with modest degradation after recovery fine-tuning; decode speedups reach ~1.4× at large batch.

Critical strengths
  1. Clean, self-contained theory. The rank bounds, the effective-rank lower bound via key conditioning, and the readout-noise amplification result directly motivate the method, and the invariance analysis honestly explains the axis-aligned restriction.

  2. Extensive experiments. Four model families (370M–9B), compression ratios 30–75%, pre-/post-RFT comparisons, calibration-robustness and per-head analyses, and kernel-level vs. end-to-end efficiency breakdowns (Tables 17–19).

  3. Useful diagnostic. The rank-utilization analysis documents that (Gated) DeltaNet models under-use their recurrent capacity.

Critical weaknesses

W1 (main): Storage cost of linear attention states is not a bottleneck in the hybrid architectures this paper targets, especially in long-context scenarios. For Qwen3.5-9B (24 GDN layers, 0.52M-scalar states each; 8 attention layers, 2,048 scalars/token/layer; native context 262K), one linear layer's full state equals the KV cache of only 256 tokens in one softmax layer; all 24 combined equal a 768-token context (~0.3% of native). At native context, KV (~8 GB/sequence) exceeds total linear state (24–48 MB) by orders of magnitude; halving every state saves <0.3% of activation memory. Qwen3.8-Flash-Next (Aug 2026) confirms: its speed comes from sparsifying its 12 softmax layers (12:1 GQA plus a sparse-attention indexer) while its 36 linear states stay full-size. The paper's own Table 4 shows hybrid recall is carried by the softmax layers (GDN-1.3B: 81.3→61.9 NIAH at 50% reduction; Qwen3.5-9B unchanged) — yet the pipeline compresses the linear side and freezes the softmax side, optimizing a non-bottleneck while leaving the bottleneck in place. No wall-clock or memory numbers are reported for Qwen3.5-9B, and the hybrid long-context evaluation stops at 4K — the only regime where linear-state traffic rivals KV traffic.

W2: The regime where the method's speedups materialize has no industrial footprint. All wall-clock gains are measured on purely recurrent 1.3B models (Tables 17–19), but no maintained production linear-attention LLM is purely recurrent: every deployed system — Qwen3.5 (4B–9B, down to edge scale), Qwen3.8-Flash-Next, Kimi Linear, Nemotron — is a hybrid, and even Mamba-3 (Mar 2026) reports its best results in hybrid configurations (one attention layer per five Mamba-3 layers). The sole purely recurrent lineage still trained at scale, RWKV-7 (community-driven), has minimal industrial adoption.

W3 (secondary): Generality is narrower than the title promises. The key ingredient, joint key/query orthogonal invariance, is specific to the DeltaNet-style rank-1 erase-write recurrence (and Mamba-2 via SSD duality); empirical support beyond the DeltaNet family is a single set of pre-RFT Mamba-2 results where DRRQR no longer beats the gradient baseline (App. C.3), and on Qwen3.5-9B only DRRQR vs. L1 is reported.

Questions for authors
  1. Can you report end-to-end memory and latency for Qwen3.5-9B (baseline vs. 50% reduction) at representative contexts (ideally ≥32K), with a linear-state vs. KV-cache breakdown? If supplemented experiments demonstrate an effective end-to-end speedup in the hybrid setting, or otherwise refute the bottleneck analysis above, I would be willing to raise my rating to Accept.

  2. Since state capacity is not a memory/speed pressure in hybrids, the more valuable direction is strengthening linear attention from my perspective: how would your theory transfer to state expansion (MoM, arXiv:2502.13685; SSE, arXiv:2507.16577, ICLR 2026)? Could rank utilization guide efficient expansion — where and how much capacity to add, and how to raise hidden-state utilization — and what would it mean for hybrid and purely recurrent models, respectively?

  3. Section 5 states Grad is compared on all evaluated models including Qwen3.5-9B, yet Tables 12–13 report only DRRQR and L1 without explanation — despite Grad being the strongest selector on Mamba-Codestral (Table 3). Why the omission? Does DRRQR remain competitive with Grad included?

6 comments

We thank Reviewer SrWP for their thoughtful review. We appreciate their comments highlighting our “Clean, self-contained theory” and “Extensive experiments”, and hope to address the remaining questions below.

We have uploaded a revised version of the paper, with all changes highlighted in blue. All references to pages, sections, figures, tables, and equations use the revised version's numbering. Citations in square brackets refer to this response.

Positioning

We want to start by clarifying the positioning of our paper. One of its central contributions is to shed light on the low-rank structure of pretrained linear recurrences. We explore pruning as one direct application of this analysis as our second contribution. The latter connects to the research communities interest in post-training methods for better efficiency of linear attention hybrids. Concurrent state-quantization work, including DAMP [1], STEPQuant [2], LeapQuant [3], and QReS [4] also targets recurrent states in hybrid models. However, they quantize the state, while we analyze its recurrent capacity from the perspective of effective rank. Our work is therefore complementary to that branch of research.

W1: Long Context

The table below reports Qwen3.5-9B retrieval accuracy (%) at longer contexts under 50% DRRQR state reduction. It shows that, even at long context, the softmax layers seem to maintain a long-range path.

ContextRULER BaselineRULER 50%NIAH BaselineNIAH 50%
16K91.090.6100.099.9
32K91.491.0100.0100.0

W1/W2/Q1: Efficiency

We acknowledge the reviewer's concerns about the relevance of state reduction in linear attention layers, especially for hybrid models. Before providing explicit data, we again want to highlight the research community's interest in adjacent problems. In particular, QReS [4] (Figure 6) motivates state quantization in hybrids by showing that state- and weight quantization allow for consistent speedups across a range of batch sizes. The benefit of either state- or weight-quantization alone would be smaller for small or large batch-sizes.

We have updated Figure 4 in the revised paper with end-to-end decode measurements for Qwen3.5-9B and Nemotron-H-8B. State reduction yields speedups with both BF16 and FP8 KV caches up to 32K, with the largest gains at larger batch sizes and lower KV-cache precision. The table below reports Qwen's per-sequence cache memory at 50% state reduction with a BF16 KV cache. Memory saved is relative to the combined baseline state and KV cache.

Context2K4K8K32K
State, baseline (MiB)49.149.149.149.1
State, reduced (MiB)24.824.824.824.8
KV (MiB)641282561024
Memory saved21.5%13.7%8.0%2.3%

Compressing the KV cache increases the relative memory savings from state reduction. A natural follow-up is the reviewer's Q2, which we address below.

W3/Q3: Generality.

The rotational invariance holds for linear recurrences with scalar or no decay, as well as those with and without delta-correction. We have generalized Proposition 3.1 accordingly in the revised submission. We thank the reviewer for pointing out that Grad was missing in Table 13. The table below provides the Qwen3.5-9B results before RFT.

ReductionSelectorWiki ppl ↓\downarrowLAMBADA ppl ↓\downarrowZero-shot (%) ↑\uparrow
0%Baseline9.03.873.3
25%DRRQR9.74.373.0
25%Grad11.55.269.4
50%DRRQR11.65.771.4
50%Grad26.527.158.0

DRRQR leads on Qwen in the selector comparison above, while Grad leads on Mamba-Codestral (Table 9). We have added the missing results to Table 13.

Q2: State expansion.

Our analysis diagnoses underutilization, and pruning tests whether capacity can be removed. We agree that the opposite approach is also a promising avenue for future work. Both state expansion and increased state utilization are promising directions for optimizing linear recurrences. Beyond MoM and SSE, which the reviewer mentioned, our analysis points to well-conditioned keys for increased memory utilization in linear recurrences. Well-conditioned keys allow for more directed indexing into the linear associative memory.

We again thank the reviewer for their thoughtful review. We hope that positioning our paper within concurrent work on hybrid efficiency, reflecting the research community's interest, together with the additional empirical evidence above, addresses the reviewer's concerns.

References

[1] Anonymous. DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization. ICLR 2027.

[2] Anonymous. STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization. ICLR 2027.

[3] Anonymous. LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization. ICLR 2027.

[4] Anonymous. QReS: Quantizing Recurrent Language Models Is Easier Than You Think. ICLR 2027.

I appreciate the authors for their contributions, including the additional experimental results and the preservation of most of the performance under state reduction, both of which are strengths of the model. However, as the authors mentioned in the rebuttal, when the context length reaches 32k, state reduction yields only a 2.3% GPU memory saving for the entire Qwen hybrid architecture. This suggests that state reduction may be effective only for state sizes within 16k/8k.

Therefore, I am more inclined to view this paper as a systematic analysis of the redundancy phenomenon in linear attention's memory capacity, rather than as a further breakthrough along a major research direction in linear attention.

Based on my experience in linear attention, I believe that the efficient expansion of the memory capacity of linear attention is becoming the mainstream direction, and has been increasingly emphasized by recent frontier works [1][2][3][4]. My reason for raising questions related to state expansion is to see whether the authors’ theory of state reduction can be analogically transferred to the setting of state expansion, and whether some general methods and philosophies can be extracted from this work, and be used to deeply analyze memory utilization efficiency, memory redundancy, and the gap between the mathematically effective capacity expansion and the physical capacity expansion under various forms of state expansion techniques. I think the "well-conditioned keys" mentioned in the authors' initial response is not detailed nor abundant, and suggest the author to take two representative strategies into consideration: One is represented by SSE[1] and MoM[2], which leverages a router to determine the activated memory partition for key-value updates, thus alleviating the collision and interference between different key-value associations. The other is represented by March[3] and HLA[4], which applies direct attention to state checkpoints at boundaries of previous sequential chunks, where the analysis strategy in DRRQR may need some modification, since they directly retrieve the earlier state checkpoints, rather than strike to recover old information from a single matrix memory. I think utilizing DRRQR's strategy to identify the redundancy of storing historical checkpoints and improve the corresponding efficiency is very interesting.

My other concerns have all been addressed. I understand that the rebuttal period is time-constrained, so I did not ask the authors to add analytical experiments related to state expansion. If the authors can provide a direct and in-depth theoretical response to my questions about state expansion, I believe this would make the paper’s phenomenon analysis more complete, and could provide meaningful insights, inspirations and guidance for frontier research directions including state expansion. If this remaining issue can be well addressed, I would be glad to raise my score.

[1] https://arxiv.org/abs/2507.16577
[2] https://arxiv.org/abs/2502.13685
[3] https://arxiv.org/abs/2608.12435
[4] https://arxiv.org/abs/2610.05842

We thank Reviewer SrWP for their thoughtful follow up and for acknowledging that we have addressed almost all of your concerns. We hope to resolve the remaining question about state expansion below, which we agree is an interesting and complementary research direction to state reduction. Consistent with the interpretation of our work of poor rank utilization, state expansion is a way of increasing the absolute available memory at low relative utilization.

SSE We thank the reviewer for pointing out this very interesting connection. In fact, SSE [1] already explicitly connects memory expansion to memory utilization: its Figure 5 reports higher singular value entropy than GLA and describes a "less compressible and more effectively utilized state composition". This closely connects to the question studied in our paper. We will make sure to include this connection in the final version of the submission.

Our framework could shed light on that observation. Our analysis in Section 2.2 suggests studying how routing changes the diversity of representations entering each memory.

SSE's row sparse formulation uses top-kk selection followed by softmax to build the keys indexing the linear associative memory. We now study the top-1 case. Let K=[k1,…,kT]K=[k_1,\ldots,k_T] and nin_i be the number of writes to coordinate ii up to time T, then KK⊤=diag⁡(n1,…,ndk)KK^\top=\operatorname{diag}(n_1,\ldots,n_{d_k}). If every coordinate is visited, κ(K)=nmax⁡/nmin⁡\kappa(K)=\sqrt{n_{\max}/n_{\min}}, so equal usage gives perfectly conditioned keys, supporting better memory utilization.

SSE's state expanded instantiation selects top-kk sub memories. Its load balancing loss encourages uniform memory usage. We again investigate a special case theoretically. For top-1 routing among NN routed memories, excluding the always selected shared memory partition, let mtm_t denote the selected partition. At time TT, let kˉt∈Rdk\bar{k}_t\in\mathbb{R}^{d_k} be the within partition softmax key after applying its write gate and decay. The expanded key k~t∈RNdk\widetilde{k}_t\in\mathbb{R}^{Nd_k} places kˉt\bar{k}_t in block mtm_t and zeros elsewhere. With K~=[k~1,…,k~T]\widetilde{K}=[\widetilde{k}_1,\ldots,\widetilde{k}_T] and V=[v1,…,vT]V=[v_1,\ldots,v_T], the full expanded state satisfies

\widetilde{S}&=[S^{(1)}\\ \cdots\\ S^{(N)}]=V\widetilde{K}^\top. \end{aligned}$$ Let $d_{\mathrm{exp}}=\min(d_v,Nd_k)$ and $P_{\widetilde{K}}$ project onto the row space of $\widetilde{K}$, and let $G_m=\sum_{t:\\,m_t=m}\bar{k}_t\bar{k}_t^\top$. Applying our bound in Section 2.2 to the full expanded state gives $$u(\widetilde{S})\geq \frac{\operatorname{er}(VP_{\widetilde{K}})}{d_{\mathrm{exp}}} \frac{\min_m\lambda_{\min}(G_m)}{\max_m\lambda_{\max}(G_m)}.$$ The minimum and maximum compare the weakest effective key direction across all partitions with the strongest, potentially in a different partition. The ratio therefore captures imbalance both within and between partitions. The load balancer targets one possible source of the above mentioned imbalance by encouraging similar numbers of writes to each submemory. To quantify its role, let $n_m>0$ denote the number of writes to partition $m$ and $C_m=G_m/n_m$. Suppose there is a matrix $C\succ0$ and $0\leq\varepsilon<1$ such that $$(1-\varepsilon)C\preceq C_m\preceq(1+\varepsilon)C.$$ (Here, $A\preceq B$ means $v^\top Av\leq v^\top Bv$ for every vector $v$.) Then $$u(\widetilde{S})\geq \frac{\operatorname{er}(VP_{\widetilde{K}})}{d_{\mathrm{exp}}} \frac{n_{\min}}{n_{\max}} \frac{1-\varepsilon}{1+\varepsilon} \frac{1}{\kappa(C)}.$$ This bound connects load balancing directly to utilization of the expanded state. In the above limit case, a balanced router improves the ratio $n_{\min}/n_{\max}$ and thus strengthens the memory utilization bound. In summary, increasing state size adds theoretical capacity. Our above analysis identifies how key geometry and balanced writes support the effective utilization of that capacity. **MoM** Similar to the SSE's state expanded formulation, MoM [2] maintains a mixture of states (memories), with a router assigning tokens to memories. We provide another, slightly different perspective on utilization of routed memories. For a routed memory $m$, let $K_m$ contain its assigned keys and $U_m$ the corresponding effective write vectors (either plain linear attention or decayed, delta corrected writes), giving $S_m=U_mK_m^\top$. With $P_{K_m}$ denoting the orthogonal projector onto the row space of $K_m$, our bound in Section 2.2 gives $$u(S_m)\geq \frac{\operatorname{er}(U_mP_{K_m})} {d_m\\,\kappa(K_m)^2}, \qquad d_m=\min(d_v^{(m)},d_k^{(m)}).$$ From the perspective of our paper, this could suggest considering key alignment during routing to promote diverse keys within each memory and support higher rank utilization.
Reviewer_GHahOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies redundancy in the recurrent states of pretrained linear attention models. It relates state rank to the keys and values and analyzes how an uneven state spectrum can amplify query perturbations. The proposed compression framework jointly prunes query and key channels, reducing the state dimension while preserving compatibility with depthwise convolutions and existing kernels. Within this framework, DRRQR uses strong rank-revealing QR to select channels. Experiments across DeltaNet, Gated DeltaNet, Mamba-2, and a hybrid model demonstrate substantial state reduction and inference speedups, with task-dependent accuracy losses and recovery fine-tuning where applicable.

Critical strengths

The compression targets a useful practical bottleneck. Joint Q/K channel pruning directly reduces recurrent-state storage and update costs, and the paper demonstrates end-to-end speedups using existing kernels. The results showing that state size can be halved with relatively modest changes in language-modeling performance, particularly after recovery fine-tuning, make this a worthwhile direction.

The evaluation goes beyond perplexity to examine calibration robustness and retrieval. DRRQR is competitive with gradient-based selection and is more stable across the tested calibration corpora. The retrieval experiments also reveal an important qualification: similar language-modeling performance does not imply preserved recall, and the pure recurrent and hybrid models respond differently to state reduction.

Critical weaknesses

The methodological novelty needs clearer positioning. QR-based neuron and channel selection has been studied in Efficient Inference of CNNs via Channel Pruning, Model Preserving Compression for Neural Networks, and LinDeps. Algorithm 2 also explicitly adapts a classical strong RRQR algorithm. The paper should explain what the linear-attention formulation adds beyond these existing selection methods. A comparison with ordinary QR with column pivoting would help establish whether the additional refinement in strong RRQR matters in practice.

The pruning comparisons mainly evaluate generic selection criteria within the proposed framework. They do not establish the tradeoff against other pruning approaches for linear attention, such as AMAP or the head-level method in State Rank Dynamics in Linear Attention LLMs.

Questions for authors

Check above.

3 comments

We thank Reviewer GHah for their thoughtful review. We appreciate their assessment that our work targets “a useful practical bottleneck” and represents “a worthwhile direction”, and hope to address the remaining questions below.

We have uploaded a revised version of the paper, with all changes highlighted in blue. All references to pages, sections, figures, tables, and equations use the revised version's numbering. Citations in square brackets refer to this response.

Positioning

We thank the reviewer for pointing out these papers and want to clarify the positioning of our paper. Our paper first analyzes how the inputs of linear recurrences shape state rank (Equation 4) and how rank utilization affects readout sensitivity (Proposition 2.1). It then develops a state-reduction framework that jointly reduces key and query channels while preserving depthwise convolutions and existing kernels. DRRQR instantiates channel selection within this framework using strong RRQR.

We appreciate that QR-based channel selection has been used in the CNN literature [1, 2, 3], and we have included these works in the revised related work section.

LinDeps, for example, uses pivoted QR to prune dependent CNN filters and updates the next layer to compensate for their removal. In comparison, the goal of our pruning is to reduce the linear attention recurrent state during language-model inference.

W1: Column pivoting and swaps

We thank the reviewer for this question. DRRQR verifies the strong rank-revealing condition and refines the column selection only when this condition is not satisfied. In the evaluated settings on real data, we find that the QR with column pivoting (QRCP) initialization already satisfied this condition, so QRCP and DRRQR selected the same channels. Satisfying the condition establishes the singular-value bounds on the selected calibration columns (Appendix B.2). We have clarified in the revised version that the swap step serves to enforce these guarantees when the initialization does not satisfy the condition.

W2: Other pruning approaches

We appreciate the reviewer's questions about comparisons to other pruning methods and address them below. State Rank Dynamics [4] studies rank and norm stability across heads and proposes Joint Rank-Norm Pruning (JRNP) to remove entire heads. As the authors did not publish code, we implemented the method ourselves for our setting, using its scoring rule based on state rank and nuclear norm. The table below compares DRRQR with JRNP before RFT at matched recurrent-state budgets, using uniform per-layer head removal and α=0.5\alpha=0.5 for JRNP.

ModelState removedDRRQR WikiText-2 ppl ↓\downarrowDRRQR Average zero-shot (%) ↑\uparrowJRNP WikiText-2 ppl ↓\downarrowJRNP Average zero-shot (%) ↑\uparrow
DeltaNet 370M50%31.444.71642.029.6
DeltaNet 370M75%42.043.14889.129.9
DeltaNet 1.3B50%20.545.7423.635.6
DeltaNet 1.3B75%43.941.835,45130.7
Gated DeltaNet 370M50%31.644.0147.233.7
Gated DeltaNet 370M75%45.942.3527.130.6
Gated DeltaNet 1.3B50%17.357.5109.235.6
Gated DeltaNet 1.3B75%22.554.0385.631.5

DRRQR gives lower perplexity and higher average zero-shot accuracy throughout the head-pruning comparison above. Removing entire heads also removes their value/output channels, while our approach reduces the state-space dimension and preserves the output dimension.

AMAP [5] learns pruning masks jointly with vision transformers, including models with linear attention. Our setting is post-training reduction of recurrent language-model states.

We have incorporated the related-work discussion and QRCP clarification into the revised manuscript. We again thank the reviewer for their thoughtful review and hope to have addressed their remaining questions.

References

[1] Zhang et al. Efficient Inference of CNNs via Channel Pruning.

[2] Chee et al. Model Preserving Compression for Neural Networks.

[3] Henry et al. LinDeps: A Fine-tuning Free Post-Pruning Method to Remove Layer-Wise Linear Dependencies with Guaranteed Performance Preservation.

[4] Sun et al. State Rank Dynamics in Linear Attention LLMs.

[5] Lee and Hwang. AMAP: Automatic Multihead Attention Pruning by Similarity-Based Pruning Indicator.

Thank you for the additional JRNP comparison and the clarification that QRCP and DRRQR selected the same channels in the evaluated settings. These address my main concerns, and I maintain my weak accept recommendation.

We thank you for the positive assessment and your insightful comments, which have helped further strengthen our submission. We are glad our clarifications have addressed your concerns.

Reviewer_NGwCOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides a theoretical perspective on inefficient memory utilization in linear attention models, attributing it to the low effective rank of their recurrent states. From this observation, the authors propose a structured pruning framework that jointly reduces key and query dimensions, adapting existing pruning methods and introducing a rank-revealing QR-based approach. Experiments across various recurrent and hybrid language models demonstrate substantial state reduction with relatively modest performance degradation and improved inference efficiency.

Critical strengths

This paper proposes an interesting theoretical perspective on memory utilization in linear attention models. From their theoretical analysis, the proposed structured pruning framework is practical, as it consider the architectural constraints of linear attention and compatibility with existing kernels. The authors also conduct extensive experiments across different model scales and architectures, demonstrating the practical effectiveness of state reduction in terms of both model performance and inference efficiency.

Critical weaknesses

The theoretical analysis provides useful insights into rank utilization, but i think the connection to structured pruning remains unclear. In particular, low effective rank does not imply that the state can be safely compressed through axis-aligned channel selection.

Broader evaluations across different types or categrioes are needed to know which type of information is dominantly lost from compression. For example, the long-context results show notable performance degradation in recurrent models. Although hybrid architectures may compensate for this limitation, a broader evaluation would help identify what types of information are most affected by state reduction.

The paper reports inference speedups, but the computational costs of channel selection and recovery fine-tuning are not considered in the overall efficiency trade-off.

Questions for authors
  1. Please further explain how low effective rank justifies axis-aligned channel pruning. How can low effective rank imply that the state can be safely compressed through axis-aligned channel selection? Is there any relationshp between rank utilization and performance degradation after compression?

  2. This can be slightly extended question from this paper, but could the authors evaluate a broader range of tasks to better understand what types of information are lost during state reduction, beyond the existing long-context evaluations?

  3. Could the authors report the computational costs of channel selection and recovery fine-tuning, and discuss when the inference speedups outweigh these costs?

3 comments

We thank Reviewer NGwC for their thoughtful review. We appreciate their recognition of our “interesting theoretical perspective” and “extensive experiments”, and hope to address the remaining questions below.

We have uploaded a revised version of the paper, with all changes highlighted in blue. All references to pages, sections, figures, tables, and equations use the revised version's numbering. Citations in square brackets refer to this response.

Q1: Effective Rank

We thank the reviewer for the questions and want to clarify the connection between low-rank structure and state pruning. We hypothesize that the observed low-rank structure permits a reduction in state dimension with limited degradation in predictive performance. Consistent with this hypothesis, halving the state of GDN 1.3B using Grad results in only modest increases in perplexity before RFT (Table 24).

When key activations concentrate in a low-dimensional subspace, retaining their dominant principal components yields an optimal low-rank approximation, with reconstruction error determined by the discarded spectral energy. The Eckart–Young–Mirsky theorem provides a guarantee for low-rank approximation of the key activation matrix (Section 3.2). However, the corresponding projection generally mixes channels and cannot be absorbed into the model while preserving its depthwise convolution structure (Appendix B.3). We therefore restrict our study to selecting the same key and query channels, preserving the convolutions and existing kernels.

Q1: Rank Utilization

The quality of this selection also depends on the retained energy besides just pure rank utilization. The table below reports per-head medians of the joint key/query spectra for DeltaNet 370M at 75% reduction before RFT: DRRQR and Grad largely retain the dominant singular value and have much lower perplexity, while S-Wanda and L1 obtain smaller condition numbers by discarding high-energy directions. In particular, high rank utilization alone is not sufficient for good performance.

Selectorσmin⁡\sigma_{\min}σmax⁡\sigma_{\max}κ\kappaWikiText-2 ppl ↓\downarrow
Baseline (unpruned)0.1049.452929.8
DRRQR0.9649.46542.0
Grad0.6248.58946.7
S-Wanda0.6522.54233,512
L10.6415.332100,083
Random0.1613.3116136,955

Q2: Affected capabilities

The table below reports task accuracy (%) at 50% DRRQR state reduction across reading comprehension, science QA, and causal reasoning. Pre/Post denote before/after RFT. On Qwen, the largest losses before RFT occur in context-dependent prediction (LAMBADA) and passage-based QA (BoolQ). RFT recovers most of the LAMBADA loss, while the BoolQ loss persists.

TaskDN 1.3B BaselineDN 1.3B PostGDN 1.3B BaselineGDN 1.3B PostQwen3.5-9B BaselineQwen3.5-9B PreQwen3.5-9B Post
LAMBADA48.448.251.547.471.464.169.5
BoolQ61.060.261.762.586.382.281.6
RACE32.231.536.535.640.940.241.4
OpenBookQA32.832.641.042.044.643.644.0
SciQ76.773.188.787.795.195.595.7
COPA75.073.078.078.091.089.091.0

The purely recurrent model shows a larger recall loss (Table 11). The table below reports Qwen3.5-9B retrieval accuracy (%) at 50% state reduction, extending Table 12's protocol to 32K. Performance remains close to baseline. This again highlights that hybrids seem to maintain a long-context path in their softmax layers. We consider the study of what different layers in a hybrid learn an interesting venue for future work.

TaskContextBaselineReduced
RULER average16K91.090.6
RULER average32K91.491.0
Multikey-116K98.494.0

Q3: Cost

We thank the reviewer for the question. Pruning and RFT incur a one-time preparation cost that enables more efficient inference. GPTQ [1] follows the same principle, spending compute on layer-wise weight compensation before deployment to enable faster inference. For GDN 1.3B at 50% state reduction, loading, selecting, and saving the model takes 5.6 minutes with DRRQR and 6.2 minutes with Grad on one H100. RFT at 50% DRRQR state reduction takes 62 minutes on one GPU.

We again thank the reviewer for their thoughtful review and hope to have addressed their questions.

References

[1] Frantar et al. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

Thank you for the insightful and candid response. My concerns have been addressed, and I have decided to maintain my current score.

Dear reviewer, we thank you for the positive assessment and your insightful comments, which have helped further strengthen our submission. We are glad our clarifications and experiments have addressed your remaining concerns.

Comments on the paper

1 comment

We would like to thank all three reviewers for their thorough reading of our paper and for recognizing our “Clean, self-contained theory” (SrWP) and “interesting theoretical perspective on memory utilization in linear attention models” (NGwC), and for describing state reduction as “a worthwhile direction” (GHah). We are also glad that the reviewers appreciated the practical relevance of our framework, noting that it targets “a useful practical bottleneck” (GHah) and considers “the architectural constraints of linear attention and compatibility with existing kernels” (NGwC), as well as our “Extensive experiments” (SrWP),

The constructive suggestions allowed us to improve the paper, and we hope that the revisions satisfactorily address the reviewers' concerns and questions. We have uploaded a revised version of the paper incorporating the reviewers' suggestions. All changes are highlighted in blue. Please note that all references to sections, figures, tables, and equations refer to those in the revised version. We address each reviewer's specific points in the corresponding responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.