acceptodds
Under review as a conference paper at ICLR 2027

Multi-State Linear Attention with Generalized Second-Order Routing

Abstract

Self-attention plays a key role in long-sequence modeling, but its quadratic time complexity fundamentally limits the scaling of Large Language Models. Recently, Multi-State Linear Attention (MSLA) mechanisms have been proposed to make sequence modeling more efficient by splitting the context into segments and compressing each in its own memory state. MSLA aggregates the information extracted from each memory state using only the mean alignment between a query and each segment's keys, ignoring how the alignment is distributed within the segment. In order to model the relevance of a segment to a query more comprehensively and incorporate the variance of the alignment in the aggregation, we propose MERLIN (omnt outing for multi-state ear attention), a method that tracks the first and second moments of each segment's keys. Using the moments of keys, a query can recover both the mean and the variance of its alignment with each segment to perform a more content-aware aggregation of per-state readouts. Experiments on multi-state variants of Gated DeltaNet-1.3B and Mamba-2-780M show that MERLIN consistently outperforms existing aggregation methods on various long-context tasks. In particular, MERLIN outperforms baselines on RULER needle-in-a-haystack and long-context QA (+8.3% for Gated DeltaNet-1.3B and +7.0% for Mamba-2-780M) and diverse subtasks in LongBench (+13.5% and +12.0%). It even outperforms baselines that consume 4x more memory. MERLIN provides a simple and general routing mechanism for improving long-context capabilities of MSLA models. Code will be available upon acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Linear Attention, Recurrent Memory, Multi-State

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_q4NeOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MERLIN, a routing mechanism for multi-state linear attention that uses the mean and variance of query-key alignment to aggregate segment-level readouts. Experiments on Gated DeltaNet and Mamba-2 show improvements over position-based and mean-based aggregation.

Critical strengths

The method is straightforward and has a statistical motivation. Comparisons with Meanpool support the benefit of second-order routing on both backbones, with similar parameter counts.

Critical weaknesses

My main concern is that the experiments do not establish a competitive quality-efficiency trade-off.

  1. With fixed segment length m, the model caches L=T/m states and reads all visible states for every query. Consequently, prefill costs O(T^2(d^2+dr^2))/m), and persistent memory grows linearly with context length (Table 1). The evaluated architecture therefore loses the linear-time and constant-memory advantages of single-state GDN while retaining lossy compression within each segment.

Despite sharing these asymptotic scaling properties with full attention, the paper provides neither a matched full-attention quality baseline nor wall-clock prefill/decode comparisons against an optimized implementation. Comparisons with single-state GDN cannot establish resource efficiency: the default GDN variant uses approximately 72× its recurrent cache at 16K (Table 9).

The Meanpool results demonstrate a routing improvement, but do not establish why this growing bank of compressed states should be preferred over full attention. Without matched comparisons and measured quality-memory-latency trade-offs, I find the evidence insufficient for acceptance.

Questions for authors

see weakness

2 comments

We thank the reviewer for their constructive feedback and comments. We will address all the concerns raised by the reviewer and revise our paper before next Wednesday.

[W1] Quality-memory-latency trade-offs

Multi-state linear attention (MSLA) aims to improve the memory capacity of recurrent models like GDN and Mamba-2. Unlike standard transformers which store each token’s keys and values (KV) explicitly in a growing KV-cache, recurrent models compress the entire sequence into a fixed-size memory state, achieving greater memory efficiency. MSLA extends this approach by dividing the sequence into segments and compressing each segment’s KV into a separate fixed-size memory state. This allows them to achieve a balance between constant memory of linear attention and linear growing memory KV-cache of transformers [1].

To show this experimentally, we report the prefill throughput and peak memory of MERLIN and baselines below. For the attention baseline, we use the fla-hub/transformer-1.3B-100B model in huggingface and its optimized kernels. For each sequence length, we use a batch size of 1 and run it on a single L40 GPU. Table R3.1 shows that MERLIN’s throughput exceeds attention at about 16K, and the gap widens at 32K. Table R3.2 shows that MERLIN’s peak memory above weights stays below attention across all sequence lengths, with the gap increasing as sequence length increases. This demonstrates the practical efficiency of MERLIN compared to standard transformers.

Table R3.1 Prefill throughput (tokens/s)

ModelConfiguration2K4K8K16K32K
Attention40,16633,00826,73819,65613,152
GDN37,06235,00134,29133,81633,722
GDN+Meanpoolm=51228,51128,89626,31123,55020,533
GDN+MERLINm=512/drd_r=12823,31326,77024,65722,19518,864

Table R3.2. Prefill peak memory above weights (GiB)

ModelConfiguration2K4K8K16K32K
Attention0.480.961.913.837.66
GDN0.210.360.661.252.43
GDN+Meanpoolm=5120.340.631.232.404.76
GDN+MERLINm=512/drd_r=1280.360.671.292.535.00

References

  • [1] Behrouz et al. (2026): Memory Caching: RNNs with Growing Memory

Dear Reviewer,

We sincerely appreciate your thoughtful review. We uploaded a revision, where the revised contents are highlighted in blue. We list the revised contents corresponding to your comments and questions below.

[W1] Quality-memory-latency trade-offs: We report prefill throughput and peak memory in Section 4.3 and Figure 5. We report decode latency in Appendix D. We report evaluation results for standard Transformers in Section 4.1, Table 2 and Table 3.

Please let us know if the revision is insufficient. We will update the draft as soon as possible.

Reviewer_tGWhOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MERLIN for aggregating segment-level readouts in multi-state linear attention. It stores first and second moments of each segment’s routing keys, computes the mean and variance of their alignment with a query, and combines the resulting segment-weight distributions using a learned gate. Experiments on Gated DeltaNet-1.3B and Mamba-2-780M report gains over position-based and mean-pooled aggregation on RULER and LongBench at 16K context length.

Critical strengths

The paper identifies segment aggregation as a concrete bottleneck and presents a clear, query-dependent routing method.

MERLIN achieves consistent improvements in the reported average scores on two distinct backbones. The controlled gold-passage-position analysis is also informative.

Critical weaknesses

The main second-order claim is not established by a controlled comparison. MERLIN and the Meanpool baseline differ in more than the variance term, while the ablations do not include an otherwise identical mean-only MERLIN router. As a result, the reported gains cannot be attributed to second-order information. This is decisive for my recommendation because the variance-based router is the paper’s principal contribution. The cumulant expansion motivates using variance, but the implemented mixture of two softmax distributions does not follow directly from that expansion.

The evidence does not establish a sufficiently broad practical advantage over existing routing choices. The experiments compare against position and mean-pooled routing at 16K, without a matched comparison to more expressive content-based routing alternatives or measured prefill and decode costs. The paper reports cache-size calculations, but not the runtime trade-off incurred by reading a growing number of states. These omissions limit the significance of the reported accuracy gains.

Questions for authors

Under identical projections, normalization, parameter budget, and training, how do mean-only, variance-only, and combined routing compare?

How does MERLIN compare with a stronger content-based router under matched cache and compute budgets, including measured prefill throughput and decoding latency?

4 comments

We thank the reviewer for their constructive feedback and comments. We will address all the concerns raised by the reviewer and revise our paper before next Wednesday.

[W1, Q1] Additional ablation study on the design choice

We conduct additional ablations on Gated DeltaNet (GDN) 1.3B and MERLIN to isolate the effect of the mean term and the variance term. We fix the segment length (m=256) and the dimension of moments (drd_r=128). Table R2.1 shows that 40% of the gains between the Meanpool baseline and original MERLIN come from the mean term, and the remaining 60% come from the variance term. Interestingly, variance-only MERLIN outperforms mean-only MERLIN, although their interaction leads to much higher performance. These results suggest that the variance term, which uses the second moment of keys, is crucial for content-aware aggregation.

Table R2.1.

ModelAggregationSingle-doc QA AVGMulti-doc QA AVGSummarization AVGFew-shot AVGAll AVG
GDNMeanpool14.012.811.437.219.3
GDNMERLIN mean only14.313.911.639.320.3
GDNMERLIN var only15.014.312.240.821.1
GDNMERLIN mean + var (ours)17.915.412.240.221.9

[W2, Q2] Latency, throughput, and stronger baseline comparisons

Multi-state linear attention (MSLA) aims to improve the memory capacity of recurrent models like GDN and Mamba-2. Unlike standard transformers which store each token’s keys and values (KV) explicitly in a growing KV-cache, recurrent models compress the entire sequence into a fixed-size memory state, achieving greater memory efficiency. MSLA extends this approach by dividing the sequence into segments and compressing each segment’s KV into a separate fixed-size memory state. This allows them to achieve a balance between constant memory of linear attention and growing memory of transformers [1].

Existing routing methods for recurrent models such as RAT [2] and Sparse State Expansion (SSE) [3] make structurally different assumptions about the model architecture compared to MERLIN. As a result, these works cannot directly be applied to Gated DeltaNet and Mamba-2 without significant changes to the base model.

Therefore, we conduct additional experiments with Mixture-of-Memories [4] (MoM), which serves as the motivating work for SSE, is open-sourced, and is compatible with GDN. We set MoM to use 16 memory states, in addition to an always-read shared state and adjust MERLIN’s chunk size (m=1024) and dimension of moments (drd_r = 64) to match MoM’s cache size at 16K input sequence. We also evaluate MERLIN models in top-k mode (only read from states with k highest aggregation weights) to match MoM’s selective readout. Table R2.2 shows that GDN-1.3B with MERLIN outperforms MoM on LongBench tasks, suggesting that MERLIN is a stronger aggregation mechanism.

Table R2.2

Method# Memory StatesTop-kSingle-doc QA AVGMulti-doc QA AVGSummarization AVGFew-shot AVGAll AVG
MERLIN161611.419.913.345.820.9
MERLIN16810.719.213.645.720.6
MERLIN16410.318.813.944.120.1
MoM16 + 1 shared89.98.512.345.219.8
MoM16 + 1 shared49.48.611.642.818.9

We report the prefill throughput and peak memory of MERLIN and baselines below. For the attention baseline, we use the fla-hub/transformer-1.3B-100B model in huggingface and its optimized kernels. For each sequence length, we use a batch size of 1 and run it on a single L40 GPU. Table R2.3 shows that MERLIN’s throughput exceeds attention at about 16K, and the gap widens at 32K. Table R2.4 shows that MERLIN’s peak memory above weights stays below attention across all sequence lengths, with the gap increasing as sequence length increases. This demonstrates the practical efficiency of MERLIN compared to standard transformers.

Table R2.3. Prefill throughput (tokens/s)

ModelConfiguration2K4K8K16K32K
Attention40,16633,00826,73819,65613,152
GDN37,06235,00134,29133,81633,722
GDN+Meanpoolm=51228,51128,89626,31123,55020,533
GDN+MERLINm=512/drd_r=12823,31326,77024,65722,19518,864

Table R2.4. Prefill peak memory above weights (GiB)

ModelConfiguration2K4K8K16K32K
Attention0.480.961.913.837.66
GDN0.210.360.661.252.43
GDN+Meanpoolm=5120.340.631.232.404.76
GDN+MERLINm=512/drd_r=1280.360.671.292.535.00

References

  • [1] Behrouz et al. (2026): Memory Caching: RNNs with Growing Memory
  • [2] Wei et al. (2025): Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling
  • [3] Scaling Linear Attention with Sparse State Expansion: https://arxiv.org/abs/2507.16577
  • [4] Du et al. (2025): MoM: Linear Sequence Modeling with Mixture-of-Memories

Thank you for the additional experiments, especially the new ablations which address my main concern. MoM comparison is also helpful. I am thinking about raising scores and I would appreciate further clarification on decoding efficiency.

We appreciate the reviewer for the response.

As the reviewer requested, we show the decoding latency for baselines and MERLIN at various sequence lengths. Using the same model configuration as the prefill measurements in Tables R2.3 and R2.4, we measure the runtime for decoding a single token given a 2K-32K context on a single L40 GPU. We repeat the experiment 5 times and report the average results. Table R2.5 shows that the decode latency (lower is better) for MERLIN outperforms the Attention baseline from 8K and beyond, demonstrating its practical efficiency during the decoding process.

We will include the efficiency analysis in the revision before Tuesday.

Table R2.5. Decode latency (ms)

ModelConfiguration2K4K8K16K32K
Attention6.768.5512.0520.4535.32
GDN5.005.045.035.045.02
GDN+Meanpoolm=5127.377.598.018.9010.75
GDN+MERLINm=512/drd_r=1289.369.6810.2011.3313.66

Dear Reviewer,

We sincerely appreciate your thoughtful review. We uploaded a revision, where the revised contents are highlighted in blue. We list the revised contents corresponding to your comments and questions below.

[W1, Q1] Additional ablation study on the design choice: We report mean-only ablation results in Section 4.4 and Table 4.

[W2, Q2] Latency, throughput, and stronger baseline comparisons: We report prefill throughput and peak memory in Section 4.3 and Figure 5. We report decode latency in Appendix D. We report additional comparison with MoM in Appendix D.

Please let us know if the revision is insufficient. We will update the draft as soon as possible.

Reviewer_asLhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MERLIN improves output aggregation in multi-state linear attention. It stores the first and second moments of segment keys to compute query-dependent mean and variance scores. A learned gate combines the two routing distributions to aggregate state readouts. Experiments on Gated DeltaNet-1.3B and Mamba-2-780M report average improvements over Vanilla, Position, and Meanpool on long-context retrieval and understanding tasks.

Critical strengths
  • S1: The paper addresses an important problem: retaining useful long-range information under compressed memory. The solution is clear and well motivated. Variance complements mean alignment when segments have similar averages but different internal distributions. Key moments provide this information without revisiting every key, making the routing change a useful contribution.
  • S2: Results on two different recurrent backbones provide encouraging evidence that the approach is useful beyond one architecture. The average gains, segment-size analysis, and routing visualizations support the design. Similar added parameter counts for MERLIN and Meanpool also help control for parameter overhead.
Critical weaknesses
  • W1: The experimental comparisons include a single-state Vanilla baseline and two aggregation controls, Position and Meanpool, which share MERLIN’s segmentation setup. These comparisons establish gains over the tested baselines, but leave the broader advantage over relevant memory systems unclear. RAT and Sparse State Expansion address closely related memory and retrieval limitations. Sparse State Expansion is cited and discussed, but its full system is not compared experimentally. A representative comparison under matched training and memory budgets would strengthen the contribution's positioning.
  • W2: The validation leaves practical efficiency and broader generalization uncertain. There are no wall-clock prefill or decoding measurements, so lower cache cost does not establish actual speed gains. Experiments use only 1.3B and 780M backbones at 16K context, with a length-filtered LongBench subset. Larger-scale evaluation and a length-compatible LongBench v2 subset or another demanding benchmark would help establish how widely the gains apply.

References

Questions for authors
  • Q1: How does MERLIN compare with a representative alternative such as RAT or Sparse State Expansion under matched training and memory budgets? If this comparison is not feasible, which differences in assumptions or implementation prevent it?
  • Q2: What are the measured prefill and decoding latency, throughput, and peak memory relative to the existing baselines on the same hardware, precision, batch size, and context length, including moment updates and routing?
3 comments

[W1, W2] Comparison with RAT and SSE

We thank the reviewer for their constructive feedback and comments. We will address all the raised concerns by the reviewer and revise our paper before next Wednesday.

Although RAT and Sparse State Expansion both build on recurrent architectures, their architectural assumptions differ from those underlying MERLIN. Specifically, RAT applies an RNN to each chunk in the sequence, compressing it into a representative token. However, MERLIN is designed for models like Gated DeltaNet (GDN) and Mamba-2, which compress each chunk’s keys and values into a memory matrix. This architectural difference leads to distinct chunk-size regimes, where RAT uses shorter chunks of 16 tokens [1], while MERLIN supports much larger chunks with 512 tokens or more.

Furthermore, Sparse State Expansion (SSE) builds on the GLA [2] architecture and extension to GDN is nontrivial. Therefore, we conduct additional experiments with Mixture-of-Memories [3] (MoM), which serves as the motivating work for SSE, is open-sourced, and is compatible with GDN. We set MoM to use 16 memory states, in addition to an always-read shared state and adjust MERLIN’s chunk size (m=1024) and dimension of moments (drd_r = 64) to match MoM’s cache size at 16K input sequence. We also evaluate MERLIN models in top-k mode (only read from states with k highest aggregation weights) to match MoM’s selective readout. Table R1.1 shows that GDN-1.3B with MERLIN outperforms MoM on LongBench tasks, suggesting that MERLIN is a stronger aggregation mechanism.

Table R1.1.

Method# Memory StatesTop-kSingle-doc QA AVGMulti-doc QA AVGSummarization AVGFew-shot AVGAll AVG
MERLIN161611.419.913.345.820.9
MERLIN16810.719.213.645.720.6
MERLIN16410.318.813.944.120.1
MoM16 + 1 shared89.98.512.345.219.8
MoM16 + 1 shared49.48.611.642.818.9

[W2, Q2] Latency, throughput and peak memory comparisons

We report the prefill throughput and peak memory of MERLIN and baselines below. For the attention baseline, we use the fla-hub/transformer-1.3B-100B model in huggingface and its optimized kernels. For each sequence length, we use a batch size of 1 and run it on a single L40 GPU. Table R1.2 shows that MERLIN’s throughput exceeds attention at about 16K, and the gap widens at 32K. Table R1.3 shows that MERLIN’s peak memory above weights stays below attention across all sequence lengths, with the gap increasing as sequence length increases. This demonstrates the practical efficiency of MERLIN compared to standard transformers.

LongBench v2 requires training at 32K-64K context length for meaningful evaluations, which is beyond our hardware constraints. However, we already observe MERLIN’s efficiency advantage at 16K and beyond from Tables R1.2 and R1.3.

Table R1.2. Prefill throughput (tokens/s)

ModelConfiguration2K4K8K16K32K
Attention40,16633,00826,73819,65613,152
GDN37,06235,00134,29133,81633,722
GDN+Meanpoolm=51228,51128,89626,31123,55020,533
GDN+MERLINm=512/drd_r=12823,31326,77024,65722,19518,864

Table R1.3. Prefill peak memory above weights (GiB)

ModelConfiguration2K4K8K16K32K
Attention0.480.961.913.837.66
GDN0.210.360.661.252.43
GDN+Meanpoolm=5120.340.631.232.404.76
GDN+MERLINm=512/drd_r=1280.360.671.292.535.00

References

  • [1] Wei et al. (2025): Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling
  • [2] Yang et al. (2023): Gated Linear Attention Transformers with Hardware-Efficient Training
  • [3] Du et al. (2025): MoM: Linear Sequence Modeling with Mixture-of-Memories

Thank you for the helpful response. The additional MoM comparison and prefill measurements address my requests for a broader comparison and measured efficiency evidence. The new ablations also strengthen the motivation for second-order routing. These additions reinforce my positive assessment, and I would like to keep my current score, which is already positive.

Dear Reviewer,

We sincerely appreciate your thoughtful review. We uploaded a revision, where the revised contents are highlighted in blue. We list the revised contents corresponding to your comments and questions below.

[W1, Q1] Comparison with RAT and SSE: We report additional comparison with MoM in Appendix D.

[W2, Q2] Latency, throughput and peak memory comparisons We report prefill throughput and peak memory in Section 4.3 and Figure 5. We report decode latency in Appendix D.

Please let us know if the revision is insufficient. We will update the draft as soon as possible.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.