- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes MERLIN, a routing mechanism for multi-state linear attention that uses the mean and variance of query-key alignment to aggregate segment-level readouts. Experiments on Gated DeltaNet and Mamba-2 show improvements over position-based and mean-based aggregation.
Critical strengths
The method is straightforward and has a statistical motivation. Comparisons with Meanpool support the benefit of second-order routing on both backbones, with similar parameter counts.
Critical weaknesses
My main concern is that the experiments do not establish a competitive quality-efficiency trade-off.
- With fixed segment length m, the model caches L=T/m states and reads all visible states for every query. Consequently, prefill costs O(T^2(d^2+dr^2))/m), and persistent memory grows linearly with context length (Table 1). The evaluated architecture therefore loses the linear-time and constant-memory advantages of single-state GDN while retaining lossy compression within each segment.
Despite sharing these asymptotic scaling properties with full attention, the paper provides neither a matched full-attention quality baseline nor wall-clock prefill/decode comparisons against an optimized implementation. Comparisons with single-state GDN cannot establish resource efficiency: the default GDN variant uses approximately 72× its recurrent cache at 16K (Table 9).
The Meanpool results demonstrate a routing improvement, but do not establish why this growing bank of compressed states should be preferred over full attention. Without matched comparisons and measured quality-memory-latency trade-offs, I find the evidence insufficient for acceptance.
Questions for authors
see weakness
2 comments
Dear Reviewer,
We sincerely appreciate your thoughtful review. We uploaded a revision, where the revised contents are highlighted in blue. We list the revised contents corresponding to your comments and questions below.
[W1] Quality-memory-latency trade-offs: We report prefill throughput and peak memory in Section 4.3 and Figure 5. We report decode latency in Appendix D. We report evaluation results for standard Transformers in Section 4.1, Table 2 and Table 3.
Please let us know if the revision is insufficient. We will update the draft as soon as possible.
We thank the reviewer for their constructive feedback and comments. We will address all the concerns raised by the reviewer and revise our paper before next Wednesday.
[W1] Quality-memory-latency trade-offs
Multi-state linear attention (MSLA) aims to improve the memory capacity of recurrent models like GDN and Mamba-2. Unlike standard transformers which store each token’s keys and values (KV) explicitly in a growing KV-cache, recurrent models compress the entire sequence into a fixed-size memory state, achieving greater memory efficiency. MSLA extends this approach by dividing the sequence into segments and compressing each segment’s KV into a separate fixed-size memory state. This allows them to achieve a balance between constant memory of linear attention and linear growing memory KV-cache of transformers [1].
To show this experimentally, we report the prefill throughput and peak memory of MERLIN and baselines below. For the attention baseline, we use the
fla-hub/transformer-1.3B-100Bmodel in huggingface and its optimized kernels. For each sequence length, we use a batch size of 1 and run it on a single L40 GPU. Table R3.1 shows that MERLIN’s throughput exceeds attention at about 16K, and the gap widens at 32K. Table R3.2 shows that MERLIN’s peak memory above weights stays below attention across all sequence lengths, with the gap increasing as sequence length increases. This demonstrates the practical efficiency of MERLIN compared to standard transformers.Table R3.1 Prefill throughput (tokens/s)
Table R3.2. Prefill peak memory above weights (GiB)
References