- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper analyzes multi-slot linear attention from the perspective of query-conditioned selectivity, and identifies two limitations in existing methods: cross-token selectivity leakage (ABC/GSA) and tied selectivity (SSE/MoM). Based on this, the authors propose SimpleRoute, which decouples read/write routing with multi-head competition and can be used to expand the state size. Experiments at the 340M and 1B scales with expanded states demonstrate improved performance, particularly on retrieval tasks.
Critical strengths
-
The paper identifies meaningful limitations in existing multi-slot linear attention methods, including mixed token associations in ABC/GSA and coupled read/write routing in SSE/MoM.
-
The paper presents a clear motivation and a coherent, easy-to-follow method design within its proposed framework.
Critical weaknesses
- Insufficiently rigorous motivation and limited novelty. Several key arguments in the motivation and theoretical analysis are not fully justified or convincing, as detailed in Questions (1)–(4). The methodological novelty also needs stronger justification.
- Insufficient and confounded experimental comparisons. The experiments lack important baselines and rigorous state-size-matched comparisons, making it difficult to isolate the benefits of the proposed method from state capacity expansion. See Questions (5)–(6) for details.
Questions for authors
-
Section 3.1 mischaracterizes query-conditioned competition in linear attention (LA). LA already provides query-conditioned selection through . For positive , its normalized attention weights are equivalent to . The key difference from softmax attn lies in the feature map and resulting selection sharpness, rather than the absence of competition. Moreover, the authors incorrectly attribute competition to softmax normalization. Actually, softmax normalization only introduces a common scaling factor without changing relative attention weights (which is exactly the same as in linear attn), and it is the exponential function, rather than normalization, that sharpens the distribution. Moreover, LA is often followed by head RMSNorm, making this scaling normalization factor largely irrelevant to the output. Therefore, attributing GLA's faint needle peak to a lack of query-conditioned competition is misleading. Could the authors clarify what exactly they mean by the ``Competition'' that LA supposedly lacks, and how it fundamentally differs from the query-conditioned association already present in LA?
-
Section 3.2 similarly misinterprets the role of routing. The authors describes the routing factor () as introducing the missing competition mechanism into LA, but this explanation also has a logical issue. In fact, the routing factor depends only on the current query and the corresponding historical key, rather than on other historical tokens. Therefore, can be rewritten as , where and . Thus, SimpleRoute can be viewed as LA with augmented feature maps. Routing may produce sharper selection patterns through slot assignments, but it does not introduce a fundamentally different token-level competition mechanism. In particular, its softmax operates over slots rather than historical tokens, and slot competition is not fully equivalent to explicit token competition, which is already present in vanilla LA.
-
About Section 3.3: While I agree that ABC/GSA lack explicit token-wise independent associations, two points need clarification. First, follows from the authors' chosen factorization, while already incorporates QK associations through (i.e., ). Even when Eq. (5) reduces to standard softmax attn (with one slot per token), this factorization still gives . Thus, why does imply the absence of token-specific association?
-
Although SimpleRoute preserves token-specific associations, increasing a slot's read-routing weight can still amplify the attention weights of all tokens assigned to that slot, including irrelevant ones. The token-specific association may suppress such interference, but cannot fully prevent it, as standard linear attention also relies on yet still suffers from interference. Could the authors clarify in what sense SimpleRoute avoids cross-token selectivity leakage rather than merely mitigating it?
-
Insufficiently controlled comparisons in the main experiments. SimpleRoute expands the recurrent state by 4$\times$ and 16$\times$ at 340M and 1B, respectively, with proportional increases in computation and state I/O due to dense routing. State expansion is already known to improve LA performance. Indeed, Table 2 shows that state-neutral SimpleRoute performs worse than GLA on RW Retrieval (21.05 vs. 23.05), making it difficult to attribute the main gains to the proposed routing design.
More importantly, the paper lacks rigorous state-matched comparisons with key multi-slot baselines, especially SSE and MoM. The MoM results are cited without specifying its state expansion ratio, while SSE is entirely absent from the main comparisons. Both methods use sparse routing, offering substantially better efficiency at the same state capacity. SSE also shares attention parameters across slots, avoiding additional parameter growth and making it a particularly relevant baseline. Given that SSE and MoM are central to the paper's motivation, controlled comparisons of performance, state capacity, and prefill/decode efficiency are essential to establish SimpleRoute's actual advantages. -
Limited methodological novelty. The core design of SimpleRoute can be viewed as dense input/output gating with softmax competition across heads or slots. Without softmax competition (e.g., using sigmoid gates), the input gating resembles the -controlled state updates in GDN, while the output scalar gating can be canceled by the commonly used head-wise RMSNorm in the state-neutral setting. Therefore, a matched sigmoid gating baseline is essential to demonstrate the actual benefit of the proposed softmax competition. Moreover, the benefits of state expansion have already been extensively explored in SSE and MoM (both are sparse expansion), as well as prior work on dense, parameter-efficient state expansion (e.g., Scaling up the State Size of RNN LLMs for Long-Context Scenarios, ACL 2025). While I acknowledge the value of decoupling input and output routing, the paper should better clarify its methodological novelty beyond existing gating and state-expansion techniques.
7 comments
Response to Reviewer CtcW (2/3): Q3-Q4
Q3. for ABC/GSA in Sec. 3.3 follows from the authors' chosen factorization, while already incorporates QK associations through . Even when Eq. (5) reduces to standard softmax attention with one slot per token, this factorization still gives ; why does this imply the absence of token-specific association?
Response:
The reviewer has misunderstood the role of Eq. (5): Eq. (5) analyzes multi-slot routing within the framework of Eq. (2). Equation (2) decomposes retrieval weights into token-specific association and slot-mediated selection . Here, can only represent slot-mediated selection; therefore, means that ABC lacks token-specific association.
Furthermore, and in Eq. (2) are defined over a fixed number of slots. In the one-slot-per-token limit, the slot count grows with sequence length, taking this case outside the framework of Eq. (2). It therefore does not constitute a counterexample.
Q4. Does SimpleRoute "avoid" or "mitigate" leakage? Increasing a slot's read-routing weight can still amplify all tokens assigned to that slot, including irrelevant ones, and token-specific association cannot fully prevent such interference. In what sense does SimpleRoute avoid cross-token selectivity leakage rather than merely mitigate it?
Response:
Slots can indeed represent only low-rank token-level selectivity, but our experiments directly validate the effectiveness of this representation. In Sec. 5.3, we measure the attention mass assigned to the needle sentence using normalized token-level read coefficients, including the routing factor (see Eq. (15)). Across 1,500 examples, this mass is only 0.44% for GLA, compared with 4.56% for SimpleRoute (approximately 10 times that of GLA) and 8.07% for Transformer++ (Fig. 3 and Table 5). In the single-example visualization in Fig. 1(b), GLA's needle peak is overwhelmed by the final tokens, whereas SimpleRoute forms a clear peak at the needle, as does Transformer++. In other words, slot-level selectivity allows the model to attend to the needle, which the original GLA fails to do.
Response to Reviewer CtcW (3/3): Q5-Q6
Q5. State-matched comparisons Could gains reflect added state, given the worse state-neutral retrieval result? Please compare SSE/MoM performance and prefill/decode efficiency at matched state size, specifying MoM's state budget and accounting for sparse routing and SSE's shared parameters.
Response:
1. Our conclusions about routing design come from controlled experiments matched in both state and parameters. In Table 3, state, parameters, slot count, and training recipe are all held constant. Removing key–value binding (Unbound, corresponding to ABC/GSA) reduces RW Avg. from 20.73 to 11.80, while forcing tied read/write routing (Exact-Tied, corresponding to SSE) reduces it to 18.57. These differences are unrelated to state expansion.
2. Comparison with GLA using directly expanded state. We would like to clarify that the motivation for multi-slot routing is to increase state size with fewer parameters and lower peak memory. We therefore compare against a baseline that directly increases GLA's state size: it increases the key dimension (\texttt{expand_k}=2), giving wide-GLA and SimpleRoute () the same per-layer state size of 524,288. SimpleRoute achieves results comparable to wide-GLA with 18.2% fewer parameters, validating the effectiveness of its design.
| Metric | wide-GLA (\texttt{expand_k}=2) | SimpleRoute () |
|---|---|---|
| State elements per layer | 524,288 | 524,288 |
| Parameters (M) ↓ | 417.832 | 341.807 |
| Peak decoding memory, batch=8 (MiB) ↓ | 1250.526 | 1098.613 |
| WikiText PPL ↓ | 24.86 | 24.27 |
| LAMBADA PPL ↓ | 26.04 | 24.64 |
| Common Avg.-6 ↑ | 43.76 | 44.33 |
| Real-world Retrieval Avg.-4 ↑ | 26.26 | 26.17 |
3. Why performs worse than GLA on real-world retrieval. The configuration is a degenerate case with only one slot per head and does not represent multi-slot routing. As the reviewer notes in Q6, the read route then reduces to a scalar per head, which is canceled by head-wise RMSNorm. The model effectively retains only write-side routing and lacks the query-conditioned selectivity analyzed in our paper. Even with only write-side routing, WikiText PPL (26.32→24.81), Common Avg. (43.69→44.35), and S-NIAH (50.64→53.93) still improve. When , the readouts of multiple slots within each head are mixed according to before entering RMSNorm, preserving read-side selection. Table 3 () tests routing design precisely in this setting. We will clarify in the revision that only write-side routing remains effective when .
4. SSE and MoM. SSE and MoM belong to the same class of designs with tied read/write routing (tied selectivity). In terms of routing constraints, MoM relaxes SSE's restriction: SSE uses identical read/write routing weights, making symmetric positive semidefinite, whereas MoM ties only the read/write supports, so need not be symmetric (Sec. A.2). We therefore use the less constrained MoM as a representative of this class.
The MoM results in Table 1 come from the setting of Du et al. (2026). Since MoM and SimpleRoute have different individual state shapes, we compare capacity using the number of state elements stored per layer per sequence, including shared memory. MoM stores 1,310,720 state elements per layer and activates 786,432 of them for each token; SimpleRoute () has 524,288 state elements per layer. Despite using a smaller state size, SimpleRoute still outperforms the MoM baseline.
Q6. Methodological novelty How does SimpleRoute differ from dense input/output gating and prior state expansion? Please clarify GDN's beta updates and RMSNorm-cancelled output gating, and provide a matched sigmoid-gating baseline.
Response:
The reviewer views softmax over slots as an interchangeable gating function, but in our framework, softmax is a natural choice for representing query-conditioned selectivity. Query-conditioned selectivity requires the query to highlight relevant tokens while suppressing distractors (see our response to Q1). Softmax over slots entails a trade-off: the more a token writes to certain slots, the less it writes to others. Consequently, tokens written to slots that the query does not read are suppressed (see our response to Q2). Independent sigmoid gates for individual slots lack this constraint, failing to suppress irrelevant tokens.
Difference from in GDN. This is also the similarity between sigmoid write gates and GDN's noted by the reviewer: both determine only the strength of a token's write. The coefficient is independent of subsequent queries and uniformly increases or decreases token 's contribution for all queries. It cannot highlight the same token for one query while suppressing it for another, and therefore cannot capture the suppression of distractors required by query-conditioned selectivity.
Response to Reviewer CtcW (1/3): Q1-Q2
Q1. "Competition" in linear attention in Sec. 3.1 Section 3.1 mischaracterizes query-conditioned competition in LA: LA already provides query-conditioned selection through bij, and for positive bij, its normalized weights are equivalent to softmaxj(logbij). Please clarify what "competition" LA supposedly lacks, given that exponentiation rather than normalization sharpens relative weights, and head-wise RMSNorm makes the common scaling factor largely irrelevant.
Response:
The disagreement stems from a conceptual conflation: the reviewer treats the query-conditioned association already present in LA as the query-conditioned competition discussed in our paper, whereas we distinguish the two. By query-conditioned competition, we mean a trade-off between relevant and distracting tokens after the query arrives: relevance differences are amplified, and any weight gained by one token necessarily comes at the expense of other tokens, thereby highlighting relevant historical tokens while suppressing distractors. The resulting capability is what we call query-conditioned selectivity in lines 50–51 and 144, consistent with Zhang et al. (2026, MHLA). LA has query-conditioned association bij=ϕ(qi)⊤ϕ(kj), as explicitly labeled in Eq. (1); we do not deny this. Our point is that LA lacks this competition, which is also a central argument of MHLA (termed query-conditioned selectivity there; see its Sec. 3.2, Appendix B, and Table 1).
Why LA lacks query-conditioned competition. MHLA attributes this to all queries sharing the same summary (MHLA Sec. 3.2 and Appendix B). Softmax attention applies exponentiation to individual tokens after the query arrives. The weight ratio between a relevant token j and a distracting token ℓ is exp(qi⊤(kj−kℓ)/d), which exponentially amplifies their difference in relevance to the query. LA, in contrast, compresses the history into a summary shared by all queries before the query arrives:
yi=S⊤ϕ(qi)=∑jbijvj,S=∑jϕ(kj)vj⊤.
To compute the output recurrently through S, no nonlinearity can then be applied to the individual token coefficients bij after the query arrives (lines 155–156). The weight ratio bij/biℓ is determined entirely by the feature map, with no subsequent amplification stage. We agree with the reviewer that sharpness comes from exponentiation and that normalization introduces only a common scaling factor; Sec. 3.1 already makes this distinction (lines 146–148 and 157–158). For precisely this reason, even with normalization or head-wise RMSNorm, LA yields a distribution with a fixed budget but unamplified weight ratios: the budget is spread across the entire history, preventing relevant tokens from capturing a dominant share and thus failing to suppress distractors. The measurements in Fig. 3 are consistent with this: only 0.44% of GLA's attention falls on the needle, compared with 8.07% for Transformer++.
Q2. The role of routing in Sec. 3.2 The routing factor depends only on the current query and corresponding historical key, so SimpleRoute can be viewed as LA with augmented feature maps. Routing may produce sharper selection patterns, but it does not introduce a fundamentally different token-level competition mechanism: its softmax operates over slots rather than historical tokens.
Response:
We would like to clarify that we do not claim that softmax over slots is equivalent to token-level softmax competition. Our claim is that competition at the slot level propagates to historical tokens through C=ΠR(ΠW)⊤, forming token-level competition of rank at most r (lines 179–181 and Eq. (4)) that allows the query to highlight relevant tokens and suppress distractors. The reviewer also notes that routing can produce sharper selection patterns, which is precisely what we call selectivity. As explained in our response to Q1, vanilla LA has association but lacks this selectivity.
The ability to express the method using augmented feature maps does not mean that no new selection mechanism has been introduced: the kernel exp(q⊤k) of softmax attention can itself be written as an inner product of infinite-dimensional feature maps. What matters is whether the augmentation factor implements selection among tokens.
Our experiments validate the effectiveness of approximating token selectivity with low-rank slot selectivity. Please see our response to Q4 for the experimental analysis, which verifies that slot-level selectivity allows the model to attend to the needle, which the original GLA fails to do.