acceptodds
Under review as a conference paper at ICLR 2027

Beyond State Capacity: Diagnosing and Improving Multi-Slot Linear Attention through Selectivity

Abstract

Multi-slot linear attention is commonly understood as expanding recurrent-state capacity, but this view alone cannot explain why routing designs with comparable state capacity differ in retrieval performance. We show that multi-slot routing is not merely state-capacity expansion: it factorizes a query-conditioned selection operator over historical tokens. Under this view, routing design determines the operator's usable expressivity even at fixed state capacity. We identify two structural restrictions in existing methods: cross-token selectivity leakage, where selecting one token also emphasizes others sharing its slot, and tied selectivity, where coupled read and write routing limits which selection patterns can be expressed. These diagnoses lead to SimpleRoute, a simple routing mechanism that removes both restrictions to improve selective retrieval while retaining linear complexity in sequence length. Trained from scratch with 1B parameters on 50B tokens, SimpleRoute achieves the highest average commonsense and real-world retrieval scores among the compared large-scale recurrent models, including 1.3B models trained on twice as many tokens, and improves real-world retrieval by 6.35 average points over the strongest baseline at the same scale and token budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
linear attention, multi-slot memory, routing, query-conditioned selectivity, long-context retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CtcWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper analyzes multi-slot linear attention from the perspective of query-conditioned selectivity, and identifies two limitations in existing methods: cross-token selectivity leakage (ABC/GSA) and tied selectivity (SSE/MoM). Based on this, the authors propose SimpleRoute, which decouples read/write routing with multi-head competition and can be used to expand the state size. Experiments at the 340M and 1B scales with expanded states demonstrate improved performance, particularly on retrieval tasks.

Critical strengths
  1. The paper identifies meaningful limitations in existing multi-slot linear attention methods, including mixed token associations in ABC/GSA and coupled read/write routing in SSE/MoM.

  2. The paper presents a clear motivation and a coherent, easy-to-follow method design within its proposed framework.

Critical weaknesses
  1. Insufficiently rigorous motivation and limited novelty. Several key arguments in the motivation and theoretical analysis are not fully justified or convincing, as detailed in Questions (1)–(4). The methodological novelty also needs stronger justification.
  2. Insufficient and confounded experimental comparisons. The experiments lack important baselines and rigorous state-size-matched comparisons, making it difficult to isolate the benefits of the proposed method from state capacity expansion. See Questions (5)–(6) for details.
Questions for authors
  1. Section 3.1 mischaracterizes query-conditioned competition in linear attention (LA). LA already provides query-conditioned selection through bij=ϕ(qi)⊤ϕ(kj)b_{ij}=\phi(q_i)^\top\phi(k_j). For positive bijb_{ij}, its normalized attention weights are equivalent to softmaxj(log⁡bij)softmax_j(\log b_{ij}). The key difference from softmax attn lies in the feature map and resulting selection sharpness, rather than the absence of competition. Moreover, the authors incorrectly attribute competition to softmax normalization. Actually, softmax normalization only introduces a common scaling factor without changing relative attention weights (which is exactly the same as in linear attn), and it is the exponential function, rather than normalization, that sharpens the distribution. Moreover, LA is often followed by head RMSNorm, making this scaling normalization factor largely irrelevant to the output. Therefore, attributing GLA's faint needle peak to a lack of query-conditioned competition is misleading. Could the authors clarify what exactly they mean by the ``Competition'' that LA supposedly lacks, and how it fundamentally differs from the query-conditioned association already present in LA?

  2. Section 3.2 similarly misinterprets the role of routing. The authors describes the routing factor (cijc_{ij}) as introducing the missing competition mechanism into LA, but this explanation also has a logical issue. In fact, the routing factor cij=πR(qi)⊤πW(kj)c_{ij} = \pi^R(q_i)^\top \pi^W(k_j) depends only on the current query and the corresponding historical key, rather than on other historical tokens. Therefore, aij=bijcija_{ij}=b_{ij}c_{ij} can be rewritten as ϕ~Q(qi)⊤ϕ~K(kj)\tilde{\phi}_Q(q_i)^\top\tilde{\phi}_K(k_j), where ϕ~Q(q)=ϕ(q)⊗πR(q)\tilde{\phi}_Q(q)=\phi(q)\otimes\pi^R(q) and ϕ~K(k)=ϕ(k)⊗πW(k)\tilde{\phi}_K(k)=\phi(k)\otimes\pi^W(k). Thus, SimpleRoute can be viewed as LA with augmented feature maps. Routing may produce sharper selection patterns through slot assignments, but it does not introduce a fundamentally different token-level competition mechanism. In particular, its softmax operates over slots rather than historical tokens, and slot competition is not fully equivalent to explicit token competition, which is already present in vanilla LA.

  3. About Section 3.3: While I agree that ABC/GSA lack explicit token-wise independent associations, two points need clarification. First, bij=1b_{ij}=1 follows from the authors' chosen factorization, while cijc_{ij} already incorporates QK associations through himh_{im} (i.e., qi⊤klq_i^\top k_l). Even when Eq. (5) reduces to standard softmax attn (with one slot per token), this factorization still gives bij=1b_{ij}=1. Thus, why does bij=1b_{ij}=1 imply the absence of token-specific association?

  4. Although SimpleRoute preserves token-specific associations, increasing a slot's read-routing weight πiR\pi^R_{i} can still amplify the attention weights of all tokens assigned to that slot, including irrelevant ones. The token-specific association bijb_{ij} may suppress such interference, but cannot fully prevent it, as standard linear attention also relies on bijb_{ij} yet still suffers from interference. Could the authors clarify in what sense SimpleRoute avoids cross-token selectivity leakage rather than merely mitigating it?

  5. Insufficiently controlled comparisons in the main experiments. SimpleRoute expands the recurrent state by 4$\times$ and 16$\times$ at 340M and 1B, respectively, with proportional increases in computation and state I/O due to dense routing. State expansion is already known to improve LA performance. Indeed, Table 2 shows that state-neutral SimpleRoute performs worse than GLA on RW Retrieval (21.05 vs. 23.05), making it difficult to attribute the main gains to the proposed routing design.
    More importantly, the paper lacks rigorous state-matched comparisons with key multi-slot baselines, especially SSE and MoM. The MoM results are cited without specifying its state expansion ratio, while SSE is entirely absent from the main comparisons. Both methods use sparse routing, offering substantially better efficiency at the same state capacity. SSE also shares attention parameters across slots, avoiding additional parameter growth and making it a particularly relevant baseline. Given that SSE and MoM are central to the paper's motivation, controlled comparisons of performance, state capacity, and prefill/decode efficiency are essential to establish SimpleRoute's actual advantages.

  6. Limited methodological novelty. The core design of SimpleRoute can be viewed as dense input/output gating with softmax competition across heads or slots. Without softmax competition (e.g., using sigmoid gates), the input gating resembles the β\beta-controlled state updates in GDN, while the output scalar gating can be canceled by the commonly used head-wise RMSNorm in the state-neutral setting. Therefore, a matched sigmoid gating baseline is essential to demonstrate the actual benefit of the proposed softmax competition. Moreover, the benefits of state expansion have already been extensively explored in SSE and MoM (both are sparse expansion), as well as prior work on dense, parameter-efficient state expansion (e.g., Scaling up the State Size of RNN LLMs for Long-Context Scenarios, ACL 2025). While I acknowledge the value of decoupling input and output routing, the paper should better clarify its methodological novelty beyond existing gating and state-expansion techniques.

7 comments
AuthorsOpenReview ↗

Response to Reviewer CtcW (1/3): Q1-Q2

Q1. "Competition" in linear attention in Sec. 3.1 Section 3.1 mischaracterizes query-conditioned competition in LA: LA already provides query-conditioned selection through bijb_{ij}, and for positive bijb_{ij}, its normalized weights are equivalent to softmaxj(log⁡bij)\mathrm{softmax}_j(\log b_{ij}). Please clarify what "competition" LA supposedly lacks, given that exponentiation rather than normalization sharpens relative weights, and head-wise RMSNorm makes the common scaling factor largely irrelevant.

Response:

The disagreement stems from a conceptual conflation: the reviewer treats the query-conditioned association already present in LA as the query-conditioned competition discussed in our paper, whereas we distinguish the two. By query-conditioned competition, we mean a trade-off between relevant and distracting tokens after the query arrives: relevance differences are amplified, and any weight gained by one token necessarily comes at the expense of other tokens, thereby highlighting relevant historical tokens while suppressing distractors. The resulting capability is what we call query-conditioned selectivity in lines 50–51 and 144, consistent with Zhang et al. (2026, MHLA). LA has query-conditioned association bij=ϕ(qi)⊤ϕ(kj)b_{ij}=\phi(q_i)^\top\phi(k_j), as explicitly labeled in Eq. (1); we do not deny this. Our point is that LA lacks this competition, which is also a central argument of MHLA (termed query-conditioned selectivity there; see its Sec. 3.2, Appendix B, and Table 1).

Why LA lacks query-conditioned competition. MHLA attributes this to all queries sharing the same summary (MHLA Sec. 3.2 and Appendix B). Softmax attention applies exponentiation to individual tokens after the query arrives. The weight ratio between a relevant token jj and a distracting token ℓ\ell is exp⁡(qi⊤(kj−kℓ)/d)\exp\big(q_i^\top(k_j-k_\ell)/\sqrt d\big), which exponentially amplifies their difference in relevance to the query. LA, in contrast, compresses the history into a summary shared by all queries before the query arrives:

yi=S⊤ϕ(qi)=∑jbijvj,S=∑jϕ(kj)vj⊤.y_i=S^\top\phi(q_i)=\sum_j b_{ij}v_j,\qquad S=\sum_j\phi(k_j)v_j^\top.

To compute the output recurrently through SS, no nonlinearity can then be applied to the individual token coefficients bijb_{ij} after the query arrives (lines 155–156). The weight ratio bij/biℓb_{ij}/b_{i\ell} is determined entirely by the feature map, with no subsequent amplification stage. We agree with the reviewer that sharpness comes from exponentiation and that normalization introduces only a common scaling factor; Sec. 3.1 already makes this distinction (lines 146–148 and 157–158). For precisely this reason, even with normalization or head-wise RMSNorm, LA yields a distribution with a fixed budget but unamplified weight ratios: the budget is spread across the entire history, preventing relevant tokens from capturing a dominant share and thus failing to suppress distractors. The measurements in Fig. 3 are consistent with this: only 0.44% of GLA's attention falls on the needle, compared with 8.07% for Transformer++.

Q2. The role of routing in Sec. 3.2 The routing factor depends only on the current query and corresponding historical key, so SimpleRoute can be viewed as LA with augmented feature maps. Routing may produce sharper selection patterns, but it does not introduce a fundamentally different token-level competition mechanism: its softmax operates over slots rather than historical tokens.

Response:

We would like to clarify that we do not claim that softmax over slots is equivalent to token-level softmax competition. Our claim is that competition at the slot level propagates to historical tokens through C=ΠR(ΠW)⊤C=\Pi^R(\Pi^W)^\top, forming token-level competition of rank at most rr (lines 179–181 and Eq. (4)) that allows the query to highlight relevant tokens and suppress distractors. The reviewer also notes that routing can produce sharper selection patterns, which is precisely what we call selectivity. As explained in our response to Q1, vanilla LA has association but lacks this selectivity.

The ability to express the method using augmented feature maps does not mean that no new selection mechanism has been introduced: the kernel exp⁡(q⊤k)\exp(q^\top k) of softmax attention can itself be written as an inner product of infinite-dimensional feature maps. What matters is whether the augmentation factor implements selection among tokens.

Our experiments validate the effectiveness of approximating token selectivity with low-rank slot selectivity. Please see our response to Q4 for the experimental analysis, which verifies that slot-level selectivity allows the model to attend to the needle, which the original GLA fails to do.

AuthorsOpenReview ↗

Response to Reviewer CtcW (2/3): Q3-Q4

Q3. bij=1b_{ij}=1 for ABC/GSA in Sec. 3.3 bij=1b_{ij}=1 follows from the authors' chosen factorization, while cijc_{ij} already incorporates QK associations through himh_{im}. Even when Eq. (5) reduces to standard softmax attention with one slot per token, this factorization still gives bij=1b_{ij}=1; why does this imply the absence of token-specific association?

Response:

The reviewer has misunderstood the role of Eq. (5): Eq. (5) analyzes multi-slot routing within the framework of Eq. (2). Equation (2) decomposes retrieval weights into token-specific association bijMSb^{\mathrm{MS}}_{ij} and slot-mediated selection cijc_{ij}. Here, cijABC=∑m=1rπjmW,ABC,softmax(hi)mc^{\mathrm{ABC}}_{ij}=\sum_{m=1}^{r}\pi^{W,\mathrm{ABC}}_{jm}\\,\mathrm{softmax}(h_i)_m can only represent slot-mediated selection; therefore, bijABC=1b^{\mathrm{ABC}}_{ij}=1 means that ABC lacks token-specific association.

Furthermore, πR\pi^R and πW\pi^W in Eq. (2) are defined over a fixed number of rr slots. In the one-slot-per-token limit, the slot count grows with sequence length, taking this case outside the framework of Eq. (2). It therefore does not constitute a counterexample.

Q4. Does SimpleRoute "avoid" or "mitigate" leakage? Increasing a slot's read-routing weight can still amplify all tokens assigned to that slot, including irrelevant ones, and token-specific association cannot fully prevent such interference. In what sense does SimpleRoute avoid cross-token selectivity leakage rather than merely mitigate it?

Response:

Slots can indeed represent only low-rank token-level selectivity, but our experiments directly validate the effectiveness of this representation. In Sec. 5.3, we measure the attention mass assigned to the needle sentence using normalized token-level read coefficients, including the routing factor (see Eq. (15)). Across 1,500 examples, this mass is only 0.44% for GLA, compared with 4.56% for SimpleRoute (approximately 10 times that of GLA) and 8.07% for Transformer++ (Fig. 3 and Table 5). In the single-example visualization in Fig. 1(b), GLA's needle peak is overwhelmed by the final tokens, whereas SimpleRoute forms a clear peak at the needle, as does Transformer++. In other words, slot-level selectivity allows the model to attend to the needle, which the original GLA fails to do.

AuthorsOpenReview ↗

Response to Reviewer CtcW (3/3): Q5-Q6

Q5. State-matched comparisons Could gains reflect added state, given the worse state-neutral retrieval result? Please compare SSE/MoM performance and prefill/decode efficiency at matched state size, specifying MoM's state budget and accounting for sparse routing and SSE's shared parameters.

Response:

1. Our conclusions about routing design come from controlled experiments matched in both state and parameters. In Table 3, state, parameters, slot count, and training recipe are all held constant. Removing key–value binding (Unbound, corresponding to ABC/GSA) reduces RW Avg. from 20.73 to 11.80, while forcing tied read/write routing (Exact-Tied, corresponding to SSE) reduces it to 18.57. These differences are unrelated to state expansion.

2. Comparison with GLA using directly expanded state. We would like to clarify that the motivation for multi-slot routing is to increase state size with fewer parameters and lower peak memory. We therefore compare against a baseline that directly increases GLA's state size: it increases the key dimension (\texttt{expand_k}=2), giving wide-GLA and SimpleRoute (r=16r=16) the same per-layer state size of 524,288. SimpleRoute achieves results comparable to wide-GLA with 18.2% fewer parameters, validating the effectiveness of its design.

Metricwide-GLA (\texttt{expand_k}=2)SimpleRoute (r=16r=16)
State elements per layer524,288524,288
Parameters (M) ↓417.832341.807
Peak decoding memory, batch=8 (MiB) ↓1250.5261098.613
WikiText PPL ↓24.8624.27
LAMBADA PPL ↓26.0424.64
Common Avg.-6 ↑43.7644.33
Real-world Retrieval Avg.-4 ↑26.2626.17

3. Why r=4r=4 performs worse than GLA on real-world retrieval. The r=4r=4 configuration is a degenerate case with only one slot per head and does not represent multi-slot routing. As the reviewer notes in Q6, the read route then reduces to a scalar per head, which is canceled by head-wise RMSNorm. The model effectively retains only write-side routing and lacks the query-conditioned selectivity analyzed in our paper. Even with only write-side routing, WikiText PPL (26.32→24.81), Common Avg. (43.69→44.35), and S-NIAH (50.64→53.93) still improve. When r>Hr>H, the readouts of multiple slots within each head are mixed according to πR\pi^R before entering RMSNorm, preserving read-side selection. Table 3 (r=16r=16) tests routing design precisely in this setting. We will clarify in the revision that only write-side routing remains effective when r=Hr=H.

4. SSE and MoM. SSE and MoM belong to the same class of designs with tied read/write routing (tied selectivity). In terms of routing constraints, MoM relaxes SSE's restriction: SSE uses identical read/write routing weights, making CC symmetric positive semidefinite, whereas MoM ties only the read/write supports, so CMoM=GZ⊤C^{\mathrm{MoM}}=GZ^\top need not be symmetric (Sec. A.2). We therefore use the less constrained MoM as a representative of this class.

The MoM results in Table 1 come from the setting of Du et al. (2026). Since MoM and SimpleRoute have different individual state shapes, we compare capacity using the number of state elements stored per layer per sequence, including shared memory. MoM stores 1,310,720 state elements per layer and activates 786,432 of them for each token; SimpleRoute (r=16r=16) has 524,288 state elements per layer. Despite using a smaller state size, SimpleRoute still outperforms the MoM baseline.

Q6. Methodological novelty How does SimpleRoute differ from dense input/output gating and prior state expansion? Please clarify GDN's beta updates and RMSNorm-cancelled output gating, and provide a matched sigmoid-gating baseline.

Response:

The reviewer views softmax over slots as an interchangeable gating function, but in our framework, softmax is a natural choice for representing query-conditioned selectivity. Query-conditioned selectivity requires the query to highlight relevant tokens while suppressing distractors (see our response to Q1). Softmax over slots entails a trade-off: the more a token writes to certain slots, the less it writes to others. Consequently, tokens written to slots that the query does not read are suppressed (see our response to Q2). Independent sigmoid gates for individual slots lack this constraint, failing to suppress irrelevant tokens.

Difference from β\beta in GDN. This is also the similarity between sigmoid write gates and GDN's β\beta noted by the reviewer: both determine only the strength of a token's write. The coefficient βt\beta_t is independent of subsequent queries and uniformly increases or decreases token tt's contribution for all queries. It cannot highlight the same token for one query while suppressing it for another, and therefore cannot capture the suppression of distractors required by query-conditioned selectivity.

Reviewer_hpvgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SimpleRoute, a routing design for multi-slot linear attention. It argues that existing designs limit how a query selects past tokens, because they either lose the binding between each key and its value (ABC, GSA) or use the same routing to write and to read (SSE, MoM). SimpleRoute adds two independent softmax routers over the per-head states of Gated Linear Attention (GLA). Keys decide where a token is written, and queries decide where the output is read from. Extra states can be added per head at almost no parameter cost. The models are trained from scratch at 340M and 1B parameters and evaluated on language modeling, commonsense reasoning and retrieval. The paper claims the best retrieval among the compared recurrent models at 1B, and that the gains come from the routing design and not only from a larger state.

Critical strengths
  1. Simple, natural design: SimpleRoute adds two independent softmax routers over GLA's per-head states, one on the keys for writing and one on the queries for reading. It is a minimal, natural change that drops into GLA with 0.03% more parameters at 340M (99,072 router parameters), so it is easy to adopt. I like the idea. Its simplicity is also why its novelty is modest. Prior designs already have each half: ABC and GSA route writes and reads separately, and SSE and MoM keep the key–value binding, as the paper's own Sec. 3.3 shows. SimpleRoute combines the two.
Critical weaknesses
  1. State is not matched, so the routing claim is untested: Every comparison in which SimpleRoute beats a non-routed model also gives it more state, both in number of states (recurrent states per layer, heads × slots: 16 for the main 340M model, r = 16, against 4 for GLA) and in state size (state elements per layer: 524,288 against 131,072). The paper defines "state capacity" with both (Sec. 1) but measures only state size (Table 2). More state retrieving better is the expected result (Arora et al., 2024), so these gains do not show that the routing design is better. The only comparison with a non-routed model at equal state (GLA against r = 4) gives mixed results: WikiText perplexity and S-NIAH improve, but real-world retrieval drops 2.00 points. The paper concedes the confound (Sec. 4.2 and 5.4) but still credits the gains to routing (Sec. 5.2 and the conclusion). This is the paper's central claim, and the experiments do not separate it from added state.
  2. The headline results are not controlled: All 1B baselines and 10 of the 12 recurrent baselines at 340M are copied from other papers (App. B.2), which trained and evaluated them with their own setups. These setups differ by as much as the claimed margins. The same 340M GLA scores 23.05 on real-world retrieval in-house and 18.39 in Du et al. (+4.66), against the 6.35-point margin at 1B. At 1B there is no in-house model at all, not even GLA. So the 1B margin, the paper's main result, cannot be attributed to the method.
Questions for authors
  1. [Experiment] Please compare against a non-routed GLA trained with the same recipe at the same state size as r = 16, for example expand_k = 2 (4 heads, 512-dim keys and 256-dim values, 524,288 per layer, the same total key dimension as r = 16). Please also train SimpleRoute (r = 16) with frozen random router codes, and run each over at least 3 seeds. This separates the routing design from the added state.

  2. [Clarification] Please give GSA-s256's heads, slots per head and key/value widths, and its state per layer counted as in Table 3. Its 347.416M parameters exactly match a GSA configuration (fla implementation) with 4 heads, 64 slots per head and 256-dim keys and values, which stores 131,072 per layer: GLA's state size and a quarter of r = 16's. Other layouts also fit the count. B.2 states that it matches SimpleRoute's state capacity, and Sec. 5.2 uses its 3.83-point gap as evidence that the gains come from routing.

  3. [Experiment] Please train an in-house 1B GLA or GDN with the same recipe, or re-score the public 1.3B GLA and GSA checkpoints with your evaluation pipeline. Otherwise, the 1B block should be labelled as uncontrolled.

  4. [Experiment] Please report the mean and standard deviation over at least 3 seeds for GLA, r = 4 and r = 16. Differences between neighbouring configurations are as large as the claimed effects (retrieval falls 1.79 points from r = 8 to r = 16).

5 comments
AuthorsOpenReview ↗

Response to Reviewer hpvg

W1 / Q1. State-matched comparisons State is not matched, so the routing claim is untested. Please compare with equal-state GLA and frozen random routers over at least three seeds.

Response: We would like to clarify that multi-slot routing with shared projections can expand state size with fewer parameters and lower peak memory overhead. The parameter-efficiency is the major motivation of multi-slot routing. In contrast, increasing GLA's key dimension to expand its state also widens the corresponding Q/K projections, adding parameters and storage costs.

Following the reviewer's suggestion, we add a non-routed wide-GLA baseline that directly increases state size, using the same training recipe as SimpleRoute. This baseline uses expand_k=2. It contains 4×512×256=524,2884\times512\times256=524{,}288 state elements per layer, matching the 340M SimpleRoute configuration (r=16r=16) in Table 1.

Metricwide-GLA (equal state size)SimpleRoute (r=16r=16)
State elements per layer524,288524,288
Parameters (M) ↓417.832341.807
Peak decoding memory, batch=8 (MiB) ↓1250.5261098.613
WikiText PPL ↓24.8624.27
LAMBADA PPL ↓26.0424.64
Common Avg.-6 (%) ↑43.7644.33
RW Avg.-4 (%) ↑26.2626.17

At the same total state-storage budget, SimpleRoute achieves performance comparable to wide-GLA with 18.20% fewer parameters and 12.15% lower overall peak decoding memory (batch=8): both perplexities and Common Avg. are better, while the RW averages are close (26.17 vs. 26.26). This validates the effectiveness of organizing states into multiple slots with shared projections to expand state size with lower parameter and memory costs.

W2 / Q3. Controlled 1B comparisons The headline results are not controlled. Please add an in-house 1B GLA/GDN baseline or re-score public checkpoints.

Response: Following the reviewer's suggestion, we add a 1B GDN trained in-house to directly examine the original 1B results against an in-house baseline. The comparison between this additional GDN and the original SimpleRoute model is as follows:

MetricAdditional GDN 1B (in-house)SimpleRoute 1B (original paper)
FDA ↑15.1529.04
SWDE ↑52.5762.83
SQuAD ↑36.2942.23
DROP ↑22.4221.95
Real-World Retrieval Avg.-4 (%) ↑31.6139.01
WikiText PPL ↓17.9616.29
LAMBADA PPL ↓12.8712.54
Common Avg.-6 (%) ↑50.3452.61

Compared with the additional in-house GDN, SimpleRoute improves the real-world retrieval average by 7.40 percentage points, with lower perplexity on both datasets. Among the retrieval tasks, SimpleRoute performs better on FDA, SWDE, and SQuAD. This addition provides direct in-house baseline support for the original 1B results.

Q2. GSA-s256 configuration Please specify heads, slots per head, K/V widths, and state per layer to clarify the claimed capacity match.

Response: SimpleRoute follows GLA's architecture, with the addition of routers, while GSA-s256 is based on GSA. Original GSA has only 64 slots per head; for a fair comparison, we increase this to 256. The architectural differences between GSA and SimpleRoute also require us to control their parameter budgets. GSA-s256 projects a 1024-dimensional input to 4×2564\times256 slot scores, requiring 1024×4×256=1,048,5761024\times4\times256=1{,}048{,}576 projection parameters per layer. SimpleRoute instead uses the existing Q/K representations to compute write and read routing, requiring only 2×16×(128+1)=4,1282\times16\times(128+1)=4{,}128 routing parameters per layer. Therefore, to keep GSA-s256's parameter count close to SimpleRoute's for a fair comparison, we adjust its FFN width from 2816 to 2560. The complete architectural hyperparameters are listed below:

ConfigurationGSA (Original)GSA-s256SimpleRoute (r=16r=16)GLA-340M
Hidden size1024102410241024
Layers24242424
Attention heads4444
Key width per head256256128128
Value width per head256256256256
Slots / routes per head64 slots256 slots4 routes1 state
FFN intermediate width2816256028162816
Parameters per model layer13,110,27213,110,27212,876,57612,872,448
Total parameters347,415,552347,415,552341,806,848341,707,776
State elements per layer per sequence131,072524,288524,288131,072

Dear Reviewer hpvg,

Thank you again for your time and constructive review! As the author–reviewer discussion period is ending soon, we wanted to follow up with an additional controlled experiment directly addressing W2/Q3: our in-house 1B GLA baseline (seed 42, same training recipe), complementing the in-house 1B GDN baseline reported in our earlier response.

Summary: Across both in-house 1B baselines (GDN and GLA), SimpleRoute 1B (m=16m=16) achieves the best performance on almost all reported metrics, including language modeling perplexity, commonsense reasoning, and real-world retrieval averages.

MetricGDN 1B (in-house)GLA 1B (in-house)SimpleRoute 1B (m=16m=16)
FDA ↑15.1523.4129.04
SWDE ↑52.5756.0862.83
SQuAD ↑36.2937.0042.23
DROP ↑22.4222.2821.95
Real-World Retrieval Avg.-4 (%) ↑31.6134.6939.01
WikiText PPL ↓17.9617.1316.29
LAMBADA PPL ↓12.8716.6812.54
Common Avg.-6 (%) ↑50.3451.0552.61

We hope these complete in-house 1B controls (both GDN and GLA) address your concerns, and we would be grateful to know whether any questions remain before the deadline.

Best regards, Authors

Hi Reviewer,

The authors have responded, and the paper is currently borderline. Could you please review their responses, follow up as needed, and revise your review if appropriate? The discussion deadline is in 20 hours.

Thanks!

Reviewer_MEDzOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits the interpretation of multi-slot linear attention mechanisms, arguing that their effectiveness depends not only on recurrent-state capacity but also on the expressivity of the implicit query-conditioned selection operator.

The authors formulate slot routing as a factorization of a token-selection operator and identify two limitations in existing approaches: cross-token selectivity leakage and restrictions induced by tied read/write routing.

Based on this analysis, they introduce SimpleRoute, a routing mechanism designed to alleviate both limitations while preserving linear complexity in sequence length.

Experiments with language models trained from scratch, including models at approximately one billion parameters, demonstrate improvements in retrieval-intensive tasks and competitive performance against existing recurrent architectures.

Critical strengths
  1. Useful conceptual analysis of multi-slot linear attention.

The paper provides a meaningful perspective on the relationship between recurrent memory capacity and selective retrieval.

The factorization of the routing operator clarifies why architectures with comparable state capacity may exhibit different retrieval capabilities. This analysis is potentially valuable independently of the proposed implementation.

  1. Direct connection between architectural diagnosis and empirical improvements.

The proposed modifications follow naturally from the identified limitations. The experiments provide evidence that improved routing selectivity can translate into better retrieval performance, including at a nontrivial model scale.

The connection between the analytical framework, architectural design, and empirical validation is a strength.

Critical weaknesses
  1. The theoretical analysis does not fully characterize the expressive advantage of the proposed architecture.

The factorization of the routing operator provides useful structural intuition, but the relationship between operator rank, routing constraints, and achievable retrieval performance remains only partially characterized.

In particular, removing symmetry or coupling restrictions expands the set of possible selection patterns, but does not by itself establish that these patterns are learnable, efficiently representable, or sufficient for improved retrieval.

A more precise characterization of the expressivity gains would strengthen the theoretical contribution.

  1. The practical efficiency trade-off requires closer examination.

Although SimpleRoute preserves linear complexity in sequence length, asymptotic complexity alone does not establish practical efficiency.

The additional routing mechanisms may introduce computational overhead, memory-access costs, or implementation constraints. The practical significance of the retrieval improvements depends on their relationship to throughput, latency, and recurrent-state memory consumption.

A more comprehensive comparison under matched computational and memory budgets would strengthen the claims.

Questions for authors
  1. Can the authors characterize more precisely the class of selection operators representable by SimpleRoute compared with the tied-routing alternatives, beyond the rank and symmetry observations?
  2. What are the actual training throughput, inference latency, and memory costs relative to the strongest baselines under matched hardware and implementation conditions?
  3. How robust are the reported improvements to changes in the number of slots, sequence length, and retrieval-task difficulty?
4 comments
AuthorsOpenReview ↗

Response to Reviewer MEDz

W1 / Q1. Expressivity and its connection to retrieval performance The theoretical analysis does not fully characterize the expressive advantage of the proposed architecture. Please characterize the class of selection operators representable by SimpleRoute beyond rank and symmetry, including whether these patterns are learnable, efficiently representable, and sufficient for improved retrieval.

Response: We first give a direct construction showing that tied routing with identical read and write assignments cannot represent even a simple symmetric selection matrix that SimpleRoute can represent. Consider the slot-mediated selection factor in Eq. (2), before causal masking:

\begin{pmatrix} 0.18 & 0.82\\\\ 0.82 & 0.18 \end{pmatrix}.$$ This matrix is symmetric and has rank 2. For tied routing, $C=\Pi\Pi^\top$ must be a Gram matrix and therefore satisfy $c_{21}^{2}\le c_{11}c_{22}$. The matrix above instead satisfies $0.82^{2}>0.18^{2}$ and thus cannot be represented by tied routing; increasing the number of slots does not remove this constraint. In contrast, SimpleRoute needs only two slots, with $$\Pi^{\mathrm W}= \begin{pmatrix} 0.9 & 0.1\\\\ 0.1 & 0.9 \end{pmatrix}, \qquad \Pi^{\mathrm R}= \begin{pmatrix} 0.1 & 0.9\\\\ 0.9 & 0.1 \end{pmatrix},$$ to obtain $C=\Pi^{\mathrm R}(\Pi^{\mathrm W})^\top$. This example shows that independent read and write routing allows high cross-token retrieval compatibility to coexist with low self-compatibility, a pattern excluded by tied Gram routing. Under the $j\le i$ causal mask used in the paper, $c_{11},c_{21},c_{22}$ are all retained, so this distinction also survives causal masking. We further test this design choice directly in Table 3. Full SimpleRoute and Exact-Tied use the same slot count and state size, nearly identical parameter counts. Full achieves an RW average of 20.73, compared with 18.57 for Exact-Tied. Its WikiText/LAMBADA perplexities are also better. These results show that the expressivity gained by removing the tie can be exploited during training and translated into practical gains in this controlled setting. **W2 / Q2. Practical efficiency and computational and memory costs** Practical efficiency under matched computational and memory budgets: what are the actual training throughput, inference latency, and memory costs relative to the strongest baselines under matched hardware and implementation conditions? **Response:** We analyze computational and memory overhead in Sec. 4.2 and report the corresponding comparisons in Appendix C, Figure 4. Our efficient routing formulation uses the multi-head architecture, with backbone projection parameters shared across slots. At the 340M scale with $r=16$, the routers add only 99,072 parameters, or 0.03% of the model. Appendix C also separately analyzes recurrent-state expansion and its associated computation. Figure 4 compares Transformer++, GLA, and SimpleRoute under the same hardware conditions. From 2K to 16K, GLA and SimpleRoute retain 98.4%-99.3% of their respective 2K training throughput, whereas Transformer++ retains only 46.8%. Meanwhile, SimpleRoute's overall decoding FLOPs and peak memory remain close to GLA and nearly constant as context length increases. These results show that routing preserves linear attention's linear-time training complexity and context-independent decoding computation and storage costs. Its efficiency advantage over Transformer increases further as context length grows. **Q3. Slot count, sequence length, and retrieval-task settings** Robustness of the reported improvements to changes in the number of slots, sequence length, and retrieval-task difficulty. **Response:** **(1) Slot count.** Table 2 compares $r=4,8,16,64$, showing how language modeling, commonsense reasoning, real-world retrieval, and S-NIAH performance change with the slot budget. Increasing the number of slots improves WikiText perplexity and overall S-NIAH performance. **(2) Sequence length.** Table 2 and Appendix B.7, Table 9 evaluate 1K, 2K, and 4K contexts. With the main configuration fixed at $r=16$, SimpleRoute achieves higher average S-NIAH scores than GLA at all three lengths, demonstrating its effectiveness across these sequence lengths. **(3) Retrieval-task settings.** Table 9 separately reports three S-NIAH configurations with different background texts (noise and natural essays) and retrieved value types (numbers and UUIDs). Our evaluation therefore covers multiple retrieval conditions, and the full results show how the gains vary across task configurations. These results provide evidence across retrieval settings, but do not separately control difficulty factors such as the number of distractors.

Dear Reviewer MEDz,

Thank you again for your time and thoughtful review! As the author–reviewer discussion period is ending soon, we wanted to gently follow up and share additional single-GPU decoding benchmarks addressing W2/Q2 under matched hardware and execution settings.

Setup: Single GPU, BF16 weights, FP32 recurrent states (median across 8 independent GPUs). All four linear-attention models use CUDA Graph; Transformer++ uses Eager execution with dynamic KV cache at 2K context. Latency/throughput are measured at batch = 8 (decode step only, excluding prefill), and peak allocated memory at batch = 1.

ModelState / LayerLatency (ms, ↓)Throughput (tok/s, ↑)Memory (GiB, ↓)
Transformer++— (KV cache)16.6113481.600.9090
GLA131,0722.74842910.810.7557
State-matched524,288
wide-GLA (expand_k=2)524,2883.05942614.930.9395
GSA-s256524,2883.30262422.340.7976
SimpleRoute (r=16)524,2883.05952614.770.7911

Key takeaway: Under the same state capacity (524,288 state elements per layer), SimpleRoute is the most efficient overall—matching wide-GLA in decoding latency/throughput while using 15.8% less peak memory, and running 7.4% faster than GSA-s256 with lower memory.

We hope these measurements resolve your remaining questions, and we would be grateful to know if anything else needs clarification before the deadline.

Best regards, Authors

Hi Reviewer,

The authors have responded, and the paper is currently borderline. Could you please review their responses, follow up as needed, and revise your review if appropriate? The discussion deadline is in 20 hours.

Thanks!

Thank you for the detailed responses and the additional decoding benchmarks. The rank-2 construction gives a concrete example of a selection pattern that SimpleRoute can represent but tied routing cannot. The controlled comparison in Table 3 provides useful evidence that this added flexibility can translate into gains in the evaluated setting.

The new single-GPU measurements also substantially address my efficiency concern. They report latency, throughput, and memory at a matched state capacity. I would interpret them within the stated setup, since the linear-attention models use CUDA Graphs while Transformer++ uses eager execution, and peak memory is measured at a different batch size than latency and throughput.

The responses on slot count, sequence length, and retrieval configurations provide useful robustness evidence. The authors also acknowledge that retrieval difficulty factors such as distractor count were not separately controlled. The theoretical example establishes a specific expressivity difference, while a fuller characterization of the operator class and its learnability remains open.

Overall, the responses strengthen the paper’s empirical and practical case while leaving some limits on the scope of the theoretical and robustness claims. I will maintain my rating of 3 (Weak Accept).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.