acceptodds
Under review as a conference paper at ICLR 2027

Gated Slot Attention-2: Two-Sided Associative Memory Correction in Linear Attention

Abstract

Linear attention models have emerged as efficient alternatives to standard attention, but effectively managing their fixed-size recurrent memory remains challenging. To improve memory, recent work has explored two distinct directions: delta-rule variants for precise correction of values associated with keys, and slot-based architectures such as Gated Slot Attention for modeling key and value memories in two stages. We observe that these directions are complementary–the delta rule provides effective memory correction, while the two-stage structure provides a natural way to operate on both sides of an association. Building on this insight, we introduce a new Gated Oja Rule for key-side correction and extend it with decoupled erase and write control to obtain Gated Oja Rule-2. We then introduce Gated Slot Attention-2 (GSA2), which combines Gated Oja Rule-2 for key-side correction with Gated Delta Rule-2 for value-side correction through shared latent slots. We further derive a hardware-efficient chunkwise algorithm for parallel training. Experiments demonstrate that GSA2 consistently improves over strong linear-attention baselines across benchmarks while retaining linear-time sequence modeling and constant-memory recurrent decoding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Linear Attention, Associative Memory, Delta Rule, Oja Rule

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PeYuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Gated Slot Attention-2 (GSA2), a two-stage recurrent linear-attention architecture that uses a shared latent-slot representation to connect a key-slot memory and a slot-value memory. The method applies a proposed Gated Oja Rule-2 update to the key-side memory and Gated Delta Rule-2 to the value-side memory. In other words, GSA2 aims at combining Gated Oja Rule-2 for key-side correction with Gated Delta Rule-2 for value-side correction through shared latent slots. Furthermore, this work derives a chunkwise training formulation. GSA2 is evaluated at approximately 1.3B parameters on language modeling, commonsense reasoning, retrieval, and long-context benchmarks.

Critical strengths

S1 - The core architectural idea is coherent and well motivated. The paper identifies an intuitively meaningful distinction between correcting a slot-value association and correcting a key-slot association. Equations 11 and 12 make the intended two-stage design explicit: the shared wt\boldsymbol{w}_t is the value-like vector for the first memory and the addressing key for the second memory. This is a clean way to turn the slot bottleneck of GSA into an editable associative interface rather than only a gated accumulator.

S2 - The mathematical exposition is generally strong at the recurrence level. Equation 8 is a valid gradient step for the stated objective in Equation 7, and Equations 9 and 10 consistently introduce value-side decay and decoupled erase/write control. The dimensions in the GSA2 instantiation are also compatible: St(1)∈RM×Dk\boldsymbol{S}^{(1)}_t \in \mathbb{R}^{M \times D_k} is updated using wt∈RM\boldsymbol{w}_t \in \mathbb{R}^M and kt∈RDk\boldsymbol{k}_t \in \mathbb{R}^{D_k}, while St(2)∈RDv×M\boldsymbol{S}^{(2)}_t \in \mathbb{R}^{D_v \times M} uses the same slots as addressing keys.

S3 - The experiments are broad for a single architecture paper. Table 2 covers perplexity, LAMBADA accuracy, and eight commonsense tasks; Tables 3 and 4 extend evaluation to retrieval and LongBench; Table 5 directly targets the claimed components. The ablation is particularly useful because it compares coupled versus decoupled gates, correction directions, readout nonlinearities, and slot counts within a common setup.

S4 - The efficiency evidence is informative, even if incomplete. Figure 3 directly shows the intended scaling trade-off: GSA2 drops from 11.82 to 11.32 Kt/s as length rises from 2K to 16K, whereas the Transformer declines from 14.37 to 8.60 Kt/s. The figure supports the narrower claim that the proposed recurrent/hybrid design retains favorable training-throughput scaling with sequence length, while incurring a moderate throughput penalty in the tested settings.

Critical weaknesses

W1. (Important) The proposed "Oja" correction is algebraically a transposed Gated Delta Rule-2 update, which substantially weakens the claimed novelty of the update rule itself.
Equation 8 is correctly derived as gradient descent on 12∥kt−S⊤vt∥22\frac{1}{2}\|\boldsymbol{k}_t-\boldsymbol{S}^{\top}\boldsymbol{v}_t\|_2^2, but this is simply the usual delta-rule update for the transposed state and reversed association direction. More importantly, this remains true for the central GSA2 update. Define Tt=(St(1))⊤.\boldsymbol{T}_t = \left(\boldsymbol{S}^{(1)}_t\right)^\top. Transposing Equation 11 gives

(sorry for not displaying correctly in openreview latex support!)

\left[ \boldsymbol{I} - (\boldsymbol{b}^{(1)}_t \odot \boldsymbol{w}_t) \boldsymbol{w}_t^\top \right] + (\boldsymbol{c}^{(1)}_t \odot \boldsymbol{k}_t) \boldsymbol{w}_t^\top$$, which has exactly the GDN2 form in Equation 3 with addressing key $\boldsymbol{w}_t$ and value $\boldsymbol{k}_t$. The important design choice is therefore applying a delta-style associative correction in the reverse direction, from slots to keys, not introducing a fundamentally distinct correction rule. The contribution should be positioned more precisely as applying a GDN2-equivalent recurrence to the transposed first-stage memory, thereby enabling key-side and value-side correction within the shared-slot architecture. W2. **The ablation labels in Table 5 do not adequately disentangle correction direction from update family.** Given the transpose equivalence above, Table 5 should clarify the association directions and exact implementations of the “GDN2 in both updates” and “OJA2 in both updates” ablations. Moreover, GDN2 in both updates achieves better LAMBADA perplexity (10.27 vs. 10.68) and MK-NIAH (13.1 vs. 11.4) than GSA2. These results suggest a task-dependent advantage of the proposed pairing rather than a consistent superiority of complementary correction rules. W3. **The language that GSA2 "consistently improves" over baselines in abstract is stronger than the presented tables support.** Table 2 contains a conspicuous counterexample: hybrid GSA2 has WikiText perplexity 17.23, which is substantially worse than hybrid GSA, 15.98, and worse than every other listed hybrid/recurrent-mixer baseline in that portion of the table. Table 3 also shows that hybrid GSA2, 36.70 average real-world retrieval accuracy, is behind OJA, 37.49, and OJA2, 37.19. The text partly notes that OJA/OJA2 become competitive on the 2K retrieval tasks, but the abstract, introduction, and conclusion should qualify the central claim. The evidence supports that GSA2 is frequently strong, rather than uniformly superior. W4. **State capacity and compute are not controlled tightly enough to isolate the source of gains.** GSA2 has two states with total capacity $M D_k + D_v M = M(D_k+D_v),$ whereas a single-memory baseline has capacity $D_vD_k$. Parameter matching at 1.3B does not imply matched recurrent-state capacity, memory bandwidth, or update cost. The slot-count ablation is useful, but a state-capacity-matched comparison against single-state baselines such as GDN2 would better isolate the contribution of the proposed two-stage memory organization. W5. **Several implementation-critical details are insufficiently specified in the main method.** The standard $1/\sqrt{d}$ scaling is central enough to cause severe degradation in Table 5, yet its exact location and the definition of $d$ are absent from Equations 11 to 13 and the main block description. The paper also does not fully specify gate nonlinearities, gate initializations, low-rank projection rank, chunk size, head/state dimensions, or whether all baselines, including inherited starred results, use exactly matched architectural and optimization settings. W6. The used 1.3B model is rather small in parameters. It would be better if any larger model is tested to support this work.
Questions for authors
  1. Equation 11 appears exactly equivalent to applying GDN2 to the transposed first-stage state with wt\boldsymbol{w}_t as the addressing key and kt\boldsymbol{k}_t as the written value. Do the authors agree with this equivalence? If so, what is the precise conceptual and technical distinction between Gated Oja Rule-2 and a transposed/reversed GDN2 update?

  2. Why does hybrid GSA2 have notably worse WikiText perplexity than recurrent GSA2, 17.23 versus 15.69, and worse than hybrid GSA, 17.23 versus 15.98, in Table 2? Is this a reproducible behavior across seeds, an optimization issue, or a trade-off associated with the hybrid layer schedule?

  3. Where exactly is the 1/d1/\sqrt{d} factor applied in the GSA2 block, and what does dd denote for multi-head/multi-slot configurations? Since removing this factor has one of the largest degradations in Table 5, the mathematical specification should make it explicit.

9 comments
AuthorsOpenReview ↗

Response to W1/Q1: Algebraic Duality and the Distinction Between OJA2 and GDN2

We thank the reviewer for recognizing the soundness of our derivations (S2) and the coherent two-stage architecture (S1). We agree with the precise algebraic observation: the OJA2 state update is equivalent to a transposed GDN2 recurrence under exchanged key/value roles. However, this equivalence does not extend to the complete input–output operator. We clarify this distinction below.

(1) State-update equivalence versus complete-operator equivalence.

Following Eq. 11, let St∈RM×DkS_t\in\mathbb{R}^{M\times D_k} denote the first-stage memory, wt∈RMw_t\in\mathbb{R}^M the shared slot representation, and kt∈RDkk_t\in\mathbb{R}^{D_k} the input key. Writing Dt=Diag⁡(αt(1))D_t=\operatorname{Diag}(\alpha_t^{(1)}), OJA2 updates:

+w_t(c_t\odot k_t)^\top.$$ Setting $T_t=S_t^\top$ gives: $$T_t=T_{t-1}D_t[I-(b_t\odot w_t)w_t^\top] +(c_t\odot k_t)w_t^\top.$$ This is exactly the GDN2 state-update form with $w_t$ as the addressing key and $k_t$ as the written value, under the corresponding gate mapping. We acknowledge this identity. However, the complete operators differ in their readout. OJA2 computes the key-space query-to-slot mapping: $$z_t=S_tq_t=T_t^\top q_t\in\mathbb{R}^M,$$ where $q_t\in\mathbb{R}^{D_k}$. In contrast, standard GDN2 applied to $T_t$ reads: $$y_t=T_tr_t\in\mathbb{R}^{D_k},$$ where $r_t\in\mathbb{R}^M$ is a query in the addressing-key space. **Thus, the same transposed recurrence does not produce the same input–output function.** A one-token example makes this distinction explicit. Let $M=D_k=2$, $S_0=0$, $D=I$, all erase/write gates equal to 1, and scale $=1$. Choose $q=k=(1,0)^\top$ and $w=(0,1)^\top$. Both updates produce the corresponding states $S_1=wk^\top$ and $T_1=kw^\top=S_1^\top$. Nevertheless: $$z_{\mathrm{OJA2}}=S_1q=(0,1)^\top,$$ $$y_{\mathrm{GDN2}}=T_1q=(0,0)^\top.$$ The outputs differ even when all dimensions match. Consequently, **simply exchanging GDN2's key/value inputs cannot reproduce OJA2's forward readout or its output-dependent gradients**. In GSA2, the OJA2 readout additionally passes through SiLU before addressing the second-stage memory (Eq. 13). **(2) Distinct hardware-efficient execution.** The algebraic relationship allows both mechanisms to exploit WY/UT principles, but does not make their complete GPU implementations interchangeable. As derived in Sec. 3.3 and Apps. A.1–A.2, OJA/OJA2 require slot-side causal Gram matrices, gate-aware WY/UT factors, triangular solves, chunk-state propagation, query-to-slot output computation, and corresponding backward gradients. The FLA Triton implementations reflect this distinction: | Component | GDN2 | OJA2 | |---|---|---| | Causal correction | Key-side Gram/WY | Slot-side Gram/WY | | State computation | Delta state kernel | OJA state kernel | | Output | GLA-style readout | Dedicated OJA intra-/inter-chunk readout | | Backward | Delta-specific gradients | OJA readout, state, and Gram/WY gradients | Specifically, GDN2 uses `chunk_gated_delta_rule_fwd_h` and `chunk_gla_fwd_o_gk`, whereas OJA2 uses `chunk_oja2_kkt_fwd`, `recompute_w_u_fwd_oja2`, `chunk_oja_fwd_h`, and `chunk_oja_fwd_o`, with corresponding backward computations. These specialized kernels implement the required reverse-association readout and its gradients while retaining hardware-efficient chunkwise parallelism. **(3) Precise contribution and positioning.** We do not claim that OJA2 introduces a state-transition family algebraically unrelated to GDN2. Its novelty lies in **formulating and efficiently implementing the complementary key-side associative operator**, and integrating it with value-side GDN2 through shared latent slots. The first stage learns key–slot associations and maps input queries into slot space; the second corrects slot–value associations and retrieves the corresponding values. This coordinated two-sided memory organization is the central architectural contribution recognized in S1. We will clarify the transpose relationship explicitly in the manuscript while distinguishing the recurrence-level identity from the complete operator, its GPU algorithms, and its role in GSA2.
AuthorsOpenReview ↗

Response to W2: Ablation Interpretation and Correction Directions

We thank the reviewer for this precise observation. We agree that the labels in Table 5 should distinguish the correction direction from the update-rule name, particularly given the transposition relationship discussed in W1/Q1.

The ablations preserve GSA2's two-stage memory organization, shared latent slots, and sequential readout, while changing the recurrent correction applied to each stage.

ConfigurationStage 1 (Key–Slot)Stage 2 (Slot–Value)
GSA2OJA2: slot → keyGDN2: slot → value
GDN2 in bothGDN2: key → slotGDN2: slot → value
OJA2 in bothOJA2: slot → keyOJA2: value → slot

Here, the arrows denote the association being corrected, not the direction of the final output. All variants retain the same query-to-slot and slot-to-value readout structure. Because the OJA2/GDN2 update families are related by transposition, changing the update rule in a fixed memory orientation also changes the correction direction. We will make this interpretation explicit rather than claim to isolate these two aspects independently.

We also agree that the complementary pairing does not dominate every individual metric. In particular, GDN2 in both stages achieves better LAMBADA perplexity (10.27 vs. 10.68) and MK-NIAH (13.1 vs. 11.4).

However, the proposed pairing achieves stronger results on the broader set of evaluated capabilities:

MetricGSA2GDN2 in bothOJA2 in both
WikiText PPL ↓15.6915.9216.97
LAMBADA PPL ↓10.6810.2712.46
Commonsense ↑54.0853.0352.35
S-NIAH ↑72.872.060.2
MK-NIAH ↑11.413.18.4
RULER Other ↑31.829.520.9
LongBench ↑17.316.813.8

Thus, GSA2 outperforms both alternatives on five of these seven metrics, including Commonsense, RULER Other, and LongBench. The advantage is therefore not uniform task-level superiority, but a stronger overall balance across language modeling, reasoning, retrieval, and long-context understanding.

Our central claim is that complementary key-side and value-side correction is more effective overall than applying the same correction direction to both stages, while retaining task-dependent trade-offs. We will clarify both the ablation configurations and this conclusion in the revised manuscript.

AuthorsOpenReview ↗

Response to Q2/W3: Hybrid Performance and Calibration of Claims

We thank the reviewer for carefully identifying these task-specific exceptions. We agree that the phrase "consistently improves over strong linear-attention baselines" could be interpreted as claiming superiority over every baseline on every metric, which is broader than our intended conclusion. We clarify the experimental results and will refine this wording.

(1) Hybrid WikiText perplexity and experimental controls (Q2).

We acknowledge that hybrid GSA2 has higher WikiText perplexity (17.23) than recurrent GSA2 (15.69) and hybrid GSA (15.98).

All our GSA2 training runs use the same fixed seed of 42 and identical optimization settings: AdamW (β1=0.9\beta_1=0.9, β2=0.95\beta_2=0.95), peak learning rate 4×10−44\times10^{-4}, cosine decay, 1B-token warmup, and 100B training tokens. The results are reported under this fixed-seed protocol, rather than as multi-seed estimates.

The hybrid model follows GDN2's recurrent/SWA interleaving schedule with a 2K sliding window. Thus, the WikiText discrepancy is not caused by different seed settings or optimizer configurations. The architectural change is associated with task-dependent performance differences, rather than uniformly worse modeling capability.

Table A. Language modeling and downstream comparisons.

ModelWiki. PPL ↓LMB. PPL ↓Retrieval ↑NIAH ↑RULER Other ↑
GSA2 (Recurrent)15.6910.6831.2642.131.8
GSA2 (Hybrid)17.2310.8036.7050.636.2
GSA (Hybrid)15.9811.3632.8944.733.0
GDN2 (Hybrid)16.2611.9335.4147.728.5
OJA (Hybrid)16.5312.1237.4942.327.8
OJA2 (Hybrid)16.3312.4737.1946.228.6

Despite worse WikiText perplexity, hybrid GSA2 improves LAMBADA accuracy over recurrent GSA2 (51.76 vs. 50.07), NIAH (50.6 vs. 42.1), and RULER Other (36.2 vs. 31.8). Its LAMBADA perplexity also remains better than all evaluated hybrid recurrent-mixer baselines.

These findings support a task-specific trade-off in the evaluated hybrid configuration, rather than a systematic degradation. We do not attribute the WikiText result to a different optimizer or random-seed setting.

(2) Hybrid real-world retrieval and overall performance (W3).

We also acknowledge that hybrid OJA (37.49) and OJA2 (37.19) outperform hybrid GSA2 (36.70) on the 2K real-world retrieval average.

As already discussed in Sec. 4, the 2K input length equals the SWA window, allowing local attention to access relevant evidence directly. This evaluation therefore does not exclusively measure long-range recurrent memory.

Importantly, GSA2 leads both OJA and OJA2 on the broader RULER benchmarks: NIAH averages of 50.6 vs. 42.3/46.2 and RULER Other averages of 36.2 vs. 27.8/28.6. It also outperforms both on LongBench (17.3 vs. 16.2/15.9).

Furthermore, GSA2 improves all five major benchmark-group averages over GSA and GDN2 in both recurrent and hybrid settings, spanning commonsense reasoning, real-world retrieval, RULER NIAH, RULER Other, and LongBench.

These results demonstrate broad improvements against our principal architectural baselines, without implying superiority over every specialized alternative on every individual task.

(3) Clarification of the paper's claims.

We agree that this distinction should be made explicit. We will revise the Abstract, Introduction, and Conclusion to replace the broad "consistently improves" language with:

"At the 1.3B/100B scale, GSA2 improves all five major benchmark-group averages over GSA and GDN2 in both recurrent and hybrid settings, while exhibiting task-specific trade-offs."

We will also explicitly acknowledge the hybrid WikiText perplexity and 2K retrieval exceptions in the experimental discussion.

Thus, our intended claim is broad effectiveness over the principal GSA/GDN2 baselines—not uniform superiority over every method on every metric.

Reviewer_maqSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

GSA2 adds error-correcting updates to both stages of GSA’s slot memory. The key–slot stage uses a new Gated Oja Rule-2, and the slot–value stage uses GDN2’s Gated Delta Rule-2, with the two linked by shared latent slots.

Critical strengths
  • Ablations that target the central claim.
Critical weaknesses
  1. The empirical evidence is too narrow to support the generality of the claims.

    • All results come from a single scale (1.3B/100B tokens) and no seeds/variance are reported. Many reported margins are 0.3–0.8 points on averages, which may be within run-to-run noise.
    • The ablation for slot count is non-monotonic (both 64 and 256 slots), which suggests either sensitivity to hyperparameter tuning or noise. This is especially concerning given the single-scale training.
    • Recurrent state size does not appear to be matched across models. GSA2's state is M×Dk+Dv×MM \times D_k + D_v \times M versus Dv×DkD_v \times D_k for GDN2, and "parameter-matched" does not imply memory-matched.
    • In the hybrid setting the results are mixed. GSA2 has the worst WikiText perplexity of all hybrid models (17.23, worse than recurrent GSA2 at 15.69), despite the claim of "remaining strong on language modeling."
    • The real-world retrieval context (2K) equals the SWA window, so the hybrid numbers barely exercise the recurrent state.
    • RULER only goes up to 8K, 2× the training length, which does not strongly stress long-context memory.
  2. The novelty is limited relative to the added complexity and cost.

    • The proposed "Oja correction" (Eq. 8) is a gradient step on 12∣k−S⊤v∣2\frac{1}{2}|k - S^\top v|^2, which is the standard delta rule applied to S⊤S^\top with the roles of key and value swapped. The chunkwise derivation therefore closely follows existing DeltaNet/GDN2 algorithms.
    • The erase/write decoupling is taken directly from GDN2. The contribution is therefore mainly a composition of GSA, a transposed delta rule, and GDN2.
    • This composition adds a second state, a low-rank slot branch, and six gate types, and costs lower training throughput relative to GDN2.
    • The introduction also does not explain why key-side correction should help a slot memory, so the motivation for this added machinery remains unclear.
Questions for authors

check weaknesses

11 comments
AuthorsOpenReview ↗

Response to W1.1/1.2: Experimental Scale, Run-to-Run Variability, and Slot-Count Sensitivity

We clarify our protocol and contextualize its scale against prior efficient-architecture studies.

(1) Training scale and random seeds.

Our GSA2 experiments follow exactly the same experimental setup as GDN2 [1], as stated in Section 4 and Appendix A.3: approximately 1.3B parameters, 100B FineWeb-Edu tokens, AdamW (peak LR 4×10⁻⁴, weight decay 0.1), gradient clipping 1.0, cosine decay, 1B-token warm-up, and a 0.5M-token global batch. Both use 4K contexts and a 2K SWA window in hybrid models.

Furthermore, our GSA2 and in-house GDN2 runs use the same fixed random seed (42), following the default training protocol of Flash Linear Attention (FLA), a widely used and influential open-source implementation repository for efficient attention and sequence-model architectures. FLA provides hardware-efficient kernels and training-ready implementations of major architectures, including GLA, DeltaNet, GDN, KDA, GDN2, and many others. Its official training launcher defaults to seed=42, and its training parser explicitly applies transformers.set_seed(args.seed). Fixed-seed evaluation is common in compute-intensive architecture research. DeltaNet [4] states in its NeurIPS reproducibility checklist that it omits error bars because repeated large-model training is computationally expensive. NHA [3] reports a representative seed-42 run and notes consistent repeated runs; Comba [2] also uses seed 42. Multi-seed full-scale pretraining would require substantial additional compute.

Importantly, our ablations use the same 1.3B/100B scale as our main comparisons. GSA2 and GDN2 therefore have the largest ablation scale in this comparison; several other works use substantially smaller models and budgets. Our architectural choices are tested directly in the target model regime.

Model (venue)Ablation scaleMain comparison
GLA (ICML'24) [5]340M/7B1.3B/100B
DeltaNet (NeurIPS'24) [4]340M/15B340M/15B; 1.3B/100B
GSA (NeurIPS'24) [6]340M/10B1.3B/100B; 2.7B/100B
Mamba-2 (ICML'24) [7]125M–380M / 2.5B–7B1.3B/2.7B
GDN (ICLR'25) [8]400M/15B1.3B/100B
Comba (NeurIPS'25) [2]340M/15B; selected 1.3B/100B340M/15B; 1.3B/100B
NHA (ACL'26) [3]340M/15B340M/15B; 1.3B/100B
Mamba-3 (ICLR'26) [9]440M/8.8B–17.6B180M/440M/880M/1.5B; 100B
GDN2 (NeurIPS'26) [1]1.3B/100B (gate ablations)1.3B/100B
GSA2 (ours)1.3B/100B1.3B/100B

(2) Magnitude and consistency of improvements.

The claim that improvements are mostly 0.3–0.8 points does not reflect the full results. Across five multi-task benchmark groups, GSA2 improves over GDN2 [1]:

Benchmark groupTasks/settingsRecurrent ΔHybrid Δ
Commonsense reasoning10+0.97+1.31
Real-world retrieval6+1.38+1.29
RULER NIAH24+2.4+2.9
RULER other tasks28+5.8+7.7
LongBench11+0.5+0.9

In particular, the 5.8–7.7-point gains on RULER non-NIAH tasks, alongside NIAH and real-world retrieval gains, demonstrate benefits beyond marginal average differences.

(3) Slot-count sensitivity.

We investigate slot count because prior work shows that memory capacity affects quality and efficiency. GSA [6], for example, reports:

SlotsWikiText PPL ↓Throughput (K tokens/s)
3213.7446.7
6413.5144.1
12813.4637.1

Slot count is therefore an important architectural choice, not an arbitrary tuning variable. There is no theoretical requirement for memory performance to increase monotonically with slot count: additional capacity changes memory representation, optimization, and interference, and need not improve every task.

Our 1.3B/100B ablation confirms this trade-off:

SlotsCommon Avg.S-NIAHMK-NIAHLongBench
6453.4862.68.216.9
12854.0872.811.417.3
25652.8761.711.115.3

128 slots gives the best overall results. This non-monotonic outcome is precisely why capacity ablation is necessary; it is not evidence of uncontrolled tuning or noise.

References

[1] Hatamizadeh et al. Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention. NeurIPS 2026.

[2] Hu et al. Comba: Improving Bilinear RNNs with Closed-Loop Control. NeurIPS 2025.

[3] Native Hybrid Attention. ACL 2026.

[4] Yang et al. Parallelizing Linear Transformers with the Delta Rule over Sequence Length. NeurIPS 2024.

[5] Yang et al. Gated Linear Attention Transformers with Hardware-Efficient Training. ICML 2024.

[6] Zhang et al. Gated Slot Attention for Efficient Linear-Time Sequence Modeling. NeurIPS 2024.

[7] Dao and Gu. Transformers are SSMs. ICML 2024.

[8] Yang et al. Gated Delta Networks: Improving Mamba2 with Delta Rule. ICLR 2025.

[9] Mamba-3: Improved Sequence Modeling Using State Space Principles. ICLR 2026.

AuthorsOpenReview ↗

Response to W1.3: Memory Capacity and Comparison Fairness

We additionally trained GDN2-256 at 1.3B/100B, using the same training and evaluation settings as GSA2, to directly test whether greater recurrent-state capacity explains our gains.

(1) Memory organization.

GDN2 uses a single key-addressed state. GSA2 builds on GSA's two-stage memory: a key--slot state S(1)∈RM×DkS^{(1)} \in \mathbb{R}^{M \times D_k} and a slot--value state S(2)∈RDv×MS^{(2)} \in \mathbb{R}^{D_v \times M}, enabling complementary error correction on both sides.

We recognize that GSA2 stores more recurrent-state elements than GDN2-128. GDN2-256 provides an expanded-capacity control to investigate whether enlarging a single-stage associative memory can reproduce GSA2's improvements.

(2) Experimental results.

Table A. Overall benchmark averages (↑).

BenchmarkGDN2-128GDN2-256GSA2Δ vs. 256
Commonsense (9)53.1152.6754.08+1.41
Retrieval (6)29.8830.8431.26+0.42
RULER NIAH (24)39.741.3242.1+0.78
RULER other (28)26.030.3631.8+1.4
LongBench (11)16.816.517.3+0.8

Table B. Language modeling (PPL ↓) and reasoning (accuracy ↑).

MetricGDN2-256GSA2
WikiText PPL16.4415.69
LAMBADA PPL10.7710.68
LAMBADA acc50.2850.07
PIQA72.6373.56
HellaSwag acc_norm57.6859.28
WinoGrande57.7060.77
ARC-e71.7672.77
ARC-c36.0138.48
OBQA29.4030.00
SIQA37.4141.30
BoolQ61.1660.46
Avg (9)52.6754.08

Table C. Real-world retrieval, 2K (↑).

TaskGDN2-256GSA2
SWDE29.0729.97
SQuAD38.5736.29
FDA18.1519.42
TriviaQA61.9763.09
NQ16.2517.26
DROP21.0321.51
Avg30.8431.26

Table D. LongBench (↑).

TaskGDN2-256GSA2
MultiFieldQA14.9513.0
Qasper6.645.8
HotpotQA6.985.5
2WikiMQA9.278.3
GovReport10.7910.0
MultiNews12.6411.6
SAMSum29.9827.8
TREC32.5046.5
TriviaQA16.0514.7
LCC19.8022.4
RepoBench-P21.9524.2
Avg (11)16.517.3

Tables E–F. RULER accuracy (%), each cell GDN2-256 / GSA2.

Table E. NIAH (24 settings).

Task1K2K4K8K
S-NIAH-1100/100100/10099.8/99.690/86
S-NIAH-296/10099.6/99.278.6/70.435.6/30.2
S-NIAH-380.4/9247/59.69.6/276.2/9.6
MK-NIAH-148.4/45.239/39.828/2627.6/23.2
MK-NIAH-24.6/1.81/0.60.2/00/0.2
MK-NIAH-30/00/00/00/0

Mean (24): 41.32 / 42.10.

Table F. Other RULER tasks (28 settings).

Task1K2K4K8K
MQ82.55/7767.75/57.424.55/30.69.95/10.4
MV79.70/73.364.45/60.931.95/43.616.70/14.7
CWE59.78/66.319.54/41.132.32/40.710.82/19.5
FWE39.53/42.547.93/4443.53/39.339.93/33
HPQA25.4/2017/16.412/11.414.6/12.8
SQuAD30.2/22.623.8/15.418.4/13.615.4/10.8
VT12.08/30.26.36/22.73.64/19.20.32/0.4

Mean (28): 30.36 / 31.8.

(3) Implications for memory-capacity fairness.

Expanding GDN2 improves some retrieval scores, yet GSA2 retains higher averages across all five benchmark groups. It also achieves lower WikiText and LAMBADA perplexity, higher accuracy on seven of nine reasoning metrics, and improvements on five of six real-world retrieval tasks.

These results demonstrate that simply enlarging GDN2's single-stage recurrent memory does not reproduce GSA2's overall performance. The key distinction is memory organization: GDN2 corrects values within a single key-addressed state, whereas GSA2 performs complementary key-side and value-side correction through shared latent slots.

Together with our existing correction-rule ablations (Table 5), the new full-scale experiments provide further evidence that two-sided associative correction offers benefits beyond increasing recurrent-state capacity alone.

AuthorsOpenReview ↗

Response to W1.4/1.5/1.6: Hybrid Language Modeling and Long-Context Evaluation

We clarify the evaluation scope and compare our protocol with established efficient-architecture studies.

(1) Hybrid WikiText perplexity.

We acknowledge that hybrid GSA2 has higher WikiText perplexity (17.23) than its recurrent counterpart (15.69). However, this single metric does not characterize the architecture's overall language modeling and downstream performance. Hybrid GSA2 achieves the best LAMBADA perplexity, LAMBADA accuracy, and commonsense average among the evaluated hybrid models, alongside consistent improvements on retrieval and long-context benchmarks.

Table A. Hybrid GSA2 versus GDN2.

MetricGDN2GSA2Improvement
WikiText PPL ↓16.2617.23−
LAMBADA PPL ↓11.9310.80−1.13
LAMBADA Acc ↑49.4951.76+2.27
Commonsense Avg ↑52.6553.96+1.31
Real-world retrieval ↑35.4136.70+1.29
RULER NIAH ↑47.750.6+2.9
RULER Other ↑28.536.2+7.7
LongBench ↑16.417.3+0.9

These results demonstrate that GSA2's benefits extend well beyond WikiText, particularly in associative retrieval and long-context understanding. We will refine the language-modeling description to distinguish individual perplexity results from overall performance.

(2) Real-world retrieval at 2K.

Our evaluation follows the protocol of GDN2 [1], including the same six real-world retrieval tasks, 2K context length, and 2K SWA window in hybrid models.

This setting is well established in efficient-architecture research. GDN [2] and GSA [3] also evaluate real-world retrieval with 2K inputs. Mamba-3 [4] likewise adopts a 2K retrieval protocol, evaluating both recurrent and hybrid architectures.

The purpose of this benchmark is to evaluate real-world retrieval capability under standardized conditions, not exclusively to isolate information beyond the local attention window. SWA can contribute to retrieval within 2K, which is precisely why we additionally evaluate recurrent-only models and longer-context RULER tasks.

Notably, recurrent GSA2, without SWA, improves real-world retrieval over GDN2 from 29.88 to 31.26. The separate RULER experiments further assess memory beyond the 2K window. These evaluations provide complementary evidence rather than duplicating the same capability.

(3) RULER evaluation length and coverage.

Our RULER protocol is consistent with established efficient-architecture research. Importantly, the original papers use different long-context evaluations, and 8K is already a standard extrapolation target for comparable 1.3B recurrent models.

Table B. Representative original-paper evaluation protocols.

ModelRetrieval / long-context evaluation
Mamba [5]Selective Copying; Induction Heads
Mamba-2 [6]MQAR: 256–1024
GSA [3]MQAR: 512; real-world retrieval: 2K
GDN [2]RULER S-NIAH up to 8K; PPL extrapolation to 20K
Mamba-3 [4]NIAH up to 4K
GDN2 [1]Selected RULER NIAH up to 8K
NHA [7]Four RULER tasks, up to 8K
GSA2 (ours)13 RULER tasks, all evaluated at 1K–8K

These are different experimental protocols; synthetic MQAR, perplexity extrapolation, and RULER accuracy are not interchangeable measures.

In particular, GDN2 evaluates selected S-NIAH and MK-NIAH cases, with only S-NIAH-1/2 reaching 8K. NHA evaluates four RULER task groups, with only Hotpot reaching 8K.

Our GSA2 evaluation is substantially broader, covering:

  • 24 NIAH settings: Six single-/multi-key variants × four lengths.
  • 28 additional RULER settings: MQ, MV, CWE, FWE, HPQA, SQuAD, and VT × four lengths.

Thus, GSA2 systematically evaluates 52 task-length settings at 1K, 2K, 4K, and 8K. The maximum length is twice the 4K training context and four times the 2K SWA window, providing direct evidence of zero-shot length extrapolation and retrieval beyond local attention.

Across these settings, GSA2 outperforms GDN2 on average in both recurrent and hybrid configurations (Table A; Appendix Tables 6–7).

Overall, our evaluation follows established protocols while providing broader RULER task coverage. The results substantiate GSA2's improvements at the evaluated lengths without requiring claims about untested extreme-context regimes.

References

[1] Hatamizadeh et al. Gated DeltaNet-2. NeurIPS 2026.

[2] Yang et al. Gated Delta Networks. ICLR 2025.

[3] Zhang et al. Gated Slot Attention. NeurIPS 2024.

[4] Lahoti et al. Mamba-3. ICLR 2026.

[5] Gu and Dao. Mamba. 2023.

[6] Dao and Gu. Transformers are SSMs (Mamba-2). ICML 2024.

[7] Du et al. Native Hybrid Attention. ACL 2026.

Reviewer_pewNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Gated Slot Attention-2 (GSA2), combining Gated Oja Rule-2 for key-side correction and Gated Delta Rule-2 for value-side correction through shared latent slots. Both stages employ independent memory gates, while a WY/UT chunkwise formulation enables parallel training.

Experiments on 1.3B-parameter models trained with 100B tokens cover language modeling, reasoning, retrieval, and long-context tasks. GSA2 improves several aggregate metrics and maintains favorable throughput scaling, but incurs additional computation relative to GDN2.

Critical strengths
  • GSA2 combines key- and value-side correction through shared slots, outperforming GSA and GDN2 on several language modeling and retrieval metrics (Tables 2–3).

  • The WY/UT chunkwise formulation supports parallel training, while ablations and throughput measurements demonstrate useful design choices and favorable sequence-length scaling (Table 5, Figure 3).

Critical weaknesses
  • GSA2 changes memory updates, gates, and readout activation simultaneously, making their individual contributions unclear. Table 5 shows substantial gains from SiLU, but lacks GSA + SiLU and single-sided correction controls under the same backbone.

  • Long-context evaluation settings are insufficiently specified. Models train at 4K, yet Transformer scores zero across all six 8K NIAH variants, raising questions about length extrapolation and evaluation configurations. Moreover, GSA2's LongBench advantage over GDN2 (17.3 vs. 16.8) is largely driven by TREC; excluding it reverses the ranking (14.33 vs. 14.77).

  • Reported retrieval results are inconsistent: recurrent GSA2 scores 31.26 in Table 3 but 31.51 under the default configuration in Table 5. The discrepancy requires clarification.

Questions for authors
  • Could the authors compare no correction, first-stage-only, second-stage-only, and two-sided correction under matched architectures and training budgets, including GSA + SiLU?

  • Why does the Transformer score zero on all 8K NIAH settings (Table 6)? Please clarify positional encoding, length extrapolation, truncation policies, and results with length-adapted attention baselines.

  • Why do Tables 3 and 5 report different GSA2 retrieval scores (31.26 vs. 31.51)? Are the GDN2 baselines evaluated under identical training and evaluation conditions?

6 comments
AuthorsOpenReview ↗

Response to W1/Q1: Isolating update-rule and readout contributions

We thank the reviewer for pointing out the importance of disentangling readout and memory-update contributions. We address these two aspects separately.

(1) Readout activation selection.

We retain the best-supported readout configuration for each architecture rather than imposing the same activation on both. The original GSA [1] study (Table 2, Sec. 4.1.3) explicitly investigated the effect of replacing Softmax after the first recurrent pass. Its ablation, conducted on 340M models trained with 10B SlimPajama tokens, reports:

GSA readout configurationWikiText PPL ↓
Original GSA (Softmax)13.51
Without Softmax14.03
Replace Softmax with Swish (SiLU)13.71
Replace Softmax with ReLU13.69
Replace Softmax with ReLU²13.95

These results support retaining Softmax as the canonical GSA readout. Accordingly, we do not introduce a separate GSA+SiLU training run, as this activation substitution has already been investigated in the original GSA work. For GSA2, our own Table 5 independently establishes SiLU as the better choice: it outperforms Softmax in common-sense reasoning (54.08 vs. 50.66), MK-NIAH (11.4 vs. 9.7), and LongBench (17.3 vs. 14.0). Therefore, we adopt the empirically validated best configuration for each architecture to ensure a fair comparison.

(2) Isolating associative correction mechanisms.

We agree that single-sided controls provide a more direct assessment of the contributions of the two correction mechanisms. Our existing 1.3B/100B experiments already provide the no-explicit-correction GSA baseline and the full two-sided GSA2 configuration. We are currently conducting two additional 1.3B/100B ablations:

VariantStage 1 (Key–Slot)Stage 2 (Slot–Value)Status
GSA (no explicit correction)GSA gated accumulationGSA gated accumulationCompleted
Stage-1-onlyOJA2GSA gated accumulationIn progress
Stage-2-onlyGSA gated accumulationGDN2In progress
Full GSA2OJA2GDN2Completed

The two additional variants retain the GSA2 backbone, SiLU readout, and the same 1.3B/100B training budget, changing only the update rule in the designated stage. These comparisons will help isolate the individual contributions of key-side and value-side correction and assess their complementarity. We will report the results once both experiments are completed.

References

[1] Zhang et al. Gated Slot Attention for Efficient Linear-Time Sequence Modeling. NeurIPS 2024.

AuthorsOpenReview ↗

Response to W2/Q2: 8K Transformer Evaluation and LongBench

We sincerely thank the reviewer for the constructive suggestions. We clarify the extrapolation protocol and examine LongBench task sensitivity.

(1) Transformer length extrapolation.

The Transformer was trained at 4,096 tokens with full-dimensional RoPE (base 10,000). At 8K, we extend the RoPE cache without changing its frequencies or performing long-context training. This deliberately evaluates zero-shot length extrapolation beyond the training context.

The evaluator retains the entire input prompt, using full causal attention and KV-cached decoding, without truncation or sliding-window restrictions. The RULER NIAH 8K budget reserves 128 tokens for generation; over-budget inputs raise errors rather than being silently truncated.

Following the reviewer's suggestion, we additionally tested inference-only 2× linear position interpolation using the same checkpoint and evaluation data (500 examples per task and length). Scaling RoPE positions from pp to p/2p/2 compresses the 8K positional range into approximately the original 4K training range, without discarding any input tokens. This reduces positional extrapolation beyond the training range while preserving full 8K attention.

Table A. Transformer accuracy (%) on six 8K NIAH tasks.

RoPES-NIAH-1S-NIAH-2S-NIAH-3MK-NIAH-1MK-NIAH-2MK-NIAH-3
Original000000
2× interpolation000000

Interpolation did not recover 8K accuracy. These results reflect the limited length-generalization ability of this particular 4K-trained Transformer under the tested extrapolation settings, rather than an evaluation artifact or an intrinsic limitation of attention.

(2) LongBench performance and TREC sensitivity.

We appreciate the reviewer's observation and acknowledge that TREC substantially influences the recurrent average. However, TREC evaluates a meaningful aspect of long-context modeling closely related to our motivation. LongBench [1] (Sec. 3.2) identifies few-shot in-context learning as an important long-context application and uses TREC with 100–600 in-context examples across 50 fine-grained question classes. Unlike simple retrieval, this task requires learning from numerous example–label associations distributed across a long context and applying the relevant associations to unseen questions despite competing examples. This makes TREC particularly relevant to evaluating associative memory and interference under many in-context examples, which are central motivations of GSA2.

Thus, GSA2's TREC improvements (recurrent: 46.5 vs. 37.0; hybrid: 49.0 vs. 46.0) are meaningful, although they do not alone establish the mechanism's effectiveness.

Table B. LongBench mean scores (↑).

SettingGDN2GSA2
Recurrent, 11 tasks16.8017.30
Recurrent, excluding TREC14.7714.33
Hybrid, 11 tasks16.4017.30
Hybrid, excluding TREC13.4514.12

We agree that excluding TREC reverses the recurrent ranking. However, hybrid GSA2 retains its advantage, outperforming GDN2 on 7 of the remaining 10 tasks.

Table C. LongBench tasks favoring GSA2, excluding TREC.

SettingTaskGDN2GSA2
RecurrentGovReport7.910.0
RecurrentMultiNews11.411.6
RecurrentTriviaQA14.214.7
HybridMultiFieldQA15.115.8
HybridHotpotQA5.66.5
Hybrid2WikiMultiHopQA9.610.4
HybridGovReport5.16.4
HybridSAMSum23.030.1
HybridTriviaQA13.715.0
HybridLCC21.222.2

The hybrid improvements span single- and multi-document QA, summarization, few-shot learning, and code completion.

Importantly, GSA2's improvements also extend beyond LongBench. Across other benchmark suites, GSA2 outperforms GDN2 in both recurrent and hybrid settings:

Table D. Broader benchmarks: average accuracy (%) ↑.

BenchmarkRec. GDN2Rec. GSA2Hyb. GDN2Hyb. GSA2
Commonsense53.1154.0852.6553.96
Real-world retrieval29.8831.2635.4136.70
RULER NIAH (24 settings)39.742.147.750.6
RULER other (28 settings)26.031.828.536.2

These results show that GSA2's benefits are not confined to TREC, but extend across commonsense reasoning, synthetic retrieval, and real-world retrieval. We will clarify the TREC sensitivity of the recurrent LongBench average without claiming uniform superiority.

Reference

[1] Bai et al. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. ACL 2024.

AuthorsOpenReview ↗

Response to W3/Q3: Retrieval-score Discrepancy and GDN2 Baseline Fairness

We sincerely thank the reviewer for carefully identifying this inconsistency. Upon rechecking our experimental records, we confirmed that the discrepancy resulted from an inadvertent error when preparing Table 5. The correct results are:

Stage 1Stage 2Real-world Retrieval Avg.
CoupledDecoupled31.51
DecoupledDecoupled (GSA2)31.26

Therefore, the 31.26 reported for GSA2 in Table 3 is correct, while the corresponding entries in Table 5 require correction. We apologize for this oversight and will correct Table 5 in the revised manuscript. This is solely a reporting error and does not affect the underlying experimental results.

Regarding GDN2 baseline fairness, our all experiments follow the same training and evaluation protocols as GDN2 [1], as specified in Section 4 and Appendix A.3. Both use approximately 1.3B parameters, 100B FineWeb-Edu tokens, 4K training sequences, AdamW with a peak learning rate of 4×10⁻⁴, weight decay of 0.1, gradient clipping at 1.0, cosine scheduling with a 1B-token warm-up, and a global batch size of 0.5M tokens. Hybrid models also use the same 2K SWA window. The GDN2 baseline is compared under the same benchmark settings and evaluation protocols, ensuring consistent training budgets and comparable evaluation conditions.

Reference

[1] Hatamizadeh et al. Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention. NeurIPS, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.