- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Gated Slot Attention-2 (GSA2), a two-stage recurrent linear-attention architecture that uses a shared latent-slot representation to connect a key-slot memory and a slot-value memory. The method applies a proposed Gated Oja Rule-2 update to the key-side memory and Gated Delta Rule-2 to the value-side memory. In other words, GSA2 aims at combining Gated Oja Rule-2 for key-side correction with Gated Delta Rule-2 for value-side correction through shared latent slots. Furthermore, this work derives a chunkwise training formulation. GSA2 is evaluated at approximately 1.3B parameters on language modeling, commonsense reasoning, retrieval, and long-context benchmarks.
Critical strengths
S1 - The core architectural idea is coherent and well motivated. The paper identifies an intuitively meaningful distinction between correcting a slot-value association and correcting a key-slot association. Equations 11 and 12 make the intended two-stage design explicit: the shared is the value-like vector for the first memory and the addressing key for the second memory. This is a clean way to turn the slot bottleneck of GSA into an editable associative interface rather than only a gated accumulator.
S2 - The mathematical exposition is generally strong at the recurrence level. Equation 8 is a valid gradient step for the stated objective in Equation 7, and Equations 9 and 10 consistently introduce value-side decay and decoupled erase/write control. The dimensions in the GSA2 instantiation are also compatible: is updated using and , while uses the same slots as addressing keys.
S3 - The experiments are broad for a single architecture paper. Table 2 covers perplexity, LAMBADA accuracy, and eight commonsense tasks; Tables 3 and 4 extend evaluation to retrieval and LongBench; Table 5 directly targets the claimed components. The ablation is particularly useful because it compares coupled versus decoupled gates, correction directions, readout nonlinearities, and slot counts within a common setup.
S4 - The efficiency evidence is informative, even if incomplete. Figure 3 directly shows the intended scaling trade-off: GSA2 drops from 11.82 to 11.32 Kt/s as length rises from 2K to 16K, whereas the Transformer declines from 14.37 to 8.60 Kt/s. The figure supports the narrower claim that the proposed recurrent/hybrid design retains favorable training-throughput scaling with sequence length, while incurring a moderate throughput penalty in the tested settings.
Critical weaknesses
W1. (Important) The proposed "Oja" correction is algebraically a transposed Gated Delta Rule-2 update, which substantially weakens the claimed novelty of the update rule itself.
Equation 8 is correctly derived as gradient descent on , but this is simply the usual delta-rule update for the transposed state and reversed association direction. More importantly, this remains true for the central GSA2 update. Define Transposing Equation 11 gives
(sorry for not displaying correctly in openreview latex support!)
\left[ \boldsymbol{I} - (\boldsymbol{b}^{(1)}_t \odot \boldsymbol{w}_t) \boldsymbol{w}_t^\top \right] + (\boldsymbol{c}^{(1)}_t \odot \boldsymbol{k}_t) \boldsymbol{w}_t^\top$$, which has exactly the GDN2 form in Equation 3 with addressing key $\boldsymbol{w}_t$ and value $\boldsymbol{k}_t$. The important design choice is therefore applying a delta-style associative correction in the reverse direction, from slots to keys, not introducing a fundamentally distinct correction rule. The contribution should be positioned more precisely as applying a GDN2-equivalent recurrence to the transposed first-stage memory, thereby enabling key-side and value-side correction within the shared-slot architecture. W2. **The ablation labels in Table 5 do not adequately disentangle correction direction from update family.** Given the transpose equivalence above, Table 5 should clarify the association directions and exact implementations of the “GDN2 in both updates” and “OJA2 in both updates” ablations. Moreover, GDN2 in both updates achieves better LAMBADA perplexity (10.27 vs. 10.68) and MK-NIAH (13.1 vs. 11.4) than GSA2. These results suggest a task-dependent advantage of the proposed pairing rather than a consistent superiority of complementary correction rules. W3. **The language that GSA2 "consistently improves" over baselines in abstract is stronger than the presented tables support.** Table 2 contains a conspicuous counterexample: hybrid GSA2 has WikiText perplexity 17.23, which is substantially worse than hybrid GSA, 15.98, and worse than every other listed hybrid/recurrent-mixer baseline in that portion of the table. Table 3 also shows that hybrid GSA2, 36.70 average real-world retrieval accuracy, is behind OJA, 37.49, and OJA2, 37.19. The text partly notes that OJA/OJA2 become competitive on the 2K retrieval tasks, but the abstract, introduction, and conclusion should qualify the central claim. The evidence supports that GSA2 is frequently strong, rather than uniformly superior. W4. **State capacity and compute are not controlled tightly enough to isolate the source of gains.** GSA2 has two states with total capacity $M D_k + D_v M = M(D_k+D_v),$ whereas a single-memory baseline has capacity $D_vD_k$. Parameter matching at 1.3B does not imply matched recurrent-state capacity, memory bandwidth, or update cost. The slot-count ablation is useful, but a state-capacity-matched comparison against single-state baselines such as GDN2 would better isolate the contribution of the proposed two-stage memory organization. W5. **Several implementation-critical details are insufficiently specified in the main method.** The standard $1/\sqrt{d}$ scaling is central enough to cause severe degradation in Table 5, yet its exact location and the definition of $d$ are absent from Equations 11 to 13 and the main block description. The paper also does not fully specify gate nonlinearities, gate initializations, low-rank projection rank, chunk size, head/state dimensions, or whether all baselines, including inherited starred results, use exactly matched architectural and optimization settings. W6. The used 1.3B model is rather small in parameters. It would be better if any larger model is tested to support this work.Questions for authors
-
Equation 11 appears exactly equivalent to applying GDN2 to the transposed first-stage state with as the addressing key and as the written value. Do the authors agree with this equivalence? If so, what is the precise conceptual and technical distinction between Gated Oja Rule-2 and a transposed/reversed GDN2 update?
-
Why does hybrid GSA2 have notably worse WikiText perplexity than recurrent GSA2, 17.23 versus 15.69, and worse than hybrid GSA, 17.23 versus 15.98, in Table 2? Is this a reproducible behavior across seeds, an optimization issue, or a trade-off associated with the hybrid layer schedule?
-
Where exactly is the factor applied in the GSA2 block, and what does denote for multi-head/multi-slot configurations? Since removing this factor has one of the largest degradations in Table 5, the mathematical specification should make it explicit.
9 comments
Response to W2: Ablation Interpretation and Correction Directions
We thank the reviewer for this precise observation. We agree that the labels in Table 5 should distinguish the correction direction from the update-rule name, particularly given the transposition relationship discussed in W1/Q1.
The ablations preserve GSA2's two-stage memory organization, shared latent slots, and sequential readout, while changing the recurrent correction applied to each stage.
| Configuration | Stage 1 (Key–Slot) | Stage 2 (Slot–Value) |
|---|---|---|
| GSA2 | OJA2: slot → key | GDN2: slot → value |
| GDN2 in both | GDN2: key → slot | GDN2: slot → value |
| OJA2 in both | OJA2: slot → key | OJA2: value → slot |
Here, the arrows denote the association being corrected, not the direction of the final output. All variants retain the same query-to-slot and slot-to-value readout structure. Because the OJA2/GDN2 update families are related by transposition, changing the update rule in a fixed memory orientation also changes the correction direction. We will make this interpretation explicit rather than claim to isolate these two aspects independently.
We also agree that the complementary pairing does not dominate every individual metric. In particular, GDN2 in both stages achieves better LAMBADA perplexity (10.27 vs. 10.68) and MK-NIAH (13.1 vs. 11.4).
However, the proposed pairing achieves stronger results on the broader set of evaluated capabilities:
| Metric | GSA2 | GDN2 in both | OJA2 in both |
|---|---|---|---|
| WikiText PPL ↓ | 15.69 | 15.92 | 16.97 |
| LAMBADA PPL ↓ | 10.68 | 10.27 | 12.46 |
| Commonsense ↑ | 54.08 | 53.03 | 52.35 |
| S-NIAH ↑ | 72.8 | 72.0 | 60.2 |
| MK-NIAH ↑ | 11.4 | 13.1 | 8.4 |
| RULER Other ↑ | 31.8 | 29.5 | 20.9 |
| LongBench ↑ | 17.3 | 16.8 | 13.8 |
Thus, GSA2 outperforms both alternatives on five of these seven metrics, including Commonsense, RULER Other, and LongBench. The advantage is therefore not uniform task-level superiority, but a stronger overall balance across language modeling, reasoning, retrieval, and long-context understanding.
Our central claim is that complementary key-side and value-side correction is more effective overall than applying the same correction direction to both stages, while retaining task-dependent trade-offs. We will clarify both the ablation configurations and this conclusion in the revised manuscript.
Response to Q2/W3: Hybrid Performance and Calibration of Claims
We thank the reviewer for carefully identifying these task-specific exceptions. We agree that the phrase "consistently improves over strong linear-attention baselines" could be interpreted as claiming superiority over every baseline on every metric, which is broader than our intended conclusion. We clarify the experimental results and will refine this wording.
(1) Hybrid WikiText perplexity and experimental controls (Q2).
We acknowledge that hybrid GSA2 has higher WikiText perplexity (17.23) than recurrent GSA2 (15.69) and hybrid GSA (15.98).
All our GSA2 training runs use the same fixed seed of 42 and identical optimization settings: AdamW (, ), peak learning rate , cosine decay, 1B-token warmup, and 100B training tokens. The results are reported under this fixed-seed protocol, rather than as multi-seed estimates.
The hybrid model follows GDN2's recurrent/SWA interleaving schedule with a 2K sliding window. Thus, the WikiText discrepancy is not caused by different seed settings or optimizer configurations. The architectural change is associated with task-dependent performance differences, rather than uniformly worse modeling capability.
Table A. Language modeling and downstream comparisons.
| Model | Wiki. PPL ↓ | LMB. PPL ↓ | Retrieval ↑ | NIAH ↑ | RULER Other ↑ |
|---|---|---|---|---|---|
| GSA2 (Recurrent) | 15.69 | 10.68 | 31.26 | 42.1 | 31.8 |
| GSA2 (Hybrid) | 17.23 | 10.80 | 36.70 | 50.6 | 36.2 |
| GSA (Hybrid) | 15.98 | 11.36 | 32.89 | 44.7 | 33.0 |
| GDN2 (Hybrid) | 16.26 | 11.93 | 35.41 | 47.7 | 28.5 |
| OJA (Hybrid) | 16.53 | 12.12 | 37.49 | 42.3 | 27.8 |
| OJA2 (Hybrid) | 16.33 | 12.47 | 37.19 | 46.2 | 28.6 |
Despite worse WikiText perplexity, hybrid GSA2 improves LAMBADA accuracy over recurrent GSA2 (51.76 vs. 50.07), NIAH (50.6 vs. 42.1), and RULER Other (36.2 vs. 31.8). Its LAMBADA perplexity also remains better than all evaluated hybrid recurrent-mixer baselines.
These findings support a task-specific trade-off in the evaluated hybrid configuration, rather than a systematic degradation. We do not attribute the WikiText result to a different optimizer or random-seed setting.
(2) Hybrid real-world retrieval and overall performance (W3).
We also acknowledge that hybrid OJA (37.49) and OJA2 (37.19) outperform hybrid GSA2 (36.70) on the 2K real-world retrieval average.
As already discussed in Sec. 4, the 2K input length equals the SWA window, allowing local attention to access relevant evidence directly. This evaluation therefore does not exclusively measure long-range recurrent memory.
Importantly, GSA2 leads both OJA and OJA2 on the broader RULER benchmarks: NIAH averages of 50.6 vs. 42.3/46.2 and RULER Other averages of 36.2 vs. 27.8/28.6. It also outperforms both on LongBench (17.3 vs. 16.2/15.9).
Furthermore, GSA2 improves all five major benchmark-group averages over GSA and GDN2 in both recurrent and hybrid settings, spanning commonsense reasoning, real-world retrieval, RULER NIAH, RULER Other, and LongBench.
These results demonstrate broad improvements against our principal architectural baselines, without implying superiority over every specialized alternative on every individual task.
(3) Clarification of the paper's claims.
We agree that this distinction should be made explicit. We will revise the Abstract, Introduction, and Conclusion to replace the broad "consistently improves" language with:
"At the 1.3B/100B scale, GSA2 improves all five major benchmark-group averages over GSA and GDN2 in both recurrent and hybrid settings, while exhibiting task-specific trade-offs."
We will also explicitly acknowledge the hybrid WikiText perplexity and 2K retrieval exceptions in the experimental discussion.
Thus, our intended claim is broad effectiveness over the principal GSA/GDN2 baselines—not uniform superiority over every method on every metric.
Response to W1/Q1: Algebraic Duality and the Distinction Between OJA2 and GDN2
We thank the reviewer for recognizing the soundness of our derivations (S2) and the coherent two-stage architecture (S1). We agree with the precise algebraic observation: the OJA2 state update is equivalent to a transposed GDN2 recurrence under exchanged key/value roles. However, this equivalence does not extend to the complete input–output operator. We clarify this distinction below.
(1) State-update equivalence versus complete-operator equivalence.
Following Eq. 11, let St∈RM×Dk denote the first-stage memory, wt∈RM the shared slot representation, and kt∈RDk the input key. Writing Dt=Diag(αt(1)), OJA2 updates:
+w_t(c_t\odot k_t)^\top.$$ Setting $T_t=S_t^\top$ gives: $$T_t=T_{t-1}D_t[I-(b_t\odot w_t)w_t^\top] +(c_t\odot k_t)w_t^\top.$$ This is exactly the GDN2 state-update form with $w_t$ as the addressing key and $k_t$ as the written value, under the corresponding gate mapping. We acknowledge this identity. However, the complete operators differ in their readout. OJA2 computes the key-space query-to-slot mapping: $$z_t=S_tq_t=T_t^\top q_t\in\mathbb{R}^M,$$ where $q_t\in\mathbb{R}^{D_k}$. In contrast, standard GDN2 applied to $T_t$ reads: $$y_t=T_tr_t\in\mathbb{R}^{D_k},$$ where $r_t\in\mathbb{R}^M$ is a query in the addressing-key space. **Thus, the same transposed recurrence does not produce the same input–output function.** A one-token example makes this distinction explicit. Let $M=D_k=2$, $S_0=0$, $D=I$, all erase/write gates equal to 1, and scale $=1$. Choose $q=k=(1,0)^\top$ and $w=(0,1)^\top$. Both updates produce the corresponding states $S_1=wk^\top$ and $T_1=kw^\top=S_1^\top$. Nevertheless: $$z_{\mathrm{OJA2}}=S_1q=(0,1)^\top,$$ $$y_{\mathrm{GDN2}}=T_1q=(0,0)^\top.$$ The outputs differ even when all dimensions match. Consequently, **simply exchanging GDN2's key/value inputs cannot reproduce OJA2's forward readout or its output-dependent gradients**. In GSA2, the OJA2 readout additionally passes through SiLU before addressing the second-stage memory (Eq. 13). **(2) Distinct hardware-efficient execution.** The algebraic relationship allows both mechanisms to exploit WY/UT principles, but does not make their complete GPU implementations interchangeable. As derived in Sec. 3.3 and Apps. A.1–A.2, OJA/OJA2 require slot-side causal Gram matrices, gate-aware WY/UT factors, triangular solves, chunk-state propagation, query-to-slot output computation, and corresponding backward gradients. The FLA Triton implementations reflect this distinction: | Component | GDN2 | OJA2 | |---|---|---| | Causal correction | Key-side Gram/WY | Slot-side Gram/WY | | State computation | Delta state kernel | OJA state kernel | | Output | GLA-style readout | Dedicated OJA intra-/inter-chunk readout | | Backward | Delta-specific gradients | OJA readout, state, and Gram/WY gradients | Specifically, GDN2 uses `chunk_gated_delta_rule_fwd_h` and `chunk_gla_fwd_o_gk`, whereas OJA2 uses `chunk_oja2_kkt_fwd`, `recompute_w_u_fwd_oja2`, `chunk_oja_fwd_h`, and `chunk_oja_fwd_o`, with corresponding backward computations. These specialized kernels implement the required reverse-association readout and its gradients while retaining hardware-efficient chunkwise parallelism. **(3) Precise contribution and positioning.** We do not claim that OJA2 introduces a state-transition family algebraically unrelated to GDN2. Its novelty lies in **formulating and efficiently implementing the complementary key-side associative operator**, and integrating it with value-side GDN2 through shared latent slots. The first stage learns key–slot associations and maps input queries into slot space; the second corrects slot–value associations and retrieves the corresponding values. This coordinated two-sided memory organization is the central architectural contribution recognized in S1. We will clarify the transpose relationship explicitly in the manuscript while distinguishing the recurrence-level identity from the complete operator, its GPU algorithms, and its role in GSA2.