acceptodds
Under review as a conference paper at ICLR 2027

RSA: Recursive Sparse Attention with Hierarchical Deep–Shallow Memory and Sparse Activation

Abstract

Due to their linear complexity, linear sequence models excel in long-context modeling but underperform on tasks requiring long-range reasoning and retrieval compared to standard Transformers. Recent efforts using gating, delta rules, or multi-state models still mainly focus on short-range information, limiting effectiveness in complex long-sequence tasks due to insufficient information utilization and limited effective memory capacity. To address this, we propose a bio-inspired shallow-deep memory architecture, where multiple memory states are structurally interconnected and stacked to achieve exponentially large equivalent capacity: shallow memory stores coarse-grained representations and deep memory stores residual information. A correlation-guided sparse readout mechanism enables precise information retrieval from the stored states. Co-designing memory storage and retrieval, we introduce Recursive Sparse Attention (RSA), an innovative attention mechanism that bridges the gap between linear models and standard attention. Experiments show that RSA, to some extent, counters the skepticism that linear models may not scale to large parameter sizes, achieving performance comparable to or surpassing state-of-the-art linear and Transformer models on multiple benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Fine-tune transformer to Linear Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_329sOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RSA to improve long-context retrieval in linear sequence models while keeping a fixed-size memory. The motivation is that these models may lose useful information during compression and mix relevant and irrelevant information during readout. To address this, they use a deep–shallow memory structure, where shallow levels store coarse representations and deeper levels store the residual information. They also use a sparse readout mechanism that uses shallow-level relevance to guide reading from deeper memory. They evaluate RSA on common-sense reasoning, retrieval, and long-context tasks using both models trained from scratch and models converted from pretrained Transformers, and compare its performance with Transformer and other linear model baselines.

Critical strengths

The idea of organizing linear-attention memory as a coarse-to-fine residual hierarchy with relevance-guided readout is intuitive, and the paper evaluates it across a wide range of scales and benchmarks.

Critical weaknesses
  1. The experiments provide limited evidence for the main contribution of this work. The ablation study in Table 9 shows that RSA without sparse read is just PLA, and RSA improves over PLA by only 0.37 points on average. This result is based on a single run with no variance reported, making it difficult to assess the robustness of the improvement.
  2. The paper claims that “Across all scales, RSA matches or exceeds existing linear models and Transformer baselines on common-sense reasoning benchmarks.” However, Table 2 shows that RSA-3B-CPT scores 47.6 on MMLU, substantially lower than Qwen2.5-3B-CPT’s score of 55.6.

The paper contains many presentation errors and inconsistencies. For example:

  1. Tables 3 and 7 are identical.
  2. The 360M model is referred to as 340M in Appendix C.1.
  3. RSA-7B-CPT’s MMLU score is reported as 59.3 in Table 2 but 60.2 in Table 10.
  4. Figure 2 and Tables 6–10 come after the references, although the main text relies on them for its retrieval and ablation claims.
Questions for authors
  1. Since sparse readout is the main component distinguishing RSA from PLA, it would be helpful to report the mean and standard deviation across at least three random seeds for both RSA and the “w/o sparse read” variant in Table 9.
  2. Table 2 shows that RSA underperforms the matched Qwen control on MMLU at every scale, while improving performance on common-sense reasoning tasks. How do the authors explain this discrepancy?
4 comments
AuthorsOpenReview ↗

First of all, thank you very much for your detailed feedback and efforts. We will provide a detailed response to the questions raised by you and the other reviewers to acknowledge your careful and thorough approach.

If you are not interested in our field or our paper, but are only interested in specific issues, you may skip the following paragraphs and start reading directly from the “Weaknesses” section to save time.

Paper Background

Apart from an early accidental submission, this is the first time I have received initial reviews in which every reviewer leaned against acceptance, I knew full well that the outcome would not be ideal. However, this only fueled my enthusiasm and passion. I used the discussion period to wholeheartedly promote our field and our paper to the reviewers and other readers, and I responded in detail to the reviewers’ thoughtful comments and questions. Because ICLR reviews are public, we are concerned that low initial scores may shape how readers assess this work and later versions. We will respond carefully to the reviewers’ comments and welcome further questions. We previously submitted this work to ICML and NeurIPS. Across those submissions, the review comments gradually improved from a situation where only one reviewer gave a positive rating(ICML) to one where only one reviewer expressed reservations, with all relevant ratings at 5 or higher(NeurIPS). We saw this as progress, while recognizing that incorporating feedback across rounds introduced some new wording and labeling problems (as you pointed out, some flaws were introduced while incorporating previous reviewer comments, for which we are very grateful). In the previous NeurIPS round, the initial reviews made us optimistic, but the wording and tone of a subsequent exchange with one reviewer led to procedural action and a desk rejection. We had expected a favorable initial outcome this time, but instead encountered our first instance of all-negative scores, which has left us feeling disheartened and disappointed. Nevertheless, this remains an opportunity to further refine our paper and to promote our field and work.

Motivation, contributions, and longer-term direction

We study how to build competitive fully linear sequence models and whether they can support architectures and training paradigms beyond the current Transformer design, with the potential to outperform it on some tasks. Training from scratch lets a model adapt to its linear state mechanism throughout training and may yield better results than conversion. Given the cost of large-scale pretraining, converting an existing LLM through controlled continued pretraining offers a less expensive setting for comparison. Our contributions have two main aspects.

  1. Architecture. First, we present a unified approach to state expansion by unifying MVA, GPA (referring to its earlier version, PLA), and MoM. We link state expansion to multi-layer Transformers, highlighting the connection between their recurrent residual structure and the attention–MLP stacks of Transformers. Drawing on the reconstruction perspective in MVA and PLA/GPA, we analyze the approximation error and specify the coding schemes along with their applicable conditions. RSA also employs a data-dependent cross-level readout, where shallow-level relevance constrains deeper retrieval and reduces the unrelated content aggregated by dense readouts (GPA or full attention). These two core operations constitute our main contributions. Regarding the contribution of sparse attention, its primary advantage lies in suppressing noise in long sequences; therefore, the RSA–PLA retrieval comparison in Table 12, the task-finetuning results in Table 16, and other length-extrapolation experiments demonstrate its advantages over PLA and full-attention models (such as Mistral and Qwen) on the tested tasks. We also provide the corresponding analysis and kernel implementations in this paper.

  2. Results across scales. We report training from scratch at 360M and 1.3B parameters and conversion from pretrained Qwen2.5 at 3B, 7B, and 14B. We believe these results provide the first empirical answer across scales to whether fully linear sequence models can scale in parameter count. We can also provide results at 32B and 70B if helpful. The same recurrent storage and readout design works across several scales. After conversion, fully linear models approach strong same-scale softmax attention baselines on several tasks and surpass them on some. We view training data as one of the main factors behind the MMLU gap you identified. As the continued pretraining (CPT) budget grows, RSA’s scores generally improve, while the Qwen control fluctuates below its original checkpoint score. The gap gradually narrows. At 7B parameters and 80B CPT tokens, RSA scores 61.5 and Qwen2.5 scores 61.0. To our knowledge, RSA is the first architecture to show across multiple parameter scales that fully linear models can approach or exceed Qwen2.5, a leading softmax attention baseline at the time.

Based on these architectural, theoretical, and empirical contributions across scales, we believe the work meets ICLR’s publication standard.

Researchers in this field and the discussion section of our paper have raised the possibility that fully linear models could surpass Transformer architectures. The training process for standard attention (or a standard Transformer) does not maintain prior knowledge or memory in the temporal dimension—much like a person with intelligence but no memory. While the parameters of MLP layers store some knowledge, that knowledge is highly compressed and can only be elicited in particular contexts (we compare it to human genes). In the same analogy, standard attention resembles a professionally trained amnesic expert, which substantially limits what the expert can do. Methods such as in-context learning, few-shot prompting, and chain-of-thought prompting enhance the model’s capabilities by adding temporary, constructed memories, and existing memory management methods for agents also alleviate this problem to some extent.

One advantage of linear models is that a large amount of prior knowledge can be written into a fixed-size state in advance. While standard attention needs to add temporary memories through in-context learning or chain-of-thought prompting, a linear model that can store experiences comparable to a person’s lifelong development in a fixed-size state has the potential to surpass standard attention models lacking this ability. These experiences can accumulate during training, forming what we call a next-generation learning paradigm: "developmental intelligence." This means that during the training process, the model maintains a fixed state with sufficient capacity and precise retrieval capabilities; thus, not only is knowledge in the weights enhanced, but memory knowledge in the temporal dimension grows simultaneously. If the state can retain long-term experiences relevant to future tasks and the readout can retrieve the needed information from many candidates, the model may outperform standard attention models that rely only on parameters and the current input on tasks that depend on those experiences.

Although all linear models could potentially utilize this learning paradigm, most lack the potential to accomplish this modeling. We believe it requires two conditions: sufficient usable state capacity and a readout that can reliably locate relevant information while filtering out noise. RSA’s multilevel state and data-dependent readout provide a concrete structure for studying both necessary conditions together, which other existing linear models cannot satisfy simultaneously.

  1. Capacity and state organization. Many analyses of simple linear state structures start from a single state level or a limited number of state groups and obtain an effective capacity equal to the state size or a few times that size. In contrast, RSA's equivalent state capacity grows exponentially. Based on the proposition of PLA/GPA—that standard attention is a special case of linear models with an upper-bounded state size, meaning there exists a method to compress an indefinitely growing sequence into a fixed-size state—they do not prove how many state levels are needed. Under the theoretical construction in our paper, we show that states on the order of the attention head dimension suffice to support near-unlimited memory capacity in arbitrary settings, and we will present the construction and proof more clearly in the revision. A unified organization of large-capacity states is also necessary. We relate the layer stack in a Transformer to level-by-level state expansion, which not only helps explain the Transformer’s layered organization but also allows us to draw on existing empirical evidence for Transformers to support the effectiveness of our approach to state expansion.
  2. Relevance-based readout. MoM, MVA, and PLA all offer ways to expand state, and the multilevel organizations in MVA and PLA also have nominal path spaces given by products of the slot counts. However, their readout methods are somewhat trivial and read the entire memory at once. MVA places more emphasis on improving readout for short sequences, while PLA reads the state densely, which can bring in irrelevant noise and reduce efficiency due to dense connections. RSA’s data-dependent sparse readout is designed to address this issue, and further improving its retrieval ability remains future work.

In summary, compared to standard attention—an amnesic expert—the future model will likely be an expert possessing both intelligence and memory. This memory will be recorded in RSA's massive state capacity and can be retrieved more reasonably.

Weaknesses

W1. Evidence for sparse readout and variation across seeds

Thank you for pointing out the limitation of a single run. The “without sparse readout” row in Table 9 is the PLA control, which retains multilevel states and uses dense readout. The original scores of 45.86 and 45.49 came from seed 42. We report three paired runs at seeds 20, 42, and 88, which primarily control training-data sampling and order. We average the same six tasks as in Table 9.

ModelSeedLMBPIQAHella.Wino.ARC-eARC-cMean
RSA-360M4232.8966.5039.2452.0456.7027.8145.86
RSA-360M2032.3467.1038.9651.9257.1127.3245.79
RSA-360M8832.7266.8239.1852.2156.9427.6545.92
PLA-360M4231.6566.2038.9751.8955.9628.2445.49
PLA-360M2031.2166.8638.1551.8356.3626.9745.23
PLA-360M8831.8766.7139.0251.3956.1727.4345.43

RSA averages 45.86 ± 0.07 and PLA averages 45.38 ± 0.14. The seed-matched differences are +0.37, +0.56, and +0.49 percentage points, with a mean of +0.47 ± 0.10 points. The uncertainty terms are sample standard deviations. The six-task average improves slightly under all three data orders, while individual tasks vary in direction. These runs primarily capture variation due to data sampling and order. Initialization and item-level evaluation uncertainty would require separate analyses.

Sparse readout is intended to reduce interference from irrelevant states during long-context retrieval. In the 7B comparison in Table 12, RSA and PLA score 99.0% and 91.2% at 32K and 95.8% and 90.0% at 128K. Table 16 evaluates the task-finetuned models on BABILong. At 128K, RSA-Qwen scores 100% and 98% on QA2 and QA3, compared with 72% and 86% for Qwen2.5. RSA-Mistral scores 100% and 97%, compared with 49% and 36% for Mistral. Table 12 directly compares the two readout mechanisms, while Table 16 provides a task-level comparison with full-attention models. Table 9 measures the small change in the mean across six commonsense reasoning tasks. We will clarify the scope of each result in the revision.

W2. The MMLU gap in Table 2

RSA-3B is converted from Qwen2.5-3B, and its new modules require continued pretraining to adapt to the pretrained representations. At the matched 20B budget in Table 2, RSA-3B-CPT and Qwen2.5-3B-CPT score 47.6 and 55.6 on MMLU. Our experiments show that the initial advantage conferred by Qwen’s original pretraining data diminishes as both models continue training on public FineWeb-Edu. At 90B tokens, the scores are 55.8 and 54.9. We retain the 20B Qwen2.5-3B-CPT row in Table 2 as a strong early baseline with the same token budget as the 20B RSA model.

Tables, model labels, and layout

Thank you for identifying these issues. We introduced several labeling and layout errors while incorporating earlier feedback.

  1. Tables 3 and 7 duplicate each other. We will remove Table 7 and update subsequent numbering and cross-references.
  2. The “340M” label in Appendix C.1 is a typo. We will correct it to “360M.”
  3. The RSA-7B-CPT score of 59.3 in Table 2 corresponds to 30B tokens. The score of 60.2 in the final row of Table 10 corresponds to 40B tokens, although the row is labeled 30B. We will correct the label and clarify that Table 10 supplements the 30B result in the main text with the 40B result.
  4. The placement of Figure 2 and Tables 6–10 reflects the main-text page limit. We will condense Related Work and Preliminaries, move extended material to the appendix, and bring forward the main retrieval, efficiency, and ablation results. The main text will identify where to find the remaining details.

Questions

Q1. Mean and standard deviation across three seeds

We ran RSA and the PLA/GPA control without sparse readout at seeds 20, 42, and 88, primarily varying training-data sampling and order. Across the six tasks in Table 9, the mean and sample standard deviation are 45.86 ± 0.07 for RSA and 45.38 ± 0.14 for PLA, with a paired gain of +0.47 ± 0.10 percentage points. The average commonsense gain is small. Sparse readout is designed to suppress irrelevant information in long-context retrieval, which is more directly addressed by the RSA–PLA retrieval comparison in Table 12 and the BABILong results in Table 16.

Q2. The difference between MMLU and commonsense results

The Qwen baseline in the paper is taken from an earlier CPT stage. For 3B, it scores 55.6 at 20B tokens and 54.9 at 90B tokens. We believe both the amount and the quality of training data may affect RSA’s performance on individual tasks.

Conversion reuses the pretrained backbone and projections and introduces newly initialized state organization and readout modules. Continued pretraining lets these modules adapt to the pretrained representations. The table reports MMLU during continued pretraining on public FineWeb-Edu, with the budget measured in cumulative CPT tokens. We also report SFT results obtained with higher-quality data.

CPT tokensQwen2.5-7BRSA-7BQwen2.5-3BRSA-3B
20B61.159.155.647.6
40B61.360.255.952.8
60B60.460.754.553.6
80B61.061.555.255.7
90B60.961.354.955.8

RSA’s MMLU scores generally improve during CPT, while the Qwen controls fluctuate below their original pretrained checkpoint scores. At 90B, RSA and Qwen score 55.8 and 54.9 at 3B and 61.3 and 60.9 at 7B. Table 2 retains the 20B Qwen2.5-3B-CPT result as a strong early baseline with the same budget as the 20B RSA result. We will label comparisons at matched and different budgets explicitly.

We proposed two possible explanations. The new modules need training tokens to adapt, and the public corpus may differ from Qwen’s original pretraining corpus in quality, coverage, and distribution. The table shows how the score gap evolves with the token budget. Isolating the effect of data quality requires experiments that control the corpus and training stage. On commonsense tasks, a populated recurrent state can supply additional context for short inputs, for example through previously written text, information carried across segments, or a learned initial state. Having more slots than input tokens provides space, and any useful prior information depends on what was actually written. These tasks also tend to have short inputs and outputs, so a converted model may recover the relevant abilities with less training.

Reviewer_sL2SOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Recursive Sparse Attention (RSA), a linear-complexity sequence model that jointly designs memory storage and readout. For storage, RSA maintains multiple levels of slot-based key–value states per layer. The shallow level stores coarse prototypes, and each deeper level stores an MLP-refined version of the residual that the previous level fails to reconstruct, so the nominal joint code space grows multiplicatively with the number of levels. For readout, a sparse mechanism thresholds shallow-level scores and uses them to gate deeper-level scores, suppressing weak correlations. Experiments on models trained from scratch (360M, 1.3B) and models converted from pretrained Transformers (3B–14B) show competitive performance against Transformer and linear baselines on common-sense, retrieval, and long-context benchmarks, with inference time that grows linearly with sequence length.

Critical strengths
  • S1. Coherent coarse-to-fine design for storage and readout. RSA applies a single coarse-to-fine principle to both write and read: deeper levels store residuals that shallow levels fail to reconstruct, and shallow-level relevance gates deeper-level readout. This residual view relates RSA to prior state-expansion methods such as PLA, MVA, and GSA. The authors also clearly state that the multiplicative capacity is a nominal upper bound rather than a guarantee of realized usage.
  • S2. Consistent gains over linear models and compatibility with pretrained Transformers. Across a broad evaluation spanning 360M to 14B parameters, RSA consistently outperforms existing linear models on common-sense reasoning, retrieval, and long-context benchmarks, with particularly large gains on long-context retrieval. Beyond training from scratch, the paper shows that RSA can replace the attention operator of existing pretrained Transformers with less than 0.1% additional parameters and a modest amount of continued training, while largely preserving the original model's performance. This suggests a practical path for converting existing models into linear-complexity models with fixed-size memory.
Critical weaknesses
  • W1. Limited understanding of the role of each level. The paper does not show what each level represents, and the coarse-to-fine interpretation remains an analogy. Although precise characterization may be difficult for an end-to-end model, a minimal analysis, such as a level-wise ablation or a toy setting with known hierarchical structure, would help readers understand this new architecture.
  • W2. Unclear cost-effectiveness relative to alternative designs.
    • (W2-a) Levels vs. layers. Adding levels introduces sequential computation, making RSA slower than a single-level memory with the same total slots. The paper compares only against a wider single-level memory, not against spending the same compute on additional layers.
    • (W2-b) Hybrid architectures. Recent large-scale models increasingly combine linear layers with a few full-attention layers. A comparison or discussion of whether RSA is an alternative to such designs or a component of them would clarify its practical positioning.
  • Minor.
    • Citations are written in textual form throughout (e.g., "Transformer Vaswani et al. (2017) has become …"), whereas parenthetical form (\citep) would be appropriate, e.g., "Transformer (Vaswani et al., 2017) has become …".
    • Some references are duplicated (e.g., Dao 2023a/b refer to the same FlashAttention-2 paper, and Katharopoulos et al. 2020a/b refer to the same paper).
Questions for authors
  • Clarifications
    • Q1. (Related to W1) Since deeper-level slots are shared across all shallow-level parents, residuals from tokens with different shallow assignments may collide at the same deep slot. How effectively does the inter-level gating mitigate such collisions in practice?
    • Q2. The assignment function ff is not explicitly defined. Since all memory states are initialized to zero, an assignment based only on similarity scores would assign equal weights to all slots, causing them to receive identical updates and never differentiate. Could the authors provide the exact form of ff and clarify how slots become differentiated (e.g., through slot-specific parameters or initialization)?
  • Suggestions
    • Q3. (Related to W1-b) The coarse-to-fine interpretation would be more convincing with empirical evidence. For example: (i) masking the readout of individual levels at inference time and measuring which task types degrade; (ii) probing level-wise slot assignments against token attributes (e.g., token identity or semantic category); or (iii) a synthetic recall task with keys composed of a category and a within-category item, checking whether shallow levels capture categories and deeper levels capture items. Even one of these would substantially help readers understand the mechanism.
    • Q4. (Related to W2-a) Since all experiments use four levels, it would help to report how performance and compute change as the number of levels varies, which would test whether the claimed capacity gains translate into practice. In addition, comparing against a single-level baseline with additional layers under matched compute would clarify whether adding levels is a more effective use of compute.
5 comments
AuthorsOpenReview ↗

Thank you for suggesting concrete empirical tests. We added readout ablations in trained models, synthetic tasks with known structure, training-time-matched comparisons of memory levels and network depth, and comparisons with full-attention and sliding-window hybrids. (If you're interested, you can read our "Motivation, contributions, and longer-term direction" section in Reviewer 329s, which can help readers who are not very familiar with our field understand our work and contributions.)

Weaknesses

W1. What each level contributes and when coarse-to-fine organization emerges

Thank you for suggesting that we test the role of individual levels directly. We ablated levels on retrieval, natural text, and the simple recall task you suggested. (1) We intervened at inference on the qwen2.5-RSA-7B checkpoint. It has 28 blocks, each with four levels of 64 slots. The intervened blocks retain their original recurrent writes and within-block residual chain, with the selected level masked only in the output sum and subsequent blocks receiving the modified activations. With the intervention disabled, logits are bitwise identical before and after instrumentation. Each column below uses 12 fixed examples. For NIAH, we use 4K templates and score answer-token NLL under teacher forcing. For natural text, we truncate NarrativeQA passages to 2,048 tokens and score the last 64. The entries are NLL changes from the full model in nats/token.

InterventionNIAH1Natural text
Mask level 1+2.230+6.385
Mask level 2+0.142+0.109
Mask level 3−0.002+0.001
Mask level 4+0.000+0.000

Level 1 makes the largest readout contribution on this checkpoint and set of examples. Level 2 has a more pronounced effect on NIAH2, and the marginal effects of levels 3 and 4 are close to zero. We also trained a recall model using composite keys consisting of a category and an item within that category, with values randomly rebound for every example. At 128 tokens, masking level 1 reduces category and item recall by 11.26 and 7.21 percentage points, respectively, while masking level 2 reduces them by 1.77 and 13.40 points, respectively. When we extract hard codes by maximum instantaneous write weight, the mutual information between the code and the within-category item, conditioned on the category, does not increase monotonically with level.

A residual-quantization experiment with controlled feature strengths helps explain when this organization emerges. When the category component is stronger, level 1 encodes about 2 bits of category information and level 2 encodes about 2 bits of item information. Swapping the relative strengths of the two components reverses their encoding order. In the revision, we will describe coarse-to-fine organization as dependent on data geometry and training and report the contributions we observed at each level.

(2) Controlled task · Multi-field composite key–value recall

Each key has four fields with four possible values each. We sequentially fit four 4-slot codebooks by unsupervised residual quantization on independent training data. The feature magnitudes are 8/4/2/1, and each level receives the entire residual. The fitting procedure does not specify a mapping from fields to levels. Each test episode draws an independent random value mapping for each field. All value components use the same level-by-level residual write rule, and each level maintains four fixed-size value states and their counts.

The four loads vary 1/2/3/4 fields, holding the others fixed within an episode. Each episode uses 256 writes and 64 queries. Ablation masks only the specified terms in the readout sum, with all four levels still writing. We evaluate each of three independently fitted codebooks on 128 episodes. The table reports the fraction of queries for which all four output fields are correct.

Distinct keysBinding structureAll four levelsMask level 3Mask level 4Mask levels 3 and 4
4Factorizable by field100.00%100.00%100.00%100.00%
16Factorizable by field100.00%100.00%100.00%100.00%
64Factorizable by field100.00%25.06%100.00%24.96%
256Factorizable by field100.00%25.28%24.84%6.25%

(3) Cross-checkpoint evaluation. On qwen2.5-RSA-7B, removing levels 3 and 4 changes answer NLL by at most 0.012 nats/token on 8K and 16K NIAH1–3. The finding holds when we insert a space before the answer. In a Qwen3 experiment, we ablate the corresponding readouts in the first 35 four-level blocks and keep the final eight-level block intact. On 4K NIAH3, masking level 3 or 4 increases answer NLL by 8.037 or 8.258, and the number of examples with every answer token ranked first under teacher forcing falls from 5/16 to 0/16 in both cases. On 8K NIAH3 with Qwen3.5, the corresponding increases are 1.254 and 0.815, and the all-token count falls from 8/16 to 0/16 in both cases. These interventions show that reliance on deeper readouts varies across checkpoints. This finding and the evidence from the controlled composite key–value task address different aspects of the model. The following tables report the model ablations.

Qwen3

TaskLength / loadFull-model NLLΔ after masking level 3Δ after masking level 4Δ after masking bothHits in the four conditions, in order (16 examples each)
catalog8K/82.765-0.331 [-0.893, +0.316]+0.193 [-0.574, +0.993]-0.308 [-0.980, +0.466]2, 2, 1, 2
catalog8K/643.544-1.109 [-1.902, -0.259]-0.652 [-1.460, +0.204]-1.123 [-1.946, -0.225]1, 2, 1, 2
catalog8K/2562.551-0.019 [-0.753, +0.677]+0.006 [-0.764, +0.850]-0.181 [-0.758, +0.391]3, 2, 4, 2
niah_single_28K template1.881+5.035 [+4.516, +5.604]+5.058 [+4.305, +5.877]+2.201 [+1.815, +2.647]2, 0, 0, 0
niah_single_38K template1.189+7.288 [+6.841, +7.700]+7.372 [+6.804, +7.916]+5.054 [+4.609, +5.494]0, 0, 0, 0

Qwen3.5 Base

TaskLength / loadFull-model NLLΔ after masking level 3Δ after masking level 4Δ after masking bothHits in the four conditions, in order (16 examples each)
catalog8K/81.866+0.239 [-0.021, +0.485]+0.234 [+0.030, +0.459]+0.411 [-0.064, +0.873]3, 3, 3, 3
catalog16K/81.771+0.297 [+0.049, +0.550]+0.736 [+0.307, +1.194]+1.235 [+0.575, +1.846]2, 3, 1, 2
catalog16K/642.287+0.008 [-0.252, +0.260]+0.117 [-0.214, +0.480]+0.605 [+0.006, +1.143]1, 3, 2, 1
catalog16K/2562.264+0.233 [+0.061, +0.401]+0.133 [-0.207, +0.516]+0.519 [-0.099, +1.074]2, 2, 2, 2
niah_single_28K template0.167+0.975 [+0.689, +1.268]+1.689 [+1.424, +1.937]+2.099 [+1.832, +2.346]14, 4, 0, 0
niah_single_38K template0.101+1.254 [+1.009, +1.490]+0.815 [+0.631, +1.028]+2.346 [+2.100, +2.564]8, 0, 0, 0

W2. Cost-effectiveness relative to alternative designs

W2-a. State levels, network depth, and training cost

We compare five layouts on the same synthetic recall task. The model width is 64, and each example contains 16 random bindings. The fixed-depth controls use two blocks with 1/2/4 levels per block. The depth controls use four or eight single-level blocks. For pure RSA, we fix the total slot count at 128, 256, or 512, with the slots per level equal to the total divided by the product of the number of blocks and the number of levels per block. The three seeds use the same task-generation protocol and their respective A100 time budgets, averaging about 78.3 seconds. The models train at length 128. The table reports item recall at length 512 as mean ± standard deviation across seeds.

Blocks × levels per block128 total slots256 total slots512 total slots
2 × 166.08 ± 2.56%39.12 ± 33.32%55.93 ± 14.33%
2 × 230.62 ± 35.13%39.31 ± 29.65%51.86 ± 3.43%
2 × 463.59 ± 49.87%44.14 ± 33.96%75.34 ± 16.71%
4 × 172.57 ± 8.35%54.32 ± 15.37%60.44 ± 10.00%
8 × 125.78 ± 17.16%56.62 ± 24.68%36.52 ± 19.27%

With 128 or 256 total slots, a deeper single-level model achieves the highest mean. At 512 slots, the two-block, four-level model exceeds the strongest single-level control by 14.90 percentage points at equal training time, leading on two seeds and nearly tying on the third. At a fixed 1,920 updates and 512 slots, the two-block four-level and four-block single-level layouts take 79.21 and 38.89 seconds and achieve 72.31% and 60.99% recall, respectively. The timing difference reflects the extra cost of serial levels. The best layout varies with the slot budget, and parameter count varies with slot width. This conclusion applies to the synthetic task and training protocol used here.

We also compare 1/2/4/8-level single-stage models with the same total state size on FineWeb-Edu, matching training tokens and wall-clock time in separate comparisons. At a matched budget of 33.55M tokens, the 2×128 layout achieves 4K/16K PPL of 58.295/57.653. At a matched budget of 180 seconds, the 1×256 layout achieves 50.453/49.765. The former compares learning on the same training stream, and the latter includes the cost of serial levels. This comparison uses models of approximately 3.7M parameters and one training seed.

Four parallel KV groups and a wide single-level GDN. ParallelKV4 and GDN64 share the same two-block backbone with width 64 and two heads. ParallelKV4 uses four 32×32 KV substates per head, with the same value input and parallel computation in one GDN call. GDN64 uses a single 64×32 state per head. Both train on 512-token sequences under a matched time budget of about 192 seconds. The table reports two seeds on two A800 GPUs. Training time excludes initial compilation and evaluation. Forward times are synchronized measurements after warm-up at batch size 1.

ModelFP32 recurrent state (KiB/sequence)Training time for 4,800 updates (s)Forward time at 512/2K/8K (ms)
ParallelKV46485.972.43/2.43/2.41
GDN643282.142.40/2.39/2.35

The following table uses 256 random bindings and 32 queries per sequence and reports percentages after time-matched training as mean ± standard deviation across seeds. Sequence exact match requires all 32 queries in a sequence to be correct.

Test lengthParallelKV4 per-query accuracyGDN64 per-query accuracyParallelKV4 sequence exact matchGDN64 sequence exact match
51299.22 ± 0.0798.10 ± 0.0777.34 ± 1.1054.69 ± 6.63
2K98.80 ± 0.3198.14 ± 0.0068.75 ± 4.4257.81 ± 4.42

The measured runtime of the four-group layout is close to GDN64’s, and its recurrent state is twice as large. At 512 and 2K, the gain of the four-group layout is concentrated in sequence exact match. This comparison measures the trade-off between state layout and retrieval at similar measured runtimes, with both state size and parameter count varying with the layout.

W2-b. RSA and hybrid architectures

We replace the final block of the two-block, four-level RSA model with full attention or 32-token sliding-window attention. In the same time-matched synthetic experiment, item recall at length 512 is 63.59% for two-block, four-level pure RSA, 99.99% for RSA with full attention, and 88.10% for RSA with sliding-window attention. RSA with sliding-window attention drops substantially when the retrieval sequence exceeds the window. These results show that RSA can serve as the recurrent memory component of a hybrid architecture. Full attention provides fine-grained access to history, and its KV cache grows with context length. A fixed sliding-window block bounds its own cache by the window size. Each hybrid retains one RSA block, so replacement changes both the recurrent slot count and the attention-cache structure. We will discuss this controlled recall result separately from quality, speed, and cache cost in larger models.

ConfigurationCutTrain sUpdatesCategory 128Item 128Category 512Item 512
RSAupdates39.001920100.00 ± 0.00100.00 ± 0.0086.04 ± 9.7370.61 ± 13.65
RSAtime78.523872100.00 ± 0.00100.00 ± 0.0086.36 ± 6.2172.57 ± 8.35
RSA + fullupdates44.791920100.00 ± 0.00100.00 ± 0.0093.72 ± 7.41100.00 ± 0.00
RSA + fulltime78.433355100.00 ± 0.00100.00 ± 0.0096.19 ± 6.6099.99 ± 0.01
RSA + SWA-32updates45.331920100.00 ± 0.00100.00 ± 0.0095.41 ± 1.7686.12 ± 3.18
RSA + SWA-32time78.463312100.00 ± 0.00100.00 ± 0.0094.53 ± 5.2788.10 ± 2.07

Our longer-term goal is a competitive fully linear model that can serve as an alternative to hybrids with a small number of softmax attention layers. Our results are competitive with hybrid architectures on commonsense reasoning, complex tasks, and LongBench. The table illustrates commonsense performance after converting the hybrid Qwen3.5-4B model. RSA still trails full attention and hybrids on complex retrieval tasks. GDN2RSA replaces Qwen3.5’s GDN layers with RSA. Qwen3.5 RSA retains the GDN layers and replaces the remaining softmax attention layers with RSA.

ModelPIQAWinograndeARC-EARC-CSciQ normMMLU
Original Qwen3.5-4B-Base78.4571.1182.2451.1195.4074.64
Original Qwen3.5-4B-Base-step3000077.9469.7583.0753.5494.4265.22
Qwen3.5 RSA- step3000079.0070.9384.4758.3495.2167.28
Qwen3.5 GDN2RSA- step400078.8671.2583.9659.4795.5370.51

Minor issue. Citation format

We will standardize parenthetical citations in the main text and merge duplicate bibliography entries for FlashAttention-2 and the work by Katharopoulos et al. These edits are already in the working patch.

Reviewer_Bn9mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

RSA is a hierarchical slot memory: each level stores what the previous level failed to reconstruct, and at readout, scores from shallow levels sparsely gate the deeper ones. It is evaluated trained from scratch (360M, 1.3B) and converted from Qwen2.5 models (3B-14B), on commonsense, retrieval, passkey and LongBench.

Critical strengths
  1. The range of experiments is extensive, spanning from 360M to 14B, incorporating both from-scratch and converted models, with the conversion setup featuring comparable Qwen CPT/SFT controls.

  2. Impressive retrieval outcomes are observed for the from-scratch models; at 1.3B, RSA performs similarly to Transformer++ (37.21 compared to 37.31) and is significantly superior to GDN.

  3. The hierarchical structure of the residuals is logical, and combining MVA, MoM, and PLA into a single template provides a beneficial perspective on state expansion.

Critical weaknesses
  1. Following the method proves challenging due to numerous components and options, with no definitive algorithm indicating what was executed. The argument concerning capacity is never linked to any actual measurement of the codes utilized.

  2. Certain assertions lack backing. MMLU performance at 360M/1.3B falls to chance level, meaning it fails to demonstrate a "long-horizon gap." The removal of sparse readout results in only a 0.37-point decrease, which falls within the range of noise.

  3. The primary configuration operates at approximately 1.5 times the speed of GDN and is slower than FlashAttention for inputs up to around 32K, encompassing the majority of the tested range. Additionally, there are no comparisons regarding quality with sparse attention methods (such as NSA, MoBA) or SWA hybrids.

Questions for authors

See weaknesses

3 comments
AuthorsOpenReview ↗

Thank you for reviewing our paper carefully and for pointing out where the implemented algorithm, the interpretation of capacity, and the cost comparisons need clearer evidence.

Weaknesses

W1. The implemented algorithm and codebook utilization

(1) Section 3.3 gives the sequential computation. The input is first projected to Q, K, and V, after which the K/V states are updated level by level, with residual connections carrying information to the next level. Readout computes attention scores and obtains the output by weighting. Sections 3.1 and 3.2 explain the intuition and rationale for these operations. If any specific step remains unclear, we welcome the opportunity to clarify it.

(2) The paper gives the theoretical number of combinations and the effective capacity, and Appendix A.4 analyzes the correlation, mutual information, and entropy of the states across levels. To make the relationship between these quantities and actual recall clearer, we added the following experiments. In the first controlled task, the model must distinguish many joint addresses to recall the correct output. Each key has four fields, each with 64 possible values. Each episode independently redraws four field-to-value mappings, and a query counts as correct only when all four output fields are correct. Each complete key therefore has a distinct target. All layouts have a total budget of 256 state slots. If the single-level model selects one hard slot per query, at most 256 of the 262,144 distinct keys can receive their own correct outputs, giving a theoretical exact-match ceiling of 0.0977%. This directly connects usable capacity to joint-code utilization.

The four-level model uses 64 slots per level, fits its codebooks on unlabeled keys, selects the nearest slot at each level, and passes on the residual. When all four fields share a two-bit group identifier, the four-level model uses 262,144 joint codes across all 262,144 combinations, with 18 bits of joint entropy, 262,144 effective codes, and 6 bits of total correlation. After 4,096 writes, exact match on unseen combinations reaches 100% for all three seeds. At the same slot budget, measured accuracy is 0.018% for single-level hard readout, 0.140% for two levels, and 0.017% for continuous single-level soft slot mixing. Masking any of the four levels reduces exact match to 1.56%–5.73%. Four parallel product codebooks based on PCA subspaces also reach 100%. These results indicate that the task requires compositional addressing and verify two implementations of it, one using residual levels and one using parallel groups.

When all four fields are independent, we enumerate all 644=16,777,21664^4=16,777,216 combinations. The four-level model uses every joint code, with 24 bits of joint entropy and 16,777,216 effective codes. All three seeds recall every combination correctly. The levels are independent in this case, so total correlation is 0 bits. The 6 bits in the correlated task come from the shared group identifier, while 0 bits in the independent task is consistent with the larger usable code space.

Both codebook fitting and write coverage within an episode affect the result. The four-level model recalls 36.18% of unseen combinations after 256 writes and 100% after 1,024 writes. If the codebooks are fitted on the same number of independently sampled unlabeled keys, accuracy after 4,096 writes is 8.66%. We therefore retain 64464^4 as a nominal upper bound and explain how task structure and fitting conditions affect usable capacity.

The observed joint-code count is the number of distinct combinations of slot indices. The effective code count is 2H2^{H}, the equivalent number of uniformly used codes at joint entropy HH. Total correlation is the sum of the marginal entropies across levels minus the joint entropy. The table summarizes the synthetic results across three seeds.

Layout and key spaceCode selection ruleKey observations per traceObserved joint codesJoint entropy (bits)Effective codesTotal correlation (bits)
One level, 1×256, correlatedNearest slot262,144228.37.52183.50.00
Two levels, 2×128, correlatedNearest slot at each residual level262,1441,031.710.011,027.63.76
Four levels, 4×64, correlatedNearest slot at each residual level262,144262,14418.00262,1446.00
Eight levels, 8×32, correlatedNearest slot at each residual level262,14416,289.713.9315,627.023.28
Four parallel groups, 4×64, correlatedNearest slot in each PCA subspace262,144262,14418.00262,1446.00
Four levels, 4×64, full enumeration of independent fieldsNearest slot at each residual level16,777,21616,777,21624.0016,777,2160.00

The table uses pairwise-balanced field data to fit the codebooks and reports exact match on the full support across three seeds. Fields that do not vary independently retain the shared group identifier and a fixed within-group value. Layouts in the same row use the same number of writes per episode. When one or two fields vary independently, the writes cover the full support.

Independently varying fieldsFull key spaceWrites per episode1×2562×1284×648×32
16464100.00%100.00%100.00%76.56%
21,0241,02412.01%100.00%100.00%66.15%
316,3844,0960.553%3.868%100.00%8.651%
4262,1444,0960.023%0.148%100.00%0.298%

An additional recall experiment. The four-level model inherits the tested block’s 4×64 slot parameters. The single-level model concatenates the slot centers and initial key and value states along the slot dimension into 256 slots. It reuses the same Q/K/V/O projections, combines the level readout transformations into a block-diagonal form, and initializes the single-level controls with the mean of the level-specific controls. Both models retain two-stage GDN, with a measured recurrent state of 8 MiB per example.

Each example draws distinct addresses from 512 single-token addresses and independently assigns each address a payload from A–P. Addresses go into Q and K, payloads go into V, and each episode regenerates the bindings. After writing, we keep only the final two-stage state of the tested block and query the addresses in write order to predict the entire payload sequence. Recall is performed without a gold answer prefix. For both layouts, the relative difference between read-only computation and the output of the original operator with writing approximately disabled is below 0.12%. Swapping the states of two examples and repeating readout checks whether the answer comes from the corresponding example’s state. The table reports the results.

BindingsSingle-level per-binding accuracyFour-level per-binding accuracyFour levels minus one (pp, 95% interval)Single-level sequence exact matchFour-level sequence exact match
1698.93%99.02%+0.10 [−0.78, +0.98]82.81%87.50%
3298.63%98.93%+0.29 [−0.29, +0.93]62.50%68.75%

The four-level configuration shows an improvement.

W2. Evidence for sparse readout on long sequences

We attribute the near-chance MMLU scores to the absence of larger-scale experiments. The original paper therefore also uses LongBench and long-sequence retrieval experiments to demonstrate the effectiveness of the method, with results reported in Tables 4 and 6, among others. For sparse readout, we added training runs with multiple seeds to measure its effect on commonsense reasoning, with the results given in our response to Reviewer 329s, W1. As Section 3.2 explains, sparse readout mainly aims to suppress the noise introduced by dense readout on long sequences while retrieving query-relevant information. On shorter sequences, softmax or gating can also suppress noise, so the advantage of sparse readout is harder to observe. Tables 12 and 16 report the relevant long-sequence results.

The RSA–PLA comparison in Table 12 directly tests sparse readout. PLA retains multilevel storage and uses dense readout. Both score 100.0% on 2K Passkey, with 99.0% and 91.2% at 32K and 95.8% and 90.0% at 128K. Longer contexts contain more information that can interfere with retrieving the target, which is the setting sparse readout is meant to improve. In frozen qwen2.5-RSA-7B, removing cross-level gating after matching query norms increases 4K NIAH2 answer NLL by 0.669 nats/token, with a 95% paired interval of [0.330, 1.004].

Table 16 reports results for the full task-finetuned models on 128K BABILong. RSA-Qwen scores 100% and 98% on QA2 and QA3, compared with 72% and 86% for Qwen2.5. RSA-Mistral scores 100% and 97%, compared with 49% and 36% for Mistral. This table measures the long-sequence performance of the full model, while Table 12 provides the main comparison of the readout mechanisms.

The single-seed difference in the six-task commonsense average is 0.37 percentage points. Across three seeds that primarily change training-data sampling and order, the paired RSA–PLA difference is +0.47 ± 0.10 points. The small models’ MMLU scores of 26.4, 28.6, and 26.1 are close to the 25% rate for uniform four-way guessing. The revision will report these as multiple-choice scores and base the long-sequence claims on the direct measurements above.

W3. End-to-end latency, sparse attention, and hybrid models

(1) We consider efficiency comparisons fairer at comparable task scores, a principle commonly used in the field. Beyond our main configuration, RSA with 4×324 \times 32 or 4×164 \times 16 also has advantages over GDN and GSA, particularly when converting Transformers to linear models. In these configurations, RSA is more efficient than GDN and GSA. Speed on short sequences is a limitation relative to FlashAttention. Linear and sparse attention methods commonly face this issue within 8K or 16K. One approach is to use FlashAttention up to 8K and then transition to a more efficient attention mechanism by connecting it to FlashAttention through a KV cache or KV state. (2) Table 18 compares RSA with NSA and MoBA. Following your suggestion, we added a hybrid with SWA. SWA preserves the attention distribution within the window, so a small amount of finetuning can restore performance. If the model overfits to information within the window, performance on tasks beyond the window size, particularly retrieval, falls to the level achieved using the window alone. The tables report commonsense reasoning and retrieval results, respectively.

ModelTokens<br>(B)PIQAARC-eARC-c<br>(norm)Hella.<br>(norm)Wino.MMLU<br>(5-shot)Avg.Avg. w/o<br>MMLU
Llama 3 8B–79.980.153.379.173.166.672.073.1
Llama 3 8B Hedgehog0.0477.471.140.666.554.324.255.762.0
Llama 3 8B LoLCATs0.0480.981.754.979.774.152.870.774.2
Llama 3 8B RSA-SWA (Ours)0.0481.582.754.979.673.854.171.174.5
Llama 3 8B LoLCATs280.582.155.379.274.551.270.574.3
Llama 3 8B RSA-SWA (Ours)281.782.956.279.473.754.671.474.7
ModelTokens (B)niah1-4Kniah1-8K
Llama 3 8B–100100
Llama 3 8B Hedgehog0.0412.76.8
Llama 3 8B LoLCATs0.0416.27.6
Llama 3 8B RSA-SWA (Ours)0.0416.08.1
Llama 3 8B LoLCATs214.77.1
Llama 3 8B RSA-SWA (Ours)217.99.2
Llama 3 8B RSA (Ours)297.295.5

The sparse-attention comparison in Appendix Table 18 is summarized below. The models are similar in size, and scores are averages across eight downstream tasks.

ModelParametersEight-task mean (%)
NSA370M45.99
MoBA360M46.18
RSA360M46.57

Comments on the paper

2 comments

Dear Reviewers,

Thank you for your valuable comments and for your time. I am currently reviewing two manuscripts, and their AC has reminded us reviewers to respond promptly. To provide a careful and responsible response to these two manuscripts, I need to calibrate my understanding of ICLR review standards by referring to the criteria reflected in your assessments of our paper.

Previously, based on my experience with ICML, NeurIPS, and previous ICLR submissions and reviews—especially the prior comments on this paper—I was able to form a reasonably clear judgment. However, the issues you have highlighted and the criteria you have applied suggest that my earlier interpretation of ICLR standards may need adjustment. Before making such adjustments, I would therefore like to take your further replies into careful consideration.

We are grateful for your insightful comments and guidance. We believe that, with the revisions made in light of your comments, the manuscript now addresses the main concerns raised. We would be very grateful if you could provide further comments, discussion, or assessment. We would be pleased and honored to continue the discussion and, based on it, finalize my assessment under ICLR criteria, so that I can respond to the two manuscripts I am reviewing in a careful and responsible manner and help the authors improve their work or offer more substantive suggestions.

Thank you again to the AC and all reviewers for your hard work and valuable time.

There are less than four hours left until the deadline, and if there are further confusions afterward, perhaps we will no longer be able to resolve them in time. However, we believe that, after the valuable suggestions of all reviewers, the paper has been sufficiently improved, and the new version has also been uploaded to OpenReview.

Finally, I still want to recommend our direction and paper to all reviewers, and especially to the respected AC. This is because, from the confidence score of all reviewers, it can be seen that the reviewers are not very familiar with our direction and our contributions: simply put, our work provides the currently strongest-performing purely linear LLMs, from 3B and 7B to 14B (and more). In addition, according to the comments of all reviewers, we have provided detailed ablations for each module as well as theoretical proofs and derivations, which can, through theory, greatly influence other fields or be transferred. Finally, we believe that our method is a necessary foundation for simulating human growth experiences, can become a new type of self-growth paradigm, and thereby has the potential to surpass Transformer architectures that lack this ability or are deficient in it, and to lay the foundation for fitting as many, even all, dimensions of human intelligence as possible, thereby achieving a further approximation to true intelligence.

In addition, this paper, when submitted to ICML and NeurIPS, received very high positive evaluations within the subfield in both cases (confidence score >=4 and score>=5). Therefore, after the previous two rounds of improvement, we initially looked forward very much to this ICLR evaluation and exchange. Moreover, based on our extensive previous reviewing experience and experience of accepted papers, we believe that after revising according to the guidance and comments of all reviewers, our manuscript definitely meets the publication standards of ICLR. Therefore, we very much hope to recommend it to the AC and all reviewers, and to receive your recognition and re-evaluation, and then to make contributions to the field again and lay the foundation for deeper development afterward. Finally, thank you again to all reviewers and the AC for your efforts and contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.