- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces RSA to improve long-context retrieval in linear sequence models while keeping a fixed-size memory. The motivation is that these models may lose useful information during compression and mix relevant and irrelevant information during readout. To address this, they use a deep–shallow memory structure, where shallow levels store coarse representations and deeper levels store the residual information. They also use a sparse readout mechanism that uses shallow-level relevance to guide reading from deeper memory. They evaluate RSA on common-sense reasoning, retrieval, and long-context tasks using both models trained from scratch and models converted from pretrained Transformers, and compare its performance with Transformer and other linear model baselines.
Critical strengths
The idea of organizing linear-attention memory as a coarse-to-fine residual hierarchy with relevance-guided readout is intuitive, and the paper evaluates it across a wide range of scales and benchmarks.
Critical weaknesses
- The experiments provide limited evidence for the main contribution of this work. The ablation study in Table 9 shows that RSA without sparse read is just PLA, and RSA improves over PLA by only 0.37 points on average. This result is based on a single run with no variance reported, making it difficult to assess the robustness of the improvement.
- The paper claims that “Across all scales, RSA matches or exceeds existing linear models and Transformer baselines on common-sense reasoning benchmarks.” However, Table 2 shows that RSA-3B-CPT scores 47.6 on MMLU, substantially lower than Qwen2.5-3B-CPT’s score of 55.6.
The paper contains many presentation errors and inconsistencies. For example:
- Tables 3 and 7 are identical.
- The 360M model is referred to as 340M in Appendix C.1.
- RSA-7B-CPT’s MMLU score is reported as 59.3 in Table 2 but 60.2 in Table 10.
- Figure 2 and Tables 6–10 come after the references, although the main text relies on them for its retrieval and ablation claims.
Questions for authors
- Since sparse readout is the main component distinguishing RSA from PLA, it would be helpful to report the mean and standard deviation across at least three random seeds for both RSA and the “w/o sparse read” variant in Table 9.
- Table 2 shows that RSA underperforms the matched Qwen control on MMLU at every scale, while improving performance on common-sense reasoning tasks. How do the authors explain this discrepancy?
4 comments
Based on these architectural, theoretical, and empirical contributions across scales, we believe the work meets ICLR’s publication standard.
Researchers in this field and the discussion section of our paper have raised the possibility that fully linear models could surpass Transformer architectures. The training process for standard attention (or a standard Transformer) does not maintain prior knowledge or memory in the temporal dimension—much like a person with intelligence but no memory. While the parameters of MLP layers store some knowledge, that knowledge is highly compressed and can only be elicited in particular contexts (we compare it to human genes). In the same analogy, standard attention resembles a professionally trained amnesic expert, which substantially limits what the expert can do. Methods such as in-context learning, few-shot prompting, and chain-of-thought prompting enhance the model’s capabilities by adding temporary, constructed memories, and existing memory management methods for agents also alleviate this problem to some extent.
One advantage of linear models is that a large amount of prior knowledge can be written into a fixed-size state in advance. While standard attention needs to add temporary memories through in-context learning or chain-of-thought prompting, a linear model that can store experiences comparable to a person’s lifelong development in a fixed-size state has the potential to surpass standard attention models lacking this ability. These experiences can accumulate during training, forming what we call a next-generation learning paradigm: "developmental intelligence." This means that during the training process, the model maintains a fixed state with sufficient capacity and precise retrieval capabilities; thus, not only is knowledge in the weights enhanced, but memory knowledge in the temporal dimension grows simultaneously. If the state can retain long-term experiences relevant to future tasks and the readout can retrieve the needed information from many candidates, the model may outperform standard attention models that rely only on parameters and the current input on tasks that depend on those experiences.
Although all linear models could potentially utilize this learning paradigm, most lack the potential to accomplish this modeling. We believe it requires two conditions: sufficient usable state capacity and a readout that can reliably locate relevant information while filtering out noise. RSA’s multilevel state and data-dependent readout provide a concrete structure for studying both necessary conditions together, which other existing linear models cannot satisfy simultaneously.
- Capacity and state organization. Many analyses of simple linear state structures start from a single state level or a limited number of state groups and obtain an effective capacity equal to the state size or a few times that size. In contrast, RSA's equivalent state capacity grows exponentially. Based on the proposition of PLA/GPA—that standard attention is a special case of linear models with an upper-bounded state size, meaning there exists a method to compress an indefinitely growing sequence into a fixed-size state—they do not prove how many state levels are needed. Under the theoretical construction in our paper, we show that states on the order of the attention head dimension suffice to support near-unlimited memory capacity in arbitrary settings, and we will present the construction and proof more clearly in the revision. A unified organization of large-capacity states is also necessary. We relate the layer stack in a Transformer to level-by-level state expansion, which not only helps explain the Transformer’s layered organization but also allows us to draw on existing empirical evidence for Transformers to support the effectiveness of our approach to state expansion.
- Relevance-based readout. MoM, MVA, and PLA all offer ways to expand state, and the multilevel organizations in MVA and PLA also have nominal path spaces given by products of the slot counts. However, their readout methods are somewhat trivial and read the entire memory at once. MVA places more emphasis on improving readout for short sequences, while PLA reads the state densely, which can bring in irrelevant noise and reduce efficiency due to dense connections. RSA’s data-dependent sparse readout is designed to address this issue, and further improving its retrieval ability remains future work.
In summary, compared to standard attention—an amnesic expert—the future model will likely be an expert possessing both intelligence and memory. This memory will be recorded in RSA's massive state capacity and can be retrieved more reasonably.
Weaknesses
W1. Evidence for sparse readout and variation across seeds
Thank you for pointing out the limitation of a single run. The “without sparse readout” row in Table 9 is the PLA control, which retains multilevel states and uses dense readout. The original scores of 45.86 and 45.49 came from seed 42. We report three paired runs at seeds 20, 42, and 88, which primarily control training-data sampling and order. We average the same six tasks as in Table 9.
| Model | Seed | LMB | PIQA | Hella. | Wino. | ARC-e | ARC-c | Mean |
|---|---|---|---|---|---|---|---|---|
| RSA-360M | 42 | 32.89 | 66.50 | 39.24 | 52.04 | 56.70 | 27.81 | 45.86 |
| RSA-360M | 20 | 32.34 | 67.10 | 38.96 | 51.92 | 57.11 | 27.32 | 45.79 |
| RSA-360M | 88 | 32.72 | 66.82 | 39.18 | 52.21 | 56.94 | 27.65 | 45.92 |
| PLA-360M | 42 | 31.65 | 66.20 | 38.97 | 51.89 | 55.96 | 28.24 | 45.49 |
| PLA-360M | 20 | 31.21 | 66.86 | 38.15 | 51.83 | 56.36 | 26.97 | 45.23 |
| PLA-360M | 88 | 31.87 | 66.71 | 39.02 | 51.39 | 56.17 | 27.43 | 45.43 |
RSA averages 45.86 ± 0.07 and PLA averages 45.38 ± 0.14. The seed-matched differences are +0.37, +0.56, and +0.49 percentage points, with a mean of +0.47 ± 0.10 points. The uncertainty terms are sample standard deviations. The six-task average improves slightly under all three data orders, while individual tasks vary in direction. These runs primarily capture variation due to data sampling and order. Initialization and item-level evaluation uncertainty would require separate analyses.
Sparse readout is intended to reduce interference from irrelevant states during long-context retrieval. In the 7B comparison in Table 12, RSA and PLA score 99.0% and 91.2% at 32K and 95.8% and 90.0% at 128K. Table 16 evaluates the task-finetuned models on BABILong. At 128K, RSA-Qwen scores 100% and 98% on QA2 and QA3, compared with 72% and 86% for Qwen2.5. RSA-Mistral scores 100% and 97%, compared with 49% and 36% for Mistral. Table 12 directly compares the two readout mechanisms, while Table 16 provides a task-level comparison with full-attention models. Table 9 measures the small change in the mean across six commonsense reasoning tasks. We will clarify the scope of each result in the revision.
W2. The MMLU gap in Table 2
RSA-3B is converted from Qwen2.5-3B, and its new modules require continued pretraining to adapt to the pretrained representations. At the matched 20B budget in Table 2, RSA-3B-CPT and Qwen2.5-3B-CPT score 47.6 and 55.6 on MMLU. Our experiments show that the initial advantage conferred by Qwen’s original pretraining data diminishes as both models continue training on public FineWeb-Edu. At 90B tokens, the scores are 55.8 and 54.9. We retain the 20B Qwen2.5-3B-CPT row in Table 2 as a strong early baseline with the same token budget as the 20B RSA model.
Tables, model labels, and layout
Thank you for identifying these issues. We introduced several labeling and layout errors while incorporating earlier feedback.
- Tables 3 and 7 duplicate each other. We will remove Table 7 and update subsequent numbering and cross-references.
- The “340M” label in Appendix C.1 is a typo. We will correct it to “360M.”
- The RSA-7B-CPT score of 59.3 in Table 2 corresponds to 30B tokens. The score of 60.2 in the final row of Table 10 corresponds to 40B tokens, although the row is labeled 30B. We will correct the label and clarify that Table 10 supplements the 30B result in the main text with the 40B result.
- The placement of Figure 2 and Tables 6–10 reflects the main-text page limit. We will condense Related Work and Preliminaries, move extended material to the appendix, and bring forward the main retrieval, efficiency, and ablation results. The main text will identify where to find the remaining details.
Questions
Q1. Mean and standard deviation across three seeds
We ran RSA and the PLA/GPA control without sparse readout at seeds 20, 42, and 88, primarily varying training-data sampling and order. Across the six tasks in Table 9, the mean and sample standard deviation are 45.86 ± 0.07 for RSA and 45.38 ± 0.14 for PLA, with a paired gain of +0.47 ± 0.10 percentage points. The average commonsense gain is small. Sparse readout is designed to suppress irrelevant information in long-context retrieval, which is more directly addressed by the RSA–PLA retrieval comparison in Table 12 and the BABILong results in Table 16.
Q2. The difference between MMLU and commonsense results
The Qwen baseline in the paper is taken from an earlier CPT stage. For 3B, it scores 55.6 at 20B tokens and 54.9 at 90B tokens. We believe both the amount and the quality of training data may affect RSA’s performance on individual tasks.
Conversion reuses the pretrained backbone and projections and introduces newly initialized state organization and readout modules. Continued pretraining lets these modules adapt to the pretrained representations. The table reports MMLU during continued pretraining on public FineWeb-Edu, with the budget measured in cumulative CPT tokens. We also report SFT results obtained with higher-quality data.
| CPT tokens | Qwen2.5-7B | RSA-7B | Qwen2.5-3B | RSA-3B |
|---|---|---|---|---|
| 20B | 61.1 | 59.1 | 55.6 | 47.6 |
| 40B | 61.3 | 60.2 | 55.9 | 52.8 |
| 60B | 60.4 | 60.7 | 54.5 | 53.6 |
| 80B | 61.0 | 61.5 | 55.2 | 55.7 |
| 90B | 60.9 | 61.3 | 54.9 | 55.8 |
RSA’s MMLU scores generally improve during CPT, while the Qwen controls fluctuate below their original pretrained checkpoint scores. At 90B, RSA and Qwen score 55.8 and 54.9 at 3B and 61.3 and 60.9 at 7B. Table 2 retains the 20B Qwen2.5-3B-CPT result as a strong early baseline with the same budget as the 20B RSA result. We will label comparisons at matched and different budgets explicitly.
We proposed two possible explanations. The new modules need training tokens to adapt, and the public corpus may differ from Qwen’s original pretraining corpus in quality, coverage, and distribution. The table shows how the score gap evolves with the token budget. Isolating the effect of data quality requires experiments that control the corpus and training stage. On commonsense tasks, a populated recurrent state can supply additional context for short inputs, for example through previously written text, information carried across segments, or a learned initial state. Having more slots than input tokens provides space, and any useful prior information depends on what was actually written. These tasks also tend to have short inputs and outputs, so a converted model may recover the relevant abilities with less training.
First of all, thank you very much for your detailed feedback and efforts. We will provide a detailed response to the questions raised by you and the other reviewers to acknowledge your careful and thorough approach.
If you are not interested in our field or our paper, but are only interested in specific issues, you may skip the following paragraphs and start reading directly from the “Weaknesses” section to save time.
Paper Background
Apart from an early accidental submission, this is the first time I have received initial reviews in which every reviewer leaned against acceptance, I knew full well that the outcome would not be ideal. However, this only fueled my enthusiasm and passion. I used the discussion period to wholeheartedly promote our field and our paper to the reviewers and other readers, and I responded in detail to the reviewers’ thoughtful comments and questions. Because ICLR reviews are public, we are concerned that low initial scores may shape how readers assess this work and later versions. We will respond carefully to the reviewers’ comments and welcome further questions. We previously submitted this work to ICML and NeurIPS. Across those submissions, the review comments gradually improved from a situation where only one reviewer gave a positive rating(ICML) to one where only one reviewer expressed reservations, with all relevant ratings at 5 or higher(NeurIPS). We saw this as progress, while recognizing that incorporating feedback across rounds introduced some new wording and labeling problems (as you pointed out, some flaws were introduced while incorporating previous reviewer comments, for which we are very grateful). In the previous NeurIPS round, the initial reviews made us optimistic, but the wording and tone of a subsequent exchange with one reviewer led to procedural action and a desk rejection. We had expected a favorable initial outcome this time, but instead encountered our first instance of all-negative scores, which has left us feeling disheartened and disappointed. Nevertheless, this remains an opportunity to further refine our paper and to promote our field and work.
Motivation, contributions, and longer-term direction
We study how to build competitive fully linear sequence models and whether they can support architectures and training paradigms beyond the current Transformer design, with the potential to outperform it on some tasks. Training from scratch lets a model adapt to its linear state mechanism throughout training and may yield better results than conversion. Given the cost of large-scale pretraining, converting an existing LLM through controlled continued pretraining offers a less expensive setting for comparison. Our contributions have two main aspects.
Architecture. First, we present a unified approach to state expansion by unifying MVA, GPA (referring to its earlier version, PLA), and MoM. We link state expansion to multi-layer Transformers, highlighting the connection between their recurrent residual structure and the attention–MLP stacks of Transformers. Drawing on the reconstruction perspective in MVA and PLA/GPA, we analyze the approximation error and specify the coding schemes along with their applicable conditions. RSA also employs a data-dependent cross-level readout, where shallow-level relevance constrains deeper retrieval and reduces the unrelated content aggregated by dense readouts (GPA or full attention). These two core operations constitute our main contributions. Regarding the contribution of sparse attention, its primary advantage lies in suppressing noise in long sequences; therefore, the RSA–PLA retrieval comparison in Table 12, the task-finetuning results in Table 16, and other length-extrapolation experiments demonstrate its advantages over PLA and full-attention models (such as Mistral and Qwen) on the tested tasks. We also provide the corresponding analysis and kernel implementations in this paper.
Results across scales. We report training from scratch at 360M and 1.3B parameters and conversion from pretrained Qwen2.5 at 3B, 7B, and 14B. We believe these results provide the first empirical answer across scales to whether fully linear sequence models can scale in parameter count. We can also provide results at 32B and 70B if helpful. The same recurrent storage and readout design works across several scales. After conversion, fully linear models approach strong same-scale softmax attention baselines on several tasks and surpass them on some. We view training data as one of the main factors behind the MMLU gap you identified. As the continued pretraining (CPT) budget grows, RSA’s scores generally improve, while the Qwen control fluctuates below its original checkpoint score. The gap gradually narrows. At 7B parameters and 80B CPT tokens, RSA scores 61.5 and Qwen2.5 scores 61.0. To our knowledge, RSA is the first architecture to show across multiple parameter scales that fully linear models can approach or exceed Qwen2.5, a leading softmax attention baseline at the time.