- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper shows a connection between attention and Gaussian mixture models (GMMs). Using the GMM perspective, maximum a posteriori (MAP) estimation, and conditional likelihood, the authors introduce a framework that allows them to derive a family of linear RNNs with memory routing, such as ABC, Gated Slot Attention (GSA), and the recent Raven architecture. Based on the proposed framework, they derive a new architecture, Urchin, focusing on improved recall.
The authors evaluate Urchin on synthetic and real-world recall tasks, as well as their own variations of needle-in-a-haystack (NIAH) tasks. Urchin demonstrates length extrapolation of up to 500× on NIAH-1 from RULER. Urchin and its hybrid variant with Kimi Delta Attention outperform other baselines on synthetic NIAH tasks.
Critical strengths
-
The proposed framework provides a useful interpretation of existing memory-based linear RNNs and enables the design of new linear RNN architectures with more sophisticated memory-update and readout rules.
-
Urchin achieves perfect performance at 500× length extrapolation on NIAH-1 from RULER. Even though the task is very simple (the haystack consists of repetitions of five short sentences), this result shows that the model is capable of effectively avoiding the storage of repetitive information.
Critical weaknesses
-
The authors claim: "We design a recall-focused learning objective around three principles: 1) fit to incoming data, 2) retain previous information from context, and 3) encourage parameter plasticity over time." However, these three principles are already part of the Miras framework (Behrouz et al., 2025). It might be more appropriate to present the proposed framework as an extension of existing work.
-
The paper is built around the idea that the derived architecture has improved recall capabilities, but the results are clearly superior mainly on NIAH-1, the simplest task, where the haystack consists of repeated text. The improvements are less consistent on more difficult synthetic tasks, such as NIAH-2 and NIAH-3, and real-world recall tasks, such as SWDE, FDA, and SQuAD. For example, the 850M-parameter Urchin model performs worse than KDA on average across the real-world recall tasks. The paper provides relatively limited discussion of the observed performance differences and failure cases.
-
The authors state: "We dive deeper into how these RNN language models perform recall." However, they mainly introduce additional tasks that are variations of existing RULER tasks. The authors do not provide an analysis or insights into which mechanisms in Urchin are responsible for the observed improvements on these new tasks.
-
Some choices made in the derivation of Urchin lack sufficient explanation: (1) The theoretically derived memory update differs from the one that includes the decay factor . The relationship between these two formulations is not clearly explained. (2) The authors do not explain why is parameterized as a low-rank linear projection of .
Questions for authors
NA
4 comments
W3
We are adding a new appendix section to our revision to analyze the impact of each design choice in our layer. We ran separate routing and gating ablations, along with a joint-versus-conditional update comparison with routing, gating, state size, and training held fixed. These experiments allow us to discuss how each component contributes to recall across the different task settings.
All ablations use a 125M-scale Urchin model trained on 15B tokens.
| Experiment | SN1-128K | SN2-4K* | SN3-4K* | Lambada ppl / acc |
|---|---|---|---|---|
| Base Urchin | 100% | 41.4% → 98.0% | 5.2% → 45.6% | 113.8 / 22.3% |
| Router Sparsity: Top-k = 4 | 100% | 4.0% → 91.0% | 0.8% → 36.8% | 126.2 / 20.5% |
| Dense Routing | 2.0% | 23.0% → 36.8% | 8.6% → 10.0% | 135.3 / 20.0% |
| No | 0% | 3.4% → 37.2% | 0.4% → 10.4% | 134.5 / 19.3% |
| State-Agnostic Routing | 100% | 26.4% → 69.4% | 0% → 27.8% | 114.9 / 22.4% |
| Joint Likelihood | 0% | 2.8% → 12.6% | 0% → 0% | 176.94 / 16.9% |
*For S-NIAH-2 and S-NIAH-3, we report accuracy with the default evaluation prompt and the colon-added prompt as [default → colon].
Dense-vs-Sparse routing) From Urchin’s default top-16 routing, we evaluate top-4 and dense routing. Similar to Raven, we find that sparsity helps extreme-context recall (SN1 128K), as components that are unlikely to generate the current key-value pair are not disturbed. Routing that is too sparse may restrict the update expressivity, as observed with a drop in SN2/3.
Gating) We remove (e.g. setting ), deteriorating recall. Without decay, as discussed in Section 3, the accumulated state magnitudes can make components rigid, moving less freely to incoming updates. Shrinking the magnitude over time allows new observations to have a greater influence on the direction of these components. For example, at extreme context lengths as in SN1, the state norms may be too large by the time the needle needs to be stored.
State-agnostic routing) Urchin constructs from similarities between the incoming key-value pair and the stored component means, together with the input-dependent mixture weights . In this ablation, we remove both the key and value similarity terms, constructing using only the mixture weights. S-NIAH-1 and Lambada performance remain similar, while S-NIAH-2/3 performance decreases. State-dependent routing lets the model choose where to write based on what each component currently stores. Without this feedback, the mixture weights must select components from the current input alone, potentially directing updates to components that already contain unrelated information. This result is consistent with our belief that state-dependent routing is particularly useful when the haystack contains varied information.
Joint versus conditional update) Replacing the conditional key-update rule with the joint rule substantially degrades performance, including complete failure at extreme context lengths (SN1 128K) and declines across shorter context length recall tasks (SN2/3 4K). Similarly, Lambada performance also drops. All routing, gating, training, and other design choices were held fixed to the original Urchin. Together with our original comparison against Raven—which uses different choices designed for joint modeling—our experiments and ablations establish the benefit of conditional modeling.
W4
We will add these clarifications to the revision, with changes marked in red.
-
In Appendix J, we replaced the derived gate so that we could reuse the well-studied gate parameterization of Mamba2 and Gated DeltaNet. The gate still serves the Gaussian prior’s purpose of shrinking the stored memories. Additionally, this provides a common update rule term that other SSM researchers can relate to. We will further emphasize this in our revision.
-
Since there are mixture weights at each time step, a full projection from the -dimensional input requires weights, whereas a rank- projection requires . As we increase the number of components, we want to limit the accompanying increase in model size and compute. We use to help keep the model’s FLOPs and parameter count comparable to the other methods in our table. This strategy is also used for other SSM parameters; for example, GLA similarly uses a low-rank projection for its gate parameterization to reduce parameter overhead.
Thank you for your response. The new ablations and your clarifications address most of my concerns, so I will raise my score. It would be helpful to include the real-world recall score in the ablation table to see whether the same design choices improve performance on those tasks. I also suggest briefly discussing the key findings from the ablations in the main text.
Thank you, we have added these to the revision & included the real-world recall scores in Table 16.
Thank you for your time. We appreciate your feedback and suggestions. We plan to incorporate this discussion into a revised draft shortly.
W1
Thank you for this suggestion. We did not intend our use of these three principles to be viewed as our contribution, but rather to show that Urchin builds on foundational principles of online learning. These principles have roots extending beyond modern deep learning [2,3], as well as connections to modern sequence modeling and continual learning [1,4,5,6]. In our revision, we will acknowledge prior work when first introducing each principle, including explicit discussions of Miras [1] and other work, rather than leaving this discussion to the related work.
W2
We agree that S-NIAH-1 is a simple task. However, all of our tested baselines fail to maintain reliable recall on sufficiently long sequences, highlighting a critical failure mode of modern SSMs. By achieving 93% accuracy on 10 million token sequences, we view Urchin as a major step toward recall over arbitrarily long contexts. For more difficult synthetic tasks, Table 5 demonstrates that Urchin provides robust gains across more complicated haystacks and needle types. However, we recognize that further work is needed to push performance on these harder formats to million-token sequences.
We also recognize that KDA is a very strong short-context language model, especially noticeable on commonsense reasoning tasks. The real-world recall tasks use much shorter contexts (e.g. 2K tokens), meaning strong short-context models can still perform these well. On much longer sequences, however, we observe that KDA forgets information that Urchin retains. This behavior originally motivated our hybrid methods, as KDA acts as a short-context modeler while Urchin contributes extreme-context retention. We observe that this hybrid can help close the gap between Urchin and KDA on real-world recall.
To investigate the exact performance differences, we analyzed individual predictions from Urchin, KDA, and Urchin-KDA on real-world recall tasks (SWDE, FDA, SQuAD) at the 850M scale. By comparing examples where Urchin fails when KDA is correct, we identified recurring errors where Urchin retrieves the value for a similar, but slightly different, field. For example, in SWDE movie examples, Urchin often confuses the movie’s “genre” with “category / type.” These errors suggest that Urchin may struggle with recall when competing fields have similar semantic meanings. In our revision, we are adding a new appendix section to discuss these failure modes, along with presenting exact errors on these tasks.
References
[1] Behrouz et al. (2026). It’s All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization.
[2] Widrow, B., and Hoff, M. E. (1960). Adaptive Switching Circuits.
[3] Kivinen, J., and Warmuth, M. K. (1997). Exponentiated Gradient versus Gradient Descent for Linear Predictors. Information and Computation.
[4] Liu, B et al. (2025). Longhorn: State Space Models are Amortized Online Learners.
[5] Qin, Z., Yang, S., and Zhong, Y. (2023). Hierarchically Gated Recurrent Neural Network for Sequence Modeling.
[6] Dohare, S et al. (2024). Loss of Plasticity in Deep Continual Learning.