acceptodds
Under review as a conference paper at ICLR 2027

Urchin: Parametrizing Attention for Long-Context Recall

Abstract

Transformers remain the dominant architecture for recall as they cache every key-value pair from context; however, the memory requirements for this cache limit scalability to ultra-long sequences. This has motivated work into finite-state relaxations of attention, such as linear attention and state-space models (SSMs). The test-time regression perspective offers insight into such layers by viewing attention as an online nonparametric regressor from keys to values, and proposing linear key-value regressors as parametric alternatives. Linear-state methods, such as Mamba-2, Gated DeltaNet, and Kimi Delta Attention, have seen widespread success; however, while competitive with attention on language modeling, they fall short on long-context recall. Recent dual-state SSMs such as Raven address this by introducing nonlinearity into the retrieval process, yet they sit outside the test-time regression framework, leaving it unclear how to extend or improve these methods in a principled manner. In this work, we view softmax attention as a nonparametric Gaussian mixture model over the key-value space, with a mixture component centered at each observed key-value pair. By parameterizing this density estimate with a fixed number of mixture components, this framework yields a design space of fixed-state models with the same softmax nonlinearity as attention. We unify past dual-state SSMs as particular instantiations of this framework, and use our framework to derive Urchin, a new sequence layer for long-context recall and language modeling. Urchin demonstrates strong recall across tasks and reaches near-perfect passkey retrieval on 1M-token contexts, a 500x extrapolation beyond its training length.

Primary area:
foundation or frontier models, including LLMs
Keywords:
State Space Models, Long Context Recall, Attention, Gaussian Mixture Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bQG6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows a connection between attention and Gaussian mixture models (GMMs). Using the GMM perspective, maximum a posteriori (MAP) estimation, and conditional likelihood, the authors introduce a framework that allows them to derive a family of linear RNNs with memory routing, such as ABC, Gated Slot Attention (GSA), and the recent Raven architecture. Based on the proposed framework, they derive a new architecture, Urchin, focusing on improved recall.

The authors evaluate Urchin on synthetic and real-world recall tasks, as well as their own variations of needle-in-a-haystack (NIAH) tasks. Urchin demonstrates length extrapolation of up to 500× on NIAH-1 from RULER. Urchin and its hybrid variant with Kimi Delta Attention outperform other baselines on synthetic NIAH tasks.

Critical strengths
  • The proposed framework provides a useful interpretation of existing memory-based linear RNNs and enables the design of new linear RNN architectures with more sophisticated memory-update and readout rules.

  • Urchin achieves perfect performance at 500× length extrapolation on NIAH-1 from RULER. Even though the task is very simple (the haystack consists of repetitions of five short sentences), this result shows that the model is capable of effectively avoiding the storage of repetitive information.

Critical weaknesses
  • The authors claim: "We design a recall-focused learning objective around three principles: 1) fit to incoming data, 2) retain previous information from context, and 3) encourage parameter plasticity over time." However, these three principles are already part of the Miras framework (Behrouz et al., 2025). It might be more appropriate to present the proposed framework as an extension of existing work.

  • The paper is built around the idea that the derived architecture has improved recall capabilities, but the results are clearly superior mainly on NIAH-1, the simplest task, where the haystack consists of repeated text. The improvements are less consistent on more difficult synthetic tasks, such as NIAH-2 and NIAH-3, and real-world recall tasks, such as SWDE, FDA, and SQuAD. For example, the 850M-parameter Urchin model performs worse than KDA on average across the real-world recall tasks. The paper provides relatively limited discussion of the observed performance differences and failure cases.

  • The authors state: "We dive deeper into how these RNN language models perform recall." However, they mainly introduce additional tasks that are variations of existing RULER tasks. The authors do not provide an analysis or insights into which mechanisms in Urchin are responsible for the observed improvements on these new tasks.

  • Some choices made in the derivation of Urchin lack sufficient explanation: (1) The theoretically derived memory update differs from the one that includes the decay factor αt\alpha_t. The relationship between these two formulations is not clearly explained. (2) The authors do not explain why log⁡πt\log \pi_t is parameterized as a low-rank linear projection of xtx_t.

Questions for authors

NA

4 comments

Thank you for your time. We appreciate your feedback and suggestions. We plan to incorporate this discussion into a revised draft shortly.

W1

Thank you for this suggestion. We did not intend our use of these three principles to be viewed as our contribution, but rather to show that Urchin builds on foundational principles of online learning. These principles have roots extending beyond modern deep learning [2,3], as well as connections to modern sequence modeling and continual learning [1,4,5,6]. In our revision, we will acknowledge prior work when first introducing each principle, including explicit discussions of Miras [1] and other work, rather than leaving this discussion to the related work.

W2

We agree that S-NIAH-1 is a simple task. However, all of our tested baselines fail to maintain reliable recall on sufficiently long sequences, highlighting a critical failure mode of modern SSMs. By achieving 93% accuracy on 10 million token sequences, we view Urchin as a major step toward recall over arbitrarily long contexts. For more difficult synthetic tasks, Table 5 demonstrates that Urchin provides robust gains across more complicated haystacks and needle types. However, we recognize that further work is needed to push performance on these harder formats to million-token sequences.

We also recognize that KDA is a very strong short-context language model, especially noticeable on commonsense reasoning tasks. The real-world recall tasks use much shorter contexts (e.g. 2K tokens), meaning strong short-context models can still perform these well. On much longer sequences, however, we observe that KDA forgets information that Urchin retains. This behavior originally motivated our hybrid methods, as KDA acts as a short-context modeler while Urchin contributes extreme-context retention. We observe that this hybrid can help close the gap between Urchin and KDA on real-world recall.

To investigate the exact performance differences, we analyzed individual predictions from Urchin, KDA, and Urchin-KDA on real-world recall tasks (SWDE, FDA, SQuAD) at the 850M scale. By comparing examples where Urchin fails when KDA is correct, we identified recurring errors where Urchin retrieves the value for a similar, but slightly different, field. For example, in SWDE movie examples, Urchin often confuses the movie’s “genre” with “category / type.” These errors suggest that Urchin may struggle with recall when competing fields have similar semantic meanings. In our revision, we are adding a new appendix section to discuss these failure modes, along with presenting exact errors on these tasks.


References

[1] Behrouz et al. (2026). It’s All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization.

[2] Widrow, B., and Hoff, M. E. (1960). Adaptive Switching Circuits.

[3] Kivinen, J., and Warmuth, M. K. (1997). Exponentiated Gradient versus Gradient Descent for Linear Predictors. Information and Computation.

[4] Liu, B et al. (2025). Longhorn: State Space Models are Amortized Online Learners.

[5] Qin, Z., Yang, S., and Zhong, Y. (2023). Hierarchically Gated Recurrent Neural Network for Sequence Modeling.

[6] Dohare, S et al. (2024). Loss of Plasticity in Deep Continual Learning.

W3

We are adding a new appendix section to our revision to analyze the impact of each design choice in our layer. We ran separate routing and gating ablations, along with a joint-versus-conditional update comparison with routing, gating, state size, and training held fixed. These experiments allow us to discuss how each component contributes to recall across the different task settings.

All ablations use a 125M-scale Urchin model trained on 15B tokens.

ExperimentSN1-128KSN2-4K*SN3-4K*Lambada ppl / acc
Base Urchin100%41.4% → 98.0%5.2% → 45.6%113.8 / 22.3%
Router Sparsity: Top-k = 4100%4.0% → 91.0%0.8% → 36.8%126.2 / 20.5%
Dense Routing2.0%23.0% → 36.8%8.6% → 10.0%135.3 / 20.0%
No αt\alpha_t0%3.4% → 37.2%0.4% → 10.4%134.5 / 19.3%
State-Agnostic Routing100%26.4% → 69.4%0% → 27.8%114.9 / 22.4%
Joint Likelihood0%2.8% → 12.6%0% → 0%176.94 / 16.9%

*For S-NIAH-2 and S-NIAH-3, we report accuracy with the default evaluation prompt and the colon-added prompt as [default → colon].

Dense-vs-Sparse routing) From Urchin’s default top-16 routing, we evaluate top-4 and dense routing. Similar to Raven, we find that sparsity helps extreme-context recall (SN1 128K), as components that are unlikely to generate the current key-value pair are not disturbed. Routing that is too sparse may restrict the update expressivity, as observed with a drop in SN2/3.

Gating) We remove αt\alpha_t (e.g. setting αt=1\alpha_t=1), deteriorating recall. Without decay, as discussed in Section 3, the accumulated state magnitudes can make components rigid, moving less freely to incoming updates. Shrinking the magnitude over time allows new observations to have a greater influence on the direction of these components. For example, at extreme context lengths as in SN1, the state norms may be too large by the time the needle needs to be stored.

State-agnostic routing) Urchin constructs ztz_t from similarities between the incoming key-value pair and the stored component means, together with the input-dependent mixture weights πt\pi_t. In this ablation, we remove both the key and value similarity terms, constructing ztz_t using only the mixture weights. S-NIAH-1 and Lambada performance remain similar, while S-NIAH-2/3 performance decreases. State-dependent routing lets the model choose where to write based on what each component currently stores. Without this feedback, the mixture weights must select components from the current input alone, potentially directing updates to components that already contain unrelated information. This result is consistent with our belief that state-dependent routing is particularly useful when the haystack contains varied information.

Joint versus conditional update) Replacing the conditional key-update rule with the joint rule substantially degrades performance, including complete failure at extreme context lengths (SN1 128K) and declines across shorter context length recall tasks (SN2/3 4K). Similarly, Lambada performance also drops. All routing, gating, training, and other design choices were held fixed to the original Urchin. Together with our original comparison against Raven—which uses different choices designed for joint modeling—our experiments and ablations establish the benefit of conditional modeling.

W4

We will add these clarifications to the revision, with changes marked in red.

  1. In Appendix J, we replaced the derived gate so that we could reuse the well-studied gate parameterization of Mamba2 and Gated DeltaNet. The gate still serves the Gaussian prior’s purpose of shrinking the stored memories. Additionally, this provides a common update rule term that other SSM researchers can relate to. We will further emphasize this in our revision.

  2. Since there are DD mixture weights πt\pi_t at each time step, a full projection from the dd-dimensional input requires dDdD weights, whereas a rank-rr projection requires r(d+D)r(d+D). As we increase the number of components, we want to limit the accompanying increase in model size and compute. We use r=128r=128 to help keep the model’s FLOPs and parameter count comparable to the other methods in our table. This strategy is also used for other SSM parameters; for example, GLA similarly uses a low-rank projection for its gate parameterization to reduce parameter overhead.

Thank you for your response. The new ablations and your clarifications address most of my concerns, so I will raise my score. It would be helpful to include the real-world recall score in the ablation table to see whether the same design choices improve performance on those tasks. I also suggest briefly discussing the key findings from the ablations in the main text.

Thank you, we have added these to the revision & included the real-world recall scores in Table 16.

Reviewer_qn43OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes a way to interpret softmax attention as a conditional mean of Gaussians and replaces the growing collection of components with fixed-size key/value memories. Urchin combines state-dependent top-16 routing, gated memory updates, and softmax readout motivated by conditional-likelihood objective with retention and parameter regularization. The authors train models with three scales and show single-passkey retention gains, million-token extrapolation, natural-text retrieval, language-modeling and prompting experiments.

Critical strengths
  1. Substantial retention gains under comparable pretraining budgets.
  • At 400M parameters, Urchin achieves 97.3% accuracy at 1M tokesn versus 5.3% for raven and 2.7% for KDA after training on 2K tokens.
  • Appendix F matches training tokens and memory slot counts among the dual-state models.
  1. The evaluation identifies prompting confound.
  • At 4K tokens, adding a colon raises the accuracy of Urchin from 47.3% to 95.2%; the paper reports both prompts for competing methods.
Critical weaknesses
  • The experiments do not establish why Urchin improves recall. Relative to Raven, Urchin changes routing, key updates, and gating. The reported ablations vary prompts and task configurations without the architectural changes. To establish the benefit of conditional modeling, the paper needs a joint-versus-conditional update comparison with routing, gating, state size, and training held fixed.

  • The derivation changes the objective and does not justify the final updates. Appendix I, Eq. 26 gives term proportional to KL(zt∣pt)\mathrm{KL}(z_t|p_t), where pt=P(ct∣kt,θt(1))p_t=P(c_t\mid k_t,\theta_t^{(1)}). Appendix J, Eq. 39 reverses this to KL(pt∣zt)\mathrm{KL}(p_t|z_t), so Eq. 46 solves a different problem. The reversed KL also creates a support mismatch. Eq. 13 sets unselected entries of ztz_t to zero, but Eq. 44 defines ptp_t as softmax. The divergence is therefore infinite for finite logits unless the formulation explicitly restricts the support. Appendix J then replaces the derived gate and drops a correction term in Eqs. 43 and 49. These changes leave the claim that Eqs. 14–15 minimize Eq. 9 unsupported.

Questions for authors

See weakness.

3 comments

Thank you for your detailed feedback.

W1

We directly ran your suggested joint-versus-conditional update comparison with routing, gating, state size, and training held fixed. Beyond this, we also ran separate routing and gating ablations to explore our design choices and discuss why specific elements of Urchin were used. We will incorporate these ablations into the revision. Together, these results show that Urchin’s extreme-context recall relies on conditional updates, sparse routing, and gating, while state-dependent routing further improves recall in more complex haystacks.

All ablations use a 125M-scale Urchin model trained on 15B tokens.

ExperimentSN1-128KSN2-4K*SN3-4K*Lambada ppl / acc
Base Urchin100%41.4% → 98.0%5.2% → 45.6%113.8 / 22.3%
Router Sparsity: Top-k = 4100%4.0% → 91.0%0.8% → 36.8%126.2 / 20.5%
Dense Routing2.0%23.0% → 36.8%8.6% → 10.0%135.3 / 20.0%
No αt\alpha_t0%3.4% → 37.2%0.4% → 10.4%134.5 / 19.3%
State-Agnostic Routing100%26.4% → 69.4%0% → 27.8%114.9 / 22.4%
Joint Likelihood0%2.8% → 12.6%0% → 0%176.94 / 16.9%

*For S-NIAH-2 and S-NIAH-3, we report accuracy with the default evaluation prompt and the colon-added prompt as [default → colon].

Joint versus conditional update) Replacing the conditional key-update rule with the joint rule substantially degrades performance, including complete failure at extreme context lengths (SN1 128K) and declines across shorter context length recall tasks (SN2/3 4K). Similarly, Lambada performance also drops. As suggested, all routing, gating, training, and other design choices were held fixed to the original Urchin. Together with our original comparison against Raven—which uses different choices designed for joint modeling—our experiments and ablations establish the benefit of conditional modeling.

Dense-vs-Sparse routing) From Urchin’s default top-16 routing, we also evaluate top-4 and dense routing. Similar to Raven, we find that sparsity helps extreme-context recall (SN1 128K), as components that are unlikely to generate the current key-value pair are not disturbed. Routing that is too sparse may restrict the update expressivity, as observed with a drop in SN2/3.

Gating) We remove αt\alpha_t (e.g. setting αt=1\alpha_t=1), deteriorating recall. Without decay, as discussed in Section 3, the accumulated state magnitudes can make components rigid, moving less freely to incoming updates. Shrinking the magnitude over time allows new observations to have a greater influence on the direction of these components. For example, at extreme context lengths as in SN1, the state norms may be too large by the time the needle needs to be stored.

State-agnostic routing) Urchin constructs ztz_t from similarities between the incoming key-value pair and the stored component means, together with the input-dependent mixture weights πt\pi_t. In this ablation, we remove both the key and value similarity terms, constructing ztz_t using only the mixture weights. S-NIAH-1 and Lambada performance remain similar, while S-NIAH-2/3 performance decreases. State-dependent routing lets the model choose where to write based on what each component currently stores. Without this feedback, the mixture weights must select components from the current input alone, potentially directing updates to components that already contain unrelated information. This result is consistent with our belief that state-dependent routing is particularly useful when the haystack contains varied information.

W2

Thank you for pointing this out. We should have explained this further, and we will update the claims in our revision.

With Eq. 39, we placed ptp_t--the quantity we aim to optimize--on the left side of the “Fit to New Data” KL term to match the “Retention” KL term. By placing ptp_t on the left side in both KL terms, the update is based on the difference between logits, rather than a probability error computed after softmax. This motivates our delta-style recurrence for θ(1)\theta^{(1)}, allowing us to use fast Triton kernels from FLA’s open-source library. While the reverse KL provides a different penalty, both fitting terms encourage ptp_t to match ztz_t. This preserves our overall fitting goal.

Regarding the support mismatch, we originally derived this using dense softmax routing for ztz_t. Sparse routing was an empirical design choice made after the dense derivation, motivated in part by its benefits in layers such as Raven and supported by our experiments, including the ablations above. As described in our method section, our implementation updates only selected components, avoiding arithmetic involving masked logits. To be more precise in our revision, we will define the sparse surrogate over the selected components, normalizing both the new and previous routing distributions on this set.

In Appendix J, we replaced the derived gate so that we could reuse the well-studied gate parameterization of Mamba2 and Gated DeltaNet. The gate still serves the Gaussian prior’s purpose of shrinking the stored memories. We will better explain this update simplification in the revision.

Our aim is to explain how the fitting, retention, and regularization terms guide the construction of a sequence layer. We will identify Eq. 39 as a surrogate, explain why we aligned the “Fit to New Data” and “Retention” KL terms to have ptp_t on the same side, clarify the sparse objective and the subsequent update design choices, and update the claims in our revision in red.

I would like to thank the authors for running the ablations and adding clarification remarks about the derivation. I still believe that the abstract and the introduction has to be toned down, as it seems that Equations 14 and 15 are motivated/inspired from Equation 9 instead of being a direct derivation, which is absolutely fine, as long as the claims and statements are accurate.

The empirical retention results are strong, and the ones provided in the rebuttal directly answer my concerns. However, the theory serves as motivation only. I still feel this paper is borderline closer to reject, but I am open to discussions if other reviewers think this paper deserves an accept.

I raise my rating and soundness score to 2 accordingly.

Reviewer_7qjZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new probabilistic framework for interpreting attention beyond the test-time regression point of view. Specifically, the authors view attention as a nonparametric Gaussian mixture model (GMM) over the key-value, with a mixture component centered at each observed key-value pair. Based on this interpretation, they proposed the finite-state version (which they call Urchin) by parameterizing the GMM with a fixed number of components, which are updated with an online learning objective. Unlike finite-state deep SSMs, Urchin is able to perform well in long-context recall tasks. They demonstrate it on RULER’s S-NIAH-1 task across 1M and 10M context length, and across three model sizes (120M, 400M, 850M parameters).

Critical strengths
  1. The GMM point of view on attention is well-motivated for improving long-context recall since it performs a nonlinear function of the query, allowing for sharper retrieval for highly similar keys.
  2. Through the lens of online EM over a fixed-component GMM, it unifies previous “dual-state” SSMs (e.g., ABC, GSA, and Raven) in one framework, different from the "test-time regression" point of view.
  3. Empirically, they demonstrate Urchin can consistently achieve good recall performance on the experiments and with the model scales considered in the paper.
Critical weaknesses
  1. Line 354 "Further training and evaluation details, as well as additional ablation and interpretability experiments and full result tables, are in Appendix F". However, Appendix F only contains a single paragraph of training hyperparameters. I suggest including these details and ablations promised during the rebuttal as it would provide a more complete picture on which parts of the method are more crucial.
  2. I think it would be better to move the Related Work section (currently it's in Appendix E) to the main text. As the authors mentioned there, "The connection between attention and Gaussian mixture models has been explored in a variety of context", proper attribution to previous works is important for novelty assessment. Currently, purely from the main text, it gives me an impression that viewing attention as a GMM and online EM for fixed-state attention is a novel contribution of this paper, which is not accurate.
  3. Line 102 "Furthermore, we show that Urchin layers can be used in combination with state-of-the-art SSMs such as KDA to achieve strong performance on both language modeling and long-context recall." - Does it mean the bad recall performance of deep SSMs is due to stacking just SSM layers? If so, hybrid architectures combining both attention and SSM layers may also be free from the degraded recall performance (how do you contract Urchin with these hybrid architectures)?
  4. Would it be possible to turn a pretrained Transformer to a Urchin model, so that Urchin doesn't need to be trained from scratch every time? It's a bit unclear to me if a model proposed specifically for improving recall (a specific type of downstream task) deserves a costly pretraining procedure.
Questions for authors

See Weakness.


After reading the authors' rebuttal, I decided to raise my score from 2 to 3.

3 comments

Thank you for feedback and thorough review. We will incorporate your feedback and this discussion into the revision, which will be updated soon.

W1

Thank you for pointing this error out. We meant to direct readers to Appendix F for training details, Appendix D for the task and prompting ablations, and Appendices A–C for the full result tables. We will correct this in the revision.

We are also adding a new architectural ablation section to analyze which design choices contribute to Urchin’s recall performance. This includes routing, gating, and joint-versus-conditional update rule experiments. Together, these results show that Urchin’s extreme-context recall relies on conditional updates, sparse routing, and gating, while state-dependent routing further improves recall in more complex haystacks.

All ablations use a 125M-scale Urchin model trained on 15B tokens.

ExperimentSN1-128KSN2-4K*SN3-4K*Lambada ppl / acc
Base Urchin100%41.4% → 98.0%5.2% → 45.6%113.8 / 22.3%
Router Sparsity: Top-k = 4100%4.0% → 91.0%0.8% → 36.8%126.2 / 20.5%
Dense Routing2.0%23.0% → 36.8%8.6% → 10.0%135.3 / 20.0%
No αt\alpha_t0%3.4% → 37.2%0.4% → 10.4%134.5 / 19.3%
State-Agnostic Routing100%26.4% → 69.4%0% → 27.8%114.9 / 22.4%
Joint Likelihood0%2.8% → 12.6%0% → 0%176.94 / 16.9%

*For S-NIAH-2 and S-NIAH-3, we report accuracy with the default evaluation prompt and the colon-added prompt as [default → colon].

Dense-vs-Sparse routing) From Urchin’s default top-16 routing, we evaluate top-4 and dense routing. Similar to Raven, we find that sparsity helps extreme-context recall (SN1 128K), as components that are unlikely to generate the current key-value pair are not disturbed. Routing that is too sparse may restrict the update expressivity, as observed with a drop in SN2/3.

Gating) We remove αt\alpha_t (e.g. setting αt=1\alpha_t=1), deteriorating recall. Without decay, as discussed in Section 3, the accumulated state magnitudes can make components rigid, moving less freely to incoming updates. Shrinking the magnitude over time allows new observations to have a greater influence on the direction of these components. For example, at extreme context lengths as in SN1, the state norms may be too large by the time the needle needs to be stored.

State-agnostic routing) Urchin constructs ztz_t from similarities between the incoming key-value pair and the stored component means, together with the input-dependent mixture weights πt\pi_t. In this ablation, we remove both the key and value similarity terms, constructing ztz_t using only the mixture weights. S-NIAH-1 and Lambada performance remain similar, while S-NIAH-2/3 performance decreases. State-dependent routing lets the model choose where to write based on what each component currently stores. Without this feedback, the mixture weights must select components from the current input alone, potentially directing updates to components that already contain unrelated information. This result is consistent with our belief that state-dependent routing is particularly useful when the haystack contains varied information.

Joint versus conditional update) Replacing the conditional key-update rule with the joint rule substantially degrades performance, including complete failure at extreme context lengths (SN1 128K) and declines across shorter context length recall tasks (SN2/3 4K). Similarly, Lambada performance also drops. All routing, gating, training, and other design choices were held fixed to the original Urchin. Together with our original comparison against Raven—which uses different choices designed for joint modeling—our experiments and ablations establish the benefit of conditional modeling.

W2

We will bring relevant discussions of prior work from the appendix, including connections between attention and GMMs and EM-based methods, into the introduction and preliminaries to better clarify our contributions and novelty. Our contribution extends the GMM perspective to designing novel SSMs for storing and recalling past memories. Moreover, we are expanding our related-work section to better present where we draw inspiration for our work. We believe these changes will provide sufficient support in the main body, while retaining an in-depth related-work section in the appendix that is not restricted by the main-text page limit.

W3

Yes, a model composed entirely of SSM layers must compress the past context into a finite-sized state, which is the central challenge for recall. Hybrid attention-SSM models bypass this because their global attention layers retain direct access to past key-value pairs in a cache. However, these architectures fall outside of our focus on models that can process arbitrarily long sequences with a fixed memory budget. While hybrids with fewer full-attention layers are cheaper than standard Transformers, those layers still maintain growing KV caches. This limits the maximum context length that can fit onto the hardware.

Hybrid models, however, bring up an interesting separation between “short-context” and “long-context” layers. Here, Urchin represents a strong long-context layer with constant memory costs. This motivated our use of Urchin-KDA hybrids, as KDA represents a strong short-context language modeler. With this work, we hope to encourage the development of other long-context layers with constant memory footprints, potentially co-designed with short-context layers such as KDA.

W4

Yes, in future projects, we plan to distill Urchin from pretrained Transformers for more efficient training and model scaling. Existing approaches such as MOHAWK [1] and LoLCATs [2] have demonstrated cross-architecture distillation into Mamba and linear-attention layers. However, LoLA [3] shows that the evaluated LoLCATs-8B model struggles with recall, achieving only 3.2% accuracy on S-NIAH-1 at 1K context, despite being distilled from Llama-3.1-8B. We hypothesize that Urchin’s more attention-like structure, particularly its softmax readout, may make it easier to mimic a Transformer teacher than layers with linear readouts.

While we believe this line of work is promising, we first needed to establish whether Urchin’s design improves recall when trained under the same conditions as other sequence layers. Our from-scratch experiments address this without making the comparison depend on how well each architecture imitates a particular teacher. This paper lays the foundation of Urchin, which opens doors to more efficient training methods like cross-architecture distillation.

References

[1] Bick et al. Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models.

[2] Zhang et al. LoLCATs: On Low-Rank Linearizing of Large Language Models.

[3] McDermott et al. LoLA: Low-Rank Linear Attention with Sparse Caching.

Thanks for the detailed rebuttal and my concerns are mostly addressed. I have raised my score to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.