acceptodds
Under review as a conference paper at ICLR 2027

Beyond State Capacity: Why Retrieval Breaks in Mamba

Abstract

Mamba models underperform Transformers on in-context retrieval. This gap is usually attributed to Mamba's fixed-size recurrent state, which loses information as the context grows. We challenge this assumption by focusing on query-first retrieval, a setup in which the query precedes the context and state capacity should not be the binding constraint. We show that even in this setting Mamba lags substantially behind Transformers, despite its selection mechanism, which was designed to distinguish relevant content from distractors. We then demonstrate that the Mamba selection gates have sufficient control over the state updates to influence the model output: intervening on specific gates alone can swing accuracy between extremes. However, as gates are input-dependent, they require the incoming token's representation to encode its relevance to the query. We therefore probe token representations across layers, and find that the relevance signal emerges only gradually with model depth, and that the gates begin to reflect it only several layers deeper. Crucially, we find that gates in retrieval-critical layers are not yet aligned with relevance. The selection mechanism is thus expressive enough to solve the task, but it commonly operates on a relevance signal that has not yet arrived. Mamba's retrieval bottleneck therefore extends beyond state capacity: even when capacity is sufficient, the delayed emergence of relevance information across depth often prevents successful retrieval.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mamba, In-context retrieval, Interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wshUOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies why state-space models such as Mamba perform poorly at in-context retrieval, studying query-first retrieval with and without a target token applied on the given item. The authors show that (1) adding this target token to the needle improves retrieval accuracy, (2) controlling the gating controls the accuracy change between flagged and unflagged prompts, and (3) flagged samples are linearly decodable earlier.

Critical strengths
  • Studying in-context retrieval is an interesting problem, and the effect of transformers/state-space models/hybrid architectures on how models learn to perform in-context retrieval is underexplored.
Critical weaknesses
  • The evaluation throughout is limited. While there does seem to be few open-weight pure Mamba models, this does limit the broader applicability of the results especially given the weaker results on Codestral-Mamba compared to Nvidia-Mamba. In addition, the main results are limited to a single synthetic query-first phonebook dataset that only differs in the use of a local relevance signal 'target', and it is unclear how robust these findings are to prompt variations in general. Finally, all results seem to be based on a single run and no measure of variance is provided.
  • The long-term utility of the research findings in this work seems limited. The authors claim "that fixed state capacity does not fully account for Mamba’s retrieval failures" but it is unclear how the new explanation for retrieval failures makes any new predictions about how to design or train SSMs differently or whether these results would hold on hybrid models.
Questions for authors
  • Could additional probing analysis be provided? 1M x 20 dimension probes with 12K independent prompts is overparameterized. Do these results hold with PCA/regularization?
  • Why can't hybrids be run with this exact same experimental setup? Would this gap hold? And if these results do not hold what can still be taken away from this work?
  • The dataset used seems to differ from prior work? The paper uses the suffix "Okay, that is the end of the phonebook. I am interested in the phone number I asked for. The phone number I asked for above is:" but Waleffe et al. (2024) uses "Okay, that is the end of the phonebook. Remember when I asked you to memorize the phone number for Jake Barron? What is the phone number for Jake Barron?"
    • Does the accuracy boost from the target token hold under this matched setup? I assume the new setup is harder than the old one.
    • The main text also says "followed by a phonebook and a request to recall the phone number associated with a target name" which follows the old setup but Figure 5 does not repeat the target's name.
  • Could error bars be calculated throughout this work? Especially on Codestral-Mamba where the effects are weaker.
  • I am confused by these gating results. It seems that intervening on the gate mechanisms would work almost by design? The tokens being gated are the tokens that the model needs to retrieve and forcing/preventing them from being written should directly track with accuracy. Could the authors provide more information about why this result is surprising?
3 comments
AuthorsOpenReview ↗

Response to reviewer wshU

Thank you for your review. We’re glad you find in-context retrieval across architectures an interesting and underexplored problem. Following your comments, we ran several new experiments (error bars across seeds and a hybrid model), and we address each point below.

Evaluation scope and error bars (W1, Q4). As you note, few open-weight pure Mamba models exist. We chose two modern, large checkpoints that share an architecture but differ in training: NVIDIA-Mamba is a base model, while Codestral-Mamba was instruction-tuned on code. We’d respectfully push back on Codestral’s results being weaker. Its accuracy is higher due to more extensive training, so effect sizes are on a different scale, but every pattern we identify (flag impact, gate importance, delayed relevance, gate miscalibration) appears in it as well. We see this as strengthening our conclusions. On task variety, we agree this is a limitation, and our Limitations section discusses it along with our reasons for using a controlled synthetic task. Finally, following your suggestion, we reran all experiments with 5 seeds, varying both the dataset content (names, numbers) and probe training (initialization, mini-batch). Variance is small, and every phenomenon reported in the paper holds across all seeds, including on Codestral-Mamba. The revised paper will include error bars throughout.

Long-term utility (W2). Our account makes a prediction that a capacity explanation doesn’t: recalibrating a handful of gates at fixed state size should recover retrieval, which our gate scaling confirms. It also points to two concrete directions. Architecturally, LSTM and GRU gates are computed from both the input and the recurrent state, while Mamba’s are computed from the token representation alone. Relevance must therefore be built into the representation before any gate can act on it, which matches the delay we observe. On the training side, recipes that encourage earlier contextualization could address that delay. We’ll expand the conclusions to make these directions explicit.

Why the gating results are informative (Q5). Our methodology is to isolate which of Mamba’s mechanisms work and which don’t, tracking both the gates and the relevance signal that should drive them. Query-first retrieval is a task Mamba’s selective design should handle, so its failure means some component isn’t doing its job. You’re right that a well-functioning gate should control what gets written to the state, and our patching confirms exactly that. Transplanting the model’s own flagged gate values at only the 10 target tokens, with everything else unchanged, recovers most of the flagged accuracy, and amplifying them yields near-perfect retrieval. The fact the gates work this well once calibrated is the key finding. It rules out the gating mechanism itself and narrows the failure to what comes before it, which is where tracking the relevance signal pays off: our probing shows it emerges in the representations late and reaches the gates later still.

Hybrids (Q2). Following your question, we ran the NVIDIA Mamba-2-Hybrid 8B on our task. Its early Mamba layers, which precede the first attention layer at layer 11, show the same delayed emergence of relevance in both representations and gates. Accuracy, however, is near-perfect up to slightly beyond the training context length, consistent with Waleffe et al. (2024), since attention carries most of the retrieval in hybrids (Bick et al., 2025). What remains is that Mamba’s selection mechanism was designed for exactly this kind of filtering, yet it fails at it, whether used alone or inside a hybrid. In hybrids, Mamba layers that filter as designed could share part of the retrieval load rather than leaving it to attention alone.

Probing (Q1). Each probe was trained on ~500K samples, not 12K: we sampled up to 64 states per prompt per layer. More importantly, our conclusions rest on comparing flagged and unflagged probes that were trained identically and evaluated on held-out data, not on absolute accuracy, so any overfitting affects both equally. Across more than 100 probe pairs (every layer of both models), flagged accuracy was always greater than or equal to unflagged. PCA at this dimensionality wasn’t feasible under our storage constraints, and data-independent reductions proved too lossy.

Prompt format (Q3). We changed the suffix deliberately. Waleffe et al. state the query both first and last, while our setting is strictly query-first, which is what removes limited state capacity as the explanation for failure. Repeating the name at the end would let the model rely on the query-last cue, which could introduce artifacts. In early experiments, accuracy was very similar under both formats. We’ll also clarify the wording in the main text, which currently suggests the name is repeated.

I agree with Reviewer BVdx's point that the hybrid results presented are not very convincing. If attention carries the retrieval in hybrids, then finding the same delayed emergence for the early Mamba layers seems not particularly useful since accuracy remains high. The authors also claim that "Mamba’s selection mechanism was designed for exactly this kind of filtering" and that "Mamba layers that filter as designed could share part of the retrieval load rather than leaving it to attention alone." However, the results seem to just be supporting the claim that pure Mamba models cannot do retrieval well.

Additionally, ICLR allows for updated PDFs if the authors would be able to share the results shared in this response (error bars, hybrid results, prompt format results, etc.) Most notably, I do not understand the answer to Q3 (why does "query-last cue" introduce "artifacts"?) and if accuracy was similar then there should be no need to change formats from prior work.

Thank you for the follow-up. As noted in our general comment, the revised PDF now includes error bars over seeds and a new appendix with the hybrid results.

Importance for hybrids. The selection mechanism is Mamba’s defining component. S4’s input-independent dynamics proved limiting, and making the update input-dependent is what made Mamba more competitive. Our results suggest that input-dependence alone is insufficient, and identifying this is a step toward the next improvement. Recent recurrent blocks with better state management (Gated DeltaNet-2, Raven) improve in-context abilities in hybrids as well as in pure models.

Our results on hybrids. The new appendix shows that attention compensates for the delayed relevance signal rather than resolving it. In the hybrid’s Mamba layers, relevance emerges in the representations with a similar delay as in pure Mamba. The gates lag further behind, even after the first attention layer. The hybrid solves our task because attention performs the retrieval, not because attention helps the Mamba layers overcome the delay. This is consistent with the previous paragraph. If attention resolved the problem, improving the recurrent layers would add little to hybrids. Instead, the results suggest it could add, on harder in-context tasks where hybrids are far from perfect, unlike our task, which was designed to isolate the mechanism in pure Mamba.

On the prompt format (Q3). The slight prompt change is essential to test our hypotheses. Our study is defined by the query-first setting, in which the query appears only before the context. This is what lets us rule out state capacity as the explanation for failure. Waleffe et al.’s prompt states the query both before and after the phonebook. Restating it at the end would reintroduce the query-last regime, where the model can attempt recall at output time and the capacity explanation applies again. Failures could then come from either source, so we could no longer attribute them to the query-first mechanism. This confound is what we mean by “artifacts.”

We’d also note that this prompt is not a standard benchmark. The phonebook task was introduced by Jelassi et al. in a query-last form. Waleffe et al. adapted it with a natural-language template suited to their analysis, and we adapted it in turn to isolate the query-first setting. We document the exact prompt in Figure 5 and release our code. We checked early on that removing the repeated query does not meaningfully change accuracy, which is consistent with your intuition that the formats would behave similarly. That check is also why we don’t see the format change as affecting our conclusions. Importantly, as mentioned above, we could not use the original prompt to test our hypotheses. Finally, we’ve corrected the main-text wording that implied the name is repeated at the end.

Reviewer_Z97BOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why Mamba models still struggle with in-context retrieval in a query-first setting, where the limitations of fixed-size state capacity are expected to be less significant.

The authors show that selection gates heavily influence the retrieval performance with controlled retrieval setups, states probing, and gate patching experiments. They also find that target information becomes decodable from token representations before it could be reflected from the gate representation. This delay in gate representation is argued to contribute to the weak retrieval performance of Mamba models in addition to the limited state capacity.

Critical strengths

-The paper investigates retrieval failures of Mamba models beyond the commonly discussed state capacity limitation. This is an interesting problem to investigate. -The query-first retrieval task and the flagged/unflagged comparison provide a controlled setting for studying Mamba's behavior. The experiment design is easy to follow and clear. -The gate patching experiments show that changing gate values can significantly improve or reduce retrieval accuracy. This is a strong result. -The representation and gate probing experiments provide insights into how relevance information changes across layers. This is interesting behaviour analysis to understand how Mama models behave.

Critical weaknesses

The explanation of delayed relevance information is not fully established. Figure 4 shows that the relevance information already exists in token representations in early layers while it cannot be reflected in corresponding gates. This could also be interpreted that the gate projections fail to effectively use relevance information available in the token representations. The paper does not clearly distinguish between these two explanations. The effect of local target markers is unclear. In the flagged setting, the word “target” is always and only placed before the desired entry. Therefore it is unclear whether the model identifies information based on its relevance to the query or the model simply responds to the marker. State capacity issue is not fully isolated. Apart from providing queries before content, the authors do not show that the limited state capacity is no longer a problem in the setting.

Questions for authors

Q1. Delayed relevance information vs. gate calibration In Figure 4, we can see that the relevance information already exists in the token representations in some early layers, but the gate representations do not reflect it well. Can authors show that this is due to delayed relevance information rather than poorly calibrated gate projections? Could the authors test whether adjusting only the gate projections with frozen model parameters improves retrieval performance?

Q2. State capacity Although the query-first setting reduces the amount of information that must be remembered by the model, the target information still needs to survive through subsequent distractors. Could the authors further analyze the contribution of state decay and interference to retrieval failure, for example by varying the number of distractors after the target entry?

1 comment
AuthorsOpenReview ↗

Response to reviewer Z97B

Thank you for the careful reading and thoughtful comments. We’re glad you found the problem interesting, the experimental design clear, and the gate patching results strong. Your questions go straight to the core of our argument, and we address each below.

Delayed relevance vs. gate calibration (W1, Q1). The bidirectional patching shows that the gates clearly modulate retrieval: patching in either direction moves accuracy accordingly. We see delayed relevance and gate calibration as two sides of the same coin rather than competing explanations. A stronger relevance signal is more readily decoded by the gates, as our flagged accuracy results show, and better-calibrated gates make better use of the relevance signal, as our patching results show. Figure 4 reflects both. While relevance is not yet decodable from the token representations, the gates are at chance (probe AUC ≈ 0.5). Even once it is decodable, it reaches the gates only a few layers later. Following your suggestion, we looked into training only the gate projections with the rest of the model frozen. It’s a promising direction, and we’ll include it as future work. Because the two effects are intertwined, though, we expect it would show both at play rather than isolate one: better projections can read relevance more effectively, but only where the representation already carries it. A careful design also involves many choices and substantial compute, which put it beyond the rebuttal period. We’ll rewrite this section to make the relationship between the two explicit.

Effect of the local marker (W2). To answer your question directly: the model most likely responds to the marker itself rather than relating the entry to the query. We tested this by varying the number of entries before the target while keeping those after it fixed. Unflagged accuracy decays steadily, consistent with the query fading from the state, while flagged accuracy is barely affected. This is by design, and it doesn’t affect our conclusions, because the flag isn’t meant to test query matching. It serves as a reference: it shows how Mamba behaves when relevance is available locally, at the target itself, rather than having to be inferred from a query seen much earlier, and the gap between the two measures the cost of that inference. The flag’s benefit also runs through the gates rather than the marker token: patching flagged gate values into unflagged prompts, which contain no marker at all, recovers the flagged accuracy. We’ll revise Section 3 to state this motivation clearly.

State capacity (W3, Q2). Our claim isn’t that capacity never hurts accuracy, but that it doesn’t prevent success here: scaling the gates of just 10 target tokens yields near-perfect accuracy at a fixed state size (Section 5.2). Following your suggestion, we varied the number of entries after the target. Accuracy declines slowly and steadily in both datasets, and re-analyzing the state data from Section 4.2 shows the target information decaying accordingly, because distractors are still written rather than skipped. So the bottleneck is state management. Capacity starts to hurt only because poor selection fills the state with irrelevant information.

Reviewer_BVdxOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

State-space models and subquadratic architectures show promising directions, but create a tradeoff by sacrificing the context, i.e., limited state capacity. This is fairly obvious in exact retrieval tasks where the query in the end asks to retrieve the corresponding value, but without the query a priori, one must store the whole context to retrieve with 100% accuracy. On the other hand, the authors investigate in this paper the reason why Mamba fails in query-first retrieval where the key is given earlier than the context. In this setting, an ideal recurrent model can filter through the context without loss of information since the model knows the requested key.

The authors proceed the study with two pre-trained Mamba-2 models, namely NVIDIA-Mamba 8B and Codestral-Mamba 7.3B, using a 10-digit synthetic phone book lookup task. They compare the general prompt with the variant prompt in which the relevant item is explicitly marked with the word “target”. This local relevance flag substantially improves accuracy from approximately 39% to 73% for NVIDIA-Mamba and 86% to 98% for Codestral-Mamba (Figure 3).

To investigate mechanistically where exactly this improvement occurs, the paper intervenes on the selection gates of Mamba that control the recurrent state updates. Their gate-patching records the gates at the target-number tokens during a flagged run, then substitutes those values at the corresponding positions during an unflagged run. The recipient model still receives the original unflagged prompt. This recovers most of the flagged accuracy, while transferring gates in the opposite way reduces the performance. Using the known target positions to amplify their gates produces near-perfect retrieval in both models. Overall, these results establish that target-token gating strongly influences retrieval.

Finally, the authors probe token representations and gate values across layers and find that, in unflagged prompts, relevance becomes highly decodable from token representations around layer 10, but becomes similarly decodable from gate vectors several layers later. Cumulative patching also show that most performance gains are found in early-to-middle layers. Such mismatch demonstrates that the selection gate becomes effective too late in the layers, contributing to retrieval failures beyond fixed state capacity alone.

Critical strengths
  1. Clear experimental setup and controlled comparison. The flagged and unflagged prompts provide a clear way to study the effect of locally available relevance information.
  2. The bidirectional gate patching and scaling provide neat interventions and demonstrate that target-token gates strongly influence retrieval, providing stornger evidence than probing alone.
Critical weaknesses
  1. Relevance to hybrid architectures in unestablished. Unless a pure attention-based transformer, modern competitive LLMs are based on hybrid architectures that incorporate some subquadratic architecture, such as Mamba or GLA. While pure Mamba models are interesting, only investigating these, the paper sacrifices timeliness and relevance. All experiments use pure Mamba-2 checkpoints, although Section 7 argues that the findings also matter for SSM–attention hybrids. This extrapolation is consequential because attention can change the contextual information available to Mamba gates or perform retrieval itself. I think this is a missed opportunity. The paper presents a mechanism beyond state capacity: the presence of attention in hybrid models don't have the state capacity problem, but this paper's provided mechanism thus may have a stronger explanatory power for hybrid models. The paper should establish whether the diagnosed gate misalignment remains a performance bottleneck once attention is present. A directly relevant NVIDIA 8B hybrid checkpoint is already available from the same earlier study. Evaluating it with the same retrieval task, relevance probes, and gate interventions would clarify whether attention resolves the proposed mechanism, compensates for it, or leaves it limiting. Without this comparison, the contribution remains a narrow diagnosis of pure Mamba models, with uncertain implications for practical architecture design.

  2. The evidence is restricted to one synthetic task family and two similar Mamba-2 checkpoints, without variation in state size or training, or a learned correction that works without oracle target identification. This limits whether the paper's proposed explanation captures a general architectural (or training recipe) limitation or the behavior of these particular pretrained models.

Questions for authors
  1. Does standard fine-tuning on the unflagged query-first retrieval task yield near-perfect accuracy, with architecture and state size held fixed and without oracle flags or gate interventions? If fine-tuning closes the gap, to what extent should the observed failures be attributed to the pretraining data or objective rather than a fundamental architectural limitation? It's unclear to me whether the paper's findings are robustly found in Mamba models or if it's just a current-slice-of-time byproduct of these models and their training recipes (including data). For example, if there's a possibility to boost Mamba performance by finetuning on synthetic tasks (similarly to https://proceedings.iclr.cc/paper_files/paper/2025/hash/15a321fbfc19fce9b325ec6e77dfb597-Abstract-Conference.html), then I am unsure of the value of the paper's takeaway.
  2. Could the claimed relevance-to-gating delay persist in the Mamba layers of hybrid models, and does it still affect final retrieval accuracy?
3 comments
AuthorsOpenReview ↗

Response to reviewer BVdx

Thank you for the thorough and insightful review. To respect your time we’ll keep our response short. We’re glad you found the work sound, with a “clear experimental setup and controlled comparison”. Where we disagree is on its importance. We fully agree that most competitive models today are hybrids, but believe this is largely because pure SSMs struggle with retrieval, so attention is added to compensate. What we find most striking is that Mamba’s selection mechanism was designed for exactly this kind of filtering: keeping what’s relevant and skipping the rest. Query-first retrieval is that task in its simplest form, yet Mamba still fails at it. A core design mechanism not doing its job is a problem whether Mamba is used alone or inside a hybrid, and understanding why it fails is the first step to fixing it. That’s precisely what this paper does.

In hybrids, prior work shows that attention carries most of the retrieval (Bick et al., 2025), and the experiments we ran following your suggestion are consistent with this. On our task, the NVIDIA Mamba-2-Hybrid 8B reaches near-perfect accuracy up to slightly beyond its training context length, in line with Waleffe et al.’s (2024) query-last results. Meanwhile, its early Mamba layers, before the first attention layer at layer 11, show the same delayed emergence of relevance in both representations and gates as pure Mamba. So attention compensates for the gate problem rather than fixing it, and the Mamba layers contribute little to retrieval. A Mamba that filters as designed could share that load. We’ll revise Section 7 to ground the hybrid discussion in these results.

Below we briefly address the remaining points:‾\underline{\text{Below we briefly address the remaining points:}}

Architecture or training? (Q1). Codestral-Mamba gives a partial answer. It shares NVIDIA-Mamba’s architecture but had substantial additional training on code, a retrieval-heavy domain. That training raises unflagged accuracy considerably, but every signature of the mechanism remains: late relevance emergence, miscalibrated gates in the critical layers, and accuracy governed by gate calibration. Whether targeted synthetic fine-tuning could close the gap entirely is an interesting open question that we’ll note as future work. Our aim is to locate where the failure happens, so that remedies, whether training-based or architectural, know what to target. On the architectural side, one plausible contributor is that Mamba’s Δ is computed from the token representation alone, unlike LSTM/GRU gates, which also see the recurrent state. Relevance therefore has to be built into the representation before any gate can act on it, which matches what we observe. We’ll expand this point, currently brief in Section 8.

Scope (W2). On state size, the task requires storing very little, and gate amplification reaches near-perfect accuracy at a fixed state size, so capacity isn’t the binding constraint here. On the single task family, we agree, and our Limitations section discusses this along with our reasons for a controlled synthetic task.

Thanks for keeping it short and readable, but I think I am missing some details.

the experiments we ran following your suggestion are consistent with this.

Is this written somewhere in more detail? The current manuscript seems unedited, so I don't have enough detail to interpret these experimental data.

A core design mechanism not doing its job is a problem whether Mamba is used alone or inside a hybrid, and understanding why it fails is the first step to fixing it.

I'm not sure if I agree with this. Yes, in a hybrid model, attention will pick up the slack and do the retrieval-related part. Then I'm not sure why we would still need to be concerned about the selection mechanism of Mamba. Does the paper also suggest a way of fixing it? Even if the delayed emergence is solved, state capacity is still a problem, no? Which is why I am unsure of the value of studying this problem.

Thank you for following up. We’ve now uploaded a revised manuscript. The hybrid experiments are presented in Appendix 2.

Why the selection mechanism still matters in hybrids. SSM history is a useful context here. S4’s input-independent dynamics turned out to be a limiting factor. Mamba (S6) addressed this by making the state update input-dependent through the selection mechanism, which is what made SSMs competitive. Our work suggests that this fix is itself incomplete (input-dependence alone might not be enough) because the input-representations and gates receive relevance information too late to act on it. Identifying that limitation is how the next, improved SSM gets designed, just as identifying S4’s limitation led to Mamba.

Recent work also suggests the quality of the recurrent block matters inside hybrids, not only in pure SSM. Variants that improve state management, such as (Gated DeltaNet-2, Raven), improve in-context abilities both as pure SSM and as hybrids compared with their Mamba counterparts. The task we used is simple enough that few attention layers can handle it alone. But its core operation, holding on to query-relevant information and filtering out the rest, is a basic building block of many harder in-context tasks. Since hybrids keep most of their layers recurrent, more capable recurrent layers could take on a larger share of the work, rather than leaving it mostly to attention. We should also understand Mamba’s flaws even when a particular system masks them, since hybrids may not be the last way SSMs are deployed.

But more broadly – yes, hybrid models are much more common nowadays. This might never change, because pure SSMs are limited in various ways. However, the only way one might conceive that it does change, is if we study their limitations and find ways to overcome them. Our paper takes an important step towards the first part.

On a proposed fix. We don’t offer a particular fix. Our contribution is to identify and characterize a failure mode not described in prior work, and to analyze its dynamics precisely enough to point at concrete remedies. In particular, we observe that unlike LSTM/GRU gates, Mamba’s Δ is computed from the token representation alone, without access to the recurrent state. This view suggests a specific design direction: a capable selection mechanism’s input should contain a more naturally contextualized component.

On state capacity. You’re right that state capacity remains a constraint even if late emergence is solved. The reverse is also true: improvements in capacity would not necessarily solve late emergence. State capacity is already extensively studied, with the implicit goal of identifying SSM limitations and finding the path to improving them. What we show is that state capacity is not the only limiting factor in SSMs, so solving it by itself would not necessarily be sufficient to make them useful in practice.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A revised manuscript

We thank all reviewers for their constructive feedback. We have uploaded a revised manuscript that includes all the changes mentioned in our responses. The main revisions are:

  • Variability and error bars. All figures now report the mean over 5 random seeds with ±1 std error bars, and each caption says so. The “Models” paragraph in Section 3 is now “Models and variability” and explains what the seeds control. Seed details have also been added to the relevant parts of the appendix and to the reproducibility statement.
  • Hybrid models. Appendix B was added to discuss our experiments on the NVIDIA Mamba-2-Hybrid 8B model. Section 7 summarizes it and refers to it.
  • Motivation for the local flag. The experimental strategy paragraph in Section 3 now explains more clearly why we use the “target” flag: it serves as a controlled reference for how Mamba behaves when relevance is available locally.
  • Representation emergence vs. gate miscalibration. The final paragraphs of Sections 6.2 and 6.3 now separate more sharply the late emergence of relevance in token representations from the further lag before the gates reflect it.
  • Long-term utility and future directions. The second paragraph of Section 8 is expanded to cover the broader relevance of the selection-mechanism bottleneck and possible training-based and architectural directions for addressing it.
  • Query-first setup. The dataset paragraph in Section 3 and the appendix now state explicitly that the query appears only before the phonebook, and not both before and after it.

We hope these revisions address the reviewers’ concerns, and we welcome any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.