- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies why state-space models such as Mamba perform poorly at in-context retrieval, studying query-first retrieval with and without a target token applied on the given item. The authors show that (1) adding this target token to the needle improves retrieval accuracy, (2) controlling the gating controls the accuracy change between flagged and unflagged prompts, and (3) flagged samples are linearly decodable earlier.
Critical strengths
- Studying in-context retrieval is an interesting problem, and the effect of transformers/state-space models/hybrid architectures on how models learn to perform in-context retrieval is underexplored.
Critical weaknesses
- The evaluation throughout is limited. While there does seem to be few open-weight pure Mamba models, this does limit the broader applicability of the results especially given the weaker results on Codestral-Mamba compared to Nvidia-Mamba. In addition, the main results are limited to a single synthetic query-first phonebook dataset that only differs in the use of a local relevance signal 'target', and it is unclear how robust these findings are to prompt variations in general. Finally, all results seem to be based on a single run and no measure of variance is provided.
- The long-term utility of the research findings in this work seems limited. The authors claim "that fixed state capacity does not fully account for Mamba’s retrieval failures" but it is unclear how the new explanation for retrieval failures makes any new predictions about how to design or train SSMs differently or whether these results would hold on hybrid models.
Questions for authors
- Could additional probing analysis be provided? 1M x 20 dimension probes with 12K independent prompts is overparameterized. Do these results hold with PCA/regularization?
- Why can't hybrids be run with this exact same experimental setup? Would this gap hold? And if these results do not hold what can still be taken away from this work?
- The dataset used seems to differ from prior work? The paper uses the suffix "Okay, that is the end of the phonebook. I am interested in the phone number I asked for. The phone number I asked for above is:" but Waleffe et al. (2024) uses "Okay, that is the end of the phonebook. Remember when I asked you to memorize the phone number for Jake Barron? What is the phone number for Jake Barron?"
- Does the accuracy boost from the target token hold under this matched setup? I assume the new setup is harder than the old one.
- The main text also says "followed by a phonebook and a request to recall the phone number associated with a target name" which follows the old setup but Figure 5 does not repeat the target's name.
- Could error bars be calculated throughout this work? Especially on Codestral-Mamba where the effects are weaker.
- I am confused by these gating results. It seems that intervening on the gate mechanisms would work almost by design? The tokens being gated are the tokens that the model needs to retrieve and forcing/preventing them from being written should directly track with accuracy. Could the authors provide more information about why this result is surprising?
3 comments
I agree with Reviewer BVdx's point that the hybrid results presented are not very convincing. If attention carries the retrieval in hybrids, then finding the same delayed emergence for the early Mamba layers seems not particularly useful since accuracy remains high. The authors also claim that "Mamba’s selection mechanism was designed for exactly this kind of filtering" and that "Mamba layers that filter as designed could share part of the retrieval load rather than leaving it to attention alone." However, the results seem to just be supporting the claim that pure Mamba models cannot do retrieval well.
Additionally, ICLR allows for updated PDFs if the authors would be able to share the results shared in this response (error bars, hybrid results, prompt format results, etc.) Most notably, I do not understand the answer to Q3 (why does "query-last cue" introduce "artifacts"?) and if accuracy was similar then there should be no need to change formats from prior work.
Thank you for the follow-up. As noted in our general comment, the revised PDF now includes error bars over seeds and a new appendix with the hybrid results.
Importance for hybrids. The selection mechanism is Mamba’s defining component. S4’s input-independent dynamics proved limiting, and making the update input-dependent is what made Mamba more competitive. Our results suggest that input-dependence alone is insufficient, and identifying this is a step toward the next improvement. Recent recurrent blocks with better state management (Gated DeltaNet-2, Raven) improve in-context abilities in hybrids as well as in pure models.
Our results on hybrids. The new appendix shows that attention compensates for the delayed relevance signal rather than resolving it. In the hybrid’s Mamba layers, relevance emerges in the representations with a similar delay as in pure Mamba. The gates lag further behind, even after the first attention layer. The hybrid solves our task because attention performs the retrieval, not because attention helps the Mamba layers overcome the delay. This is consistent with the previous paragraph. If attention resolved the problem, improving the recurrent layers would add little to hybrids. Instead, the results suggest it could add, on harder in-context tasks where hybrids are far from perfect, unlike our task, which was designed to isolate the mechanism in pure Mamba.
On the prompt format (Q3). The slight prompt change is essential to test our hypotheses. Our study is defined by the query-first setting, in which the query appears only before the context. This is what lets us rule out state capacity as the explanation for failure. Waleffe et al.’s prompt states the query both before and after the phonebook. Restating it at the end would reintroduce the query-last regime, where the model can attempt recall at output time and the capacity explanation applies again. Failures could then come from either source, so we could no longer attribute them to the query-first mechanism. This confound is what we mean by “artifacts.”
We’d also note that this prompt is not a standard benchmark. The phonebook task was introduced by Jelassi et al. in a query-last form. Waleffe et al. adapted it with a natural-language template suited to their analysis, and we adapted it in turn to isolate the query-first setting. We document the exact prompt in Figure 5 and release our code. We checked early on that removing the repeated query does not meaningfully change accuracy, which is consistent with your intuition that the formats would behave similarly. That check is also why we don’t see the format change as affecting our conclusions. Importantly, as mentioned above, we could not use the original prompt to test our hypotheses. Finally, we’ve corrected the main-text wording that implied the name is repeated at the end.
Response to reviewer wshU
Thank you for your review. We’re glad you find in-context retrieval across architectures an interesting and underexplored problem. Following your comments, we ran several new experiments (error bars across seeds and a hybrid model), and we address each point below.
Evaluation scope and error bars (W1, Q4). As you note, few open-weight pure Mamba models exist. We chose two modern, large checkpoints that share an architecture but differ in training: NVIDIA-Mamba is a base model, while Codestral-Mamba was instruction-tuned on code. We’d respectfully push back on Codestral’s results being weaker. Its accuracy is higher due to more extensive training, so effect sizes are on a different scale, but every pattern we identify (flag impact, gate importance, delayed relevance, gate miscalibration) appears in it as well. We see this as strengthening our conclusions. On task variety, we agree this is a limitation, and our Limitations section discusses it along with our reasons for using a controlled synthetic task. Finally, following your suggestion, we reran all experiments with 5 seeds, varying both the dataset content (names, numbers) and probe training (initialization, mini-batch). Variance is small, and every phenomenon reported in the paper holds across all seeds, including on Codestral-Mamba. The revised paper will include error bars throughout.
Long-term utility (W2). Our account makes a prediction that a capacity explanation doesn’t: recalibrating a handful of gates at fixed state size should recover retrieval, which our gate scaling confirms. It also points to two concrete directions. Architecturally, LSTM and GRU gates are computed from both the input and the recurrent state, while Mamba’s are computed from the token representation alone. Relevance must therefore be built into the representation before any gate can act on it, which matches the delay we observe. On the training side, recipes that encourage earlier contextualization could address that delay. We’ll expand the conclusions to make these directions explicit.
Why the gating results are informative (Q5). Our methodology is to isolate which of Mamba’s mechanisms work and which don’t, tracking both the gates and the relevance signal that should drive them. Query-first retrieval is a task Mamba’s selective design should handle, so its failure means some component isn’t doing its job. You’re right that a well-functioning gate should control what gets written to the state, and our patching confirms exactly that. Transplanting the model’s own flagged gate values at only the 10 target tokens, with everything else unchanged, recovers most of the flagged accuracy, and amplifying them yields near-perfect retrieval. The fact the gates work this well once calibrated is the key finding. It rules out the gating mechanism itself and narrows the failure to what comes before it, which is where tracking the relevance signal pays off: our probing shows it emerges in the representations late and reaches the gates later still.
Hybrids (Q2). Following your question, we ran the NVIDIA Mamba-2-Hybrid 8B on our task. Its early Mamba layers, which precede the first attention layer at layer 11, show the same delayed emergence of relevance in both representations and gates. Accuracy, however, is near-perfect up to slightly beyond the training context length, consistent with Waleffe et al. (2024), since attention carries most of the retrieval in hybrids (Bick et al., 2025). What remains is that Mamba’s selection mechanism was designed for exactly this kind of filtering, yet it fails at it, whether used alone or inside a hybrid. In hybrids, Mamba layers that filter as designed could share part of the retrieval load rather than leaving it to attention alone.
Probing (Q1). Each probe was trained on ~500K samples, not 12K: we sampled up to 64 states per prompt per layer. More importantly, our conclusions rest on comparing flagged and unflagged probes that were trained identically and evaluated on held-out data, not on absolute accuracy, so any overfitting affects both equally. Across more than 100 probe pairs (every layer of both models), flagged accuracy was always greater than or equal to unflagged. PCA at this dimensionality wasn’t feasible under our storage constraints, and data-independent reductions proved too lossy.
Prompt format (Q3). We changed the suffix deliberately. Waleffe et al. state the query both first and last, while our setting is strictly query-first, which is what removes limited state capacity as the explanation for failure. Repeating the name at the end would let the model rely on the query-last cue, which could introduce artifacts. In early experiments, accuracy was very similar under both formats. We’ll also clarify the wording in the main text, which currently suggests the name is repeated.