- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose that input-dependent selectivity is what determines when mamba hybrid architectures begin to need attention. The argumentation follows a few levels based on my read. The first 3 are in section 5 and make good sense to me - all of these are synthetic experiments.
- Selectivity is important - this is section 5.
- "Selectivity extends the range by keeping filler tokens from eroding the state"
- It isn't just the average value of delta
- Section 6 - this one felt weird to me (more notes in weaknesses section), but the authors are trying to show that recall range can predict when attention becomes "useful" and then that selectivity can affect shift attention onset. However, the authors also acknowledge that this set up is flawed (I think its a trivial fact?) - attention can only be useful in this setup when SSM recall gets bad. So, almost trivially, there is a strong correlation between the %g_50% metric and attention onset. To me this is severely limiting the work... I guess it is good that it has been acknowledged, I just can't grasp how useful the paper is given the limitation?
- Can we use the metric to change attention onset - do a few shenanigans to under two interventions and see if to attention onset relationship transfers from one intervention to another. I believe the answer is no?
Overall, the paper takes a lens at the hybrid mamba architecture. I think the experiments were interesting, I am not quite sure if the results are useful, but I think the work is largely sound, although very difficult to piece together (perhaps because there is so much)
Critical strengths
Broad and thoughtful experiments: I like the breadth of experiments here. The authors investigate the role of selectivity from several angles, including training-time ablations, evaluation-time interventions, separating the decay and write mechanisms, and testing pretrained hybrid models. The experiments feel carefully designed and go beyond simply reporting correlations.
Interesting mechanistic insights: The finding that selectivity improves recall primarily by suppressing memory decay on filler tokens, rather than preventing new information from being written, is interesting. I also like the attempt to connect this low-level SSM behavior to the usefulness of attention in hybrid architectures.
Careful analysis: The authors make a reasonable effort to control for confounders, including average timestep, information leakage through attention ablations, and the accuracy ceiling in their attention-onset metric. They are also fairly transparent about which causal claims their experiments do not establish.
Critical weaknesses
Unclear significance of the findings: My biggest concern is that I'm not sure how meaningful these results are for actual hybrid language models. The paper establishes some interesting relationships on synthetic associative-recall tasks, but it's unclear whether these insights translate to practical model design, training, or downstream performance. In particular, the authors don't show how these findings could inform architectural decisions or improve an existing model.
Weak evidence in pretrained models: The pretrained-model experiments are much less convincing than the synthetic ones. SSM-only recall is quite low even at short distances, and the proposed recall handoff is clear mainly in Granite. This makes me question how relevant the observed mechanism is to the behavior of actual hybrid LLMs.
The attention-onset relationship may be somewhat trivial: The main predictive result is potentially less interesting than it first appears. Since attention onset is defined as a 20-percentage-point improvement, it is necessarily constrained by the SSM's baseline recall accuracy. The authors acknowledge this and provide a ceiling-based baseline, but it's still unclear how much additional insight provides beyond characterizing when the SSM starts failing.
Limited causal evidence for the central relationship: While the authors convincingly show that selectivity affects recall range, they don't establish that recall range itself determines attention onset. Their within-model interventions fail to generalize, leaving the main relationship supported primarily by correlations across separately trained models.
Questions for authors
- What are the practical implications of these findings for hybrid LLM design?
- How much of the attention-onset prediction is explained by the accuracy ceiling?
- Why does the relationship fail to generalize under evaluation-time interventions?
3 comments
Response to Reviewer 66rj (2/2): pretrained evidence (W2), practical implications (Q1, W1), and presentation
W2 (pretrained evidence). A new direct test is positive in both pretrained models: in Granite-4.0 and Nemotron-H-8B, flattening tokenwise timestep variation increases the accuracy cost of replacing query-position attention (at gap 32, by 0.490 [0.365, 0.594] and 0.365 [0.250, 0.479]), while attention-intact accuracy on single-key probes stays at 0.96–1.00 (§7, Figure 4; the table in the common response, §1, gives each gap). Within each fixed checkpoint, the same replacement becomes more costly after flattening while intact recall remains high, so recurrent gating changes reliance on query-time attention. In Nemotron-H, the query-position test avoids the near-floor recall left by replacing attention at every position: attention outputs at earlier positions stay active. With query-position attention replaced, its recall is 0.21–0.45 under native timesteps and 0.03–0.10 under flattening at gaps 4–32. In the testbed, joint training from initialization yields the same signature as most pretrained hybrids we profiled: recall depends strongly on attention at every tested distance, with autonomous recall near vocabulary chance (12/12). This is consistent with the low autonomous recall of jointly pretrained checkpoints.
Q1 and W1 (practical implications). The immediate use is to measure how much recall a hybrid checkpoint retains under input-independent attention replacement, without leaking sequence content, and to compare that quantity across training choices. In our controlled testbed, this diagnostic separates hybrids whose recall depends strongly on attention from hybrids that retain recall after its replacement: all 12 trained jointly from initialization depend on attention, whereas all 24 with a long SSM head start (at least 1500 of 3900 steps) retain recall. It thus gives a measurable criterion for evaluating training schedules meant to strengthen recurrent recall.
- Selectivity also extends how far beyond its training context the recurrent path recalls: under our training recipe, for the tested horizons , 64, and 128, a non-selective SSM's range is about twice its training horizon, and a selective SSM's is 5–9 times (Appendix I, Table 8; common response §2).
These diagnostic and training-schedule implications are established on associative-recall probes.
Presentation. §1 now ends with a "Contributions and organization" paragraph that links §4–§7.
Follow-up for Reviewer 66rj
Dear Reviewer 66rj, thank you again for your detailed review. As the discussion period ends on November 18, we would like to check whether our response answers your questions.
On the pretrained evidence (W2), the new test is positive in both models: flattening timestep variation increases the cost of replacing query-position attention by 0.490 [0.365, 0.594] in Granite-4.0-h-tiny and 0.365 [0.250, 0.479] in Nemotron-H-8B at gap 32, while attention-intact recall stays at 0.96–1.00 (§7, Figure 4).
On practical implications (Q1), the leakage-free attention replacement separates testbed hybrids that depend on attention (12/12 trained jointly from initialization) from those that retain recall (24/24 after a long SSM head start; Appendix I, Table 9).
On the ceiling (Q2), with fitting and interpolation matched, reduces held-out onset error from 4.95 to 3.94 tokens relative to the fixed 0.8 crossing, while a threshold chosen by nested cross-validation has similar error (§6, Table 2).
Please let us know whether any concerns remain that we could clarify before discussion closes.
Response to Reviewer 66rj (1/2): the accuracy ceiling (Q2, W3) and evaluation-time interventions (Q3, W4)
Thank you for the detailed summary, which matches our reading of the paper's structure, and for finding the experiments broad and carefully designed. Brackets give 95% intervals: seed-block bootstrap for the tiny models (three seeds per condition) and paired prompt bootstrap for the pretrained models (96 prompts per gap for the interaction experiments).
Q2 and W3 (how much of the onset prediction the ceiling explains). With fitting and interpolation matched, g50 reduces held-out MAE from 4.95 to 3.94 tokens relative to the fixed 0.8 crossing. A threshold chosen by nested cross-validation has similar error to g50 (common response §3; §6, Table 2). What predicts onset is the SSM's measured recall curve: g50, g80, the recall area, and crossings at thresholds 0.60–0.80 reach 3.28–4.95 tokens of held-out error, against 10.29 without a recall curve (Appendix E, Table 4).
Q3 and W4 (why evaluation-time interventions do not transfer; your summary point 5). The interventions also act on the SSM layers downstream of the inserted attention, which were trained only on native Δ statistics. Pushing those layers off their training distribution plausibly explains why onset then reflects the lower recall that the hybrid reaches with attention rather than the range. §6 presents this explanation as consistent with the data.
Part 2 below covers the pretrained evidence (W2), practical implications (Q1, W1), and presentation.