acceptodds
Under review as a conference paper at ICLR 2027

Selectivity Sets the Recall Range in Hybrid Language Models

Abstract

Hybrid language models interleave a few softmax-attention layers among many state-space model (SSM) layers. It has been unclear which properties of the SSM predict the query-key distances at which attention improves recall. Replacing every attention output with an input-independent constant isolates the SSM's autonomous recall. We show that attention sustains recall at distances where this recall has declined, a handoff that is clear in Granite-4.0, weak in Nemotron-H, and absent in one parallel- and one shared-attention hybrid. Input-dependent timestep gating (Δ-selectivity) sets the SSM's recall range (g50, the gap at which chance-corrected recall falls to half its peak). It does so by keeping interfering tokens from eroding the state. On a controlled two-key associative task, turning selectivity on extends the range 2.4-fold (from 24 to 58 tokens), whereas fixing a trained selective model's timestep at each head's mean drops peak recall below query-blind guessing. Turning selectivity on during training shifts the onset of an added attention layer's benefit to larger query-key gaps. Changes in recall range quantitatively predict these shifts. Across natively trained backbones, the range predicts onset on held-out architectures with a mean absolute error of 3.9 tokens, against 10.3 for the strongest architecture-plus-selectivity baseline. In Granite-4.0 and Nemotron-H, removing input-dependent gating significantly reduces the SSM's autonomous recall while attention-on recall stays well above it. Native SSM recall depends on this gating. On Granite, fixing each sequence's timestep at its native average shrinks the range from 11 to 3 tokens without a significant drop in peak recall. Timestep gating sets how far the SSM recalls on its own, and that range predicts the query-key distance at which attention begins to help.

Primary area:
foundation or frontier models, including LLMs
Keywords:
hybrid language models, state space models, Mamba, selectivity, attention, associative recall, recall range, long-context memory

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_66rjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose that input-dependent selectivity is what determines when mamba hybrid architectures begin to need attention. The argumentation follows a few levels based on my read. The first 3 are in section 5 and make good sense to me - all of these are synthetic experiments.

  1. Selectivity is important - this is section 5.
  2. "Selectivity extends the range by keeping filler tokens from eroding the state"
  3. It isn't just the average value of delta
  4. Section 6 - this one felt weird to me (more notes in weaknesses section), but the authors are trying to show that recall range can predict when attention becomes "useful" and then that selectivity can affect shift attention onset. However, the authors also acknowledge that this set up is flawed (I think its a trivial fact?) - attention can only be useful in this setup when SSM recall gets bad. So, almost trivially, there is a strong correlation between the %g_50% metric and attention onset. To me this is severely limiting the work... I guess it is good that it has been acknowledged, I just can't grasp how useful the paper is given the limitation?
  5. Can we use the g50g_50 metric to change attention onset - do a few shenanigans to δ\delta under two interventions and see if g50g_50 to attention onset relationship transfers from one intervention to another. I believe the answer is no?

Overall, the paper takes a lens at the hybrid mamba architecture. I think the experiments were interesting, I am not quite sure if the results are useful, but I think the work is largely sound, although very difficult to piece together (perhaps because there is so much)

Critical strengths

Broad and thoughtful experiments: I like the breadth of experiments here. The authors investigate the role of selectivity from several angles, including training-time ablations, evaluation-time interventions, separating the decay and write mechanisms, and testing pretrained hybrid models. The experiments feel carefully designed and go beyond simply reporting correlations.

Interesting mechanistic insights: The finding that selectivity improves recall primarily by suppressing memory decay on filler tokens, rather than preventing new information from being written, is interesting. I also like the attempt to connect this low-level SSM behavior to the usefulness of attention in hybrid architectures.

Careful analysis: The authors make a reasonable effort to control for confounders, including average timestep, information leakage through attention ablations, and the accuracy ceiling in their attention-onset metric. They are also fairly transparent about which causal claims their experiments do not establish.

Critical weaknesses

Unclear significance of the findings: My biggest concern is that I'm not sure how meaningful these results are for actual hybrid language models. The paper establishes some interesting relationships on synthetic associative-recall tasks, but it's unclear whether these insights translate to practical model design, training, or downstream performance. In particular, the authors don't show how these findings could inform architectural decisions or improve an existing model.

Weak evidence in pretrained models: The pretrained-model experiments are much less convincing than the synthetic ones. SSM-only recall is quite low even at short distances, and the proposed recall handoff is clear mainly in Granite. This makes me question how relevant the observed mechanism is to the behavior of actual hybrid LLMs.

The attention-onset relationship may be somewhat trivial: The main predictive result is potentially less interesting than it first appears. Since attention onset is defined as a 20-percentage-point improvement, it is necessarily constrained by the SSM's baseline recall accuracy. The authors acknowledge this and provide a ceiling-based baseline, but it's still unclear how much additional insight g50g_{50} provides beyond characterizing when the SSM starts failing.

Limited causal evidence for the central relationship: While the authors convincingly show that selectivity affects recall range, they don't establish that recall range itself determines attention onset. Their within-model interventions fail to generalize, leaving the main relationship supported primarily by correlations across separately trained models.

Questions for authors
  1. What are the practical implications of these findings for hybrid LLM design?
  2. How much of the attention-onset prediction is explained by the accuracy ceiling?
  3. Why does the relationship fail to generalize under evaluation-time interventions?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 66rj (1/2): the accuracy ceiling (Q2, W3) and evaluation-time interventions (Q3, W4)

Thank you for the detailed summary, which matches our reading of the paper's structure, and for finding the experiments broad and carefully designed. Brackets give 95% intervals: seed-block bootstrap for the tiny models (three seeds per condition) and paired prompt bootstrap for the pretrained models (96 prompts per gap for the interaction experiments).

Q2 and W3 (how much of the onset prediction the ceiling explains). With fitting and interpolation matched, g50g_{50} reduces held-out MAE from 4.95 to 3.94 tokens relative to the fixed 0.8 crossing. A threshold chosen by nested cross-validation has similar error to g50g_{50} (common response §3; §6, Table 2). What predicts onset is the SSM's measured recall curve: g50g_{50}, g80g_{80}, the recall area, and crossings at thresholds 0.60–0.80 reach 3.28–4.95 tokens of held-out error, against 10.29 without a recall curve (Appendix E, Table 4).

  • The paired improvement over the 0.8 crossing is 1.01 tokens [0.14, 1.90] (discrete onset) and 0.65 [−0.01, 1.38] (interpolated), on 5/5 held-out architectures.
  • The ceiling makes a 20-point benefit possible only once SSM recall falls to 0.8, but it does not by itself make a trained attention layer deliver that benefit or fix the gap at which it does. Within the short-exposure staged protocol, attention's benefit begins where the SSM leaves headroom, and the recall curve predicts that gap on held-out architectures. §6 now presents this as a predictive characterization, with g50g_{50} as one of several recall-curve summaries that predict onset.

Q3 and W4 (why evaluation-time interventions do not transfer; your summary point 5). The interventions also act on the SSM layers downstream of the inserted attention, which were trained only on native Δ\Delta statistics. Pushing those layers off their training distribution plausibly explains why onset then reflects the lower recall that the hybrid reaches with attention rather than the range. §6 presents this explanation as consistent with the data.

  • In the training-time comparison, the held-out predictor estimates the selectivity-induced onset shift accurately: across 13 matched pairs, turning selectivity on delays onset by 40.6 tokens against a predicted 41.2 (r=0.88r = 0.88; MAE 6.0 [3.2, 9.3], against 14.4 for a constant).
  • Mediation by the range remains open, as §8 (limitation (i)) states.

Part 2 below covers the pretrained evidence (W2), practical implications (Q1, W1), and presentation.

AuthorsOpenReview ↗

Response to Reviewer 66rj (2/2): pretrained evidence (W2), practical implications (Q1, W1), and presentation

W2 (pretrained evidence). A new direct test is positive in both pretrained models: in Granite-4.0 and Nemotron-H-8B, flattening tokenwise timestep variation increases the accuracy cost of replacing query-position attention (at gap 32, by 0.490 [0.365, 0.594] and 0.365 [0.250, 0.479]), while attention-intact accuracy on single-key probes stays at 0.96–1.00 (§7, Figure 4; the table in the common response, §1, gives each gap). Within each fixed checkpoint, the same replacement becomes more costly after flattening while intact recall remains high, so recurrent gating changes reliance on query-time attention. In Nemotron-H, the query-position test avoids the near-floor recall left by replacing attention at every position: attention outputs at earlier positions stay active. With query-position attention replaced, its recall is 0.21–0.45 under native timesteps and 0.03–0.10 under flattening at gaps 4–32. In the testbed, joint training from initialization yields the same signature as most pretrained hybrids we profiled: recall depends strongly on attention at every tested distance, with autonomous recall near vocabulary chance (12/12). This is consistent with the low autonomous recall of jointly pretrained checkpoints.

Q1 and W1 (practical implications). The immediate use is to measure how much recall a hybrid checkpoint retains under input-independent attention replacement, without leaking sequence content, and to compare that quantity across training choices. In our controlled testbed, this diagnostic separates hybrids whose recall depends strongly on attention from hybrids that retain recall after its replacement: all 12 trained jointly from initialization depend on attention, whereas all 24 with a long SSM head start (at least 1500 of 3900 steps) retain recall. It thus gives a measurable criterion for evaluating training schedules meant to strengthen recurrent recall.

  • Selectivity also extends how far beyond its training context the recurrent path recalls: under our training recipe, for the tested horizons T=16T = 16, 64, and 128, a non-selective SSM's range is about twice its training horizon, and a selective SSM's is 5–9 times (Appendix I, Table 8; common response §2).

These diagnostic and training-schedule implications are established on associative-recall probes.

Presentation. §1 now ends with a "Contributions and organization" paragraph that links §4–§7.

AuthorsOpenReview ↗

Follow-up for Reviewer 66rj

Dear Reviewer 66rj, thank you again for your detailed review. As the discussion period ends on November 18, we would like to check whether our response answers your questions.

On the pretrained evidence (W2), the new test is positive in both models: flattening timestep variation increases the cost of replacing query-position attention by 0.490 [0.365, 0.594] in Granite-4.0-h-tiny and 0.365 [0.250, 0.479] in Nemotron-H-8B at gap 32, while attention-intact recall stays at 0.96–1.00 (§7, Figure 4).

On practical implications (Q1), the leakage-free attention replacement separates testbed hybrids that depend on attention (12/12 trained jointly from initialization) from those that retain recall (24/24 after a long SSM head start; Appendix I, Table 9).

On the ceiling (Q2), with fitting and interpolation matched, g50g_{50} reduces held-out onset error from 4.95 to 3.94 tokens relative to the fixed 0.8 crossing, while a threshold chosen by nested cross-validation has similar error (§6, Table 2).

Please let us know whether any concerns remain that we could clarify before discussion closes.

Reviewer_s4JUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how input-dependent timestep gating affects associative recall in state-space models and hybrid language models. It measures autonomous recall by replacing attention outputs with input-independent constants and summarizes its decline with a recall-range statistic. Controlled two-key experiments show that selectivity extends this range primarily by suppressing recurrent decay on filler tokens. In staged hybrids, the measured range predicts the distance at which an added attention layer improves recall. Pretrained-model interventions demonstrate that autonomous recall depends on input-dependent gating, with a range reduction established in Granite.

Critical strengths
  1. The controlled interventions provide useful evidence about how selectivity preserves associations. Separating the timestep’s decay and write channels shows that the observed range extension arises predominantly from suppressing filler-induced forgetting. Preserving near-perfect short-gap recall under the filler interventions helps distinguish this effect from a general loss of retrieval ability.
  2. The attention-ablation analysis addresses a meaningful methodological issue: averaging attention outputs from the evaluated sequence can leak information about its contents. The input-independent replacement avoids this leakage, and the pretrained interventions provide supporting evidence that gating matters beyond the small synthetic models. The paper also carefully distinguishes gating necessity, changes in recall range, and the unresolved question of causal mediation.
Critical weaknesses
  1. The additional explanatory value of the recall-range–attention-onset relationship remains insufficiently established. Onset is defined as the first gap where attention improves accuracy by at least 0.2, which is possible only once the frozen SSM’s accuracy falls to 0.8 or below. Every analyzed onset occurs at that first permissible gap or the next evaluated gap. Consequently, much of the relationship follows from where the recurrent model leaves room for the stipulated improvement. The authors appropriately report a ceiling-based predictor and show that the recall-range predictor is more accurate, but Appendix E notes that the comparison differs in fitting and interpolation. These results establish a useful empirical predictor without clearly establishing an additional mechanism governing attention’s benefit. The evaluation-time intervention experiments also leave mediation unresolved. The qualifications are appropriate, but they limit the conceptual advance of the central result.
  2. The extension from controlled retention to the behavior of pretrained hybrids remains narrow. The two-binding task usefully isolates preservation through irrelevant filler, and the staged hybrids explicitly train attention to improve a frozen recurrent backbone. This establishes a specific setting in which attention compensates for recurrent degradation, but its relevance to the division of labor learned during joint training is less clear. In pretrained models, the measurable range contraction is established on Granite, while the Nemotron interventions primarily establish dependence of already-low autonomous recall on gating. The other surveyed architectures do not provide comparable support for the proposed handoff. Since selectivity’s role in counteracting decay is already established, as the paper acknowledges, the remaining contribution is a useful decomposition and a qualified predictive relationship, whose broader explanatory significance is not sufficiently demonstrated.
Questions for authors
  1. After matching fitting and interpolation, what predictive information does the recall-range statistic provide beyond the threshold crossing imposed by the accuracy ceiling?
  2. What evidence connects the measured autonomous recall range to the distance-dependent benefit of attention in jointly pretrained hybrids, beyond demonstrating that removing gating reduces recall?
3 comments
AuthorsOpenReview ↗

Response to Reviewer s4JU (1/2): jointly pretrained hybrids (Q2, W2) and the ceiling crossing (Q1, W1)

Thank you for the careful review, and for recognizing the decay/write separation and the leakage-free ablation. In brief: the revision adds a direct test in both jointly pretrained models, where flattening timestep variation increases the cost of replacing query-position attention (Q2). With fitting and interpolation matched, g50g_{50} reduces held-out error relative to the fixed 0.8 crossing, and a cross-validated threshold does about as well (Q1). Brackets give 95% intervals: seed-block bootstrap for the tiny models (three seeds per condition) and paired prompt bootstrap for the pretrained models (96 prompts per gap for the interaction experiments).

Q2 and W2 (evidence in jointly pretrained hybrids). The new experiments establish a gating–attention interaction in both jointly pretrained models. In Granite-4.0-h-tiny and Nemotron-H-8B, each Mamba head's timestep is fixed to its mean over independent calibration prompts, and every attention layer's query-position output is replaced by a fixed vector calibrated once without the timestep intervention (§7, Figure 4; Appendix H). The same query-position replacement becomes more costly when timestep variation is flattened, although the intact model continues to solve the probe, so recurrent gating changes reliance on query-time attention in both models.

  • On single-key probes under the timestep intervention, attention-intact accuracy stays at 0.96–1.00. The accuracy cost of replacing query-position attention rises by 0.14–0.62 in Granite (gaps 2–32) and by 0.19–0.36 in Nemotron-H (gaps 4–32). The table in the common response, §1, gives recall in all four conditions at each gap.
  • In Nemotron-H (W2), the query-position test avoids the near-floor recall left by replacing attention at every position: attention outputs at earlier positions stay active. With query-position attention replaced, its recall is 0.21–0.45 under native timesteps and 0.03–0.10 under flattening at gaps 4–32, while attention-intact recall stays at 0.96–1.00.
  • With attention replaced, the recall loss caused by flattening is larger at gap 32 than at gap 2 in both models, and the interaction also holds under zero replacement and in log-probability.
  • An exploratory Granite sweep links lower autonomous recall area to an earlier crossing of a specified attention-replacement cost, but g50g_{50} is poorly resolved on these curves and does not track the ordering (§7; Appendix H).

Q1 and W1 (information beyond the ceiling crossing). With fitting and interpolation matched, g50g_{50} reduces held-out MAE from 4.95 to 3.94 tokens relative to the fixed 0.8 crossing (paired improvement 1.01 [0.14, 1.90] for the discrete onset and 0.65 [−0.01, 1.38] interpolated; common response §3; §6, Table 2). A cross-validated crossing threshold has similar error to g50g_{50}. The predictive information is in the measured recall curve: g50g_{50}, g80g_{80}, the recall area, and crossings at thresholds 0.60–0.80 reach 3.28–4.95 tokens, against 10.29 for the best curve-free baseline.

  • g50g_{50} is better than the 0.8 crossing on 5/5 held-out architectures and is numerically the 75% crossing.
  • On mechanism, we agree: within the short-exposure staged protocol, attention's benefit begins where the SSM leaves headroom, and the data do not identify a further mechanism. §6 now presents the result as a predictive characterization, with g50g_{50} as one summary of the recall curve.
  • Appendix E, Table 4 gives the full threshold scan, replacing the earlier comparison, which differed in fitting and interpolation.

Part 2 below covers the testbed controls under matched exposure and joint training, the other profiled architectures, novelty, and mediation.

AuthorsOpenReview ↗

Response to Reviewer s4JU (2/2): testbed controls, other architectures, novelty, and mediation

Q2 and W2, continued: testbed hybrids under matched exposure and joint training. Both controls change how much recall attention takes over (§6; Appendix I):

  • Matched exposure (common response §2): with the SSM trained on the same gaps as attention (0–128), attention improves recall by at most 0.039 at any gap ≤128\le 128 across all 30 staged hybrids. Three of the 30 hybrids, all non-selective, reach the 0.2 benefit threshold, each first at gap 256, just past the SSM's range (g50g_{50} 195–226).
  • Joint training (common response §1): from initialization, recall depends strongly on attention at every tested distance, with autonomous recall near vocabulary chance (12/12). After a long SSM head start (at least 1500 of 3900 steps), removing attention costs little recall (24/24). Intermediate head starts give mixed outcomes (Table 9).

The quantitative range–onset predictor is therefore established for the short-exposure staged protocol, as the revision now states (abstract; §6; §8).

W2: the other profiled architectures. As you note, the other profiled checkpoints provide weaker or no evidence of a distance-graded handoff, and we report these differences per model (§4, Table 1). Their autonomous recall is low or absent, and in the testbed, joint training from initialization produces the same signature (strong attention dependence, autonomous recall near vocabulary chance; 12/12), which is consistent with joint pretraining, although these controls do not identify the cause in any specific checkpoint.

Novelty relative to Huang et al. (2025). Huang et al. show that selectivity counteracts memory decay. Our contribution is to identify where that effect acts in trained networks and what it does to attention (§1, contributions):

  • The token-role interventions separate the decay and write paths, which the analytical treatment does not, and so discriminate between two explanations of retention: filler tokens writing interfering content, or existing content decaying. Raising filler Δ\Delta in the decay path alone collapses the range to 3.7 ± 1.7 tokens. Raising it in the write path alone leaves 46.8 ± 12.6, and short-gap recall stays at 0.99–1.00 under every filler intervention (§5; Appendix G).
  • The new pretrained interventions establish that changing the gating changes attention dependence in two jointly pretrained hybrids (Q2 above).
  • A leakage-free measure of autonomous recall in pretrained hybrids, and evidence that the selective advantage holds under matched exposure (2.71× [2.52, 2.92], 15/15 pairs) and across training horizons (60/60 pairs; §5; common response §2).

Mediation (W1). We agree that mediation remains open, and the revision keeps that qualifier (§6; §8, limitation (i)). Short of mediation, the range does predict the onset shift that selectivity causes during training: across 13 matched pairs, turning selectivity on delays onset by 40.6 tokens against 41.2 predicted from the change in g50g_{50} (Figure 1(b)).

AuthorsOpenReview ↗

Follow-up for Reviewer s4JU

Dear Reviewer s4JU, as the discussion period ends on November 18, we would like to check whether our response addresses your two questions. For Q2, we added a direct test of gating and query-time attention dependence in two jointly pretrained hybrids: flattening timestep variation makes the same query-position attention replacement more costly by 0.490 [0.365, 0.594] in Granite-4.0-h-tiny and 0.365 [0.250, 0.479] in Nemotron-H-8B at gap 32, while attention-intact recall stays at 0.96–1.00 (§7, Figure 4). For Q1, with fitting and interpolation matched, g50g_{50} reduces held-out onset error from 4.95 to 3.94 tokens relative to the fixed 0.8 crossing (paired improvement 1.01 [0.14, 1.90], 5/5 held-out architectures), and a threshold chosen by nested cross-validation has similar error (§6, Table 2). Please let us know whether any concerns remain that we could clarify before discussion closes.

Reviewer_UhvJOpenReview ↗
Rating
3
Confidence
2(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how input-dependent timestep gating affects associative recall in SSM–attention hybrids. It introduces an attention ablation to measure SSM recall and show selectivity could extend recall range in controlled models. They also use the recall range to predict the position where adding attention layers could improve performance.

Critical strengths
  1. The design of controlled mechanism experiments is quite interesting.
  2. The limitations and negative results are useful to evaluate autonomous recall in several pretrained models.
Critical weaknesses
  1. SSM has only been trained on tasks with tokens spaced 0–4, while attention, added later, has been trained on tasks with tokens spaced up to 128. Therefore, attention is more useful over long distances, possibly partly because it has been trained over longer distances.
  2. The attention-onset result partly follows from its definition. SSM accuracy must drop to 80% or below before attention can improve it by 20 percentage points. This makes declining SSM recall and attention onset naturally related.
Questions for authors

Plz see weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer UhvJ: matched training exposure and the accuracy ceiling

Thank you for finding the controlled mechanism experiments interesting. We ran new experiments on both points. Brackets give 95% seed-block bootstrap intervals (three seeds per condition).

W1 (SSM trained on gaps 0–4, attention on 0–128). You are right that unequal training exposure affects the onset result. The selective range advantage persists under matched exposure (common response §2).

  • With SSM and attention both trained on gaps 0–128, attention improves recall by at most 0.039 at any gap ≤128\le 128 across all 30 staged hybrids.
  • Under matched exposure, three of the 30 hybrids, all non-selective, reach the 0.2 benefit threshold, each first at gap 256, just past the SSM's range (g50g_{50} 195–226).
  • Retrained on gaps 0–128 and evaluated to gap 2048, all 30 pure-SSM backbones keep the selective advantage: median g50g_{50} 687 vs 255 tokens, geometric-mean ratio 2.71× [2.52, 2.92], longer in 15/15 seed-matched pairs.
  • Across training horizons T=4T = 4, 16, 64 and 128, selective ranges are longer in all 60 seed-matched pairs, by factors of 1.92 to 3.99 (Appendix I, Table 8).

§5 and §6 report this, and the revision states the onset result for the short-exposure staged protocol.

W2 (onset partly follows from its definition). We agree that the ceiling constrains onset: with fitting and interpolation matched, g50g_{50} improves on the 0.8 crossing by about one token, and a cross-validated threshold has similar held-out error (common response §3; §6, Table 2). What predicts onset is the SSM's measured recall curve: g50g_{50}, g80g_{80}, the recall area, and crossings at thresholds 0.60–0.80 reach 3.28–4.95 tokens, against 10.29 without it.

  • Held-out MAE is 3.94 tokens for g50g_{50}, 4.95 for a fitted 0.8 crossing (paired improvement 1.01 [0.14, 1.90], 5/5 held-out architectures), and 3.67 for a threshold chosen by nested cross-validation. Appendix E, Table 4 gives every summary we fit.
  • The ceiling makes a 20-point benefit possible only once SSM recall falls to 0.8, but it does not by itself make a trained attention layer deliver that benefit or fix the gap at which it does.

§6 now states the onset result as prediction within the short-exposure staged protocol, consistent with attention using the headroom the SSM leaves.

AuthorsOpenReview ↗

Follow-up for Reviewer UhvJ

Dear Reviewer UhvJ, as the discussion period ends on November 18, we would like to check whether our response addresses your two points. When the SSM is retrained on the same gaps as attention, the selective range advantage persists (2.71× [2.52, 2.92], 15/15 seed-matched pairs), while training exposure affects the onset pattern (W1; §5, §6; Appendix I). With fitting and interpolation matched, g50g_{50} reduces held-out onset error from 4.95 to 3.94 tokens relative to the fixed 0.8 crossing, and a threshold chosen by nested cross-validation has similar error (W2; §6, Table 2). Please let us know whether any concerns remain that we could clarify before discussion closes.

My concerns have been well addressed in the response and the revised version. Thus, I shall maintain my score.

Reviewer_41MfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how selectivity (input-dependent time-step gating) affect in hybrid models. The authors conduct controlled experiments and show that gating can affect SSM's recall range by preventing irrelevant tokens from writing irrelevant informations. The authors also show that the recall range of a frozen SSM predicts the distance at which attention can provide recall gains.

Critical strengths
  1. The authors conduct clear controlled experiment. The gate can affect both how old information decay and how new information write. They found that selectivity helps the recall mainly by reducing the decay of information.
  2. The author considered several confounding variables and had clear ablation.
  3. The author fit a linear model to predict the position where adding attention would provide gain in recall range.
Critical weaknesses
  1. Most experiments are in the setting where we fix SSM and then retrofit by adding attention, which can be learning a different mechanism then if we train hybrid models from scratch. Later the author predict the distance where adding attention would provide gain in recall range. However I am a bit unsure how this linear prediction model can be useful when we train hybrid models.
  2. The evaluation is only limited to Mamba model. How does it behave on other linear attention variants and mechanism with with selective forget like DeltaNet?
  3. The tasks are mostly synthetic tasks on associative recall. It is unclear how this would affect the more complicated downstream tasks.
Questions for authors

Check the weakness part.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 41Mf: hybrids trained from scratch, other recurrent variants, and downstream tasks

Thank you for noting the clear controlled experiments and ablations. Our new experiments address W1 most directly. Brackets give 95% intervals: seed-block bootstrap for the tiny models (three seeds per condition) and paired prompt bootstrap for the pretrained models (96 prompts per gap for the interaction experiments).

W1 (attention added to a fixed SSM vs hybrids trained from scratch; usefulness of the linear predictor). We now include controlled joint-training experiments in the testbed and interventions in two jointly pretrained hybrids.

  • Joint training in the testbed (§6; Appendix I; common response §1): from initialization, recall depends strongly on attention at every tested distance, with autonomous recall near vocabulary chance (12/12). After a long SSM head start (at least 1500 of 3900 steps), removing attention costs little recall (24/24). Intermediate head starts give mixed outcomes (Appendix I, Table 9).
  • Jointly pretrained hybrids (§7, Figure 4): in Granite-4.0 and Nemotron-H-8B, flattening tokenwise timestep variation increases the accuracy cost of replacing query-position attention (at gap 32, by 0.490 [0.365, 0.594] and 0.365 [0.250, 0.479]; the table in the common response, §1, gives each gap), while attention-intact accuracy on single-key probes stays at 0.96–1.00. Recurrent gating thus changes how much recall depends on attention outside the staged protocol.
  • Matched exposure (common response §2): with the SSM and attention both trained on gaps 0–128, attention improves recall by at most 0.039 at any gap ≤128\le 128 across 30 staged hybrids. Three of the 30 hybrids reach the 0.2 benefit threshold, each just past the SSM's range. The selective range advantage persists under matched exposure (2.71× [2.52, 2.92], 15/15 seed-matched pairs) and at every tested training horizon (60/60 pairs; §5).

On usefulness for training hybrids: the onset predictor is established for the staged protocol (abstract, §6, §8). What carries over is the measurement of autonomous recall under input-independent attention replacement, which applies to any hybrid checkpoint and, in the testbed, separates attention-dependent from recall-retaining hybrids across training schedules.

W2 (other linear-recurrent variants; DeltaNet). Our decay/write separation targets an input-dependent decay of the state, which motivates analogous interventions in Gated DeltaNet (Yang et al., 2025), whose decay gate and delta-rule update are distinct mechanisms. Our experiments cover Mamba-family SSMs, and §8 now notes that transfer to Gated DeltaNet is untested (limitation (v)).

W3 (downstream tasks). We focus on associative recall because attention-containing hybrids address long-context recall weaknesses observed in pure Mamba models (Waleffe et al., 2024), making recall a useful probe of their division of labor. The pretrained interventions show that gating matters for recall in real LMs: on Granite, fixing each sequence's timestep at its native average shrinks the range from 11 to 3 tokens. §8 now notes that downstream tasks are not evaluated (limitation (v)).

AuthorsOpenReview ↗

Follow-up for Reviewer 41Mf

Dear Reviewer 41Mf, as the discussion period ends on November 18, we would like to check whether our response addresses your concerns. For W1, the revision adds experiments on jointly trained hybrids, both pretrained and in the testbed. In Granite-4.0-h-tiny and Nemotron-H-8B, flattening timestep variation increases the cost of replacing query-position attention by 0.490 and 0.365 in recall at gap 32, while attention-intact recall stays at 0.96–1.00 (§7, Figure 4). In the testbed, all 12 hybrids trained jointly from initialization depend on attention, and all 24 given a long SSM head start lose little recall when attention is replaced (§6; Appendix I, Table 9). Please let us know whether any concerns remain that we could clarify before discussion closes.

Comments on the paper

4 comments
AuthorsOpenReview ↗

New experiments (1/2): pretrained hybrids and joint training

We thank the reviewers for recognizing the controlled interventions and the leakage-free attention ablation. The revision adds direct evidence that recurrent timestep gating changes attention dependence in two jointly pretrained hybrids: on single-key probes, flattening timestep variation increases the cost of replacing query-position attention by 0.490 [0.365, 0.594] in Granite-4.0-h-tiny and 0.365 [0.250, 0.479] in Nemotron-H-8B at gap 32, while attention-intact accuracy under flattening stays at 0.96–1.00. New training controls show that the selective range advantage persists under matched exposure and at all four tested horizons (60/60 seed-matched pairs), and joint-training experiments in the testbed show that the training schedule strongly changes how much recall survives attention replacement. With fitting and interpolation matched, the ceiling analysis clarifies the scope of the onset result: several summaries of the SSM's recall curve predict onset in the staged setting, with no demonstrated advantage unique to g50g_{50}. The revised PDF marks new text in blue, and a separate comment gives each change's location. This part covers pretrained and jointly trained hybrids (§1), and the reply below covers training exposure (§2) and the accuracy ceiling (§3).

Tiny-model sweeps use three seeds per condition and seed-block bootstrap intervals. The new pretrained interaction and graded-sweep experiments use 96 paired prompts per gap and paired prompt-bootstrap intervals. All intervals are 95%. Every new SSM training run uses learning rate 10−310^{-3}, for both selectivity settings and every horizon. The submitted backbones used 3×10−33 \times 10^{-3}, which proved too high for selective SSMs on gaps 0–128 (Appendix I).

1. Does recurrent gating affect attention dependence in jointly trained hybrids? (s4JU Q2 and W2, 66rj W2, 41Mf W1) Yes, in both pretrained hybrids we tested (§7, Figure 4). Within each fixed checkpoint, the same query-position attention replacement becomes more costly after timestep variation is flattened, while intact recall remains high, so recurrent gating changes reliance on query-time attention.

  • Each Mamba head's timestep is fixed at its mean over independent calibration prompts, and each attention layer's query-position output is replaced by one fixed vector in every condition.
  • Attention-intact accuracy on single-key probes stays at 0.96–1.00 under this intervention, and the interaction also holds under zero replacement and in log-probability.

Recall on single-key probes, native → flattened timesteps. The interaction is the cost of the replacement under flattened minus native timesteps (Appendix H, Table 6):

ModelGapAttention intactAttention replacedInteraction
Granite-4.0-h-tiny20.95 → 1.000.88 → 0.790.135 [0.052, 0.229]
40.98 → 0.970.82 → 0.630.188 [0.083, 0.292]
80.96 → 0.980.80 → 0.340.479 [0.375, 0.583]
160.99 → 0.990.85 → 0.230.625 [0.521, 0.729]
321.00 → 0.990.61 → 0.110.490 [0.365, 0.594]
Nemotron-H-8B20.93 → 0.980.36 → 0.340.073 [−0.042, 0.188]
40.95 → 1.000.24 → 0.100.188 [0.104, 0.271]
80.85 → 1.000.21 → 0.030.323 [0.229, 0.417]
160.97 → 1.000.30 → 0.080.250 [0.146, 0.354]
321.00 → 0.960.45 → 0.040.365 [0.250, 0.479]

At longer gaps the interaction comes mainly from the loss with attention replaced.

In the testbed, joint training from initialization makes recall depend strongly on attention at every tested distance, with autonomous recall near vocabulary chance (12/12 hybrids). After a long SSM head start (at least 1500 of 3900 steps), removing attention costs little recall (24/24). Intermediate head starts give mixed outcomes (§6; Appendix I, Table 9).

AuthorsOpenReview ↗

New experiments (2/2): training exposure, the accuracy ceiling, and what the revision establishes

2. Does the SSM's shorter training exposure drive the results? (UhvJ W1; also s4JU W2 and 41Mf W1) The selective range advantage persists under matched exposure, while the onset pattern depends on exposure.

  • Retrained on gaps 0–128, the distribution attention sees, and evaluated to gap 2048, all 30 pure-SSM backbones keep the selective advantage: median g50g_{50} 687 vs 255 tokens, geometric-mean ratio 2.71× [2.52, 2.92], longer in 15/15 seed-matched pairs (§5).
  • Across training horizons T=4T = 4, 16, 64 and 128, selective ranges are longer in all 60 seed-matched pairs, by factors of 1.92 to 3.99 (§5; Appendix I, Table 8).
  • With the SSM trained on the same gaps as attention, attention improves recall by at most 0.039 at any gap ≤128\le 128 across all 30 staged hybrids. Three of the 30 hybrids, all non-selective, reach the 0.2 benefit threshold, each first at gap 256, just past the SSM's range (g50g_{50} 195–226; §6).

3. How much of the range–onset relationship does the 0.8 ceiling set? (s4JU Q1 and W1, 66rj Q2 and W3, UhvJ W2) With fitting and interpolation matched, g50g_{50} reduces held-out MAE from 4.95 to 3.94 tokens relative to the fixed 0.8 crossing. A threshold chosen by nested cross-validation has similar error to g50g_{50}, so we find no demonstrated predictive advantage specific to g50g_{50} (§6, Table 2; leave-one-architecture-out fit, 28 backbones, 5 held-out architectures):

PredictorDiscrete onset MAEInterpolated onset MAE
g50g_{50}3.942.62
Fitted crossing at 0.804.953.28
Crossing, threshold by nested CV3.672.57
Width + selectivity flag (no recall curve)10.297.92
  • g50g_{50} improves on the 0.8 crossing by 1.01 tokens [0.14, 1.90] (discrete) and 0.65 [−0.01, 1.38] (interpolated), on 5/5 held-out architectures.
  • The recall-curve summaries g50g_{50}, g80g_{80}, the recall area, and crossings at thresholds 0.60–0.80 reach 3.28–4.95 tokens, against 10.29 without a recall curve (Appendix E, Table 4, gives the full scan), and g50g_{50}'s paired improvement over the latter is 6.35 [2.64, 9.41] tokens.

The ceiling makes a 20-point benefit possible only once SSM recall falls to 0.8, but it does not by itself make a trained attention layer deliver that benefit or fix the gap at which it does. Within the short-exposure staged protocol, attention's benefit begins where the SSM leaves headroom, the recall curve predicts that gap on held-out architectures, and g50g_{50} also predicts the onset shifts caused by changing selectivity during training. We present this as a predictive characterization, with g50g_{50} as one summary of the recall curve. The abstract, §1, §6, §8 and the conclusion state it for this protocol.

What the revision establishes. Together, the experiments identify how timestep gating preserves associative recall in the controlled setting (mainly by preventing filler-induced decay), provide a leakage-free diagnostic of the recall a hybrid retains under attention replacement, show that the selective retention advantage survives matched exposure and every tested horizon, and establish that changing the gating changes query-time attention dependence in two pretrained hybrids. The training controls show that exposure and schedule shape the division of recall between the components, and the staged onset experiments quantify one such regime.

AuthorsOpenReview ↗

Revised PDF uploaded: location of each change

We have uploaded the revised PDF, and the main text stays within 10 pages. New or substantially revised text is shown in blue. Each change and its location:

1. Pretrained hybrids (s4JU Q2, 66rj W2, 41Mf W1)

  • §7, Figure 4 and the paragraph "Flattening timestep variation raises dependence on query-time attention": the four conditions, the interaction with intervals, log-probability, and zero replacement.
  • §7, paragraph "An exploratory graded sweep on Granite", and Appendix H, Figure 9 and Table 7: graded timestep flattening on Granite.
  • Appendix H, Table 6: design, intervals (conditional on the checkpoint and the calibration draw), which side carries the interaction at each gap, distance, the two-key fixed-foil result, and the decay/write split.

2. Training exposure, horizon, and joint training (UhvJ W1, 41Mf W1)

  • §5, "Training exposure and horizon": the selective range advantage under matched 0–128 exposure (2.71× [2.52, 2.92], 15/15 pairs) and at every training horizon (60/60 pairs).
  • §6, "Training exposure and schedule": matched-exposure staged hybrids and joint training with an SSM head start.
  • Appendix I: the learning-rate change and its reason (Recipe), the horizon table (Table 8), the matched-exposure onsets, and the joint-training regime counts (Table 9).

3. The ceiling question (s4JU Q1, 66rj Q2, UhvJ W2)

  • §6, Table 2 and the paragraph "The ceiling comparison with fitting and interpolation matched". Figure 1(c) now includes the matched 0.8 crossing (4.95) and the nested-CV crossing (3.67).
  • Appendix E, Table 4: the full threshold scan. It replaces the earlier comparison, which differed in fitting and interpolation.

4. Staged-protocol scope (s4JU Q2, 41Mf W1, UhvJ W1): the quantitative onset result is stated for the short-exposure staged protocol (SSM trained on gaps 0–4) in the abstract; §1, paragraph 4 and contribution (iii); §6 title and opening paragraph; §8, limitation (ii); conclusion.

5. Evaluation-time interventions (66rj Q3): §6, paragraph "Evaluation-time interventions do not isolate the link", and §8, limitation (i).

6. Positioning against Huang et al. (s4JU): §1, contributions (i) and (ii).

7. Scope (41Mf W2 and W3): §8, limitation (v), on Gated DeltaNet and downstream tasks.

8. Citations: Buitrago Ruiz & Gu (2025) in §5 and §8; Bick et al. (2025) in §8.

9. Organization (66rj): §1, "Contributions and organization", links §4–§7.

10. Code: the anonymous repository now includes the scripts and saved outputs for every new experiment, and the README lists the commands behind each new table and figure.

To make room, the answer-injection depth control moved to Appendix C, the query-swap and key-selection controls to Appendix E, the calibrated-intervention figure to Appendix E (Figure 8), and the replay-accuracy check and clean recall under permute to Appendix E ("Intervention checks"). No result was removed.

AuthorsOpenReview ↗

Summary of the revision for the Area Chair

Dear Area Chair, we provide a brief map of the revised evidence and its locations.

  • Jointly pretrained hybrids (s4JU Q2, 66rj W2, 41Mf W1): in Granite-4.0-h-tiny and Nemotron-H-8B, flattening timestep variation increases the cost of replacing query-position attention by 0.490 [0.365, 0.594] and 0.365 [0.250, 0.479] at gap 32, with attention-intact recall at 0.96–1.00 (§7, Figure 4). Joint-training controls in the testbed show that the training schedule strongly changes how much recall remains after attention replacement (12/12 from initialization depend on attention, 24/24 after a long SSM head start retain recall; §6; Appendix I, Table 9).
  • Training exposure (UhvJ W1): under matched exposure the selective range advantage persists (2.71× [2.52, 2.92], 15/15 seed-matched pairs) and holds at every tested horizon (60/60 pairs), while training exposure affects the onset pattern (§5, §6; Appendix I, Table 8).
  • Accuracy ceiling (s4JU Q1, 66rj Q2 and W3, UhvJ W2): with fitting and interpolation matched, g50g_{50} reduces held-out onset error relative to the fixed 0.8 crossing from 4.95 to 3.94 tokens (lower on 5/5 held-out architectures), and a threshold chosen by nested cross-validation has similar error. The best baseline without a recall curve reaches 10.29 (§6, Table 2).
  • Practical use (66rj Q1, 41Mf W1): the leakage-free attention replacement measures how much recall a hybrid checkpoint keeps without attention and, in the testbed, separates attention-dependent from recall-retaining hybrids across training schedules (§6; Appendix I, Table 9).
  • Scope (41Mf W2, W3): §8 states the scope of the experiments (Mamba-family SSMs, associative-recall probes).

The reviewers credited the controlled interventions (all four reviewers), the decay/write separation (s4JU, 66rj, 41Mf) and the leakage-free attention ablation (s4JU, 66rj). The revision adds direct evidence in two jointly pretrained hybrids. It also adds controls for training exposure and schedule, and a ceiling comparison with fitting and interpolation matched.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.