acceptodds
Withdrawn from ICLR 2027

Selective but Not Broad: Retrieval Capacity and Accessibility in State-Space Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

State-space models (SSMs) such as Mamba solve selective retrieval (associative recall, MQAR) yet fail broad retrieval (uniform-over-positions discrimination) that requires every input position to stay recoverable (Position Recovery, PR). We argue these are two different memory problems: selective retrieval is content-addressed (recall by key), broad retrieval is position-addressed (keep every position discriminable), with distinct capacity requirements and distinct accessibility. Is the broad deficit a capacity limit or an optimization artifact? We model retrieval as distributing a fixed per-position signal-to-noise budget across the sequence. A single linear SSM channel provably cannot solve filler recall, which forces channel specialization: broad retrieval needs retrieval rank scaling with the sequence length, selective only with the number of keys, and the required timescales must be placed to match the queried ages. The theory predicts the structure of selective-retrieval collapse and the specialization signature of successful backbones; both appear in trained weights, including multi-scale RetNet tracking the theoretical 1/m line. A symmetric causal intervention then separates the two explanations: a curriculum unlocks broad retrieval on a standard SSM and transfers to selective, and unlocks selective at its own lower floor where standard SGD fails. At every scale tested, both deficits are accessibility-limited; the barrier persists through 122M parameters. A controlled Pareto frontier shows the objective is the lever: pure language modeling leaves broad at the random floor, and the PPL cost of adding broad is architecture-dependent. The theory holds for the linearized state map; every causal and theoretical claim is validated on this project's own checkpoints, and the frozen-probe absence of broad information replicates on 2.7B open pretrained checkpoints of both families.

Primary area:
learning theory
Keywords:
state-space models, associative recall, retrieval capacity, optimization accessibility, curriculum learning, sequence models

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_4MAZOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the difference between selective retrieval and broad retrieval by distinguishing the model capacity and effectiveness of training. My overall rating is 1/4 towards rejection.

Critical strengths

The problem is interesting and popular. The most valuable contribution of the paper is the curriculum training in Table 3 which shows that the PR accuracy of standard Mamba-3 improved from approximately 0.015 to 0.72 and this could be transferred to MQAR.

Critical weaknesses
  1. The presentation is difficult to follow and appears to rely heavily on AI-generated text without human explanation. In particular, section 3 need a clear mathematical formulation and well-defined notations.

  2. The insights of introducing the leverage budget are unclear. The experiments do not appear to verify the theoretical rank bounds in Section 3.

  3. Figure 6 shows that increasing model width can reduce broad retrieval accuracy. Maybe the authors could discuss possible reasons for this interesting behavior.

Questions for authors

Plz see weaknesses.

Reviewer_JqiGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that SSMs like Mamba solve selective retrieval (MQAR) but fail broad retrieval (Position Recovery) because these are distinct memory problems with different capacity requirements. Using a leverage-score “SNR budget” view of the recurrent state, it derives a per-channel 1/m ceiling and rank floors that scale with sequence length for broad retrieval but only with the number of keys for selective retrieval. It then shows that a simple curriculum can unlock both tasks when standard training fails, concluding that the deficits are optimization-limited rather than capacity-limited. Validation consists of small synthetic experiments (T ≤\leq 256, up to 122M parameters), weight-specialization analyses, and frozen probes on 2.7B pretrained checkpoints.

Critical strengths

None

Critical weaknesses
  • The paper is not readable. The exposition relies on undefined phrases (“the spine,” “moves i–v,” “P1–P9,” “floors,” “accessibility binds; the objective is the lever”). Claims are made and abandoned and reinterpreted, quantities are defined in one place, then calibrated in another. For example, θ\theta is fit from one experiment and then used to compute floors for a different quantity. The main text cannot be followed without the appendices. The paper, as it is, needs a full rewrite for the general reader to follow.

  • Weak contributions. Wherever the claims could be understood somewhat, don’t hold up:

    • The core framing (optimization rather than capacity limits recurrent retrieval; a fixed memory budget per state) is established in prior work but the paper doesn’t engage with at all [1, 2, 3].
    • Inconsistent theory: the theory’s own predictions don’t match the trained models. The minimum memory requirements it says are necessary are not met by the models that actually solve the task (App. B.8), and the results presented as confirming the theory have simpler explanations.
    • The setup is seemingly unreliable: the implemented Transformer scores 0.135 on MQAR, which it is known in prior work to solve reliably [4]. Every architecture uses one fixed learning rate despite known sensitivity to this issue [2].
    • Short contexts only: all conclusions are drawn at T ≤ 256, where capacity is unlikely to be the main factor.

References:

  • [1] Capacity and Trainability in Recurrent Neural Networks - Collins et al., 2017
  • [2] Revisiting Associative Recall in Modern Recurrent Models - Okpekpe & Orvieto, 2025
  • [3] Memory Traces in Dynamical Systems - Ganguli et al., 2008
  • [4] Zoology: Measuring and Improving Recall in Efficient Language Models - Arora et al., 2024
Questions for authors
  • Why does the Transformer reach only 0.135 on MQAR in Table 1, a task attention is known to solve reliably ([4], Sec. 4.3)? Was the learning rate tuned per architecture, and are the SSMs official implementations or re-implementations?
  • The threshold θ\theta is fit from the per-channel SNR of Theorem 1 but used to compute floors on the per-position SNRkSNR_k. How is this justified, given the paper itself calls these different quantities?
  • Broad-success models have linearized rank ~ 6.3, below the stated “necessary” floor T\theta ~ 15.4 (App. B.8). In what sense are the floors necessary conditions for the models studied?
Reviewer_FCxLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work analyzes the capacity of state-space models under two addressing mechanisms: content- and position-based addressing, from two perspectives: (1) capacity, i.e., whether the model class can represent a solution to the task; and (2) accessibility, i.e., whether training can successfully reach such a solution. The results appear to have two main findings: (1) the two addressing mechanisms may require different model structures; and (2) standard SGD has poor accessibility, whereas "curricula" can mitigate both challenges simultaneously.

Critical strengths

This work appears to address an interesting theoretical problem that has been investigated in the literature [1]. [1] Ebrahimi et al., Your context is not an array: Unveiling random access limitations in transformers

Critical weaknesses

The writing seems to be completely AI-generated. Otherwise, it fails to provide the necessary context and theoretical setup to formally define terms and guide readers through the problem and its proposed solution. To name just a few, what do “boards”, “rank floor”, ”weight signature”, actually mean? What does this work do with causal intervention? I am afraid that no researchers in this community may fully understand and evaluate the work.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.