- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces belief geometry to compare attention architectures and state-space models (SSMs) in sequential in-context learning. Using Bayesian linear regression with routed observations and cumulative Bayes regret, it studies three capabilities: belief maintenance, evidence assembly, and content addressing.
The results favor SSMs for stationary belief maintenance and memory-efficient assembly of positionally separated evidence. For content-dependent routing, fixed-depth linear attention and affine-transition SSMs incur linear regret, while softmax attention achieves sublinear regret with logarithmic key dimension; sigmoid-selective SSMs require nearly linear state dimension.
Synthetic filtering and contextual bandit experiments, including LLaMA-type Transformers and Mamba-2, qualitatively support these architectural trade-offs.
Critical strengths
Strength 1: A unified framework linking architecture to statistical performance
Belief geometry decomposes sequential Bayesian prediction into evidence assembly, belief maintenance, and content addressing, providing a common framework for comparing attention and SSMs through cumulative regret.
The framework yields concrete regret lower bounds, constructive upper bounds, and resource comparisons across three increasingly demanding settings. By identifying the operations behind architectural differences, it may also inform analyses beyond the regression setting studied.
Strength 2: Quantitative separations supported by targeted experiments
Key results include the optimal stationary-kernel comparison, SSMs’ memory advantage under positional routing, and the nearly linear recurrent-state requirement for selective SSMs under content addressing. The last result is particularly notable for bounding real-valued state dimension without finite-precision assumptions.
Experiments with practical architectures and nonlinear tasks provide complementary support. Despite empirical limitations, the combination of lower bounds, constructions, and targeted evaluations strengthens the contribution.
Critical weaknesses
Weakness 1: The theoretical conclusions are less general than some of the architectural interpretations suggest
Although the paper presents belief geometry as a general framework, its strongest comparative conclusions depend on restricted architecture classes and specific routing constructions.
For example, the stationary-kernel comparison applies to one-layer linear attention and fixed-transition SSMs, whereas richer architectures can overcome the corresponding stationary regret floor. Similarly, the content-addressing lower bounds are established for particular classes of linear attention and affine or sigmoid-selective transitions at fixed depth. They do not directly characterize arbitrary practical Transformers or nonlinear recurrent architectures.
This distinction is consequential because the principal motivation is to explain the relative capabilities of modern attention and SSM architectures. The experiments offer supporting evidence, but they cannot by themselves close the theoretical gap.
I would view this limitation as less concerning if the paper more consistently distinguished properties of the analyzed architecture classes from broader architectural conclusions.
Weakness 2: The experimental evidence does not conclusively establish the claimed scaling advantages for practical architectures
The practical experiments are carefully designed to investigate particular mechanisms, but their quantitative interpretation is complicated by architecture-specific optimization choices and limited scaling regimes.
For instance, the content-routing experiments use different training schedules and numbers of random seeds for Transformers and Mamba-2. The authors also select the best validation-performing runs from hyperparameter searches. These choices may be appropriate for demonstrating achievable performance, but make it difficult to separate representational limitations from optimization effects.
More importantly, the content-routing width-scaling experiment fixes the horizon at while increasing the codebook size to . This does not satisfy the asymptotic regime used in the corresponding theoretical comparisons. The observed scaling therefore provides suggestive finite-sample evidence, but does not directly validate the asymptotic width separation.
This matters because the practical implications of the paper rely substantially on demonstrating that the predicted resource trade-offs persist beyond the restricted theoretical classes.
Questions for authors
-
Scope of the content-addressing separation. Theorem 3 applies to restricted fixed-depth architectures. Which nonlinear mechanisms in practical Mamba-2 fall outside its assumptions, and does the state-dimension lower bound extend to nonlinear writes or reads? Appendix D.1.4 finds little performance loss from fixing transitions while retaining nonlinear writes. How does this affect the interpretation of transition selectivity as the mechanism behind the separation?
-
Interpretation of resource scaling. Figure 6 increases codebook size at fixed , while the theory requires . Does the observed width scaling persist in that regime? Jointly varying and , or clarifying which conclusions hold independently of the asymptotic theory, would help resolve this gap.
-
Optimization versus capacity. The architectures use different alignment durations, parameter-freezing strategies, and hyperparameter searches. How do the authors distinguish representational limits from optimization difficulties? Could additional training or changes to Mamba-2’s optimization substantially alter the observed width-scaling trend?
5 comments
Response to Reviewer LCKz (2/3)
(Q1) Scope of the content-addressing separation. As the reviewer notes, practical Mamba-2 goes beyond the class of Thm. 3(c): the mechanisms outside its assumptions are selectivity beyond the first layer, nonlinear writes, and nonlinear reads. Since our proof does not directly extend to these mechanisms (App. F.3.3, "Scope of the selective-SSM bound"), we complement our theory with targeted ablations that test them (App. D.1.4, Fig. 7). Our theory and both ablations point to the interpretation our paper gives: the mechanism behind the separation is sharp selection (§4.2, App. D.1.4), and transition selectivity is one route to it.
Our floors (Thm. 3(a)–(b)) identify sharp selection as necessary. The SSM construction of Prop. 4 shows that selective transitions are one way to achieve it. Our ablations show that nonlinear writes also achieve it, as in the Fixed-transitions variant the reviewer notes, but nonlinear reads alone do not. Ablated Mamba-2, which has the fixed transitions and bilinear writes of our classes but keeps nonlinear reads, stays near baseline in every tested configuration (Figs. 5 and 7). Full Mamba-2 has both routes, yet its smallest successful width still grows rapidly with the codebook size, while the Transformer's stays small (Fig. 6). As in our answer to W1, this suggests that the width separation, our theory's central prediction for content addressing, remains consequential in practice.
Changes in the revision. Section 2.1 now explicitly states that Mamba-2's transitions are selective "in every layer", and the existing "Scope of the selective-SSM bound" paragraph in App. F.3.3 now explicitly names nonlinear writes and reads.
Response to Reviewer LCKz (3/3)
(W2, Q2) Width separation and horizon condition. We thank the reviewer for this important catch: our original wording could indeed suggest that the width separation depends on the horizon condition , but it does not. The revision now makes this explicit: the condition only lets attention's construction reuse its Case 1 optimal window.
The SSM's addressing-capacity bound holds at every prediction time, with no coupling between and (Thm. 3(c); App. F.3.4, Eq. (97)), and attention addresses content with width whenever its window is long enough () to reach each key's previous occurrence (App. F.3.5, Eq. (102)). The condition's sole purpose is to make the Case 1 optimal window , which Prop. 4 reuses for belief maintenance, satisfy (App. F.3.5, after Eq. (105)). Our width-scaling experiment (Fig. 6) meets this requirement directly, with at every tested codebook size. Hence, its sweep over the codebook size measures the width needed for content addressing without relying on the horizon condition.
Changes in the revision. Section 4.2 ("Architectural lessons") now explicitly states that the width separation does not depend on the horizon condition and what the condition provides, App. F.3.4 and F.3.5 now state that the proofs use the condition only for attention's window, and the width-scaling experiment setup in App. D.1.4 now states that provides the required reach at every tested codebook size.
(W2, Q3) Optimization versus capacity. We agree that representational limits must be distinguished from optimization difficulties, and this is precisely why our training is architecture-specific: a single shared protocol would favor whichever architecture it happens to suit, so failures could be due to the training protocol rather than to capacity. The different alignment durations and freezing strategies that the reviewer notes come from our pilot experiments: the Transformer trained best with short alignment followed by joint training, and Mamba-2 with longer alignment whose layers then stay frozen (App. D.1.4). Likewise, we select the best validation-performing run from each search, since an average over runs would let optimization failures mask what an architecture can represent. Its reported score comes from independent test episodes, so the selection cannot inflate it. A success then demonstrates a capability, and a failure despite these measures is stronger evidence of a representational limit (App. D.1.2, "Capability focus and training aids").
While further optimization could lower individual widths of both models in Fig. 6, we do not expect it to substantially alter the trend, since we have devoted extensive effort to minimizing optimization difficulties. Beyond the architecture-specific protocols, we search at least 24 combinations of random seed and learning rate for every configuration of both families. Following our pilots, we search more learning rates for the Transformer and more seeds for Mamba-2, whose training was more seed-sensitive. Only Mamba-2 cells receive larger searches, which, if anything, lower Mamba-2's widths and narrow the gap we report. App. D.1.4 ("Optimization effects") also examines where optimization, rather than capacity, affects individual results.
Response to Reviewer LCKz (1/3)
We thank the reviewer for the thorough review and for recognizing belief geometry as a unified framework linking architecture to statistical performance. The points raised have led to revisions that strengthen the paper throughout, and we address each point in turn below.
(W1) Theoretical scope and practical evidence. The reviewer is correct that the architectural separation results we derive in this paper are proved for the analyzed classes of §2.1. To make this unmistakable, the revision now consistently distinguishes between these results and our practical evidence, from the abstract to the conclusion (changes listed below). Our analysis is designed to isolate the core mechanisms behind these separations, and our experiments support exactly this: the predicted trade-offs still emerge in practical models. This indicates that our theory remains consequential beyond the classes for which its results are proved.
In LLaMA-type Transformers and Mamba-2, these trade-offs emerge in all three cases: the kernel preferences of Case 1 as inductive biases (§3, Fig. 2), the positional-assembly mechanisms and memory advantage of Case 2.a (§4.1, Fig. 4), and the content-addressing mechanisms and width separation of Case 2.b (§4.2, Figs. 5–6). These mechanisms also surface on real benchmarks: Arora et al. (2023) trace most of the language-modeling gap between attention and gated-convolution models to the recall of earlier context (App. A), precisely the content-addressing operation in which our results give attention its exponential width advantage (Thm. 3(c) and Prop. 4). Although practical models differ from our analyzed classes, this evidence suggests that our analysis captures the essence of what separates the two families in practice.
Finally, we want to make sure the reach of the two results the reviewer cites is not understated. First, Case 1's stationary-kernel comparison deliberately starts with the simplest classes of both families to isolate their core belief-maintenance mechanisms, which still matter in practical models (Fig. 2). Our paper also proves sharp regret rates for richer architectures that escape the stationary floor (App. E.2, Props. 6–7). Second, the content-addressing lower bounds hold at every fixed depth for SSMs whose first layer can use any of four different selective transitions drawn from practical architectures, including those of Mamba-2, Gated DeltaNet, and RWKV-7 (§2.1, App. C.1). More broadly, belief geometry itself is general: its regret decomposition (Eq. 2) and two-factor floor (Thm. 2) hold for every predictor, not only for the analyzed classes.
Changes in the revision.