acceptodds
Under review as a conference paper at ICLR 2027

Locality and Retrieval Spend One Budget: A Conservation Law for Sequence Models

Abstract

Every causal, time-invariant sequence mixer obeys an identity its designers did not put there. With the transfer function, the first nonzero tap and the excess phase of the non-minimum-phase zeros, the frequency-averaged log-gain satisfies . It holds to across kernels drawn from S4, S4D, S5, LRU, RetNet, gated linear attention with a frozen gate, and plain causal convolutions, and to on the released weights of Mamba-130M, 1.4B and 2.8B. The identity is a conservation law. Local sharpness and long-range excess phase draw on one budget, and a retrieval head spends the whole of it on phase: we measure against , summing to the all-pass ceiling . The law forces a frontier. Nulling a fraction of the band to depth costs , which random kernels never cross. Because the frontier follows from time invariance and not from state size, an input-dependent gate does not find a better point inside it; it leaves the space where the frontier lives. That departure buys accuracy on multi-query associative recall, at the intermediate state sizes where the frontier binds and nowhere else, across matched runs. A time-invariant model initialised over a spread of timescales recovers most of the gain, which locates the operative ingredient in multi-timescale capacity rather than in gating, and makes it available with no extra parameters and no cost at inference. Beyond the boundary the selective class carries an invariant of its own, a contraction sum rule we verify to on delta-rule transitions.

Then back it, or bet against it.

Related papers

Open the market on this paper to see 7 more related papers.