Fractional-Order State Space Models with Learnable Memory Dynamics

Structured state space models (SSMs) combine parallel sequence processing with compact recurrent inference, encoding memory through matrix-exponential responses. We introduce a broader state space framework with a learnable Caputo derivative order $\alpha \in (0,1]$. The continuous formulation includes ordinary first-order SSM dynamics as the special case $\alpha =1$. For $0<\alpha <1$, the state evolution explicitly incorporates nonlocal dependence on past states, with $\alpha$ controlling the history weighting and the algebraic long-time decay of stable relaxation modes. This formulation extends integer-order SSMs with an explicit, learnable mechanism for incorporating state history. Direct evaluation of this history dependence requires a growing cache and quadratic total work. We address these costs through two computational realizations derived from a shared convolution-quadrature discretization. For parallel training, contour sampling of the discrete transfer function and an inverse FFT construct an approximate kernel, which is applied by FFT convolution. For causal streaming inference, a recent-state buffer is combined with a sum-of-exponentials approximation of the discrete history tail. Both realizations approximate the same discrete fractional operator. For fixed model dimensions and compression settings, the streaming cache size is independent of the number of processed tokens, with the tail approximation fitted over a specified horizon. Experiments on Long Range Arena, raw-waveform Speech Commands classification, and BIDMC heart-rate estimation demonstrate the applicability of the framework across diverse sequence domains. Within this framework, controlled comparisons with $\alpha =1$ test whether introducing fractional history can enhance the representational capacity of the corresponding integer-order model.