SpecOSS: Spectrally Generated Oscillatory State-Space Models

A time-dependent signal reflects the dynamics of the system that produced it, and the natural frequencies of those dynamics indicate which timescales a model of the signal should cover. Two classical theories provide us such systems in closed form, the Sturm–Liouville (SL) eigenvalues of the classical orthogonal- polynomial families (Szeg ̋o, 1939), and the memory operator that keeps an online orthogonal-function expansion of the past of a signal (Lee, 1932; Wiener, 1958). Both provide us a ladder of natural frequencies, and that ladder is all a bank of forced harmonic oscillators needs, because its state matrix is a nonnegative di- agonal that holds one squared frequency per oscillator. Existing oscillatory lay- ers leave that frequency grid free, initialising it at random and learning it, even though it is what decides which timescales a layer can remember. We build the Spectrally Generated Oscillatory State-Space model (SpecOSS), in which the grid is instead generated offline, from the SL spectrum of a polynomial family or from the function-approximation operator, and used to initialise the state matrix. The grid may then be trained further, and even leaving it fixed for the whole of training gives strong results on our accuracy and error metrics, which demonstrates the expressiveness of the physical priors it carries. Since a signal need not conform to a single physical system, we also generate grids that fuse several spectra and weight them against each other with a learned low-rank gate, at no additional cost to the recurrence. We complement the construction with an error-bound and ex- pressiveness analysis of generated and frozen grids. Across five time-dependent signal classification benchmarks, heart-rate regression from wearable sensors at a sequence length of fifty thousand, real-world weather forecasting, and WikiText- 103 and Pile language modelling, generated grids improve on a randomly ini- tialised learned grid and on several first-order baselines. We further ablate several cores of our formulation.