acceptodds
Under review as a conference paper at ICLR 2027

SpecOSS: Spectrally Generated Oscillatory State-Space Models

Abstract

A time-dependent signal reflects the dynamics of the system that produced it, and the natural frequencies of those dynamics indicate which timescales a model of the signal should cover. Two classical theories provide us such systems in closed form, the Sturm–Liouville (SL) eigenvalues of the classical orthogonal- polynomial families (Szeg ̋o, 1939), and the memory operator that keeps an online orthogonal-function expansion of the past of a signal (Lee, 1932; Wiener, 1958). Both provide us a ladder of natural frequencies, and that ladder is all a bank of forced harmonic oscillators needs, because its state matrix is a nonnegative di- agonal that holds one squared frequency per oscillator. Existing oscillatory lay- ers leave that frequency grid free, initialising it at random and learning it, even though it is what decides which timescales a layer can remember. We build the Spectrally Generated Oscillatory State-Space model (SpecOSS), in which the grid is instead generated offline, from the SL spectrum of a polynomial family or from the function-approximation operator, and used to initialise the state matrix. The grid may then be trained further, and even leaving it fixed for the whole of training gives strong results on our accuracy and error metrics, which demonstrates the expressiveness of the physical priors it carries. Since a signal need not conform to a single physical system, we also generate grids that fuse several spectra and weight them against each other with a learned low-rank gate, at no additional cost to the recurrence. We complement the construction with an error-bound and ex- pressiveness analysis of generated and frozen grids. Across five time-dependent signal classification benchmarks, heart-rate regression from wearable sensors at a sequence length of fifty thousand, real-world weather forecasting, and WikiText- 103 and Pile language modelling, generated grids improve on a randomly ini- tialised learned grid and on several first-order baselines. We further ablate several cores of our formulation.

Primary area:
learning on time series and dynamical systems
Keywords:
Time series, physical systems

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_3f7kOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SpecOSS, which replaces the random initialisation of the diagonal state matrix in LinOSS with fixed frequency grids built from orthogonal bases. It evaluates on five UEA classification tasks, PPG-DaLiA heart-rate regression, WTH forecasting, PTB-XL, and small-scale language modelling, and reports gains over LinOSS and first-order SSMs.

Critical strengths

1.The paper asks an interesting question, namely how to initialise the oscillator spectrum in second-order SSMs, and the proposed change adds no extra cost to the recurrence. 2.The theory is hard to follow, but the arguments seem sound and build on each other.

Critical weaknesses
  1. The standard long-range benchmark is only partly reported. The paper leaves out EigenWorms, the longest dataset in the six-dataset benchmark that LinOSS uses, with no reason give. Please report EigenWorms for SpecOSS and its other baseline performance, or explain why it is left out.
  2. The results do not clearly back the main claim. Several SpecOSS variants in Table 2 do worse than LinOSS or fall within its variance, especially on Motor and SCP1, so it is unclear whether a structured spectrum is reliably better than a random one. Could the authors say when a structured spectrum should beat random initialisation, and why it does not in these cases?
  3. Forecasting is tested on one setting only. The forecasting results use a single dataset at a single horizon, where the multi-basis variant does best, but the classification results show the best variant changes from dataset to dataset. With one setting, it is unclear whether multi-basis stays the best choice for forecasting or whether the ranking shifts as it does in classification. Could the authors add more forecasting datasets and horizons, compared against LinOSS, to show whether the multi-basis advantage holds?
Questions for authors

See weakness

2 comments
AuthorsOpenReview ↗

Reply to reviewer 3f7k

Thanks for the time and energy you put to review our paper. We answer each question separately:


Q The standard long-range benchmark is only partly reported. The paper leaves out EigenWorms ...

A: To address this concern we compare SpecOSS Multi FA against LinOSS with the same architecture, training budget and data splits. EigenWorms has 5 classes and sequences of 17,984 steps. As in LinOSS, each seed splits the data at random into 70/15/15 % train/validation/test. LinOSS uses AA at random, as in its original formulation. We report results on three seeds and the average:

ModelSeed 1Seed 2Seed 3Mean ± std
LinOSS (random AA)80.695.091.789.1 ± 7.3
SpecOSS (Multi FA)91.794.497.294.4 ± 2.8

In final version of paper, we provide all other variants of SpecOSS (e.g. multi-SL or single families as well).


Q SpecOSS variants in Table 2 fall within variance of LinOSS especially on Motor and SCP1...

A: The results in Table 2 mainly favor SpecOSS (ours) over LinOSS.

Regarding SCP1, all of our reported configurations outperform both LinOSS variants, except Hermite-FA being a notable exception. Although the reported variance is relatively high, please note that matched seeds show that SpecOSS outperforms in each individual seed, even when absolute performance decreases because of initialization as we inspect each individual seed (i.e. we consistently lead in each individual seed).

Regarding Motor: The advantage of structured spectra comes from their systematic frequency coverage,unlike random initialization, basis-derived spectra allocate oscillator modes according to a known frequency distribution, reducing the risk of redundant modes or poorly covered frequency regions. Motor exp (as the reviewer pointed) shows this distinction, while the multi-basis variants are weaker, the single Hermite-FA spectrum reaches our strongest result, outperforming both LinOSS variants. One explanation is that Hermite-FA already provides an appropriate distribution of temporal scales, whereas combining additional bases divides the available state capacity among potentially less relevant frequencies. Thus, our claim is not that every structured spectrum universally outperforms random initialization, but that suitable basis-derived spectra provide a more effective starting point, with the optimal choice depending on the task. In rebuttal, we also did a simple hypothesis test by increasing the number of modes mm which improved spectral resolution and model capacity at additional computational cost, giving us approximately 8% higher accuracy for Multi-FA (comparable with current Hermit-Fa performance). But, the same increase in state size did not produce comparable improvements for LinOSS (around 0.6%), further suggesting that structured frequency allocation can make more effective use of additional modes.


Q Forecasting is tested on one setting only...

A: For the rebuttal we added a second standard forecasting benchmark, ETTh1 (hourly electricity transformer data, seven channels, the standard 12/4/4-month split), at four horizons of 24, 168, 336 and 720 steps. We also evaluated WTH at the short horizon of 24 steps, in addition to the 720-step setting in the original paper (please see subsection 5.3 in rebuttal pdf). LinOSS uses the same backbone (with HPs tunning for each dataset), params, training budget and data splits are comparable with SpecOSS throughout. The table reports test MSE / MAE on standardised data (lower is better).

ModelETTh1, 24ETTh1, 168ETTh1, 336ETTh1, 720WTH, 24
LinOSS1.841 / 1.0431.029 / 0.8251.045 / 0.8230.989 / 0.8080.543 / 0.538
SpecOSS (Hermite SL)0.411 / 0.4350.491 / 0.4740.492 / 0.4740.587 / 0.5250.402 / 0.451
SpecOSS (Multi SL)0.406 / 0.4300.489 / 0.4750.513 / 0.4890.582 / 0.5250.379 / 0.435

SpecOSS outperforms LinOSS at every horizon on both datasets, so the advantage of a structured spectrum in forecasting is not tied to one dataset or one horizon. Among our variants, the multi-basis model has the lowest error at the short horizon on both datasets. Our reading is the same as for Table 2: the reliable gain comes from replacing the random spectrum with a structured one. The multi-basis union is a safe default, and its margin over the best single basis depends on the task. Or equivalently increase the size of mm as detailed in our previous answer to your query to improve the spectral resolution is a safe option.

I thank the authors for the detailed clarification and the additional experiments. My concerns have been addressed, and I will raise my score accordingly.

Reviewer_U4KxOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes SpecOSS, a method for structuring the natural frequency parameters of second-order diagonal oscillatory state-space models (SSMs). Instead of random initialization, the authors generate the diagonal squared frequencies (A) using two mathematical routes:

  • Eigenvalues of self-adjoint Sturm–Liouville (SL) differential operators associated with classical orthogonal polynomials (Hermite, Legendre, Chebyshev, Laguerre, Fourier).
  • Singular values / eigenvalues derived from the second-order differentiated continuous memory law of online function approximation (HiPPO).

The paper introduces a multi-basis formulation where distinct basis spectra occupy orthogonal subspaces of a block-diagonal transition matrix, modulated by an input-dependent learned gate. The authors analyze spectral discretization errors via numerical quadrature bounds, prove expressivity under frozen spectra, and benchmark across five UEA classification datasets, 50k-step PPG heart-rate regression, long-horizon weather forecasting (WTH), and causal language modeling on WikiText-103 and the Pile.

Critical strengths

1 - The paper establishes a sound bridge between classical Sturm–Liouville spectral theory, online function approximation operators, and oscillatory recurrences. Proposition 1 cleanly clarifies when the second-order reduction from first-order generators is exact (skew-symmetric Hermite/Fourier) versus a spectral surrogate.

2- Solid Theoretical Framing - Proposition 4 provides a rigorous motivation for spectral flooring and normalization, proving the necessity of these guards to avoid CFL instability under IMEX and runaway variance under IM.

3- The empirical scope is broad, covering diverse modalities and lengths (up to 50k steps via an exact chunked associative scan). The authors evaluate across multiple ablations: frozen vs. trainable spectra, IM vs. IMEX discretizations, and matched architectural controls.

Critical weaknesses

1- Architectural Contribution:

  • At its operational core, the backbone and recurrence are essentially LinOSS wrapped inside a standard gated residual block.
  • The primary operational novelty boils down to initializing the diagonal entries of A from closed-form polynomial formulas or precomputed matrix singular values (as reflected in lines 2430–2445). While deriving these spectra via SL theory and differentiated memory laws is mathematically elegant, the actual architectural delta over existing diagonal second-order SSMs is limited.

2- Absence of a Formal Explainability Framework:

  • Oscillatory SSMs are fundamentally motivated by their physical interpretability: modes are tied to specific frequencies. However, the paper's treatment of "interpretability" is restricted to qualitative visualizations of 1D state trajectories and 2D latent projections with decision boundaries (Figures 2 and 11).
  • The paper does not employ structured explainability or feature attribution methods (e.g., frequency-domain saliency, integrated gradients across frequency bands) to identify which modes govern predictions on physiological or mechanical time series.
  • Despite claiming that distinct bases capture complementary dynamics, there is no systematic inspection of the learned gating weights G over time or across classes to verify whether the gate actually routes specific temporal features to their corresponding basis subspaces.

3- Superficial Qualitative and Error Diagnostics:

  • The qualitative analysis in Section 5 remains largely descriptive rather than diagnostic. E.g., In WTH forecasting (Figure 4), the text notes that the model captures the diurnal cycle but misses day-to-day weather shifts, without diagnosing whether this is an inductive limitation of the frequency floor, numerical damping, or state capacity.
  • The empirical results present unexplained failure modes. In PPG-DaLiA (Table 13), Legendre fails completely (test MSE 3010-2 vs. 2.3910-2 for Hermite+Legendre and 3.1010-2 for Hermite). The manuscript dismisses this in a single sentence. Given that Legendre features a tighter theoretical quadrature bound (hm=O(m-1)) than Hermite (hm=O(m-1/2)), a rigorous failure-mode breakdown of why Legendre becomes untrainable under strided regression is helpful.
Questions for authors

1- Failure Mode of Legendre on PPG-DaLiA: Can you provide a concrete numerical or gradient-level diagnostic for why both frozen and trainable Legendre spectra failed to train under strided regression on PPG-DaLiA (Table 13), while Hermite trained stably?

2- Analysis of the Multi-Basis Gate: Could you provide a quantitative attribution analysis of the gating activations Gj across the dataset classes (e.g., in ECG or UEA classification) to substantiate the claim that the network learns to select specific bases for specific temporal features?

3- Input Mapping Decoupling: In Route 2, the derivation yields a natural input coupling matrix -AHBH. Did you evaluate using this basis-induced input map directly rather than randomly initializing and learning B?

2 comments
AuthorsOpenReview ↗

Reply 1 to Reviewer U4Kx

We appreciate the time and energy you put for our paper to review. We tried to address your concerns separately in the following


Q Absence of a Formal Explainability Framework: ...

A: To address this concern, we added a structured mode-ablation analysis to quantify the contribution of individual oscillator modes to prediction accuracy. For each mode kk of basis jj, we remove its contribution from the corresponding basis readout, while keeping all parameters fixed (please see the visualization and numbers added in rebuttal pdf subsection 5.5 and appendix K.1). The gate then operates on the modified readouts, and the model is re-evaluated on the entire test set. We report the accuracy drop in percentage points, that we derive it as ΔAccj,k=Accfull−Acc−j,k\Delta Acc_{j,k}= Acc_{full}-Acc_{-j,k}. Heree positive values indicate a greater contribution of that mode to performance.

Datasetfamilykk=12345678910
EEGHermite0.01.22.40.00.00.00.00.00.00.0
EEGFourier8.211.81.20.00.00.00.00.00.00.0
EEGLegendre10.620.01.20.05.91.23.50.01.20.0
SyntheticHermite0.10.337.84.641.01.28.51.84.20.2
SyntheticFourier39.637.62.30.00.00.30.70.00.00.1
SyntheticLegendre14.50.40.30.345.90.252.044.57.90.5
ECGHermite8.63.92.50.40.00.40.10.10.00.0
ECGLaguerre1.76.61.21.01.20.20.30.50.40.3

We ran the same experiment on a regression task, (heart-rate PPG-DaLiA). Each cell is the increase of the mean absolute error when mode kk of that basis is removed, so larger values again mean a greater contribution (visualization is in figure 5, rebuttal pdf).

familykk=12345678910
Hermite0.560.310.080.070.540.000.130.000.040.45
Legendre1.780.291.850.000.430.000.210.000.020.27

Q Analysis of the Multi-Basis Gate:...

A: We did additional analysis on SCP1 and PPG-DaLiA to examine how the gate weights each basis's contribution to the model's output (appendix K.1 and main body subsection 5.5 in rebuttal pdf). We followed the gate weights and basis readouts over time and checked how they change with the signal's behavior. In the selected SCP1 example, the smoothed gate gives the largest weight to Legendre during a signal shift, briefly to Hermite during a quiet interval, and to Fourier near a prominent peak. In the PPG-DaLiA example, the weight shifts toward Legendre during wrist motion and toward Hermite when the wrist is relatively still.

These examples show that the gate adjusts the balance between bases as the signal changes, supporting their complementary use for different temporal patterns. The associations describe the selected intervals, rather than fixed roles for each basis.


Q Failure Mode of Legendre on PPG-DaLiA:...

A: Legendre on its own is not numerically unstable and does not diverge. But, it fails to learn in this setting: within the first few hundred steps, its predictions become nearly constant and remain so as we checked the read outs. A likely cause is the frequency floor used in the initialization, Ak=max⁡(λk/max⁡iλi,amin⁡)A_k=\max(\lambda_k/\max_i\lambda_i,a_{\min}). Unlike Hermite, Legendre places several modes at this floor, creating multiple oscillators with the same lowest natural frequency (for example, at m=64m=64 with amin⁡=0.01a_{\min}=0.01, Legendre's sixth mode k=5k=5 satisfies A5=max⁡(5(5+1)/(63⋅64),0.01)=0.01A_5=\max(5(5+1)/(63\cdot64),0.01)=0.01, and all five preceding modes are also clamped to the floor. In contrast, for Hermite, only the first mode reaches the floor). These modes can respond strongly to slow drift over long recordings, potentially dominating the readout and saturating downstream nonlinearities, leading to flat predictions and weak gradients. Making the spectrum trainable does not resolve this. Besides, the parameterization Ak=amin⁡+A^k2A_k=a_{\min}+\hat A_k^2 initializes floored modes at A^k=0\hat A_k=0, where ∂Ak/∂A^k=0\partial A_k/\partial\hat A_k=0. We observed that these parameters remained zero after training.

Considering multi family of Legendre with Hermite removes this failure, suggesting that their complementary spectral distributions provide a better representation of the signal without the same optimization difficulty. Thus Legendre's tighter theoretical quadrature bound concerns kernel approximation, not the trainability of the complete model.

AuthorsOpenReview ↗

Reply 2 to reviewer U4Kx

Q In Route 2, the derivation yields a natural input coupling matrix -AHBH:...

A: We evaluated three settings for Route 2: (i) initializing BB from the basis-induced coupling −AHBH-A_H B_H and keeping it fixed, (ii) using the same initialization but allowing BB to be trained, and (iii) randomly initializing and learning BB. The fixed basis-induced coupling performed poorly, but the two trainable variants achieved comparable results with margina differences. This may be because of additional BHf′B_Hf' term in the exact second-order derivation, which is dropped in our oscillator formulation (since we dont have f′f'). Thus, the basis-induced coupling alone may be insufficient, while learning BB allows the model to adapt to the missing forcing term.


Q WTH forecasting misses day-to-day weather shifts...

A: What Figure 4 shows is set by the forecast horizon. It is the 720 step setting, in which a month of hourly values is produced from a single window, and at that lead the next weather shift is not contained in the input, so the error optimal prediction is the conditional mean, that is the periodic part plus the slow level, which is what the figure shows. For the rebuttal we trained the same model on WTH at a horizon of 24 steps, with the same spectrum, the same floor and the same state size, and there the prediction does follow the day to day deviations, because at that lead they are still visible in the input window (rebuttal subsection 5.5).

ModelWTH, 24 (MSE/MAE)
LinOSS0.543 / 0.538
SpecOSS (Hermite SL)0.402 / 0.451
SpecOSS (Multi SL)0.379 / 0.435

Q Architectural contribution...

A: SpecOSS retains the second-order recurrence system similar to LinOSS, but its contribution goes beyond replacing randomly initialized diagonal entries. Architecturally, we introduces a multi-basis state representation in which distinct orthogonal-basis spectra occupy separate state subspaces, each with its own learned readout (with two theoretical rouths to derive them). These representations can then be combined through an input-dependent, low-rank gate that dynamically weights their contributions across channels and time steps (besides ablation over complex gates). This enables multiple complementary spectral distributions to coexist and interact within a single oscillator layer, rather than restricting the model to one frequency distribution (intuitively, they are prior physical knowledge we are imposing), already tested on several exps with different behavior. Such architecture, designed in a way to preserves the diagonal structure and parallel-scan complexity at the same total state size, is a better read of what we did regarding the architectural contribution.

Reviewer_bFHfOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SpecOSS, an oscillatory SSM whose diagonal state matrix is initialised from the spectra of classical orthogonal bases. It uses either Sturm–Liouville eigenvalues or a second-order version of the HiPPO memory law. Several bases can share one block-diagonal matrix at no extra scan cost. The authors give error and expressivity bounds for the resulting spectra and report results on UEA classification, PPG-DaLiA, WTH forecasting and small-scale language modelling.

Critical strengths
  • In a second-order model the whole diagonal is a spectrum of frequencies, so drawing it from spectral theory feels natural.
  • The paper is honest about where Route 2 is exact and where it is only a surrogate.
  • The gains on PPG-DaLiA and WTH are large, although there is more room for experiments (LRA).
Critical weaknesses
  • The claim is that structured spectral initialisation improves long-range memory, and thats why we use SSM not transfomer (i.e. their linear complexity with long time) and LRA is the standard benchmark for exactly that claim. It is used by S4, S4D, S5, LRU and most work on SSM initialisation, why was LRA left out? Results on at least ListOps, Image, Pathfinder and Path-X would make the contribution much easier to place.
  • I would also like the authors to discuss more carefully why first-order diagonal SSMs appear to be a poor approximation here. First-order models with complex poles already encode structured frequency grids. S4D-Lin (the one mentioned in experiemnts) me for instance, uses evenly spaced imaginary parts close to the Fourier spectrum used in this paper.
Questions for authors

Overall this is a clear original and well-executed paper. I lean positive and would raise my score if the authors add LRA results and a fairer comparison with complex-diagonal first-order SSMs.

2 comments
AuthorsOpenReview ↗

Reply to reviewer bFHf

Thanks for the energy and time you put for our paper and positive feedback. We appreciate it! The answer to each question is provided separately:


Q discuss more carefully why first-order diagonal SSMs...

A:

Intuitively we try to attribute the observed differences to two complementary inductive biases compared to complex first-order SSMs that may explain the outperformance.

First, frequency-dependent memory retention. For a discrete oscillatory mode with pole zk=rkeiθkz_k=r_ke^{i\theta_k}, the contribution of past inputs decays proportionally to rktr_k^t. Defining the memory horizon as the number of steps required for this contribution to decrease by a factor of ee, we can deriv:

\quad\Longrightarrow\quad \tau_k=-\frac{1}{\ln r_k}.$$ For SpecOSS with implicit discretization (just for example we picked this variant), $r_k=(1+\Delta^2\omega_k^2)^{-1/2}$, giving $${\tau_k=\frac{2}{\ln(1+\Delta^2\omega_k^2)}.}$$ Thus, with a common step size, low-frequency modes retain information longer, while high-frequency modes decay faster. In contrast, S4D-Lin (again for example we picked this first ordered SSM) with fixed damping $\alpha$, and a common step size has $\tau_k=1/(\alpha\Delta)$, independent of frequency. SpecOSS therefore introduces a useful frequency dependent memory-allocation bias, although the optimal retention profile is task-dependent. **Second, spectral alignment.** Our basis-derived spectra distribute a finite number of oscillator modes differently across the frequency band. For a signal containing frequency $\Omega$, its distance to the closest available mode is $${d(\Omega,\mathcal W)=\min_k|\Omega-\omega_k|.}$$ The relevance of this distance follows from the normalized correlation between two oscillations over a window of length $T$: $$\left\lvert \frac{1}{T}\int_{0}^{T} e^{i(\Omega-\omega_{k})t}\ dt \right\rvert = \left\lvert \frac{\sin\left((\Omega-\omega_{k})T/2\right)}{(\Omega-\omega_{k})T/2} \right\rvert$$ A smaller frequency mismatch generally gives stronger alignment near the target frequency, making that oscillatory component easier to represent with limited capacity. Multi-basis SpecOSS combines complementary frequency distributions, potentially placing more modes near task-relevant patterns (intuitively it is like imposing prior frequencies resulted from known systems rather than sticking with single prior frequency (e.g. fourier aking to S4D-Lin) or random frequency grid (e.g. LinOSS)). Together, these mechanisms suggest that SpecOSS's advantage may arise from **(I)** how long each frequency is remembered and **(II)** how effectively modes are allocated (actually for this we did complementary ablation analysis in rebuttal about effectiveness of gating we proposed and importance of designed modes that you can kindly find in experiment section 5.5 and also answer A1 and A2 to reviewer U4Kx), rather than from an intrinsic limitation of complex first-order SSMs. --- **Q** *LRA results...* **A**: To address this concern, we report LRA experiments with three first-order SSM baselines S4D-Lin, S5 and LRU (S4D-Lin outperforms S4 in first few epochs in all data so we dropped S4 ), alongside second ordered LinOSS. Very huge demand on our compute nodes in this rebuttal period limited us to run 10 epochs for each approach. | Model | ListOps | Text | Retrieval | Image | Pathfinder | |---|---|---|---|---|---| | S4D-Lin | 53.85 | 81.16 | 84.17 | 66.32 | 90.69 | | S5 | 53.70 | 80.72 | 84.70 | **69.96** | 90.38 | | LRU | 44.45 | 79.92 | 83.02 | 59.12 | 91.01 | | LinOSS (IM) | 54.35 | 81.36 | 78.20 | 66.74 | 85.77 | | Ours (Multi-SL-Trainable) | **57.50** | **87.88** | **86.85** | 67.12 | **93.72** | SpecOSS already leads on four of the five tasks after this training budget with consistent outperformance from early epochs. Image is the exceptionn, inn the current configuration, the (first ordered) baseline backbones have approximately 30-40% more parameters than SpecOSS as we used their paper instructions and setting of their repo. A closer comparison therefore requires reducing their parameter counts to match ours. This may also affect the ranking on image at the end. We hope these preliminary results provide an initial response to your concern. We plan to complete the full training schedules and provide the final results (along with the parameter-matched comparison for image as well as other variants of SpecOSS e.g. single family and second route variants) in final version.
Reviewer_bFHfOpenReview ↗

Concerns regarding LRA reported results

Thank you for the additional LRA experiments, I have a few questions to help me interpret the table. You mention the " 30-40% more parameters than SpecOSS " but according to the paper (table 6) the baselines like s4d has the same number of parameters (almost), could you clarify where the 30-40% difference comes from?

Also after 10 epochs all models are still far from their published converged accuracies (Image for instance reaches~89). Since the paper argues that the benefit of SpecOSS is mainly an initialisations effect an early-training snapshot may favour it by design, could you share learning curves and ideally converged results on at least one or two tasks? Do you have results on Path-X (which most directly tests the long-range memory claim)? Thank you

Comments on the paper

2 comments

Dear Reviewers,

As we enter the final stage of the author-reviewer discussion, I would like to encourage everyone to actively engage with the authors’ responses and any additional evidence they have provided.

Please take this opportunity to reconsider your initial assessments with an open mind, particularly where the authors have offered substantive clarifications, detailed arguments, or additional experimental results. The goal of the discussion is not to defend initial positions, but to reach the fairest and most informed evaluation of the submission.

I encourage you to engage directly with the authors on any remaining concerns, clearly distinguishing between issues that have been satisfactorily addressed and those that remain unresolved. Where appropriate, please update your reviews and scores accordingly, or explain why your original assessment still stands.

Your active and constructive participation in these final exchanges is essential to reaching a fair, evidence-based decision.

Your AC

In Table 18 of the original submission (I dont know where this tabke in the new version btw) SpecOSS uses dense input and output projections, B \in C^{m×H}$$ and C \in C^{H×m}, with 4mH + m (frozen) or 4mH + 2m (trainable) parameters. This is the same form and scaling listed for S5 and LRU (4mH + 2m). S4D-Lin, and S4D-DFouT as far as I understand instead use per-channel diagonal SSMs rather than dense projections. Given this it seems that at the same state size SpecOSS and S5 should have essentially the same number of parameters in the recurrent projections. The difference in the Image experiment would then come from the state size (m = 16 vs P = 192) rather than from an architectural difference. Could you kindly clarify where the mismatch comes from? If the LRA experiments use a different channelwise realization of SpecOSS that is not described in the paper could you describe it?

Regarding the ListOps and Text results, could you confirm whether the baseline numbers are reruns in your setup or are taken from the original papers? They appear to match the published values for S5, S4D-Lin and LRU.

Finally, the SpecOSS results of 65.70 on ListOps and 92.64 on Text are as far as I can tell higher than most published SSM results on these tasks which is very encouraging. To help interpret them could you share the exact configuration used (e.g. whether the spectrum is frozen or trainable, the total parameter count?)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.