- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces LTI state space models with learnable fractional derivative order, extending standard first order dynamics. It derives an approximate streaming recurrence from a shared discretization. The authors show competitive performance compared to S4D and other models on LRA and other benchmarks.
Critical strengths
- The mathematical framework seems to be proper, even though it is hard to follow (see weaknesses)
Critical weaknesses
-
The paper uses technical language that makes it hard to follow without the necessary mathematical background. While I understand the difficulty of striking the correct balance between taking a reader along without starting from zero, especially the section on fractional operators would profit from a more careful presentation. It should provide the reader with an intuition for the method, which it currently does not.
-
The baseline results are insufficient. The field has moved on since S4. For example, LinOSS uses a different integration scheme than ZOH [1], explicitly discussing the modeling of long-range dependencies via implicit and explicit integration schemes. How does your method compare?
-
The streaming approximation lacks empirical validation: the paper does not show whether it preserves the parallel model’s performance, or how accuracy trades off against memory usage.
References
Questions for authors
- Does the method generalize to time varying systems like Mamba?
- In table 2, is FracSSM evaluated in a comparable setting (parameter count, epochs etc.) as the baselines?
7 comments
Weakness 2 & 3: Comparison with LinOSS and recent baselines and validation of streaming
The baseline results are insufficient. The field has moved on since S4. For example, LinOSS uses a different integration scheme than ZOH, explicitly discussing the modeling of long-range dependencies via implicit and explicit integration schemes. How does your method compare?
The streaming approximation lacks empirical validation: the paper does not show whether it preserves the parallel model’s performance, or how accuracy trades off against memory usage.
Weakness 2. Comparison with LinOSS and recent baselines
Response:
Thank you for the pointer.
(1) Comparison with LinOSS [R1] on LRA Text. We added the LRA Text experiment to the official LinOSS repository, using the same data preprocessing pipeline as S4 (and FracSSM), and evaluated both LinOSS variants (IM and IMEX). We swept LinOSS's learning rate over :
| Model | lr | ||
|---|---|---|---|
| LinOSS-IM | 74.63 | 83.03 | 72.48 |
| LinOSS-IMEX | 66.06 | 65.82 | 57.52 |
Table H. LinOSS on LRA Text: accuracy (%) across learning rates.
FracSSM achieves 91.12% accuracy on LRA Text with the same data preprocessing. The two models otherwise use their own architectures and training code, so this is a comparison on identical data, not a claim about LinOSS's best tuned LRA score. The IM/IMEX gap is consistent with the LinOSS analysis: IM dissipates, whereas the symplectic IMEX does not forget, which can hurt a task that needs selective forgetting.
(2) Conceptual relation. The integration scheme and the derivative order are orthogonal. LinOSS chooses how to discretize an integer-order ODE, so its per-mode memory remains geometric (damped for IM, undamped for IMEX). FracSSM changes the continuous model itself, to power-law memory; our convolution-quadrature framework then accepts any admissible discretization. The two ideas compose: LinOSS's oscillator has a fractional counterpart of order , which we have mentioned in related work.
(3) Recent baselines. We have added FRACTAL [R2], Liquid-S4 [R3], S5 [R4], LRU [R5], Mega [R6], Mamba [R7], and block-biased Mamba [R8] to Table 2, using their published per-task LRA results. FracSSM's average (88.61) is the highest among these, although some baselines (e.g., S5, Liquid-S4, and Mega) report higher Path-X accuracy.
Weakness 3. Empirical validation of streaming
Response:
We have added this validation. Trained ARFIMA models (Sec. 4; 3 seeds; trained at 1K) are evaluated token by token through the SOE recurrence, which has modes and exact recent states and is fitted once for a 64K horizon. All routes receive identical inputs. "Deviation from exact" is the relative difference from a direct time-domain evaluation of the discrete fractional operator. Results at :
| Route | states / channel | NMSE @1K | @8K | @64K | deviation from exact |
|---|---|---|---|---|---|
| Parallel (training path) | – | 6.3e-8 | 2.3e-5 | 1.28e-3 | 2e-5 – 6e-5 |
| SOE, , | 2,208 | 6.7e-8 | 2.3e-5 | 1.29e-3 | 1.5e-7 |
| SOE, , | 1,184 | 6.7e-8 | 2.3e-5 | 1.29e-3 | 2e-6 – 3e-6 |
| SOE, , | 1,056 | 2.8e-6 | 2.6e-5 | 1.24e-3 | 2e-3 |
| SOE, , | 800 | 2.4e-3 | 1.2e-2 | 3.6e-2 | 0.05 – 0.17 |
| SOE, , | 1,056 | 4.9e-2 | 7.9e-2 | 1.3e-1 | 0.22 – 0.34 |
| Integer () recurrence | 32 | 4.4e-7 | 1.1e-2 | 4.8e-2 | <1e-7 |
Table I. Streaming (SOE) versus parallel evaluation on ARFIMA ().
- Quality. With and , streaming reproduces the parallel model at every length up to the training horizon, so the advantage over is fully retained. and behave the same.
- Trade-off. Accuracy degrades gracefully down to and fails at . Without the exact head () the recurrence deviates by 22–34%.
- Memory. At equal , streaming needs the integer state. At equal deployed state, however, the fractional model wins: with and 21 states per channel it is 28× more accurate at 64K than an integer SSM with 128 states.
- Speed (trained SC09 model, H100, batch 1). Streaming runs at 129–164 tokens/s with flat per-token latency over 16,000 steps, i.e. constant cost. The one-time SOE preparation is currently unoptimized, at about 1.7 h per checkpoint.
Question 1 & 2: Extension to Mamba and Comparability of the Table 2 settings
Does the method generalize to time varying systems like Mamba?
In table 2, is FracSSM evaluated in a comparable setting (parameter count, epochs etc.) as the baselines?
Question 1. Extension to time-varying systems such as Mamba
Response:
Thank you for raising this question. As noted in the Conclusion, our current framework is limited to linear time-invariant dynamics, and extending it to linear time-varying systems is an important direction for future work. Below, we outline a possible formulation of this extension.
What carries over. The original recurrence is
Sx_n &= \Delta^\alpha Bu_n-h_n,\\ y_n = Cx_n+Du_n, \end{aligned}$$ where $S=a_0I-\Delta^\alpha A$ and $h_n=\sum_{j\ge1}a_jx_{n-j}$. With input-dependent $A_n,B_n,C_n$ on a uniform grid, it becomes $$\begin{aligned} (a_0I-\Delta^\alpha A_n)x_n &=\Delta^\alpha B_nu_n-\sum_{j\ge1}a_jx_{n-j},\\ y_n=C_nx_n+Du_n. \end{aligned}$$ The derivative weights $a_j$ do not depend on the input, and for diagonal $A_n$ the state solve is elementwise. The SOE compression therefore applies unchanged. The augmented state $(x_n,\dots,x_{n-J},g_{1,n},\dots,g_{Q,n})$ then follows a linear time-varying recurrence with one $(J+Q+1)\times(J+Q+1)$ block per state coordinate. This admits a parallel scan, but each combine step costs $O((J+Q+1)^3)$ per coordinate rather than Mamba's [H1] elementwise cost, so an efficient selective version would need a more structured scan. **What needs new work.** - The FFT training path relies on time invariance and would be replaced by a scan, as in Mamba. - Input-dependent $\Delta_n$ makes the grid non-uniform. This calls for an SOE of the continuous Caputo kernel [H2], in which each mode decays by $e^{-s_m\Delta_n}$, analogous to Mamba's decay. - An input-dependent $\alpha$ is a further step. We see a selective fractional SSM as the main follow-up. ## Question 2. Comparability of the Table 2 settings **Response:** Thank you for raising this question. Below, we clarify how FracSSM’s experimental settings compare with those of the baselines. - **Data, evaluation, and model selection.** We follow the official S4 [H3] protocols for preprocessing, data splits, tokenization, sequence lengths, and model selection. - **Model.** FracSSM modifies only the order of the S4 layer; parameterization, Cauchy kernel, and residual block follow S4. At $\alpha=1$ it reduces to S4's bilinear discretization, up to a two-tap input average (App. D.3). - **Hyperparameters.** Model depths differ on three tasks: FracSSM uses 4, 6, and 4 layers for ListOps, Text, and Retrieval, respectively, whereas S4 uses 6, 4, and 6. Both models use 6 layers for Image, Pathfinder, and Path-X. The complete FracSSM settings for every task (depth, width, state size, dropout, learning rate, batch size, epochs, and weight decay) are listed in Appendix C.2 (Table 6). - **Fixed-order controls.** We ran $\alpha=1$ controls with our FracSSM implementation on LRA Image and Pathfinder. They reach 88.39 / 93.60, compared with 88.65 / 94.20 for S4-LegS and 92.08 / 96.17 for FracSSM. On Image and Pathfinder our $\alpha=1$ implementation does not exceed S4, so FracSSM's margins there are not explained by our implementation. - **Other matched comparisons.** ARFIMA (Sec. 4) changes only $\alpha$ on the same backbone. **References:** [H1] Gu, Albert, and Tri Dao. “Mamba: Linear-time sequence modeling with selective state spaces.” COLM 2024. [H2] Jiang, Shidong, et al. “Fast evaluation of the Caputo fractional derivative and its applications to fractional diffusion equations.” Communications in Computational Physics 2017. [H3] Gu, Albert, et al. “Efficiently modeling long sequences with structured state spaces.” ICLR 2022.
Weakness 1: Intuition for the fractional operator
Weakness 1. Intuition for the fractional operator
Response:
Thank you for your suggestion. To clarify the intuition behind our method, we explain below how the fractional order controls memory decay and how the resulting model relates to S4.
In one sentence. FracSSM is S4 with one extra knob, the order α∈(0,1]. It changes how memory fades: from exponential (α=1) to power law (α<1).
1. What the fractional derivative does. In an ordinary SSM, x˙(t)=Ax+Bu: the rate of change depends only on the present. Each mode therefore forgets at a fixed rate, e−λt, and information older than a few timescales 1/λ is effectively gone. The Caputo derivative of order α<1,
CDtαx(t)=Γ(1−α)1∫0t(t−s)−αx˙(s)ds,
is a weighted combination of all past rates of change, with weights that decay slowly as a power of elapsed time. The system thus "remembers its own trajectory". At α=1 the weighting collapses to the present instant and we recover x˙ and S4.
2. What a single mode looks like. A fractional mode relaxes as the Mittag–Leffler function Eα(−λtα). This is e−λt at α=1; for α<1 it decays like t−α at long times, a heavy algebraic tail instead of an exponential cutoff. Smaller α means both faster initial decay and a heavier tail. On a log–log plot, the exponential bends down sharply, whereas the fractional response becomes a straight line.
3. Why this helps: one parameter, many timescales. For 0<α<1, Eα(−λtα) is exactly a continuous mixture of exponentials e−st with nonnegative weights spread over many orders of magnitude of s, and α controls the spread. An integer SSM can only approximate such memory by placing many separate decay rates, whether by hand, through initialization, or by learning them. A fractional mode gets the whole spectrum from one learned number.
4. The discrete picture. Take the Grünwald–Letnikov rule for concreteness. At α=1, the update uses the difference xn−xn−1, involving only the previous state. For α<1, the fractional difference is ∑j≥0ajxn−j: every past state contributes, with weights decaying like j−1−α. (Tustin weights additionally contain an alternating component; at α=1 they still recover S4's bilinear update, up to a two-tap input average.) These derivative weights depend only on α, not on the data or on A. For streaming, their tail beyond the J most recent lags is fitted by a sum of Q exponentials up to a chosen horizon T:
\qquad J<j< T,\qquad |\lambda_m|<1.$$ **5. How it is computed, in three familiar steps.** - *Training (same as S4, different evaluation points).* S4 computes its convolution kernel by evaluating a transfer function at points on the unit circle and taking an inverse FFT. In the frequency domain, a fractional derivative simply replaces $s$ by $s^\alpha$. We therefore run the *same* DPLR/Cauchy computation, only at the points $F_\alpha(\zeta_k)/\Delta^\alpha$ instead of S4's bilinear points. - *Why the radius $r<1$.* The fractional symbol has singularities on the unit circle (at zero frequency, and for Tustin also at the Nyquist frequency), so we sample on a circle slightly inside the unit circle and undo this with a known factor $r^{-n}$. The analyticity and aliasing results in the paper only guarantee that this is accurate. - *Streaming.* Because the derivative weights decay smoothly (point 4), their tail can be fitted after training by $Q$ exponential modes, while the $J$ most recent steps are kept exactly. This gives a constant-memory recurrence. Thank you again for your valuable suggestion. In the revised manuscript, we will refine the presentation of the fractional-operator section to explain the underlying intuition more clearly and make it easier to follow for readers with different mathematical backgrounds.