acceptodds
Under review as a conference paper at ICLR 2027

Fractional-Order State Space Models with Learnable Memory Dynamics

Abstract

Structured state space models (SSMs) combine parallel sequence processing with compact recurrent inference, encoding memory through matrix-exponential responses. We introduce a broader state space framework with a learnable Caputo derivative order . The continuous formulation includes ordinary first-order SSM dynamics as the special case . For , the state evolution explicitly incorporates nonlocal dependence on past states, with controlling the history weighting and the algebraic long-time decay of stable relaxation modes. This formulation extends integer-order SSMs with an explicit, learnable mechanism for incorporating state history. Direct evaluation of this history dependence requires a growing cache and quadratic total work. We address these costs through two computational realizations derived from a shared convolution-quadrature discretization. For parallel training, contour sampling of the discrete transfer function and an inverse FFT construct an approximate kernel, which is applied by FFT convolution. For causal streaming inference, a recent-state buffer is combined with a sum-of-exponentials approximation of the discrete history tail. Both realizations approximate the same discrete fractional operator. For fixed model dimensions and compression settings, the streaming cache size is independent of the number of processed tokens, with the tail approximation fitted over a specified horizon. Experiments on Long Range Arena, raw-waveform Speech Commands classification, and BIDMC heart-rate estimation demonstrate the applicability of the framework across diverse sequence domains. Within this framework, controlled comparisons with test whether introducing fractional history can enhance the representational capacity of the corresponding integer-order model.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
State Space Models, Fractional-Order Dynamics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_f9vyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces LTI state space models with learnable fractional derivative order, extending standard first order dynamics. It derives an approximate streaming recurrence from a shared discretization. The authors show competitive performance compared to S4D and other models on LRA and other benchmarks.

Critical strengths
  • The mathematical framework seems to be proper, even though it is hard to follow (see weaknesses)
Critical weaknesses
  • The paper uses technical language that makes it hard to follow without the necessary mathematical background. While I understand the difficulty of striking the correct balance between taking a reader along without starting from zero, especially the section on fractional operators would profit from a more careful presentation. It should provide the reader with an intuition for the method, which it currently does not.

  • The baseline results are insufficient. The field has moved on since S4. For example, LinOSS uses a different integration scheme than ZOH [1], explicitly discussing the modeling of long-range dependencies via implicit and explicit integration schemes. How does your method compare?

  • The streaming approximation lacks empirical validation: the paper does not show whether it preserves the parallel model’s performance, or how accuracy trades off against memory usage.

References

[1] https://arxiv.org/abs/2410.03943

Questions for authors
  • Does the method generalize to time varying systems like Mamba?
  • In table 2, is FracSSM evaluated in a comparable setting (parameter count, epochs etc.) as the baselines?
7 comments
AuthorsOpenReview ↗

Weakness 1: Intuition for the fractional operator

The paper uses technical language that makes it hard to follow without the necessary mathematical background. While I understand the difficulty of striking the correct balance between taking a reader along without starting from zero, especially the section on fractional operators would profit from a more careful presentation. It should provide the reader with an intuition for the method, which it currently does not.

Weakness 1. Intuition for the fractional operator

Response:

Thank you for your suggestion. To clarify the intuition behind our method, we explain below how the fractional order controls memory decay and how the resulting model relates to S4.

In one sentence. FracSSM is S4 with one extra knob, the order α∈(0,1]\alpha\in(0,1]. It changes how memory fades: from exponential (α=1\alpha=1) to power law (α<1\alpha<1).

1. What the fractional derivative does. In an ordinary SSM, x˙(t)=Ax+Bu\dot x(t)=Ax+Bu: the rate of change depends only on the present. Each mode therefore forgets at a fixed rate, e−λte^{-\lambda t}, and information older than a few timescales 1/λ1/\lambda is effectively gone. The Caputo derivative of order α<1\alpha<1,

CDtαx(t)=1Γ(1−α)∫0t(t−s)−αx˙(s)ds,^{C}D^\alpha_t x(t)=\tfrac{1}{\Gamma(1-\alpha)}\int_0^t (t-s)^{-\alpha}\dot x(s)ds,

is a weighted combination of all past rates of change, with weights that decay slowly as a power of elapsed time. The system thus "remembers its own trajectory". At α=1\alpha=1 the weighting collapses to the present instant and we recover x˙\dot x and S4.

2. What a single mode looks like. A fractional mode relaxes as the Mittag–Leffler function Eα(−λtα)E_\alpha(-\lambda t^\alpha). This is e−λte^{-\lambda t} at α=1\alpha=1; for α<1\alpha<1 it decays like t−αt^{-\alpha} at long times, a heavy algebraic tail instead of an exponential cutoff. Smaller α\alpha means both faster initial decay and a heavier tail. On a log–log plot, the exponential bends down sharply, whereas the fractional response becomes a straight line.

3. Why this helps: one parameter, many timescales. For 0<α<10<\alpha<1, Eα(−λtα)E_\alpha(-\lambda t^\alpha) is exactly a continuous mixture of exponentials e−ste^{-st} with nonnegative weights spread over many orders of magnitude of ss, and α\alpha controls the spread. An integer SSM can only approximate such memory by placing many separate decay rates, whether by hand, through initialization, or by learning them. A fractional mode gets the whole spectrum from one learned number.

4. The discrete picture. Take the Grünwald–Letnikov rule for concreteness. At α=1\alpha=1, the update uses the difference xn−xn−1x_n-x_{n-1}, involving only the previous state. For α<1\alpha<1, the fractional difference is ∑j≥0ajxn−j\sum_{j\ge0}a_jx_{n-j}: every past state contributes, with weights decaying like j−1−αj^{-1-\alpha}. (Tustin weights additionally contain an alternating component; at α=1\alpha=1 they still recover S4's bilinear update, up to a two-tap input average.) These derivative weights depend only on α\alpha, not on the data or on AA. For streaming, their tail beyond the JJ most recent lags is fitted by a sum of QQ exponentials up to a chosen horizon TT:

\qquad J<j< T,\qquad |\lambda_m|<1.$$ **5. How it is computed, in three familiar steps.** - *Training (same as S4, different evaluation points).* S4 computes its convolution kernel by evaluating a transfer function at points on the unit circle and taking an inverse FFT. In the frequency domain, a fractional derivative simply replaces $s$ by $s^\alpha$. We therefore run the *same* DPLR/Cauchy computation, only at the points $F_\alpha(\zeta_k)/\Delta^\alpha$ instead of S4's bilinear points. - *Why the radius $r<1$.* The fractional symbol has singularities on the unit circle (at zero frequency, and for Tustin also at the Nyquist frequency), so we sample on a circle slightly inside the unit circle and undo this with a known factor $r^{-n}$. The analyticity and aliasing results in the paper only guarantee that this is accurate. - *Streaming.* Because the derivative weights decay smoothly (point 4), their tail can be fitted after training by $Q$ exponential modes, while the $J$ most recent steps are kept exactly. This gives a constant-memory recurrence. Thank you again for your valuable suggestion. In the revised manuscript, we will refine the presentation of the fractional-operator section to explain the underlying intuition more clearly and make it easier to follow for readers with different mathematical backgrounds.
AuthorsOpenReview ↗

Weakness 2 & 3: Comparison with LinOSS and recent baselines and validation of streaming

The baseline results are insufficient. The field has moved on since S4. For example, LinOSS uses a different integration scheme than ZOH, explicitly discussing the modeling of long-range dependencies via implicit and explicit integration schemes. How does your method compare?

The streaming approximation lacks empirical validation: the paper does not show whether it preserves the parallel model’s performance, or how accuracy trades off against memory usage.

Weakness 2. Comparison with LinOSS and recent baselines

Response:

Thank you for the pointer.

(1) Comparison with LinOSS [R1] on LRA Text. We added the LRA Text experiment to the official LinOSS repository, using the same data preprocessing pipeline as S4 (and FracSSM), and evaluated both LinOSS variants (IM and IMEX). We swept LinOSS's learning rate over {10−4,3×10−4,10−3}\{10^{-4},3\times10^{-4},10^{-3}\}:

Modellr 10−410^{-4}3×10−43\times10^{-4}10−310^{-3}
LinOSS-IM74.6383.0372.48
LinOSS-IMEX66.0665.8257.52

Table H. LinOSS on LRA Text: accuracy (%) across learning rates.

FracSSM achieves 91.12% accuracy on LRA Text with the same data preprocessing. The two models otherwise use their own architectures and training code, so this is a comparison on identical data, not a claim about LinOSS's best tuned LRA score. The IM/IMEX gap is consistent with the LinOSS analysis: IM dissipates, whereas the symplectic IMEX does not forget, which can hurt a task that needs selective forgetting.

(2) Conceptual relation. The integration scheme and the derivative order are orthogonal. LinOSS chooses how to discretize an integer-order ODE, so its per-mode memory remains geometric (damped for IM, undamped for IMEX). FracSSM changes the continuous model itself, to power-law memory; our convolution-quadrature framework then accepts any admissible discretization. The two ideas compose: LinOSS's oscillator has a fractional counterpart of order 1<α≤21<\alpha\le2, which we have mentioned in related work.

(3) Recent baselines. We have added FRACTAL [R2], Liquid-S4 [R3], S5 [R4], LRU [R5], Mega [R6], Mamba [R7], and block-biased Mamba [R8] to Table 2, using their published per-task LRA results. FracSSM's average (88.61) is the highest among these, although some baselines (e.g., S5, Liquid-S4, and Mega) report higher Path-X accuracy.

Weakness 3. Empirical validation of streaming

Response:

We have added this validation. Trained ARFIMA models (Sec. 4; 3 seeds; trained at 1K) are evaluated token by token through the SOE recurrence, which has QQ modes and JJ exact recent states and is fitted once for a 64K horizon. All routes receive identical inputs. "Deviation from exact" is the relative difference from a direct time-domain evaluation of the discrete fractional operator. Results at N=64N=64:

Routestates / channelNMSE @1K@8K@64Kdeviation from exact
Parallel (training path)–6.3e-82.3e-51.28e-32e-5 – 6e-5
SOE, Q=64Q=64, J=4J=42,2086.7e-82.3e-51.29e-31.5e-7
SOE, Q=32Q=32, J=4J=41,1846.7e-82.3e-51.29e-32e-6 – 3e-6
SOE, Q=16Q=16, J=16J=161,0562.8e-62.6e-51.24e-32e-3
SOE, Q=8Q=8, J=16J=168002.4e-31.2e-23.6e-20.05 – 0.17
SOE, Q=32Q=32, J=0J=01,0564.9e-27.9e-21.3e-10.22 – 0.34
Integer (α=1\alpha=1) recurrence324.4e-71.1e-24.8e-2<1e-7

Table I. Streaming (SOE) versus parallel evaluation on ARFIMA (N=64N=64).

  • Quality. With Q≥32Q\ge32 and J≥4J\ge4, streaming reproduces the parallel model at every length up to 64×64\times the training horizon, so the advantage over α=1\alpha=1 is fully retained. N=4N=4 and N=16N=16 behave the same.
  • Trade-off. Accuracy degrades gracefully down to Q=16Q=16 and fails at Q=8Q=8. Without the exact head (J=0J=0) the recurrence deviates by 22–34%.
  • Memory. At equal NN, streaming needs (J+Q+1)×(J+Q+1)\times the integer state. At equal deployed state, however, the fractional model wins: with N=2N=2 and 21 states per channel it is 28× more accurate at 64K than an integer SSM with 128 states.
  • Speed (trained SC09 model, H100, batch 1). Streaming runs at 129–164 tokens/s with flat per-token latency over 16,000 steps, i.e. constant cost. The one-time SOE preparation is currently unoptimized, at about 1.7 h per checkpoint.
AuthorsOpenReview ↗

Question 1 & 2: Extension to Mamba and Comparability of the Table 2 settings

Does the method generalize to time varying systems like Mamba?

In table 2, is FracSSM evaluated in a comparable setting (parameter count, epochs etc.) as the baselines?

Question 1. Extension to time-varying systems such as Mamba

Response:

Thank you for raising this question. As noted in the Conclusion, our current framework is limited to linear time-invariant dynamics, and extending it to linear time-varying systems is an important direction for future work. Below, we outline a possible formulation of this extension.

What carries over. The original recurrence is

Sx_n &= \Delta^\alpha Bu_n-h_n,\\ y_n = Cx_n+Du_n, \end{aligned}$$ where $S=a_0I-\Delta^\alpha A$ and $h_n=\sum_{j\ge1}a_jx_{n-j}$. With input-dependent $A_n,B_n,C_n$ on a uniform grid, it becomes $$\begin{aligned} (a_0I-\Delta^\alpha A_n)x_n &=\Delta^\alpha B_nu_n-\sum_{j\ge1}a_jx_{n-j},\\ y_n=C_nx_n+Du_n. \end{aligned}$$ The derivative weights $a_j$ do not depend on the input, and for diagonal $A_n$ the state solve is elementwise. The SOE compression therefore applies unchanged. The augmented state $(x_n,\dots,x_{n-J},g_{1,n},\dots,g_{Q,n})$ then follows a linear time-varying recurrence with one $(J+Q+1)\times(J+Q+1)$ block per state coordinate. This admits a parallel scan, but each combine step costs $O((J+Q+1)^3)$ per coordinate rather than Mamba's [H1] elementwise cost, so an efficient selective version would need a more structured scan. **What needs new work.** - The FFT training path relies on time invariance and would be replaced by a scan, as in Mamba. - Input-dependent $\Delta_n$ makes the grid non-uniform. This calls for an SOE of the continuous Caputo kernel [H2], in which each mode decays by $e^{-s_m\Delta_n}$, analogous to Mamba's decay. - An input-dependent $\alpha$ is a further step. We see a selective fractional SSM as the main follow-up. ## Question 2. Comparability of the Table 2 settings **Response:** Thank you for raising this question. Below, we clarify how FracSSM’s experimental settings compare with those of the baselines. - **Data, evaluation, and model selection.** We follow the official S4 [H3] protocols for preprocessing, data splits, tokenization, sequence lengths, and model selection. - **Model.** FracSSM modifies only the order of the S4 layer; parameterization, Cauchy kernel, and residual block follow S4. At $\alpha=1$ it reduces to S4's bilinear discretization, up to a two-tap input average (App. D.3). - **Hyperparameters.** Model depths differ on three tasks: FracSSM uses 4, 6, and 4 layers for ListOps, Text, and Retrieval, respectively, whereas S4 uses 6, 4, and 6. Both models use 6 layers for Image, Pathfinder, and Path-X. The complete FracSSM settings for every task (depth, width, state size, dropout, learning rate, batch size, epochs, and weight decay) are listed in Appendix C.2 (Table 6). - **Fixed-order controls.** We ran $\alpha=1$ controls with our FracSSM implementation on LRA Image and Pathfinder. They reach 88.39 / 93.60, compared with 88.65 / 94.20 for S4-LegS and 92.08 / 96.17 for FracSSM. On Image and Pathfinder our $\alpha=1$ implementation does not exceed S4, so FracSSM's margins there are not explained by our implementation. - **Other matched comparisons.** ARFIMA (Sec. 4) changes only $\alpha$ on the same backbone. **References:** [H1] Gu, Albert, and Tri Dao. “Mamba: Linear-time sequence modeling with selective state spaces.” COLM 2024. [H2] Jiang, Shidong, et al. “Fast evaluation of the Caputo fractional derivative and its applications to fractional diffusion equations.” Communications in Computational Physics 2017. [H3] Gu, Albert, et al. “Efficiently modeling long sequences with structured state spaces.” ICLR 2022.
Reviewer_ueEuOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Motivated by the success of SSMs on tasks with long-range dependencies, this paper proposes fractional-order SSMs for α∈(0,1)\alpha \in (0,1), in contrast to the standard α=1\alpha=1 approach. These models have limiting power-law history dependence rather than exponential history dependence. All the required numerical integration, parallel evaluation, and sequential evaluation machinery is derived to make such an approach feasible, mirroring the structure of the S4 or S4D SSM models. A motivating auto-regressive fractional integrated moving average (ARFIMA) experiment is presented, followed by good results on Long Range Arena benchmarks compared to S4 and S4D. Lastly, good results are shown on heart rate estimation and a speech command classification datasets.

Critical strengths
  1. Generalizing the numerical integration, parallel evaluation, and sequential evaluation machinery from S4 and S4D to the α≤1\alpha \leq 1 case appears to require significant work and technical ability. These derivations appear to be correct. However, see weakness 1 and question 1.

  2. The LRA results presented in Table 2 are strong and, if further supported (see weakness 1b), support the case that fracSSM is a practically useful tool.

Critical weaknesses
  1. The paper has considerable overlap with LongSpike (particularly the fractional SSM results presented in Table E.3) and FRACTAL. This raises a couple issues and questions:

a. What is the precise relationship between FRACTAL and both of these methods? Substantially more space should be devoted to discussing these methods and identifying which proposed methods are novel and which are introduced in these earlier works.

b. The LRA evaluations for S4 and S4D appear to match those presented in FRACTAL (which are in turn taken from the S4 and S4D papers). However, these results should also compare with FRACTAL, S5, Mamba, Mamba-2, etc., and not stop with results from 2022. For comparison, the reported average LRA scores are 87.11 for FRACTAL, 87.32 for Liquid-S4, and 87.46 for S5, 87.71 for block-biased Mamba, and 88.61 for FracSSM. A similar treatment to Table 3 should be presented.

  1. There are several holes in the experimental results:

a. Since the LRA results of baseline models are taken from previous publications, the fixed α=1\alpha=1 case using the authors' exact implementation should be presented. This should control for architectural differences that do not relate to the proposed fractional-order SSM.

b. As far as I can tell, Table 4 does not identify what task or benchmark is used. I was also unable to find details on how shorter or longer timescales in the α=1\alpha=1 case are chosen. A trained timescale using the learning rate adjustments afforded to fracSSM should also be included for the α=1\alpha=1 case.

c. In the ARFIMA results, it is unsurprising that the fractional-order model outperform the integer-order model given the true fractional-order data generating process. The capacity results at extrapolated lengths is encouraging, but I would encourage the authors to also present the capacity results at the training sequence length, for a more complete experimental design. Additionally, a convincing result for evaluation length extrapolation would show improved generalization of the fractional model over the integer model relative to the training sequence length performance, which the current results in Table 2b don't support. I also have questions related to initialization relevant to these results (Question 3).

d. Overall, the experimental results as presented are promising, but they don't convincingly answer whether the proposed approach is useful, and if so, in what situations. Additional sequence modeling results on real data (text or DNA, say) could help answer this.

e. I did not find experimental results showing the retained quality of the recurrence in sequential evaluation. This should be a central result.

Questions for authors
  1. See weaknesses 1a and 1b.
  2. A central tension in this paper is the sum-of-exponential (SOE) history dependence structure for α=1\alpha=1 models versus the power law structure for α<1\alpha < 1 models. In particular, the recursive evaluation makes use of the α=1\alpha=1 SOE approach and the parallel full-sequence training approach on any finite-length sequence is also equivalent to some α=1\alpha=1 SOE model. Despite this, the fractional-order approach could have superior inductive biases for long-range history dependencies. I think this is a useful and straightforward framing of the proposal that highlights the main empirical gap in the current paper, but I am open to pushback or additional context from the authors.
  3. On a related note, one of the striking findings from the modern SSM literature is the importance of careful initialization. Could you provide plots of the α<1\alpha < 1 and α=1\alpha=1 initialization HiPPO measures? Are they roughly matched in terms of, say, mean or median value?

Minor:

  • S5 is described as a simplification of S4, which is not accurate. S4 has single-input-single-output functionality while S5 is multi-input-multi-output.
  • [1] is a recent relevant paper that proposes a multiscale HiPPO approach with an effective power-law history representation.

[1] Goffinet et al. (2026). HiPPO Zoo: Explicit Memory Mechanisms for Interpretable State Space Models. ICML.

8 comments
AuthorsOpenReview ↗

Weakness 1a & 1b & 2a: Relationship and comparison with recent baselines, alpha control experiment

1a. What is the precise relationship between FRACTAL and both of these methods? Substantially more space should be devoted to discussing these methods and identifying which proposed methods are novel and which are introduced in these earlier works.

1b. The LRA evaluations for S4 and S4D appear to match those presented in FRACTAL (which are in turn taken from the S4 and S4D papers). However, these results should also compare with FRACTAL, S5, Mamba, Mamba-2, etc., and not stop with results from 2022. For comparison, the reported average LRA scores are 87.11 for FRACTAL, 87.32 for Liquid-S4, and 87.46 for S5, 87.71 for block-biased Mamba, and 88.61 for FracSSM. A similar treatment to Table 3 should be presented.

2a. Since the LRA results of baseline models are taken from previous publications, the fixed α=1\alpha=1 case using the authors' exact implementation should be presented. This should control for architectural differences that do not relate to the proposed fractional-order SSM.

Weakness 1a. Relationship to FRACTAL and LongSpike

Response:

We have expanded related work into a dedicated comparison. The three works use fractional calculus in different places.

  • FRACTAL [D1] keeps the first-order state equation x′=Ax+Bux'=Ax+Bu. A fractional measure only shapes the HiPPO [D2] projection from which AA and BB are derived, using a fixed bank of singularity indices. Every mode therefore still decays exponentially. FracSSM instead changes the derivative order of the state equation itself, so the response family changes from exponentials to Mittag–Leffler functions with algebraic tails, and the order is learned. The two ideas are complementary: FRACTAL's (A,B)(A,B) could initialize FracSSM.
  • LongSpike [D3] uses Caputo fractional dynamics with an exponential kernel approximation; Appendix E.3 also includes a non-spiking f-SSM. FracSSM derives parallel and recurrent computations from the same convolution-quadrature operator and applies SOE compression to the α\alpha-dependent derivative weights only after training. The non-spiking f-SSM and FracSSM report LRA averages of 88.06% and 88.61%, respectively.

Our contributions relative to both:

  1. one convolution-quadrature discrete operator, from which both the parallel kernel and the recurrence are derived exactly;
  2. a spectral kernel construction for that operator, with analyticity conditions and an aliasing-error analysis, so that training uses the fractional operator itself rather than an exponential surrogate;
  3. post-hoc SOE compression of the derivative weights, which depend only on α\alpha, so streaming can be configured after training; it comes with a trajectory-error bound;
  4. an exact α=1\alpha=1 endpoint (App. D.3) that serves as a matched control;
  5. controlled order-versus-capacity experiments.

Weakness 1b. Comparison with recent LRA baselines

Response:

Thank you for highlighting these relevant baselines. We have expanded Table 2 to include FRACTAL [D1], Liquid-S4 [D4], S5 [D5], LRU [D6], Mega [D7], Mamba [D8], and block-biased Mamba [D9], using available published per-task LRA results and clarifying the corresponding experimental settings. FracSSM’s average LRA accuracy (88.61%) exceeds the baseline averages cited in your comment, although some baselines (e.g., S5, Liquid-S4, and Mega) report higher Path-X accuracy.

Weakness 2a. Same-implementation α=1\alpha=1 control

Response:

Thank you for raising this question. At the operator level, S4 [D10] is the integer-order counterpart of FracSSM. Our method retains S4’s DPLR/HiPPO-LegS parameterization, DPLR/Cauchy computational structure, and residual block while introducing the fractional derivative order α\alpha. Setting α=1\alpha=1 recovers S4’s bilinear discretization, up to a two-tap input average (App. D.3).

Fixed-order controls. We ran α=1\alpha=1 controls with our FracSSM implementation on LRA Image and Pathfinder:

ModelImagePathfinder
S4-LegS (reported)88.6594.20
Ours, α=1\alpha=188.3993.60
FracSSM92.0896.17

Table C. LRA accuracy (%) of α=1\alpha=1 controls with our implementation.

On Image and Pathfinder, our α=1\alpha=1 implementation does not exceed S4, so FracSSM's margins on these tasks are not explained by our implementation.

AuthorsOpenReview ↗

Weakness 2b & 2d & 2e: Task settings, usefulness on real data, and quality of sequential (recurrent) evaluation

2b. As far as I can tell, Table 4 does not identify what task or benchmark is used....A trained timescale using the learning rate adjustments afforded to fracSSM should also be included for the α=1\alpha=1 case.

2d. Overall, the experimental results as presented are promising, but they don't convincingly answer whether the proposed approach is useful, and if so, in what situations. Additional sequence modeling results on real data (text or DNA, say) could help answer this.

2e. I did not find experimental results showing the retained quality of the recurrence in sequential evaluation. This should be a central result.

Weakness 2b. Table 4: task and α=1\alpha=1 timescale settings

Response:

Thank you for raising this question. Table 4 reports an ablation on LRA Text. The ablation uses a separate training recipe, which differs from the one behind the Table 2 Text result (91.12%). All rows share this recipe and differ only in the order or step-size settings, so the comparisons within Table 4 are matched.

For the α=1\alpha=1 timescale variants, Δ\Delta is initialized log-uniformly in (10−4,10−2)(10^{-4},10^{-2}) ("shorter timescale", 87.36%) or (10−2,1)(10^{-2},1) ("longer timescale", 89.87%), and remains learnable throughout training.

The results show that:

  1. Fixed orders α∈{0.3,0.5,0.7}\alpha\in\{0.3,0.5,0.7\} improve accuracy by 1.25–1.47 percentage points over α=1\alpha=1, demonstrating the effectiveness of moderate fractional orders.
  2. Learnable α\alpha achieves 90.30%, improving over α=1\alpha=1 by 1.36 points and approaching the best fixed-order result of 90.41%.
  3. The best tested α=1\alpha=1 timescale adjustment reaches 89.87%, a gain of 0.93 points.

On a dedicated learning rate for Δ\Delta at α=1\alpha=1. We did not run this exact variant, but Table 4 already separates the effect of the order from that of extra optimization freedom. The fixed-order rows (α∈{0.3,0.5,0.7}\alpha\in\{0.3,0.5,0.7\}) have no trainable order, so they receive no learning-rate adjustment beyond what the α=1\alpha=1 control has, and their Δ\Delta is trained exactly as in that control. They still improve by 1.25–1.47 points. The gain therefore comes from the fractional dynamics themselves rather than from an additional trainable parameter or its learning rate, and the closest α=1\alpha=1 adjustment of Δ\Delta, retuning its range, gains at most 0.93 points.

Weakness 2d. Usefulness on real data

Response:

We add a real-data generative result: autoregressive raw-audio generation on SC09 (16,000 steps per clip), following the official SaShiMi protocol.

ModelAM ↓FID ↓IS ↑mIS ↑
SaShiMi [E1] (as reported in Mamba [E2])0.741.995.1342.57
SaShiMi [E1] (original paper)0.901.994.1224.57
FracSSM0.6361.6245.61655.13

Table A. SC09 raw-audio generation (official SaShiMi protocol).

Together with Table 4 and ARFIMA, this suggests that fractional order helps most when the relevant dependencies span a broad, scale-free range of lags.

Weakness 2e. Quality of sequential (recurrent) evaluation

Response:

Thank you for highlighting this point. We provide the additional results below. We ran the trained ARFIMA models (3 seeds, trained at 1K) token by token through the SOE recurrence. The recurrence has QQ modes and JJ exact recent states and is fitted once for a 64K horizon. All routes receive identical inputs. "Deviation from exact" is the relative difference from a direct time-domain evaluation of the discrete fractional operator. Results at N=64N=64:

Routestates / channelNMSE @1K@8K@64Kdeviation from exact
Parallel (training path)–6.3e-82.3e-51.28e-32e-5 – 6e-5
SOE, Q=32Q=32, J=4J=41,1846.7e-82.3e-51.29e-32e-6 – 3e-6
SOE, Q=16Q=16, J=16J=161,0562.8e-62.6e-51.24e-32e-3
SOE, Q=32Q=32, J=0J=01,0564.9e-27.9e-21.3e-10.22 – 0.34
Integer (α=1\alpha=1) recurrence324.4e-71.1e-24.8e-2<1e-7

Table D. Sequential (SOE) versus parallel evaluation on ARFIMA (N=64N=64).

With Q=32Q=32 and J=4J=4, sequential evaluation closely matches the parallel model at the evaluated lengths up to 64×64\times the training horizon, preserving its advantage over α=1\alpha=1. The Q=16,J=16Q=16,J=16 configuration uses fewer states but has a larger deviation from the exact reference, while removing the exact recent-state component (J=0J=0) substantially degrades prediction accuracy. Similar trends are observed for N=4N=4 and N=16N=16. In the revised manuscript, we have reported the (J,Q)(J, Q) settings used in our experiments..

References:

[E1] Goel, Karan, et al. “It's raw! Audio generation with state-space models.” ICML 2022.

[E2] Gu, Albert, and Tri Dao. “Mamba: Linear-time sequence modeling with selective state spaces.” COLM 2024.

AuthorsOpenReview ↗

Weakness 2c: ARFIMA capacity and extrapolation

2c. In the ARFIMA results, it is unsurprising that the fractional-order model outperform the integer-order model given the true fractional-order data generating process....I also have questions related to initialization relevant to these results (Question 3).

Response:

We appreciate your comments and suggestions. Appendix Table 5 already reports training-length (L=1,024L=1{,}024) results for N=2–64N=2–64. For the rebuttal, we reran the study with N∈{2,…,256}N\in\{2,\dots,256\}, 3 seeds, 2,048 training sequences (paper: 128), and 10,000 steps, so absolute values differ from Tables 1 and 5 (training-length errors are lower with more data); the conclusions are unchanged. Both arms are identical except for the order: α\alpha is initialized at 0.5 and learned, or fixed at 1. Models are trained at 1K, and the same checkpoint is evaluated at every length.

ARFIMA, d=0.4d=0.4. NMSE:

NNlearned α\alpha1K (train)2K64Kinteger / fractional @64K
20.393.0e-6 / 3.0e-13.3e-6 / 3.1e-11.8e-3 / 3.7e-1206×
40.457.6e-5 / 1.6e-21.7e-4 / 3.0e-22.3e-2 / 1.5e-16.4×
80.471.4e-6 / 1.0e-49.7e-6 / 6.3e-41.1e-2 / 4.3e-23.9×
160.487.5e-8 / 3.4e-68.4e-7 / 3.3e-32.5e-3 / 9.4e-238×
320.497.4e-8 / 3.9e-77.5e-7 / 7.9e-31.5e-3 / 8.3e-254×
640.495.7e-8 / 4.4e-75.6e-7 / 8.6e-41.3e-3 / 4.6e-236×
1280.492.7e-8 / 8.4e-72.1e-7 / 1.4e-38.6e-4 / 5.0e-258×
2560.493.1e-8 / 2.8e-61.9e-7 / 1.1e-37.9e-4 / 4.8e-261×

Table E. ARFIMA (d=0.4d=0.4): NMSE (fractional / integer, mean of 3 seeds).

  • Training length (1K column). For N≥16N\ge16, both arms fit the training window: integer 3.9×10−73.9\times10^{-7}–3.4×10−63.4\times10^{-6}, fractional 2.72.7–7.5×10−87.5\times10^{-8}. The integer model's long-horizon error is therefore not a fitting failure. At 64K, increasing NN from 16 to 256 lowers the fractional error from 2.5×10−32.5\times10^{-3} to 7.9×10−47.9\times10^{-4}. The integer error stays between 4.6×10−24.6\times10^{-2} and 9.4×10−29.4\times10^{-2}. For N≤8N\le8 the integer model does not fit even the training window.
  • Relative generalization (1K → 2K → 64K). Doubling the length (1K → 2K) increases the integer error about 400–20,000× (N≥16N\ge16; e.g. 3.9×10−7→7.9×10−33.9\times10^{-7}\to7.9\times10^{-3} at N=32N=32), but the fractional error only 6–11×. At 64K the growth factors relative to 1K are of the same order: about 22–3×1043\times10^4 (fractional) vs. 2×1042\times10^4–2×1052\times10^5 (integer). The integer error saturates at the energy of the tail it never represented. We therefore claim better relative generalization at moderate extrapolation, and 36–61× lower absolute error at 64K (last column, N≥16N\ge16).
  • The order is not given (learned-α\alpha column). Learning starts from 0.5, and no information about the generating process is supplied. The learned value settles at 0.39–0.49.

Controls. NMSE as fractional / integer:

ControlNNlearned α\alpha1K (train)64Kinteger / fractional @64K
Exp-mix target160.562.3e-4 / 6.6e-79.7e-4 / 1.6e-216×
Exp-mix target320.559.2e-5 / 7.8e-85.0e-4 / 3.1e-36.3×
Exp-mix target640.544.3e-6 / 8.3e-91.9e-4 / 8.3e-44.4×
64 training sequences160.487.5e-8 / 3.5e-62.4e-3 / 9.4e-239×

Table F. ARFIMA control experiments: NMSE (fractional / integer).

On "unsurprising":

  • Exp-mix target (purely exponential memory, timescales 4/64/512; the integer prior is correct). Inside the training window, the integer model is far better and the learned order stays at 0.54–0.56 rather than returning to 1. We report this as a real cost of the fractional prior. Beyond the window, the fractional model is still 4–16× more accurate at 64K, so the extrapolation advantage does not depend on fractional-generated data.
  • Small data. With only 64 training sequences, the gap is unchanged (39× vs. 38× with 2,048 sequences).
Reviewer_uGBuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper generalizes linear time invariant (LTI) state-spaced models (SSMs) beyond their classic derivative formulation to include learnable fractional derivatives. Motivated by the lack of representation capacity that the exponential form can provide, the authors derive an alternative formulation via fractional Caputo derivatives that can be trained in parallel while allowing exponential and algebraic decays. They find that giving the model the opportunity to learn the derivative order α\alpha, which at α=1\alpha = 1 is the classic SSM formulation, improves performance on long-context tasks.

Critical strengths

The paper provides a strong theoretical justification for its claims, walking through mathematical background, training and inference details, and motivating hypotheses in detail. Given the complexity of the mathematical knowledge required, this is extremely useful for readers with a background in classical SSMs. The motivating justification for fractional SSMs is strong as well: a weighted average of different exponential timescales may not provide enough representational capacity to understand long-range dynamics, especially those extrapolated beyond the training timescale.

Critical weaknesses

In my opinion the largest weakness of this paper is its ratio of complexity to improvement. A substantial segment of the paper is dedicated to training methods that differ from classical SSMs and rely heavily on quite complicated mathematical backgrounds. The experiments are largely restricted to LRA and synthetic tasks; the improvement (or lack thereof) on more general language tasks remains unproven. As I understand it, the training methods for fractional SSMs would require a good bit of code change, and the experimental findings in this paper may not be strong enough to justify such major changes.

Questions for authors
  1. When you say you let α\alpha be learnable, exactly how is it represented? If it is represented by a sigmoid, do you find that it tends to remain near the center of the distribution or cluster near 1? It would be useful to know whether learnable fractional order genuinely results in different SSM architectures or whether the optimizer dynamics tend to push it towards more classical α≈1\alpha \approx 1 models.
  2. Do you have any results on training or inference throughput for fractional SSMs? This applies both in theory and in terms of any specific kernels you may have developed.
  3. Can you provide results comparing SSM versions on RULER? LRA is known for relying quite heavily on more local representations. If avoiding the exponential mode is indeed useful for extrapolating past the training horizon, we should see improvements in tasks designed for long-context reasoning specifically.
  4. There are multiple formulations of fractional derivatives, each of which matches the classical derivative at integer values. Why did you choose Caputo derivatives for fractional SSMs?
5 comments
AuthorsOpenReview ↗

Weakness: Complexity relative to improvement

In my opinion the largest weakness of this paper is its ratio of complexity to improvement. A substantial segment of the paper is dedicated to training methods that differ from classical SSMs and rely heavily on quite complicated mathematical backgrounds. The experiments are largely restricted to LRA and synthetic tasks; the improvement (or lack thereof) on more general language tasks remains unproven. As I understand it, the training methods for fractional SSMs would require a good bit of code change, and the experimental findings in this paper may not be strong enough to justify such major changes.

Response:

We thank the reviewer and address implementation cost and evidence separately.

(1) The code change is small; the mathematics certifies it. FracSSM keeps the S4 [A1] training path: the same DPLR resolvent, the same Cauchy/Woodbury sums (with S4's CUDA Cauchy kernel), an inverse FFT, and the same FFT convolution. Three things change:

  • the resolvent is evaluated at wk=Fα(ζk)/Δαw_k=F_\alpha(\zeta_k)/\Delta^\alpha instead of the bilinear argument, which costs one complex power per contour point;
  • the contour radius becomes r<1r<1, followed by a known rescaling r−nr^{-n};
  • one order parameter is added per channel or per layer.

Block structure, optimizer, and data pipeline are unchanged; all datasets follow the official S4 preprocessing and splits. The appendix analysis only shows that this short computation is numerically sound. The streaming SOE construction is not used in training: it is fitted after training, and only when constant-memory recurrent inference is needed.

(2) The gains appear in controlled comparisons.

  • Same model, only α\alpha changed (Table 4, LRA Text). Fixed orders in [0.3,0.7][0.3,0.7] improve over the matched α=1\alpha=1 run (88.94) by 1.25–1.47 points, and learnable α\alpha by 1.36 points.
  • Length extrapolation (Sec. 4). For the rebuttal, we reran this study with 3 seeds, NN up to 256, and a larger training set (2,048 vs. 128 sequences), so absolute values differ from Tables 1 and 5 (training-length errors are lower with more data); the conclusions are unchanged. For N≥16N\ge16, the α=1\alpha=1 models fit the training window to 3.9×10−73.9\times10^{-7}–3.4×10−63.4\times10^{-6} NMSE but are 36–61× worse at 64×64\times the training length. Adding states does not close the gap.

(3) A new, non-classification result: autoregressive raw-audio generation (SC09). We follow the official SaShiMi protocol, including its sample screening and the released classifier.

ModelAM ↓FID ↓IS ↑mIS ↑
SaShiMi [A2] (as reported in Mamba [A3])0.741.995.1342.57
SaShiMi [A2] (original paper)0.901.994.1224.57
FracSSM0.6361.6245.61655.13

Table A. SC09 raw-audio generation (official SaShiMi protocol).

FracSSM outperforms both reported SaShiMi baselines across all four metrics, supporting its effectiveness on real-data generative modeling beyond classification.

(4) Language modeling. We will state this limitation explicitly. Our model is deliberately LTI, so that the order can be isolated against S4-type baselines. Prior work (H3 [A4], Mamba [A3]) highlights limitations of conventional LTI SSMs in content-dependent retrieval and demonstrates the benefits of input-dependent interactions and selection mechanisms for language modeling. A fair language-modeling test therefore requires a selective fractional SSM; the paper's conclusion already states the LTI limitation, and this extension is our main follow-up.

References:

[A1] Gu, Albert, et al. “Efficiently modeling long sequences with structured state spaces.” ICLR 2022.

[A2] Goel, Karan, et al. “It's raw! Audio generation with state-space models.” ICML 2022.

[A3] Gu, Albert, and Tri Dao. “Mamba: Linear-time sequence modeling with selective state spaces.” COLM 2024.

[A4] Fu, Daniel Y., et al. “Hungry Hungry Hippos: Towards language modeling with state space models.” ICLR 2023.

AuthorsOpenReview ↗

Question 1 & 2: Representation and behavior of learnable and throughput

When you say you let α\alpha be learnable, exactly how is it represented? If it is represented by a sigmoid, do you find that it tends to remain near the center of the distribution or cluster near 1? It would be useful to know whether learnable fractional order genuinely results in different SSM architectures or whether the optimizer dynamics tend to push it towards more classical α≈1\alpha \approx 1 models.

Do you have any results on training or inference throughput for fractional SSMs? This applies both in theory and in terms of any specific kernels you may have developed.

Question 1. Representation and behavior of learnable α\alpha

Response:

Representation. Each order is a bounded sigmoid, αh=αmin⁡+(αmax⁡−αmin⁡)σ(θh)\alpha_h=\alpha_{\min}+(\alpha_{\max}-\alpha_{\min})\sigma(\theta_h), either shared or set per channel, with its own learning rate. The α=1\alpha=1 control is not a saturated sigmoid: it is a separate setting in which α\alpha is fixed exactly at 1 and not trained.

The learned orders do not drift to 1. We examined the learned orders on SC09, ListOps, and ARFIMA, with the following results:

  • SC09. At the final checkpoint, the learned orders have median 0.49 and interquartile range 0.40–0.58; no channel exceeds 0.98. The resulting kernels genuinely differ from those of the α=1\alpha=1 model on the same backbone: at the same training step (40K updates), the channel-median 99% lag is 6,044 samples for the fractional model and 551 for the integer model, even though the integer model's step sizes are learned.
  • ListOps. Per-layer medians are 0.50–0.60 and maxima 0.86–0.94; the spread widens in both directions during training, and no layer clusters near 1.
  • ARFIMA (scalar α\alpha, initialized at 0.5, 3 seeds). The order converges to 0.39–0.49. Even on a control target with purely exponential memory, it stays at 0.54–0.56. So α\alpha does not default to 1; this is a genuine inductive bias, and it carries a cost when the true memory is exponential.

Question 2. Training and inference throughput

Response:

Theory (App. D.7). Parallel training costs O(Llog⁡L)O(L\log L) per layer, as in S4/S4D [B1, B2]. The extra work is one complex power Fα(ζk)F_\alpha(\zeta_k) per contour point and distinct order, which is independent of NN, plus an O(L)O(L) rescaling. Streaming with JJ exact recent states and QQ SOE modes needs O(HN(J+Q+1))O(HN(J+Q+1)) memory and O(HN(J+Q))O(HN(J+Q)) history-accumulation work per token, independent of the number of processed tokens. That is a constant factor (J+Q+1)(J+Q+1) over an integer SSM.

Measurements (SC09 model, one H100 PCIe GPU).

  • Training (batch 16). A learnable order costs 0.7–2.8% more step time and under 1% more memory than α=1\alpha=1 in the same code. For example, at length 16,000 it runs at 428K vs. 440K tokens/s.
  • Kernels. We did not write a new fused kernel. Our CUDA Cauchy kernel is ported from the official S4 extension and is simply called at the fractional arguments wkw_k; everything else is standard PyTorch. Compared with a pure-PyTorch Cauchy evaluation, it builds kernels (forward + backward) 2.9–6.3× faster with 14–16× less memory. With it, training at length 16,000 fits on one GPU, whereas the PyTorch version runs out of memory on 80 GB.
  • Streaming generation (trained SC09 checkpoint, batch 1, 16,000 steps). Speed is 158–164 tokens/s with no exact head and 129–140 tokens/s with J=16–64J=16–64, at 1.6–3.5 GiB. Latency is flat across all 16,000 steps (6.8 ms per token in the first window, 6.3 ms in the last), confirming that the cache does not grow. It is almost insensitive to Q∈[16,128]Q\in[16,128]. The current one-time SOE preparation is unoptimized, at about 1.7 h per checkpoint; it is the main practical cost of the streaming route today.
  • Streaming accuracy (ARFIMA). With Q=32Q=32 and J=4J=4, the recurrence matches the parallel model's error at all lengths up to 64×64\times the training horizon, with a deviation of 22–3×10−63\times10^{-6} from the exact operator. Accuracy degrades gracefully down to Q=16Q=16, but fails with too few modes (Q=8Q=8) or without the exact head (J=0J=0).

We have added these complexity analyses, implementation details, and empirical results in the revised manuscript.

References:

[B1] Gu, Albert, et al. “Efficiently modeling long sequences with structured state spaces.” ICLR 2022.

[B2] Gu, Albert, et al. “On the parameterization and initialization of diagonal state space models.” NeurIPS 2022.

AuthorsOpenReview ↗

Question 3 & 4: Extrapolation beyond the training horizon and Choice of the Caputo derivative

Can you provide results comparing SSM versions on RULER?....we should see improvements in tasks designed for long-context reasoning specifically.

There are multiple formulations of fractional derivatives....Why did you choose Caputo derivatives for fractional SSMs?

Question 3. Extrapolation beyond the training horizon

Response:

Thank you for sharing this perspective. For our models, however, RULER [C1] does not isolate it. RULER was designed for pretrained LLMs, and needle retrieval requires content-based (input-dependent) selection. LTI SSMs, whether integer or fractional, lack this mechanism (H3 [C2], Mamba [C3]), and a small model trained from scratch has no pretrained language ability. Its RULER score would therefore be limited by retrieval and scale rather than by the memory kernel, and could not reveal the effect of α\alpha.

We therefore test the hypothesis where the memory kernel is the only difference: ARFIMA length extrapolation (Sec. 4), expanded to 3 seeds, NN up to 256, and a larger training set. The two arms are identical except for the order (learned vs. fixed at 1). Both are trained at 1K only and evaluated with the same checkpoint up to 64×64\times. NMSE, range over N=16–256N=16–256:

Eval. length1K (train)2K8K64K
FracSSM2.7e-8 – 7.5e-81.9e-7 – 8.4e-71.1e-5 – 7.0e-50.8e-3 – 2.5e-3
α=1\alpha=13.9e-7 – 3.4e-60.9e-3 – 7.9e-31.4e-2 – 5.1e-24.6e-2 – 9.4e-2
ratio α=1\alpha=1 / FracSSM (per NN)5–89×1,500–10,500×410–1,440×36–61×

Table B. ARFIMA length extrapolation: NMSE ranges over N=16–256N=16–256 (trained at 1K).

  • Not a fitting failure. Both arms fit the training window; the α=1\alpha=1 models reach 3.9×10−73.9\times10^{-7}–3.4×10−63.4\times10^{-6} (1K column).
  • Just past the horizon. At 2×2\times, the α=1\alpha=1 error grows 400–20,000× relative to its own training-length error; the fractional error grows only 6–11×.
  • Far past the horizon. At 64×64\times the fractional error is 36–61× lower. Adding states does not close the gap for α=1\alpha=1.
  • Retained in streaming. The constant-memory SOE recurrence (Q=32Q=32, J=4J=4) reproduces these errors up to 64×64\times (Q2).

ARFIMA is a sequence-regression task rather than a reasoning benchmark. It does, however, isolate exactly the mechanism in the reviewer's hypothesis: whether a model uses information beyond the horizon it was trained on.

Question 4. Choice of the Caputo derivative

Response:

Thank you for your question. We chose Caputo for four reasons.

  1. State initialization. A Caputo system is initialized with ordinary state values, as shown by its integral formulation:

    \int_0^t (t-\tau)^{\alpha-1}f(x(\tau),u(\tau))\{d\tau}.$$ Likewise, its Laplace transform incorporates the initial state explicitly: $$\mathcal{L}\left({}^{C}D_t^{\alpha}x\right)(s) = s^{\alpha}X(s)-s^{\alpha-1}x_0.$$ Riemann–Liouville (RL) instead requires a fractional-integral initial datum rather than an ordinary state value. Caputo therefore preserves the usual interpretation of an SSM's initial latent state.
  2. Constants are equilibria. The Caputo derivative of a constant is zero; the RL derivative is ct−α/Γ(1−α)≠0ct^{-\alpha}/\Gamma(1-\alpha)\neq0. Caputo dynamics are therefore the standard relaxation/memory model (Mittag–Leffler responses), with the clean limit C!Dt1=d/dt{}^{C}\\!D_t^1=d/dt.

  3. No loss of generality. With zero prehistory, as in our model, Caputo and RL share the symbol sαs^\alpha and hence the same transfer function and kernel:

    =Ct^{\alpha-1}E_{\alpha,\alpha}(At^\alpha)B, \qquad t>0.$$
  4. Nonsingular-kernel variants such as Caputo–Fabrizio. Caputo–Fabrizio [C4] uses an exponential kernel, which would reintroduce the exponential memory that integer SSMs already have. The power-law Caputo kernel provides the algebraic decay we want, and it comes with mature numerics, including convolution quadrature and SOE compression.

References:

[C1] Hsieh, Cheng-Ping, et al. “RULER: What's the real context size of your long-context language models?” COLM 2024.

[C2] Fu, Daniel Y., et al. “Hungry Hungry Hippos: Towards language modeling with state space models.” ICLR 2023.

[C3] Gu, Albert, and Tri Dao. “Mamba: Linear-time sequence modeling with selective state spaces.” COLM 2024.

[C4] Caputo, Michele, and Mauro Fabrizio. “A new definition of fractional derivative without singular kernel.” Progress in Fractional Differentiation and Applications 2015.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal:

Dear PCs, SACs, ACs, and Reviewers,

As the author response window approaches its close, we sincerely thank the Area Chair and all reviewers for the time and effort invested in evaluating our submission. We have addressed the main concerns raised in the initial reviews through additional experiments and clarifications, which we have incorporated into the revised manuscript.


We summarize the main updates and clarifications from our rebuttal as follows:

  1. Matched controls and baselines. FracSSM outperformed same-implementation α=1\alpha=1 controls on LRA Image and Pathfinder (Rebuttal Table C). On LRA Text (Table 5), fixed fractional orders improved accuracy by 1.25–1.47 points over α=1\alpha=1 under the same training recipe. With matched preprocessing, FracSSM achieved 91.12% accuracy and LinOSS achieved 83.03%. Table 2 now includes additional published baselines.

  2. Length extrapolation. We repeated the ARFIMA study across three seeds and state sizes up to 256 (Rebuttal Table E). For N≥16N\geq16, integer-order models fit the training window, yet their absolute error at 64× the training length was 36–61 times that of FracSSM. Additional controls and an oracle sum-of-exponentials (SOE) analysis (App. B, Tables 7 and 8) support the interpretation that fractional order provides an inductive bias for extrapolation.

  3. Audio generation and language modeling. On SC09, FracSSM outperformed both published SaShiMi results on all four generation metrics under the official protocol (Table 4). The median learned order was 0.49. On WikiText-103, FracSSM achieved 1.1% lower word-level perplexity than S4 (30.73 vs. 31.07) with both models trained for 40 epochs at the 41.43M-parameter scale.

  4. Streaming accuracy and cost. Token-by-token SOE streaming closely matched parallel evaluation through 64× the training length (App. C.4, Table 11). We quantified how the number of SOE modes and retained recent states affects accuracy and memory. We also report the training overhead of learning the order (0.7–2.8%), streaming throughput and per-token latency, kernel efficiency, and the one-time SOE preparation cost (App. C.7).

  5. Motivation and scope. We described fractional memory in intuitive terms, used transfer functions to motivate the choice of derivative order, and explained why we use the Caputo derivative. We also clarified the relationship to prior work and outlined how the current linear time-invariant formulation could be extended to time-varying systems (Sec. 2).

Finally, we sincerely thank all the reviewers for the time and effort they devoted to reviewing our paper. Their thoughtful suggestions helped us improve the manuscript. We have incorporated the revisions and new results into the updated manuscript.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.