acceptodds
Under review as a conference paper at ICLR 2027

Riccati State Space Models: Non-iterative Parallelization for Nonlinear Sequence Modeling

Abstract

State space models (SSMs) achieve efficient sequence processing because their affine state updates are closed under composition and can therefore be evaluated with an associative parallel scan. Nonlinear recurrent models can provide richer, state-dependent dynamics, but generally lose this compositional structure: parallel evaluation then requires iterative methods that repeatedly linearize and scan the recurrence. We ask, what state-dependent nonlinear dynamics can be designed to remain exactly composable? We answer by introducing RiccatiSSM, a nonlinear SSM, in which each state dimension follows an input-conditioned Riccati differential equation. Its quadratic state dependence makes the local Jacobian explicitly state-dependent, while its exact per-step flow under piecewise-constant inputs is a Möbius transformation. Since Möbius maps are closed under composition and compose through matrix multiplication, the complete nonlinear state trajectory can be evaluated exactly with a single associative parallel scan, without iterative linearization. We further derive a constrained parameterization that ensures bounded, contractive dynamics, and avoids poles in the fractional-linear state update. Across long-sequence classification, regression, and forecasting tasks, RiccatiSSM achieves competitive predictive performance while reducing runtime by compared to the nonlinear LrcSSM under matched architectures. These results demonstrate that state-dependent nonlinear dynamics can retain exact composability and be evaluated efficiently within a single parallel scan.

Primary area:
learning on time series and dynamical systems
Keywords:
Riccati State Space Models, Nonlinear State Space Models, Parallel Scans, Möbius Transformations, Liquid Neural Networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hrjPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper shows that a two-dimensional parallel scan, using two-dimensional matrix multiplication as the associative operation, corresponds to a nonlinear (across time) update in one dimension. This enables nonlinear dynamics to be scan-represented. This stands in contrast with previous works, which use multiple scans to replace nonlinearity over time with nonlinearity over depth.

The full parameterization of the one-dimensional nonlinear dynamics can exhibit instability. To ensure contraction over time, To make sure these dynamics are stable, the authors develop a stable parameterization in terms of a "drive coefficient", "contraction margin", "curvature" and "speed factor".

Some experiments are shown on some relatively small datasets (sequence length < 20K, dimension < 100). In these, the new sequence model performs reasonably well.

Critical strengths
  • The main mathematical derivation of the Mobius update is clean, easy to understand, and could form the basis of useful new architectures. This is a principled application of ordinary differential equations to state-space modeling. The derivation of a stable parameterization (in the appendix) is similarly clean and principled.

  • The experiments involve a good number of competing methods.

Critical weaknesses
  • The paper presents and tests the RiccatiSSM essentially in isolation, rather than embedding it in a larger deep network. It is unclear whether RiccatiSSM would actually find any beneficial use in deep nets, as models like Mamba have found. One significant concern is: why not just keep the two-dimensional results from the two-dimensional scan?

  • Even in this isolated setup, on relatively small datasets, the experimental performance of the RiccatiSSM is not particularly impressive. RiccatiSSM is sometimes highlighted as one of best-performing method even when its mean accuracy is not close to the best accuracy. The justification for this is confidence interval overlap, but "getting credit" for the lower bound of a symmetric confidence interval is a bit dubious. There is actually not a single scenario when RiccatiSSM clearly outperforms its peers. Furthermore, these peers don't include Mamba 2, Mamba 3, or any DeltaNet variants.

I think this a nicely-written paper and would be open to changing my score if these issues are addressed.

Questions for authors
  • How does this model experimentally compare to an unconstrained two-dimensional parallel scan with a nonlinearity on top?

  • Can you provide a realistic, relevant-to-practice experimental setup where RiccatiSSM meaningfully outperforms reasonably strong baselines?

  • Does this idea generalize beyond a two-dimensional lift?

11 comments

We thank the Reviewer for the constructive feedback and suggestions. We agree that the empirical positioning of RiccatiSSM can be strengthened through comparisons with additional recent architectures, and we have conducted new experiments to address these concerns.

W1: Additional comparisons with DeltaNet, Mamba-2, and Mamba-3.

Following the Reviewer's suggestion, we evaluated these architectures on five time-series classification benchmarks. Given the limited rebuttal period, extensive hyperparameter optimization across all models was not feasible. We therefore used a consistent experimental setup with 6 layers and a state size of 64, and tuned the learning rate.

Time-series classification accuracy (% ↑)

DatasetDeltaNetMamba-2Mamba-3RiccatiSSM
Heartbeat71.9 ± 3.373.9 ± 3.772.9 ± 3.174.3 ± 4.0
Ethanol28.6 ± 5.327.8 ± 5.127.3 ± 3.834.0 ± 6.8
Motor54.7 ± 6.254.4 ± 5.0 *50.2 ± 2.9 *56.8 ± 3.8
SCP182.4 ± 2.981.9 ± 2.8 *83.5 ± 2.4 *84.8 ± 2.7
SCP254.7 ± 5.153.7 ± 4.2 *52.2 ± 4.6 *55.0 ± 5.4

Fixed setup: 6 layers, state size 64. Higher is better. Fields marked with * means that one sweep out of 3 of learning rate optimization is missing, and so their result might change a bit, but they give a good initial signal of their capabilities on these benchmarks. We will report the final results by the end of the rebuttal period.

Under this setup, RiccatiSSM achieves the highest mean accuracy on all five benchmarks. These results provide additional evidence that RiccatiSSM is competitive with recent sequence modeling architectures, including models originally developed for large-scale language modeling.

Forecasting performance: We additionally evaluated DeltaNet and Mamba-3 on the Weather forecasting benchmark, following the hyperparameter optimization protocol used in our existing experiments. While LinOSS remains the strongest model on this benchmark, these results further support RiccatiSSM's competitiveness against the DeltaNet and Mamba baselines.

Weather forecasting: 720 input steps, 720 prediction steps, lower is better.

ModelMAE (↓)
DeltaNet0.5788
Mamba-30.5702
RiccatiSSM (ours)0.5681

W1: Experimental scope and potential applications.

Our current evaluation focuses on time-series classification and forecasting, which are closely aligned with the continuous-time dynamical systems underlying RiccatiSSM. We are particularly interested in efficient sequence modeling.

We agree that evaluating RiccatiSSM within larger deep architectures, including language models, would be a valuable extension. Recent work on parallelizable nonlinear recurrent architectures, such as ParaRNN, has demonstrated that nonlinear recurrent models can also scale to billions of parameters. We therefore see large-scale evaluations as an interesting direction for future work, while keeping the present contribution focused on introducing and characterizing the RiccatiSSM recurrence.

W1, Q1: Why not just keep the two-dimensional results from the two-dimensional scan?

We thank the Reviewer for this interesting suggestion. In this ablation, we investigate whether the projective structure of RiccatiSSM provides an advantage over a generic nonlinear readout by comparing the original RiccatiSSM with a two-dimensional linear scan followed by a learned MLP.

Both models in this experiment use the same two-dimensional associative scan, with the only difference being the readout:

  • RiccatiSSM (parameter-free projective readout, original): x_t = p_t / q_t
  • LinearScanMLP readout (learned nonlinear readout): x_t = MLP([p_t, q_t])

We would like to point out that the projective division is central to the Riccati formulation. Although the lifted coordinates evolve linearly, their ratio follows a nonlinear fractional-linear recurrence. ptp_t and qtq_t are coupled through the same transition matrices, and their ratio is invariant to common rescaling. This projective structure is explicitly preserved by RiccatiSSM, whereas the learned MLP does not enforce it.

Using the PPG-DaLiA as we did for the other ablations, Test MSE (x10^-2, lower is better), reported as mean ± standard deviation over five seeds. We found that using a simple learned nonlinear readout would degrade the model's performance.

PPG-DaLiAMSE (↓)
Two-dim + nonlinear MLP readout7.36 ± 0.93
RiccatiSSM (ours)7.15 ± 1.01

We also evaluated both architectures on six time-series classification benchmarks, using the same fixed setup of 6 layers and a state size of 64, with learning rate optimization. Time-series classification accuracy (% ↑):

DatasetTwo-dim + nonlinear MLP readoutRiccatiSSM (ours)
Heartbeat75.7 ± 5.674.3 ± 4.0
Ethanol31.3 ± 2.934.0 ± 6.8
Motor57.1 ± 3.256.8 ± 3.8
SCP181.4 ± 2.284.8 ± 2.7
SCP254.3 ± 6.455.0 ± 5.4
Worms85.6 ± 5.486.1 ± 4.3
Avg.64.265.2

RiccatiSSM achieves a lower mean MSE on PPG-DaLiA and a higher average classification accuracy, outperforming the MLP readout on four of the six classification benchmarks. These results suggest that the projective readout provides a useful structural inductive bias compared with a freely learned nonlinear readout applied to the same lifted dynamics. Important for the RiccatiSSM's structure, this is achieved without introducing additional learned parameters in the readout, while preserving the exact nonlinear Riccati recurrence and its efficient parallel evaluation.

W2: RiccatiSSM is sometimes highlighted as one of best-performing method even when its mean accuracy is not close to the best accuracy.

We would also like to clarify that the original tables highlight the three best-performing models based on their mean accuracy or loss. We will make sure to include this explicitly in the manuscript for Table 3 and 4 too.

We acknowledge that RiccatiSSM does not consistently outperform all existing architectures. Given its close relationship to LrcSSM, we also do not necessarily expect large improvements in predictive performance across every benchmark. Our main contribution is the introduction of a new family of state-dependent nonlinear recurrences that admits exact, non-iterative parallelization. We believe the new experiments strengthen the empirical evidence for this approach and demonstrate that it can achieve competitive predictive performance alongside its computational advantages.

Q2: Can you provide a realistic, relevant-to-practice experimental setup where RiccatiSSM meaningfully outperforms reasonably strong baselines?

We thank the Reviewer for this question. We believe that efficient time-series modeling, particularly for resource-constrained applications, is a promising practical setting for RiccatiSSM.

Our experiments include real-world physiological signal classification tasks, such as Heartbeat and MotorImagery. In the additional comparisons with DeltaNet, Mamba-2, and Mamba-3, RiccatiSSM achieves the highest mean accuracy across all five evaluated classification benchmarks under the same architectural setup. These results demonstrate that RiccatiSSM remains competitive with strong recent sequence models.

Furthermore, RiccatiSSM achieves lower computational times than LrcSSM under our shared JAX implementation, while retaining comparable predictive performance. We believe this combination of competitive accuracy and efficient parallel evaluation makes RiccatiSSM a promising candidate for time-series applications.

We also made some additional expressivity evaluations to further investigate the expressivity of RiccatiSSM: we evaluated it on the A5 state-tracking task introduced by Merrill et al. (2024), using the original codebase and an identical experimental setup.

We find that complex-valued RiccatiSSM solves the task with a single layer for sequence lengths up to 11, requiring only two layers for lengths 12–20. At sequence length 10, this matches the single-layer capability of fully connected RNNs, while Mamba, S4, and Transformers require 2, 3, and 4 layers, respectively. Interestingly, neither real-valued nor complex-valued LrcSSM could solve the task with a single layer at this sequence length. Furthermore, a real-valued RiccatiSSM with block-diagonal state interactions can solve these tasks (3-20 lengths) with a single layer.

A5 state-tracking: Minimum number of layers required at sequence length 10:

ModelMinimum number of layers
Standard dense1
RiccatiSSM (complex)1
RiccatiSSM (real, block-diagonal)1
LrcSSM (real)>1*
LrcSSM (complex)>1*
Mamba2
S43
Transformer4

For LrcSSM, the single-layer configuration did not solve the task, the minimum successful depth has not been determined yet. Standard RNN, Mamba, S4, and Transformer results are from [1].

As discussed by [1], state-tracking tasks expose limitations of linear time-invariant (LTI) models, such as S4 and LinOSS, whose input-independent transitions restrict their ability to represent the required state updates. This is particularly interesting given the strong performance of LinOSS on other benchmarks. Our results suggest that RiccatiSSM can represent challenging state-tracking dynamics that are difficult for these LTI architectures, while retaining exact parallelizability.

Ref: [1] William Merrill, Jackson Petty, Ashish Sabharwal. The illusion of state in state-space models. ICML 2024.

Q3: Does this idea generalize beyond a two-dimensional lift?

We thank the Reviewer for this interesting question. We would like to distinguish two forms of generalization: extending RiccatiSSM to higher-dimensional states, which we investigate experimentally, and extending the underlying mathematical construction to higher-dimensional lifts.

Higher-dimensional states: Our A5 experiments reveal an interesting connection to the underlying group structure. Real scalar Möbius transformations cannot faithfully represent the non-abelian simple group A5A_5, whereas complex Möbius transformations contain A5A_5 as the icosahedral group. Consistent with this distinction, scalar real-valued RiccatiSSM does not solve the task with one layer, while its complex-valued extension does. We additionally find that a real-valued extension with block-diagonal state interactions solves A5A_5 with a single layer. These results demonstrate that extending the state representation can substantially improve expressivity.

Higher-dimensional lifts: The two-dimensional lift is not an inherent mathematical limitation. Matrix Riccati equations of the form X˙=A+BX+XC+XDX\dot X=A+BX+XC+XDX admit higher-dimensional block-linear representations, with X=VU−1X=VU^{-1}. Since composition remains associative matrix multiplication, these systems can also be evaluated through parallel scans.

However, higher-dimensional lifts bring additional computational and numerical considerations. Dense matrix composition has cubic cost in the lifted dimension, and avoiding singularities requires maintaining the invertibility of UU. We therefore view higher-dimensional RiccatiSSMs as a promising direction for future work, while the present paper focuses on the computationally efficient scalar construction.

Reviewer_rbBpOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces RiccatiSSM, a recurrent layer capable of tracking nonlinear hidden state dynamics while still being computable in parallel via a single parallel scan. The method relies on the properties of Riccati ODEs: these are quadratic differential equations which admit a linear-system representation via a projective lift. In essence, the solution to a Riccati ODE (in the form x’ = a+bx+cx^2) can be represented as x(t)=p(t)/q(t), where p and q evolve according to a coupled, 2-dimensional, linear ODE in the form [q,p]’ = A(t) [q,p], where A depends on a,b,c. Solving the quadratic system in x is then equivalent to solving the linear system in [q,p], but since the latter is linear, its solution can efficiently be computed via parallel scan, like for any modern SSM. The authors provide a parameterisation for the system which guarantees stability in the hidden state evolution, and proceed to test RiccatiSSM on a number of time-series analysis tasks. Results show accuracy comparable to a nonlinear baseline (LrcSSM), with faster runtime, although often fall short of other linear baselines (LinOSS).

Critical strengths
  • Paper is clearly presented, and motivated reasonably well
  • Mathematical derivations are on point, and the idea to leverage Riccati to allow fast training and analyze stability is quite clever
Critical weaknesses
  • The framing of the paper is misleading: I’d argue that the proposed recurrence has very little to do with actual sequence-wise application of nonlinearities
  • Empirical results aren’t particularly convincing about the overall validity of the method
  • Runtime experiments must be improved upon
Questions for authors

-W1: nonlinearity?: Consider the recurrence xt=f−1(Atf(xt−1)+bt)x_{t} = f^{-1}( A_t f(x_{t-1}) + b_t ), for any given nonlinear, invertible function ff. Is this a nonlinear recurrence? Naively, yes: ff is nonlinear, so xtx_{t} is overall a nonlinear function of xt−1x_{t-1}. A closer look at it would however reveal a much less general structure: what we actually have is a nonlinearity applied independently to each token in the sequence, yt=f(xt)y_t = f(x_t), followed by a linear recurrence sequence-wise yt=Atyt−1+bty_t = A_t y_{t-1} + b_t, and another nonlinearity applied independently to each token in the sequence. Your RiccatiSSM is a specific instantiation of this trick: ff represents the specific projective lift characterising the Riccati / Moebius correspondence. Rather than a bona-fide nonlinear recurrence, what you’re really selling with this method is a specific input-projection + read-out operation, plus a specific parameterisation of state-transition matrices in a linear SSM with 2x2 block-diagonal gates, so that the overall procedure can be interpreted as if you were evolving a quadratic RNN. I’m not saying the model itself is uninteresting (I genuinely consider the whole derivation rather insightful), but any claim regarding actual nonlinearities being applied sequence-wise is, simply put, misleading. As such, much of the paper narrative should be adjusted to better reflect this.

-W2: empirical results: In the majority of reported results, it looks like a vanilla linear SSM with 2x2 block-diagonal gates (LinOSS) actually performs better than RiccatiSSM. This begs the question of what is the extra parameterisation introduced by RiccatiSSM actually bringing to the table: do we really need the lifting projection and the ratio read-out, if the simple 2x2 block-diagonal recurrence suffices? Building up on this, you do focus on LrcSSM as the main competitor of your method, but is the comparison really fair? While LrcSSM is a parallel-trainable nonlinear cell, it also prescribes independent evolution for its hidden state components (the Jacobian is diagnal); have you tried comparing against parallel-trainable nonlinear RNNs who do allow for some actual mixing, such as ParaLSTM [https://arxiv.org/abs/2510.21450] ? This would be useful to better isolate the source of the advantage of RiccatiSSM wrt LrcSSM: is it due to its “better” nonlinearity (quadratic vs GRU-like), or just because of the richer mixing? Finally, while I appreciate the ablations in Tab5, I feel like the most relevant one is missing: what happens if you only freeze γ\gamma to 0? After all, that is the whole source of nonlinearity (see (8)), so measuring a performance difference when removing it would help justifying the whole approach.

-W3: runtime measurements: One of the main justifications supporting RiccatiSSM is given by the fact that not having to rely on Newton iterations to solve for nonlinearities effectively reduces runtime. The main proof for this is provided in Fig2 (and Tab8, although architectural differences confound measurements there), but not many details are provided regarding the setup used for the experiments. In particular, what implementation are you using for LrcSSM’s Newton iterations, and for RiccatiSSM’s associative scan? Optimised kernels exist for both, and measuring runtime under them would provide a much clearer comparison.

Minor:

  • Closure under composition: in multiple places (L14, L39, L48, L58, L94, L132, …) you repeat how, to enable parallelisability, you must ensure that the function defining the recurrence is closed under composition. This is not true: the precondition for applying associative scan (and achieving parallelisability) is associativity: closure is just a useful extra. The classical example is DeltaNet: composition of Householder reflections is not a Householder reflection in general, so the family isn’t closed under composition, and yet parallelisation is still achievable, although more annoying to implement (we need to use WY transforms and whatnot).
7 comments

We thank the Reviewer for their thorough and constructive feedback.

W1: nonlinearity?

We thank the Reviewer for this thoughtful comment. We will make the distinction between nonlinear dynamics in the original state coordinates and linear dynamics in the lifted coordinates clearer in the paper.

Our claim is that the recurrence is genuinely nonlinear with respect to the state xtx_t. In the original state coordinates, each step takes the form xt=atxt−1+btctxt−1+dtx_t = \frac{a_t x_{t-1}+b_t}{c_t x_{t-1}+d_t}, which is nonlinear in the state for ct≠0c_t \neq 0, and is applied at every one of the TT steps. In particular, the resulting state transition has a state-dependent Jacobian, allowing the local contraction rates and effective timescales to vary with the evolving state.

We agree, however, that the projective lift reveals a more specific structure: the nonlinear Riccati recurrence can be represented exactly through linear matrix evolution in a higher-dimensional space, followed by a nonlinear fractional readout. We do not claim that this construction provides general nonlinear feedback into the transition matrices. Our key contribution is identifying a useful family of state-dependent nonlinear recurrences that admits an exact, fixed-dimensional linear representation and, consequently, non-iterative parallel evaluation.

We will adjust the paper's narrative to avoid giving the impression that RiccatiSSM provides a general mechanism for nonlinear sequence-wise state evolution. We will also refine Table 1 and its discussion to better reflect this distinction, emphasizing that RiccatiSSM captures a specific family of nonlinear state transitions that admits this exact linear representation in projective coordinates. We believe this distinction is an important insight, as we aim to explore the border of nonlinearity: by exploiting the mathematical structure of Riccati dynamics, we have state-dependent nonlinear evolution in the original coordinates while getting the computational benefits of linear compositionality in the lifted space.

W2: empirical results

Thank you for these valuable suggestions. We agree that further comparisons and ablations are useful to better isolate the contribution of the RiccatiSSM.

Comparison with LinOSS: We would like to clarify that LinOSS indeed admits a 2×22\times2 block-diagonal representation when its second-order dynamics are expressed as a first-order system. However, these blocks arise from the discretization of linear oscillator dynamics and are not input-conditioned transition gates. In contrast, RiccatiSSM constructs input-conditioned 2×22\times2 transition matrices whose projective action induces nonlinear, state-dependent dynamics in the original coordinates. We acknowledge that LinOSS achieves better predictive performance on several benchmarks. Our main intention in this paper was to put the design of RiccatiSSM onto the map of the sequence modeling paradigm by showing that a specific family of nonlinear, state-dependent dynamics can retain exact compositionality and be evaluated using a single parallel scan.

We would also like to highlight that we find the competitive performance of these two models particularly interesting, given their different origins and underlying computational mechanisms. LinOSS is rooted in second-order linear oscillator dynamics, whereas RiccatiSSM builds on nonlinear Riccati dynamics and their projective representation. Despite these differences, both achieve competitive results across several benchmarks. In our view, this raises an interesting direction for future work, which would be exploring hybrid model families that combine the complementary properties of these two approaches.

W2 (cont.): Comparison with parallel-trainable nonlinear RNNs

We thank the Reviewer for highlighting the important work of ParaRNN [2], which targets general nonlinear models, in particular ParaLSTM and ParaGRU. Our focus was different, as we aimed to identify the boundary of state-dependent nonlinear recurrences that can be evaluated using a non-iterative parallel scan.

Following the Reviewer's suggestion, we also provide results for the ParaGRU and ParaLSTM variants. The results show that RiccatiSSM is competitive with these architectures, achieving a large accuracy difference on Motor, SCP1 and SCP2 compared to ParaGRU and tying for the best performance on Worms.

DatasetParaGRUParaLSTM*RiccatiSSM
Heartbeat75.7 ± 4.775.0 ± 3.574.3 ± 4.0
Ethanol34.5 ± 1.932.6 ± 5.634.0 ± 6.8
Motor49.6 ± 7.354.3 ± 3.356.8 ± 3.8
SCP180.2 ± 4.278.8 ± 3.184.8 ± 2.7
SCP252.5 ± 3.351.1 ± 9.555.0 ± 5.4
Worms86.1 ± 6.382.2 ± 4.586.1 ± 4.3
Avg.63.162.365.2

We note that the ParaGRU results are adapted from the GruSSM experiments of [1, Table 2.], as the two formulations are equivalent under a single-head input projection (see [1] Section D.1 and [2] Section 3). *The reported results are from the LstmSSM [1, Table 2.] implementation, which is not fully identical to ParaLSTM, as it does not use merged input and output gates.

We would also like to point out that the Weather forecasting experiments (Table 4) already include comparisons with established LSTM-based architectures. Specifically, LSTMa and LSTnet achieve MAE values of 1.109 and 0.757, respectively, compared with 0.5681 for RiccatiSSM. While these are not the exact parallel-trainable variants, their underlying models are LSTM-based, and so they provide additional performance comparison of RiccatiSSM against these conventional nonlinear RNN models.

Additionally, we extended Table 5 with ParaRNN models evaluated on PPG-DaLiA. Test MSE (×10−2, lower is better), reported as mean ± standard deviation over five seeds:

ModelMSE (↓)
ParaGRU7.39 ± 0.06
ParaLSTM7.37 ± 0.035
RiccatiSSM (ours)7.15 ± 1.01

We also note that the ParaRNN framework imposes structured recurrent connections to enable efficient parallelization, which are diagonal Jacobians for ParaGRU and block-diagonal Jacobians for ParaLSTM. Thus, these architectures also make specific structural assumptions about recurrent state interactions. We believe the existing comparisons provide a useful reference for positioning RiccatiSSM within this set of parallel-trainable nonlinear recurrent architectures.

W2: Ablation of the quadratic term

We additionally evaluated the effect of the quadratic contribution by setting its coefficient γ\gamma to zero, while keeping all the other terms input-dependent. We found that this results in a drop in MSE. We performed an additional experiment to analyze whether further increasing the curvature would result in better performance. Interestingly, we found that increasing the coefficient bounds to γscale=5\gamma_{\mathrm{scale}}=5 further reduces the mean. These results suggest that the quadratic contribution can improve predictive performance.

Extension of Table 5, ablations on PPG-DaLiA. Test MSE (×10−2, lower is better), reported as mean ± standard deviation over five seeds:

γscale\gamma_{\mathrm{scale}}MSE (↓)
07.24 ± 0.50
1 (default)7.15 ± 1.01
56.82 ± 0.42

We selected γscale=1\gamma_{\mathrm{scale}}=1 as the default based on preliminary experiments across different settings, which indicated that larger values such as γscale=5\gamma_{\mathrm{scale}}=5 could lead to overly strong nonlinear effects in some cases. We therefore opted for a more conservative setting that still provides a meaningful nonlinear contribution while avoiding excessively strong nonlinearities. Although γscale=5\gamma_{\mathrm{scale}}=5 achieves a lower mean MSE on PPG-DaLiA.

We aim to incorporate these additional comparisons in the revised paper.

W3: runtime measurements:

We thank the Reviewer for pointing this out. For the runtime comparisons, we used the publicly available LrcSSM GitHub implementation in JAX [1] as our starting point and modified only the recurrent cell to implement RiccatiSSM. Both models were evaluated in the same experimental setup, using 6 layers, a state size of 64, and a batch size of 4. This ensures that the runtime differences primarily reflect the recurrent computations, although the two cells have different parameter counts. For example, on Worms, RiccatiSSM has 159,623 parameters, compared with 112,391 for LrcSSM. Despite having more parameters, RiccatiSSM achieves lower computational times.

We acknowledge that neither implementation uses specialized, optimized kernels. In particular, our comparison does not rely on the CUDA kernels provided by the ParaRNN framework, which is implemented in PyTorch. We chose to retain the JAX implementations to ensure consistency with the experiments also developed in JAX [1,3,4] and to compare the two recurrent cells within the same codebase.

As shown in Figure 2, LrcSSM typically requires approximately 3 Newton iterations to converge, consistent with the findings reported in the LrcSSM paper [1] and comparable to the iteration counts reported for other recurrent models in the ParaRNN paper [2]. RiccatiSSM, in contrast, from the algorithmic point of view, requires a single associative scan without iterative convergence.

We agree that optimized CUDA implementations could change the absolute runtimes and potentially the relative speedups. We will clarify the implementation details and limitations of our runtime comparison in the revised manuscript. Nevertheless, the results demonstrate the computational advantage of avoiding iterative parallelization under a consistent JAX implementation.

Following the Reviewer's suggested direction, we will add the development of specialized GPU kernels as an important direction for future work, moving to PyTorch and leveraging the ParaRNN framework with specialized CUDA kernels and the existing ParaGRU and ParaLSTM for comparison, and potentially extending it to LrcSSM too, to further evaluate the computational efficiency and scalability of RiccatiSSM on larger-scale language modeling, which is beyond the scope of our paper.

[1] Monika Farsang, Radu Grosu. Parallelization of Non-linear State-Space Models: Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modeling. NeurIPS 2025.

[2] Federico Danieli, Pau Rodriguez, Miguel Sarabia, Xavier Suau, Luca Zappella. ParaRNN: Unlocking parallel training of nonlinear rnns for large language models. ICLR 2026.

[3] T. Konstantin Rusch, Daniela Rus. Oscillatory State-Space Models. ICLR 2025.

[4] Philipp Nazari, Francesco Maria Ruscio, Benedict Armstrong, and T. Konstantin Rusch. Discretax: A lightweight collection of state space models in JAX. 2025.

Closure under composition

We thank the Reviewer for this clarification. We agree that closure within the original transformation family is not necessary for parallelization, as illustrated by DeltaNet. However, associativity alone does not guarantee computationally efficient parallelization. The composed transformations must also admit a compact representation with an efficiently computable composition operator. For instance, general nonlinear RNNs have associative function composition, but the complexity of representing composed functions can grow with sequence length. The problem is that composing nonlinear functions generally does not produce a fixed-size representation of the same form. The nested expressions become increasingly complex, and evaluating or storing the composed transformations eliminates the computational benefit of parallelization.

For RiccatiSSM, Möbius transformations are closed under composition and admit an exact, fixed-dimensional matrix representation with constant-cost composition per channel. This enables an efficient associative scan without iterative approximation. We will revise the wording to clarify that closure within the original transformation family is a sufficient (and not a necessary condition) structural property for efficient parallelization.

We thank the Reviewer again for their thorough, constructive and insightful feedback. We believe that the additional comparisons with parallelizable nonlinear recurrent architectures, together with the clarifications of RiccatiSSM's distinct state-dependent dynamics and computational advantages, strengthen the empirical and theoretical positioning of our contribution.

Reviewer_f5C6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RiccatiSSM, a nonlinear sequence model evaluated with one parallel scan per layer. Riccati dynamics give exact Möbius updates under piecewise constant inputs, with parameter constraints ensuring bounded and contractive states. Experiments on classification, regression and forecasting report competitive prediction results and 22 to 33% lower runtime than LrcSSM under matched architectures.

Critical strengths

The paper gives a clear construction for evaluating nonlinear state dynamics without iterative solves. Sections 3.2 to 3.4 and Appendix A derive the exact flow and establish boundedness and contraction under explicit constraints.

The experiments include five seeds on six classification datasets and tests on regression and forecasting. Figure 2 compares matched architectures, supporting the reported runtime advantage over iterative LrcSSM.

Critical weaknesses

The recurrent computation has the same complexity as diagonal linear SSMs, but the predictive benefit of the nonlinear state dynamics remains unclear. A matched gamma = 0 comparison of accuracy and runtime would clarify when this design is useful.

Questions for authors

Do you have existing accuracy and runtime results for gamma = 0 under the same surrounding architecture and training setup, with the remaining coefficients still input dependent?

Which benchmark produced Figure 2, and did its timing exclude compilation and wait for GPU completion? Did it include the backward pass?

Was RiccatiSSMRev used for any reported result? Its custom backward disagrees with automatic differentiation and finite differences in a small nonlinear example; please clarify whether this path was used and whether its gradients were checked.

3 comments

We thank the Reviewer for their thorough and constructive feedback.

Do you have existing accuracy and runtime results for gamma = 0 under the same surrounding architecture and training setup, with the remaining coefficients still input dependent?

We thank the Reviewer for this suggestion. We additionally evaluated the contribution of the nonlinear quadratic term by setting its coefficient γ\gamma to zero, while keeping all other terms input-dependent. We also investigated whether increasing the magnitude of the quadratic contribution could improve predictive performance; for this, we vary the γscale\gamma_{\mathrm{scale}} parameter in our code.

Extension of Table 5: PPG-DaLiA ablation. Test MSE (x10^-2, lower is better), reported as mean ± standard deviation over five seeds.

γscale\gamma_{\mathrm{scale}}MSE (↓)
07.24 ± 0.50
1 (default)7.15 ± 1.01
56.82 ± 0.42

We observe that removing the quadratic contribution slightly increases the mean MSE, while increasing γscale\gamma_{\mathrm{scale}} value to 5 further reduces it. These results suggest that the nonlinear quadratic contribution can improve predictive performance. See also our reply to Reviewer hrjP, where we also compare with the 2-dimensional scan with nonlinearity (MLP) on top, instead of the fractional readout.

We selected γscale=1\gamma_{\mathrm{scale}}=1 as the default based on preliminary experiments across different settings, where larger values sometimes produced excessively strong nonlinear effects. We therefore chose a more conservative setting (γscale\gamma_{\mathrm{scale}}=1) that keeps a meaningful nonlinear contribution.

Runtime comparison: we additionally measured the runtime of the different γscale\gamma_{\mathrm{scale}} configurations and found that all achieve approximately 0.041 per step, with a batch size of 4, sequence length of 49920, on an NVIDIA A100 GPU, in all setups. This is expected, as setting γscale=0\gamma_{\mathrm{scale}}=0 removes the quadratic contribution but has the same computational structure: both configurations use 2×22\times2 transition matrices and the same associative scan. Thus, the nonlinear quadratic term can improve predictive performance without introducing additional runtime overhead within the RiccatiSSM implementation.

We acknowledge that a separately optimized diagonal linear SSM could be more computationally efficient, but would be a different implementation from this presented controlled ablation.

Which benchmark produced Figure 2, and did its timing exclude compilation and wait for GPU completion? Did it include the backward pass?

We thank the Reviewer for requesting these additional details. Figure 2 reports inference runtime measurements on three benchmarks with different sequence lengths: Heartbeat (405), MotorImagery (3,000), and EigenWorms (17,984). We selected these datasets to illustrate how the runtime difference between LrcSSM and RiccatiSSM changes with sequence length, with larger differences observed for longer sequences.

All experiments were performed on an NVIDIA A100 GPU. To ensure a consistent comparison, we used the original LrcSSM implementation and modified only the recurrent cell for RiccatiSSM, keeping the remaining architecture unchanged. Both models use 6 layers and a state size of 64. This results in the following number of trainable parameters:

DatasetLrcSSM parametersRiccatiSSM parameters
Heartbeat115,716162,948
MotorImagery115,844163,076
EigenWorms112,391159,623

Interestingly, RiccatiSSM has approximately 41% more parameters than LrcSSM, yet achieves lower inference runtimes. The main computational difference arises from LrcSSM's iterative parallelization, which typically requires approximately 3 Newton iterations, whereas RiccatiSSM uses a single associative scan.

Runtime measurements were collected after 500 warm-up steps to exclude JAX compilation and initialization overhead. Figure 2 reports runtimes measured during inference, which exclude the backward pass, while Table 8 reports full training runtimes, including forward and backward computations. We will clarify these experimental details in the revised paper.

Was RiccatiSSMRev used for any reported result? Its custom backward disagrees with automatic differentiation and finite differences in a small nonlinear example; please clarify whether this path was used and whether its gradients were checked.

We thank the Reviewer for carefully examining the provided implementation and identifying this. We included this piece of code unintentionally in our submission. We would like to clarify that RiccatiSSMRev was not used for any of the experiments reported in the current state of the paper. All results were obtained using the standard RiccatiSSM implementation with JAX automatic differentiation. In the paper, we focused on inference times (forward pass), as those include the iterative parallelization route, meanwhile the default RiccatiSSM's and the LrcSSM's backward passes had similar runtimes.

RiccatiSSMRev is an experimental implementation that we developed to investigate whether a custom backward pass could further improve computational efficiency during the training phase. Following the Reviewer's observation, we took the time to identify an incorrect operand ordering in the reverse associative scan (_scan_readout_bwd). Specifically, the composition formula al * br + bl should have been ar * bl + br, accounting for the ordering of the accumulated suffix and the current element.

After correcting this issue, we verified that the custom backward matches the JAX automatic differentiation to numerical precision (maximum relative difference of approximately 10−710^{-7} (consistent with float32 rounding).

We thank the Reviewer for helping us identifying this. We plan to further validate and benchmark the custom backward implementation before including it as a supported alternative (and rerun all experiments with it, as slight divergence over time might happen). Once we validate all the results, this implementation will be able to speed up the backward pass too. On the EigenWorms benchmark, for example, by around 30%: 0.041s with Riccati(Rev), compared to 0.058s with LrcSSM.

We thank the Reviewer again for their positive and thorough feedback and would welcome any additional questions or comments regarding our responses.

Comments on the paper

1 comment

We thank the Reviewers for their time, thorough feedback and engagement throughout the rebuttal period. Our submission, under the primary area Learning on Time Series and Dynamical Systems introduces RiccatiSSM, a new family of state-dependent nonlinear recurrent models whose dynamics admit exact, non-iterative parallelization through a single associative scan. Our primary goal is to establish this mathematical and architectural formulation and evaluate it on established sequence modeling benchmarks.

We would like to highlight that all three Reviewers rated the paper as "Clear and easy to follow" and indicated that they "learned something interesting or useful". Reviewer hrjP reported no substantive soundness concerns and described the mathematical derivation as “clean” and "principled", noting that it could form the basis of useful new architectures. Reviewer f5C6 highlighted the "clear construction for evaluating nonlinear state dynamics without iterative solves". Reviewer rbBp described the Riccati formulation as "quite clever" and the derivations as "on point". During the rebuttal, Reviewers also acknowledged the appropriateness of our parameter-count comparisons, the motivation for nonlinear dynamics, and the relevance of the additional layer-count experiments. We appreciate these positive assessments alongside the constructive criticisms.

During the rebuttal, we substantially extended the experimental evaluation with comparisons against DeltaNet, Mamba-2, a Mamba-3 variant, and parallelizable nonlinear RNNs; ablations of the quadratic contribution and projective readout; additional A5 state-tracking experiments demonstrating expressivity; and detailed clarifications of runtime, implementation and higher-dimensional extensions. While larger hybrid language-model experiments and hardware-optimized evaluations remain valuable future directions, we believe the additional evidence and discussions have substantially strengthened the submission and clarified its contributions. We hope the Area Chair will consider both the original contribution and the extensive additional work in response to the Reviewers' feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.