- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper shows that a two-dimensional parallel scan, using two-dimensional matrix multiplication as the associative operation, corresponds to a nonlinear (across time) update in one dimension. This enables nonlinear dynamics to be scan-represented. This stands in contrast with previous works, which use multiple scans to replace nonlinearity over time with nonlinearity over depth.
The full parameterization of the one-dimensional nonlinear dynamics can exhibit instability. To ensure contraction over time, To make sure these dynamics are stable, the authors develop a stable parameterization in terms of a "drive coefficient", "contraction margin", "curvature" and "speed factor".
Some experiments are shown on some relatively small datasets (sequence length < 20K, dimension < 100). In these, the new sequence model performs reasonably well.
Critical strengths
-
The main mathematical derivation of the Mobius update is clean, easy to understand, and could form the basis of useful new architectures. This is a principled application of ordinary differential equations to state-space modeling. The derivation of a stable parameterization (in the appendix) is similarly clean and principled.
-
The experiments involve a good number of competing methods.
Critical weaknesses
-
The paper presents and tests the RiccatiSSM essentially in isolation, rather than embedding it in a larger deep network. It is unclear whether RiccatiSSM would actually find any beneficial use in deep nets, as models like Mamba have found. One significant concern is: why not just keep the two-dimensional results from the two-dimensional scan?
-
Even in this isolated setup, on relatively small datasets, the experimental performance of the RiccatiSSM is not particularly impressive. RiccatiSSM is sometimes highlighted as one of best-performing method even when its mean accuracy is not close to the best accuracy. The justification for this is confidence interval overlap, but "getting credit" for the lower bound of a symmetric confidence interval is a bit dubious. There is actually not a single scenario when RiccatiSSM clearly outperforms its peers. Furthermore, these peers don't include Mamba 2, Mamba 3, or any DeltaNet variants.
I think this a nicely-written paper and would be open to changing my score if these issues are addressed.
Questions for authors
-
How does this model experimentally compare to an unconstrained two-dimensional parallel scan with a nonlinearity on top?
-
Can you provide a realistic, relevant-to-practice experimental setup where RiccatiSSM meaningfully outperforms reasonably strong baselines?
-
Does this idea generalize beyond a two-dimensional lift?
11 comments
W1, Q1: Why not just keep the two-dimensional results from the two-dimensional scan?
We thank the Reviewer for this interesting suggestion. In this ablation, we investigate whether the projective structure of RiccatiSSM provides an advantage over a generic nonlinear readout by comparing the original RiccatiSSM with a two-dimensional linear scan followed by a learned MLP.
Both models in this experiment use the same two-dimensional associative scan, with the only difference being the readout:
- RiccatiSSM (parameter-free projective readout, original): x_t = p_t / q_t
- LinearScanMLP readout (learned nonlinear readout): x_t = MLP([p_t, q_t])
We would like to point out that the projective division is central to the Riccati formulation. Although the lifted coordinates evolve linearly, their ratio follows a nonlinear fractional-linear recurrence. and are coupled through the same transition matrices, and their ratio is invariant to common rescaling. This projective structure is explicitly preserved by RiccatiSSM, whereas the learned MLP does not enforce it.
Using the PPG-DaLiA as we did for the other ablations, Test MSE (x10^-2, lower is better), reported as mean ± standard deviation over five seeds. We found that using a simple learned nonlinear readout would degrade the model's performance.
| PPG-DaLiA | MSE (↓) |
|---|---|
| Two-dim + nonlinear MLP readout | 7.36 ± 0.93 |
| RiccatiSSM (ours) | 7.15 ± 1.01 |
We also evaluated both architectures on six time-series classification benchmarks, using the same fixed setup of 6 layers and a state size of 64, with learning rate optimization. Time-series classification accuracy (% ↑):
| Dataset | Two-dim + nonlinear MLP readout | RiccatiSSM (ours) |
|---|---|---|
| Heartbeat | 75.7 ± 5.6 | 74.3 ± 4.0 |
| Ethanol | 31.3 ± 2.9 | 34.0 ± 6.8 |
| Motor | 57.1 ± 3.2 | 56.8 ± 3.8 |
| SCP1 | 81.4 ± 2.2 | 84.8 ± 2.7 |
| SCP2 | 54.3 ± 6.4 | 55.0 ± 5.4 |
| Worms | 85.6 ± 5.4 | 86.1 ± 4.3 |
| Avg. | 64.2 | 65.2 |
RiccatiSSM achieves a lower mean MSE on PPG-DaLiA and a higher average classification accuracy, outperforming the MLP readout on four of the six classification benchmarks. These results suggest that the projective readout provides a useful structural inductive bias compared with a freely learned nonlinear readout applied to the same lifted dynamics. Important for the RiccatiSSM's structure, this is achieved without introducing additional learned parameters in the readout, while preserving the exact nonlinear Riccati recurrence and its efficient parallel evaluation.
W2: RiccatiSSM is sometimes highlighted as one of best-performing method even when its mean accuracy is not close to the best accuracy.
We would also like to clarify that the original tables highlight the three best-performing models based on their mean accuracy or loss. We will make sure to include this explicitly in the manuscript for Table 3 and 4 too.
We acknowledge that RiccatiSSM does not consistently outperform all existing architectures. Given its close relationship to LrcSSM, we also do not necessarily expect large improvements in predictive performance across every benchmark. Our main contribution is the introduction of a new family of state-dependent nonlinear recurrences that admits exact, non-iterative parallelization. We believe the new experiments strengthen the empirical evidence for this approach and demonstrate that it can achieve competitive predictive performance alongside its computational advantages.
Q2: Can you provide a realistic, relevant-to-practice experimental setup where RiccatiSSM meaningfully outperforms reasonably strong baselines?
We thank the Reviewer for this question. We believe that efficient time-series modeling, particularly for resource-constrained applications, is a promising practical setting for RiccatiSSM.
Our experiments include real-world physiological signal classification tasks, such as Heartbeat and MotorImagery. In the additional comparisons with DeltaNet, Mamba-2, and Mamba-3, RiccatiSSM achieves the highest mean accuracy across all five evaluated classification benchmarks under the same architectural setup. These results demonstrate that RiccatiSSM remains competitive with strong recent sequence models.
Furthermore, RiccatiSSM achieves lower computational times than LrcSSM under our shared JAX implementation, while retaining comparable predictive performance. We believe this combination of competitive accuracy and efficient parallel evaluation makes RiccatiSSM a promising candidate for time-series applications.
We also made some additional expressivity evaluations to further investigate the expressivity of RiccatiSSM: we evaluated it on the A5 state-tracking task introduced by Merrill et al. (2024), using the original codebase and an identical experimental setup.
We find that complex-valued RiccatiSSM solves the task with a single layer for sequence lengths up to 11, requiring only two layers for lengths 12–20. At sequence length 10, this matches the single-layer capability of fully connected RNNs, while Mamba, S4, and Transformers require 2, 3, and 4 layers, respectively. Interestingly, neither real-valued nor complex-valued LrcSSM could solve the task with a single layer at this sequence length. Furthermore, a real-valued RiccatiSSM with block-diagonal state interactions can solve these tasks (3-20 lengths) with a single layer.
A5 state-tracking: Minimum number of layers required at sequence length 10:
| Model | Minimum number of layers |
|---|---|
| Standard dense | 1 |
| RiccatiSSM (complex) | 1 |
| RiccatiSSM (real, block-diagonal) | 1 |
| LrcSSM (real) | >1* |
| LrcSSM (complex) | >1* |
| Mamba | 2 |
| S4 | 3 |
| Transformer | 4 |
For LrcSSM, the single-layer configuration did not solve the task, the minimum successful depth has not been determined yet. Standard RNN, Mamba, S4, and Transformer results are from [1].
As discussed by [1], state-tracking tasks expose limitations of linear time-invariant (LTI) models, such as S4 and LinOSS, whose input-independent transitions restrict their ability to represent the required state updates. This is particularly interesting given the strong performance of LinOSS on other benchmarks. Our results suggest that RiccatiSSM can represent challenging state-tracking dynamics that are difficult for these LTI architectures, while retaining exact parallelizability.
Ref: [1] William Merrill, Jackson Petty, Ashish Sabharwal. The illusion of state in state-space models. ICML 2024.
Q3: Does this idea generalize beyond a two-dimensional lift?
We thank the Reviewer for this interesting question. We would like to distinguish two forms of generalization: extending RiccatiSSM to higher-dimensional states, which we investigate experimentally, and extending the underlying mathematical construction to higher-dimensional lifts.
Higher-dimensional states: Our A5 experiments reveal an interesting connection to the underlying group structure. Real scalar Möbius transformations cannot faithfully represent the non-abelian simple group , whereas complex Möbius transformations contain as the icosahedral group. Consistent with this distinction, scalar real-valued RiccatiSSM does not solve the task with one layer, while its complex-valued extension does. We additionally find that a real-valued extension with block-diagonal state interactions solves with a single layer. These results demonstrate that extending the state representation can substantially improve expressivity.
Higher-dimensional lifts: The two-dimensional lift is not an inherent mathematical limitation. Matrix Riccati equations of the form admit higher-dimensional block-linear representations, with . Since composition remains associative matrix multiplication, these systems can also be evaluated through parallel scans.
However, higher-dimensional lifts bring additional computational and numerical considerations. Dense matrix composition has cubic cost in the lifted dimension, and avoiding singularities requires maintaining the invertibility of . We therefore view higher-dimensional RiccatiSSMs as a promising direction for future work, while the present paper focuses on the computationally efficient scalar construction.
We thank the Reviewer for the constructive feedback and suggestions. We agree that the empirical positioning of RiccatiSSM can be strengthened through comparisons with additional recent architectures, and we have conducted new experiments to address these concerns.
Following the Reviewer's suggestion, we evaluated these architectures on five time-series classification benchmarks. Given the limited rebuttal period, extensive hyperparameter optimization across all models was not feasible. We therefore used a consistent experimental setup with 6 layers and a state size of 64, and tuned the learning rate.
Time-series classification accuracy (% ↑)
Fixed setup: 6 layers, state size 64. Higher is better. Fields marked with * means that one sweep out of 3 of learning rate optimization is missing, and so their result might change a bit, but they give a good initial signal of their capabilities on these benchmarks. We will report the final results by the end of the rebuttal period.
Under this setup, RiccatiSSM achieves the highest mean accuracy on all five benchmarks. These results provide additional evidence that RiccatiSSM is competitive with recent sequence modeling architectures, including models originally developed for large-scale language modeling.
Forecasting performance: We additionally evaluated DeltaNet and Mamba-3 on the Weather forecasting benchmark, following the hyperparameter optimization protocol used in our existing experiments. While LinOSS remains the strongest model on this benchmark, these results further support RiccatiSSM's competitiveness against the DeltaNet and Mamba baselines.
Weather forecasting: 720 input steps, 720 prediction steps, lower is better.
Our current evaluation focuses on time-series classification and forecasting, which are closely aligned with the continuous-time dynamical systems underlying RiccatiSSM. We are particularly interested in efficient sequence modeling.
We agree that evaluating RiccatiSSM within larger deep architectures, including language models, would be a valuable extension. Recent work on parallelizable nonlinear recurrent architectures, such as ParaRNN, has demonstrated that nonlinear recurrent models can also scale to billions of parameters. We therefore see large-scale evaluations as an interesting direction for future work, while keeping the present contribution focused on introducing and characterizing the RiccatiSSM recurrence.