- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work proposes a hidden state decomposition for LTC networks to both keep the adaptive nature of these networks and to enhance their long-term information-keeping capabilities. This new type of network is called SSD-LTC. The authors show theoretical guarantees and experimental validations with consistent improvements compared to the original LTC network.
Critical strengths
- The authors point out a very important and so far missing property of LTCs, which in previous work was only addressed by additional memory components
- The paper is well-written and easy to follow
- The experimental evaluation shows consistent improvement over the baselines
Critical weaknesses
- The presented experiments are not long-sequence modeling benchmarks; most of them have only a sequence length of 32 in Table 2, and for the irregularly sampled datasets, it ranges from 4 to 394. I am not convinced that the presented benchmarks are the most suitable for showing retention of information over time.
- While the new structure of the model shows improved performance in terms of accuracy or MSE, it also has higher computational times, which are already quite high for LTC networks by default. This makes the practical use of SSD-LTC questionable.
- It is not clear how generalizable this hidden state decomposition is beyond LTCs. The authors mention the CfC networks [1], which provide a closed-form solution for LTCs, and the Liquid-S4 paper [2], which presents an LTC-inspired state-space formulation for long-sequence modeling, as related work. Besides that, there are other models that are also in the direction of more efficient computation, like LRCs [3] and their state-space formulation, LrcSSMs [4], which are also liquid neural network-based architectures.
[1] Ramin Hasani, Mathias Lechner, Alexander Amini, Lucas Liebenwein, Aaron Ray, Max Tschaikowski, Gerald Teschl, and Daniela Rus. Closed-form continuous-time neural networks. Nature Machine Intelligence, 4(11):992–1003, 2022.
[2] Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus. Liquid structural state-space models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id= g4OTKRKfS7R.
[3] Monika Farsang, Sophie A Neubauer, and Radu Grosu. Liquid resistance liquid capacitance networks. In The First Workshop on NeuroAI@ NeurIPS2024, 2024.
[4] Monika Farsang and Radu Grosu. Parallelization of non-linear state-space models: Scaling up liquid-resistance liquid-capacitance networks for efficient sequence modeling. In The Thirtyninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=ENYvdnyhLl.
Questions for authors
- I was wondering if this learned persistent subspace can be interpreted in any way in terms of neuroscience features that LTCs have?
- In Table 5, what is the reason for SSD-LTCs having longer epochs but smaller GFLOPs compared to LTCs?
- Could the authors show results on long-range prediction tasks (ideally on lengths of 1000 and 10000)?
- Can the authors elaborate more on the generalizability of the state decomposition design? If the authors could demonstrate that the same idea is applicable to [1] and [3] as a typical RNN-like formulation, and to [2] and [4] as an SSM formulation, my concerns about both long-range prediction capabilities and computational times would be resolved.
4 comments
Continued Rebuttal
Response to W2, Q2: Longer epochs despite lower estimated GFLOPs.
We agree that wall-clock efficiency is an important limitation of the present implementation. On CESNET, SSD-LTC requires 6.43 seconds per training and validation epoch compared to 4.56 seconds for LTC. In return, SSD-LTC reduces prediction MSE from 0.7819 to 0.6806, an improvement of 13.0%.
The estimated training costs are 5.985 and 6.246 GFLOPs per batch for SSD-LTC and LTC, respectively. These estimates count arithmetic operations but do not fully capture implementation-dependent costs, such as kernel execution and memory operations. SSD-LTC additionally performs state projections, controller evaluations, and QR factorization, which may contribute to the observed runtime difference. GFLOPs per batch and time per training-and-validation epoch are different measurement scopes and should not be interpreted as directly interchangeable.
Our primary objective is to improve information retention and predictive performance rather than computational efficiency. We recognize that the additional runtime may limit the practicality of the current implementation in latency-sensitive applications. Future work will investigate more efficient implementations of the decomposition and controller. We clarify the computational trade-off and the distinction between estimated operation counts and measured runtime in the revised manuscript.
Response to W3, Q4: Applicability beyond LTC.
We thank the reviewer for suggesting these extensions. Although the present work focuses on LTC networks, the proposed state-space decomposition can also be applied to other architectures that maintain a recurrent hidden state. As an initial extension, we implemented SSD-CfC using Closed-form Continuous-time (CfC) networks. In SSD-CfC, the CfC cell produces a provisional hidden state whose adaptive component is retained through projection, while the complementary persistent coordinates follow the controlled update used in SSD-LTC.
We compare CfC and SSD-CfC on three benchmarks using a hidden dimension of and a persistent subspace rank of . Results are reported as mean ± standard deviation over five seeds.
| Model | CESNET (MSE ) | Electricity (MAE ) | UCI-HAR (Acc ) |
|---|---|---|---|
| CfC | 0.8468$\pm$0.0341 | 0.079$\pm$0.020 | 85.19$\pm$0.22 |
| SSD-CfC | 0.7023$\pm$0.0298 | 0.062$\pm$0.021 | 92.43$\pm$0.18 |
| Improvement | 17.1% MSE Reduction | 21.5 % MAE Reduction | + 7.24 percentage points |
These results provide initial evidence that the proposed state-space decomposition is applicable beyond LTC networks, improving predictive performance across all three evaluated tasks when integrated with CfC. Extending the approach to LRC and structured state-space architectures such as Liquid-S4 and LrcSSM remains a direction for future work.
Response to Q1: Neuroscience interpretation.
We thank the reviewer for the biological analogy suggestion. We briefly expanded this in the introduction. The persistent subspace provides a functional analogy to neural population activity. Murray et al. (2017) [2] reported that stable population coding of working-memory information can coexist with heterogeneous, time-varying activity in prefrontal cortex. This suggests a distinction between stable mnemonic representations and dynamically evolving neural activity. Raut et al. (2020) [3] further documented hierarchical temporal dynamics across brain regions, supporting the general relevance of heterogeneous timescales.
In SSD-LTC, the persistent subspace provides coordinates in which task-relevant information can be selectively written and retained, while the adaptive liquid subspace accommodates more rapidly changing dynamics. The delayed-recall and component-exchange experiments support this functional distinction within the trained model.
References
[2] Murray, J. D., Bernacchia, A., Roy, N. A., Constantinidis, C., Romo, R., & Wang, X. J. (2017). Stable population coding for working memory coexists with heterogeneous neural dynamics in prefrontal cortex. Proceedings of the National Academy of Sciences of the United States of America, 114(2), 394–399.
[3] Raut, R. V., Snyder, A. Z., & Raichle, M. E. (2020). Hierarchical dynamics as a macroscopic organizing principle of the human brain. Proceedings of the National Academy of Sciences of the United States of America, 117(34), 20890–20897.
I thank the authors for their detailed responses and additional experiments. Most of my concerns are addressed. The computational time for this model is still a bottleneck, but as it was shown, it could be extended to more efficient liquid models. I am raising my score to 3.
We thank the reviewer for the time and effort they put into evaluating our work. We are glad that our response addressed most of their concerns, and we appreciate the feedback that helped us improve the paper.
We thank the reviewer for recognizing the motivation, theoretical analysis, and consistent empirical improvements of our work. Below, we address the questions about longer-range retention, computational cost, applicability beyond LTC, and the neuroscience interpretation.
Core contribution: Controlled retention within the LTC hidden state.
In this work, SSD-LTC introduces a structured update mechanism that assigns different dynamics to two components of the existing LTC hidden state without increasing its total dimensionality. Our contribution is not gating or orthogonal decomposition alone, but how they are used to update different parts of the LTC hidden state. The adaptive component follows the projected LTC dynamics, while the persistent component employs an elapsed-time-dependent gated update for selective writing and retention. The write-rate controller receives the previous liquid component and current input, rather than the persistent coordinates directly. The components are subsequently recombined for the next recurrent update.
Our proposed method addresses the coupling between state attenuation and input-driven updates in the standard LTC parameterization. While LTC networks can learn slow dynamics, SSD-LTC explicitly separates the update mechanisms for the two state components. Our fixed-basis ablation shows that this structured update remains effective even without learning the subspace orientation. The new SSD-CfC results below further examine whether this design principle extends beyond LTC.
Response to W1, Q3: Long-term forecasting and retention.
Our evaluation includes benchmarks with varying sequence lengths including long-range forecasting tasks. However, we agree that the benchmarks might not provide sufficient evidence of long-range information retention. We therefore distinguish forecasting performance on longer trajectories from direct tests of memory retention over long delays.
Forecasting. Our evaluation includes the Electricity and Weather datasets from Cyranka and Haponiuk (2026) [1], with trajectories of 1,440 and 1,000 steps, respectively. These use lookback windows of 720 and 500 steps, with corresponding forecasting horizons of 720 and 500 steps. These settings are also described as long-range forecasting tasks in [1]. Under these settings, SSD-LTC reduces MAE relative to LTC by 20.3% on Electricity and 7.1% on Weather. These results support better predictive performance on the evaluated datasets.
Controlled retention under unseen delays. To more directly test memory preservation, we extended the delayed-recall evaluation to 4,096 steps. All Models are trained with K∈{16,32,64} and evaluated at longer unseen delays. SSD-LTC uses hidden state of dimension dh=64 and persistent rank r=16. The compared models are parameter-matched to SSD-LTC. Results are exact recall percentages, reported as mean ± standard deviation over five seeds.
SSD-LTC achieves 30.93% exact recall at 1,024 steps and 15.36% at 4,096 steps, compared with 11.52% and 5.24% for Mamba. Although recall decreases with delay, SSD-LTC retains useful information at delays up to 64 times the maximum training delay.
References
[1] Cyranka, Jacek, and Szymon Haponiuk. Unified long-term time series forecasting benchmark. Neurocomputing (2026): 134091.