acceptodds
Under review as a conference paper at ICLR 2027

Hidden State-Space Decomposition for Long-Term Retention in Liquid Time-Constant Networks

Abstract

Continuous-time recurrent models naturally handle irregularly sampled data by evolving hidden states as a function of elapsed time. Among these models, Liquid Time-Constant (LTC) networks adapt the rate of state evolution based on the current input, allowing their dynamics to change over time. This adaptability comes with an inherent decay of their hidden state, where information is progressively attenuated from earlier observations. Writing new information and preserving earlier information depend on the same state dynamics. Rapid writing requires the state to change quickly, while long-term retention requires the state to decay slowly. We introduce State-Space Decomposed Liquid Time-Constant Networks (SSD-LTC), where we decompose the LTC hidden state into an adaptive liquid subspace and a controlled persistent subspace. The adaptive liquid component follows the LTC dynamics, while the persistent component uses an input-dependent write rate to balance writing new information with preserving its previous state. We theoretically characterize state attenuation and the writing–retention trade-off in LTCs, and establish boundedness and stability properties of SSD-LTC. Across ten regularly sampled and eight irregularly sampled benchmarks, SSD-LTC improves classification accuracy by 2.2 to 8.0 percentage points and reduces regression and forecasting errors by up to 30% compared with the baseline LTC. We further show through controlled recall and component-exchange experiments that the learned persistent component acts as memory. These results demonstrate that separating adaptive state evolution from persistent information retention improves temporal modeling while retaining the continuous-time dynamics of LTCs.

Primary area:
learning on time series and dynamical systems
Keywords:
Continuous-time neural networks, liquid time-constant networks, multiscale dynamics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ajQuOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes a hidden state decomposition for LTC networks to both keep the adaptive nature of these networks and to enhance their long-term information-keeping capabilities. This new type of network is called SSD-LTC. The authors show theoretical guarantees and experimental validations with consistent improvements compared to the original LTC network.

Critical strengths
  • The authors point out a very important and so far missing property of LTCs, which in previous work was only addressed by additional memory components
  • The paper is well-written and easy to follow
  • The experimental evaluation shows consistent improvement over the baselines
Critical weaknesses
  • The presented experiments are not long-sequence modeling benchmarks; most of them have only a sequence length of 32 in Table 2, and for the irregularly sampled datasets, it ranges from 4 to 394. I am not convinced that the presented benchmarks are the most suitable for showing retention of information over time.
  • While the new structure of the model shows improved performance in terms of accuracy or MSE, it also has higher computational times, which are already quite high for LTC networks by default. This makes the practical use of SSD-LTC questionable.
  • It is not clear how generalizable this hidden state decomposition is beyond LTCs. The authors mention the CfC networks [1], which provide a closed-form solution for LTCs, and the Liquid-S4 paper [2], which presents an LTC-inspired state-space formulation for long-sequence modeling, as related work. Besides that, there are other models that are also in the direction of more efficient computation, like LRCs [3] and their state-space formulation, LrcSSMs [4], which are also liquid neural network-based architectures.

[1] Ramin Hasani, Mathias Lechner, Alexander Amini, Lucas Liebenwein, Aaron Ray, Max Tschaikowski, Gerald Teschl, and Daniela Rus. Closed-form continuous-time neural networks. Nature Machine Intelligence, 4(11):992–1003, 2022.

[2] Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus. Liquid structural state-space models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id= g4OTKRKfS7R.

[3] Monika Farsang, Sophie A Neubauer, and Radu Grosu. Liquid resistance liquid capacitance networks. In The First Workshop on NeuroAI@ NeurIPS2024, 2024.

[4] Monika Farsang and Radu Grosu. Parallelization of non-linear state-space models: Scaling up liquid-resistance liquid-capacitance networks for efficient sequence modeling. In The Thirtyninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=ENYvdnyhLl.

Questions for authors
  • I was wondering if this learned persistent subspace can be interpreted in any way in terms of neuroscience features that LTCs have?
  • In Table 5, what is the reason for SSD-LTCs having longer epochs but smaller GFLOPs compared to LTCs?
  • Could the authors show results on long-range prediction tasks (ideally on lengths of 1000 and 10000)?
  • Can the authors elaborate more on the generalizability of the state decomposition design? If the authors could demonstrate that the same idea is applicable to [1] and [3] as a typical RNN-like formulation, and to [2] and [4] as an SSM formulation, my concerns about both long-range prediction capabilities and computational times would be resolved.
4 comments

We thank the reviewer for recognizing the motivation, theoretical analysis, and consistent empirical improvements of our work. Below, we address the questions about longer-range retention, computational cost, applicability beyond LTC, and the neuroscience interpretation.

Core contribution: Controlled retention within the LTC hidden state. 

In this work, SSD-LTC introduces a structured update mechanism that assigns different dynamics to two components of the existing LTC hidden state without increasing its total dimensionality. Our contribution is not gating or orthogonal decomposition alone, but how they are used to update different parts of the LTC hidden state. The adaptive component follows the projected LTC dynamics, while the persistent component employs an elapsed-time-dependent gated update for selective writing and retention. The write-rate controller receives the previous liquid component and current input, rather than the persistent coordinates directly. The components are subsequently recombined for the next recurrent update.

Our proposed method addresses the coupling between state attenuation and input-driven updates in the standard LTC parameterization. While LTC networks can learn slow dynamics, SSD-LTC explicitly separates the update mechanisms for the two state components. Our fixed-basis ablation shows that this structured update remains effective even without learning the subspace orientation. The new SSD-CfC results below further examine whether this design principle extends beyond LTC.

Response to W1, Q3: Long-term forecasting and retention.

Our evaluation includes benchmarks with varying sequence lengths including long-range forecasting tasks. However, we agree that the benchmarks might not provide sufficient evidence of long-range information retention. We therefore distinguish forecasting performance on longer trajectories from direct tests of memory retention over long delays.

Forecasting. Our evaluation includes the Electricity and Weather datasets from Cyranka and Haponiuk (2026) [1], with trajectories of 1,440 and 1,000 steps, respectively. These use lookback windows of 720 and 500 steps, with corresponding forecasting horizons of 720 and 500 steps. These settings are also described as long-range forecasting tasks in [1]. Under these settings, SSD-LTC reduces MAE relative to LTC by 20.3% on Electricity and 7.1% on Weather. These results support better predictive performance on the evaluated datasets.

Controlled retention under unseen delays. To more directly test memory preservation, we extended the delayed-recall evaluation to 4,096 steps. All Models are trained with K∈{16,32,64}K\in\{16,32,64\} and evaluated at longer unseen delays. SSD-LTC uses hidden state of dimension dh=64d_h=64 and persistent rank r=16r=16. The compared models are parameter-matched to SSD-LTC. Results are exact recall percentages, reported as mean ±\pm standard deviation over five seeds.

Model1282565121,0242,0484,096
ODE-LSTM0.39 ± 0.200.42 ± 0.250.39 ± 0.260.29 ± 0.100.33 ± 0.150.23 ± 0.11
LTC0.59 ± 0.140.48 ± 0.160.43 ± 0.180.46 ± 0.150.39 ± 0.120.30 ± 0.15
LTC-MHN1.95 ± 0.391.86 ± 0.351.24 ± 0.151.04 ± 0.311.07 ± 0.340.62 ± 0.20
Mamba32.51 ± 1.5125.36 ± 1.4816.96 ± 1.5911.52 ± 1.539.31 ± 2.545.24 ± 2.18
SSD-LTC41.23 ± 0.4138.11 ± 0.3934.38 ± 0.4230.93 ± 0.3820.71 ± 0.3915.36 ± 0.53

SSD-LTC achieves 30.93% exact recall at 1,024 steps and 15.36% at 4,096 steps, compared with 11.52% and 5.24% for Mamba. Although recall decreases with delay, SSD-LTC retains useful information at delays up to 64 times the maximum training delay.

References

[1] Cyranka, Jacek, and Szymon Haponiuk. Unified long-term time series forecasting benchmark. Neurocomputing (2026): 134091.

AuthorsOpenReview ↗

Continued Rebuttal

Response to W2, Q2: Longer epochs despite lower estimated GFLOPs.

We agree that wall-clock efficiency is an important limitation of the present implementation. On CESNET, SSD-LTC requires 6.43 seconds per training and validation epoch compared to 4.56 seconds for LTC. In return, SSD-LTC reduces prediction MSE from 0.7819 to 0.6806, an improvement of 13.0%.

The estimated training costs are 5.985 and 6.246 GFLOPs per batch for SSD-LTC and LTC, respectively. These estimates count arithmetic operations but do not fully capture implementation-dependent costs, such as kernel execution and memory operations. SSD-LTC additionally performs state projections, controller evaluations, and QR factorization, which may contribute to the observed runtime difference. GFLOPs per batch and time per training-and-validation epoch are different measurement scopes and should not be interpreted as directly interchangeable.

Our primary objective is to improve information retention and predictive performance rather than computational efficiency. We recognize that the additional runtime may limit the practicality of the current implementation in latency-sensitive applications. Future work will investigate more efficient implementations of the decomposition and controller. We clarify the computational trade-off and the distinction between estimated operation counts and measured runtime in the revised manuscript.

Response to W3, Q4: Applicability beyond LTC.

We thank the reviewer for suggesting these extensions. Although the present work focuses on LTC networks, the proposed state-space decomposition can also be applied to other architectures that maintain a recurrent hidden state. As an initial extension, we implemented SSD-CfC using Closed-form Continuous-time (CfC) networks. In SSD-CfC, the CfC cell produces a provisional hidden state whose adaptive component is retained through projection, while the complementary persistent coordinates follow the controlled update used in SSD-LTC.

We compare CfC and SSD-CfC on three benchmarks using a hidden dimension of dh=64d_h=64 and a persistent subspace rank of r=16r=16. Results are reported as mean ± standard deviation over five seeds.

ModelCESNET (MSE ↓\downarrow)Electricity (MAE ↓\downarrow)UCI-HAR (Acc ↑\uparrow)
CfC0.8468$\pm$0.03410.079$\pm$0.02085.19$\pm$0.22
SSD-CfC0.7023$\pm$0.02980.062$\pm$0.02192.43$\pm$0.18
Improvement17.1% MSE Reduction21.5 % MAE Reduction+ 7.24 percentage points

These results provide initial evidence that the proposed state-space decomposition is applicable beyond LTC networks, improving predictive performance across all three evaluated tasks when integrated with CfC. Extending the approach to LRC and structured state-space architectures such as Liquid-S4 and LrcSSM remains a direction for future work.

Response to Q1: Neuroscience interpretation.

We thank the reviewer for the biological analogy suggestion. We briefly expanded this in the introduction. The persistent subspace provides a functional analogy to neural population activity. Murray et al. (2017) [2] reported that stable population coding of working-memory information can coexist with heterogeneous, time-varying activity in prefrontal cortex. This suggests a distinction between stable mnemonic representations and dynamically evolving neural activity. Raut et al. (2020) [3] further documented hierarchical temporal dynamics across brain regions, supporting the general relevance of heterogeneous timescales.

In SSD-LTC, the persistent subspace provides coordinates in which task-relevant information can be selectively written and retained, while the adaptive liquid subspace accommodates more rapidly changing dynamics. The delayed-recall and component-exchange experiments support this functional distinction within the trained model.

References

[2] Murray, J. D., Bernacchia, A., Roy, N. A., Constantinidis, C., Romo, R., & Wang, X. J. (2017). Stable population coding for working memory coexists with heterogeneous neural dynamics in prefrontal cortex. Proceedings of the National Academy of Sciences of the United States of America, 114(2), 394–399.

[3] Raut, R. V., Snyder, A. Z., & Raichle, M. E. (2020). Hierarchical dynamics as a macroscopic organizing principle of the human brain. Proceedings of the National Academy of Sciences of the United States of America, 117(34), 20890–20897.

I thank the authors for their detailed responses and additional experiments. Most of my concerns are addressed. The computational time for this model is still a bottleneck, but as it was shown, it could be extended to more efficient liquid models. I am raising my score to 3.

We thank the reviewer for the time and effort they put into evaluating our work. We are glad that our response addressed most of their concerns, and we appreciate the feedback that helped us improve the paper.

Reviewer_VsrLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the loss of information over time in liquid time-constant (LTC) networks. Since every unit leaks toward its resting value, information written in the hidden state progressively decays. The authors argue that, in an LTC unit, fast writing and long retention cannot be obtained at the same time.

The proposed model SSD-LTC divides the hidden state into two parts. In a learned subspace, the state follows a gated update with a write rate that can become very small, so that a stored value is kept almost unchanged. In the rest of the state space, the usual LTC dynamics are kept. SSD-LTC obtains the best average score on ten regularly sampled and eight irregularly sampled datasets. On a delayed-recall task, the authors show that the recalled information is stored in the memory subspace.

Critical strengths
  1. The mechanism is simple, cheap, and easy to interpret, since the write rate directly determines how much of the stored value is kept. The paper also goes further than accuracy tables. On the delayed-recall task, exchanging the memory part between two sequences also exchanges the recalled content, while exchanging the other part has no effect. This type of mechanistic test is rare, and I find it convincing.

  2. The improvement over the LTC baseline is consistent across many datasets, with standard deviations over five training seeds.

Critical weaknesses

My recommendation is based on two weaknesses. I read the paper from the point of view of dynamical systems and recurrent dynamics, and not as a specialist in time-series benchmarks.

  1. It is not clear that the gains come from the proposed decomposition. The ablation in Table 3 shows that most of the improvement over LTC is already obtained with a fixed basis, that is, by simply turning some units into gated memory units. This kind of gated memory is well known, for example from the update gate of a GRU. The closest method, ODE-LSTM, which also adds a separate memory to a continuous-time network and which the paper describes as related, is not included in the comparison. In addition, the baselines obtain lower scores than in the original LTC paper on the same tasks (Hasani et al., 2021), in most cases by 1 to 3 percentage points. On sequential MNIST, for example, that paper reports 98.4% for LSTM, 98.3% for CT-GRU, and 97.6% for LTC, compared to 95.2%, 96.0%, and 95.4% here. On activity recognition, it reports 97.3% for the neural ODE, compared to 94.2% here. With the published values, SSD-LTC (98.2% and 96.7%) would not be the best model on these two tasks.

  2. The claims about memory go further than what the analysis and the experiments show, both for the limitation of LTC networks and for the retention provided by the new memory. First, the limitation of LTC networks is presented as fundamental, but it follows from a property of the particular update in Eq. 1. In each unit, the same input determines both the value toward which the state converges and the speed of this convergence. When these two quantities can be controlled independently, fast writing and slow decay are compatible, and this is exactly what the proposed gated memory does. In addition, the argument considers a single unit in isolation. In a recurrent network, slow dynamics, and even continuous attractors, can emerge from the coupling between units. Second, the new memory does not provide the arbitrarily long retention that is claimed. Its write rate is always strictly positive, so the stored value always decays slowly. In the delayed-recall experiment, 81.5% of the stored state remains after 128 steps (Table 4). With the same rate, only about 13% would remain after 1,280 steps. Robust long-term memory usually relies on attractors (multistability), which this design does not provide. Long-term retention is only tested at twice the longest training delay.

Questions for authors

The questions are given in order of importance. The answer to each of them could change my assessment.

  1. How does SSD-LTC compare with ODE-LSTM (Lechner and Hasani, 2020) on a few representative tasks? If SSD-LTC remains clearly better, my main concern would be resolved.

  2. Why are several baseline results lower than those reported by Hasani et al. (2021) on the same tasks, and how were the baselines tuned?

  3. How does the exact recall of SSD-LTC change when the delay becomes much longer than during training, for example 10 or 100 times longer? What do a standard LTC and a selective state-space model such as Mamba, from which the task is adapted, obtain on the same task?

  4. The trade-off described in Section 2 comes from the fact that a single input determines both the target value and the speed of convergence of each unit. Is this a property of LTC dynamics in general, or of this particular parameterization? Does the trade-off still hold at the level of the network, where slow modes can emerge from the coupling between units?

Additional remarks:

  • Tables 1 and 2 are dense and difficult to read. Coloring the cells according to the rank, or adding the relative change with respect to LTC, would help.
4 comments

We thank the reviewer for the detailed assessment of our work and for the positive assessment of the component-exchange experiment. Below, we address the questions on ODE-LSTM, the state partition, baseline performance.

Core contribution: Controlled retention within the LTC hidden state.

SSD-LTC introduces a structured update mechanism that assigns distinct dynamics to two components of the existing LTC hidden state, without increasing its total dimensionality. The adaptive component follows projected LTC dynamics, while the persistent component uses an elapsed-time-dependent gated update for selective writing and retention. The controller receives the previous liquid component and current input, rather than the persistent coordinates directly. The two components are then recombined to form the state used by the subsequent LTC update. This design addresses the coupling between state attenuation and input-driven updates in the standard LTC parameterization.

Response to Q1: Comparison with ODE-LSTM.

ODE-LSTM combines continuous-time hidden-state dynamics with a separate LSTM cell state that is updated when observations arrive and remains unchanged between observations. In SSD-LTC, the persistent component occupies a subspace of the existing recurrent hidden state and follows an elapsed-time-dependent update. This distinction concerns how adaptive dynamics and persistent information are organized within the recurrent state.

To directly address the reviewer's concern, we compared SSD-LTC with ODE-LSTM on two classification and two irregularly sampled forecasting tasks. The models are parameter-matched, with less than 1% difference in parameter count. Results are reported as mean ± standard deviation over five seeds.

DatasetMetricODE-LSTMSSD-LTC
Activity RecognitionAccuracy ↑94.63 ± 0.4196.69 ± 0.35
Sequential MNISTAccuracy ↑95.84 ± 0.2898.19 ± 0.19
CESNETMSE ↓0.7898 ± 0.00240.6806 ± 0.0022
EPA-AirMSE ↓0.3856 ± 0.01370.3521 ± 0.0112

SSD-LTC improves accuracy over ODE-LSTM by 2.06 percentage points on Activity Recognition and 2.35 percentage points on Sequential MNIST. It also reduces MSE by 13.8% on CESNET and 8.7% on EPA-Air. These comparisons show consistent improvements over parameter-matched ODE-LSTM across the four evaluated tasks.

Response to W1: Contribution of the state partition and learned basis.

We agree that gated memory is an established mechanism and do not claim novelty in QR decomposition or gating itself. Our contribution is to assign distinct update dynamics to complementary subspaces of the LTC hidden state, replacing the LTC dynamics in the persistent subspace with controlled retention rather than introducing an additional memory state.

The fixed-basis ablation in Table 3 shows that this structured update is effective even without learning the subspace orientation. On UCI-HAR, accuracy increases from 84.22% for LTC to 91.72% with the fixed basis and 92.17% with the learned basis. On Gesture, the corresponding accuracies are 69.45%, 73.06%, and 76.38%. The fixed-basis variant improves over LTC on all three evaluated ablation tasks, while learning the orientation provides a further, task-dependent benefit. These results support the usefulness of assigning different dynamics to the partitioned state, with basis learning providing task-dependent improvement.

Response to Q2: Baseline performance and training protocol.

We understand the reviewer's concern regarding the differences between our baseline results in Table 1 and those reported by Hasani et al. (2021). For the datasets shared with the original LTC study, we use its dataset implementation, including the same preprocessing, train/validation/test splits, sequence construction, and evaluation procedure.

For the baselines and SSD-LTC, we used Adam and trained for 200 epochs, scanning the learning rate over {10−5,3×10−5,10−4,3×10−4,10−3,3×10−3,10−2}\{10^{-5}, 3\times10^{-5}, 10^{-4}, 3\times10^{-4}, 10^{-3}, 3\times10^{-3}, 10^{-2}\}. Other hyperparameters were kept fixed during the learning-rate scan. Learning rates and checkpoints were selected using the best validation performance. After selecting the learning rate for each model and task, we ran five independent seeds and report the mean and standard deviation of the test results.

The LTC baseline uses the publicly available NCPS PyTorch implementation [1], with a hidden dimension of 64 and a task-specific prediction head. SSD-LTC uses the same NCPS LTC cell for its underlying liquid dynamics, with the proposed state partition and controlled persistent update applied to the provisional hidden state. This provides a consistent backbone for the LTC-versus-SSD-LTC comparison. Similar baseline values are also reported in Coupling Liquid Time-Constant Encoders with Modern Hopfield Memory [2].

AuthorsOpenReview ↗

Continued Rebuttal

Response to W2 and Q3: Retention at longer unseen delays.

We realize that some of our phrasing may have caused confusion. Proposition 1 establishes a finite-horizon capability of the persistent update: for any specified writing and retention intervals, suitable controller outputs can provide rapid writing and a prescribed level of direct-state retention. The write rate remains positive but can approach zero during retention. The result does not guarantee indefinite retention for a fixed positive write rate, or that training will learn the required controller outputs for every task and horizon. We clarify this distinction in the revised manuscript.

The 81.5% reported in Table 4 measures the direct carry coefficient of the persistent state across the 128-step delay. Exact-recall accuracy measures whether the stored symbols are recovered. To examine retrieval over substantially longer delays, we extended the delayed-recall evaluation to 4,096 steps.

Each training example samples KK from {16,32,64}\{16,32,64\}. For each model and seed, checkpoint selection uses 256 fixed validation examples at K=64K=64. The selected checkpoint is then frozen and reused at every evaluation delay, without retraining or selection at longer delays. Evaluation extends the blank suffix while preserving the cue prefix and targets. Thus, K=64K=64 is a seen-delay reference, while all delays in the table below are unseen during training. SSD-LTC uses dh=64d_h=64 and r=16r=16, and the compared models are parameter-matched to SSD-LTC. Results are exact-recall percentages, reported as mean ± standard deviation over five seeds.

Model1282565121,0242,0484,096
ODE-LSTM0.39 ± 0.200.42 ± 0.250.39 ± 0.260.29 ± 0.100.33 ± 0.150.23 ± 0.11
LTC0.59 ± 0.140.48 ± 0.160.43 ± 0.180.46 ± 0.150.39 ± 0.120.30 ± 0.15
LTC-MHN1.95 ± 0.391.86 ± 0.351.24 ± 0.151.04 ± 0.311.07 ± 0.340.62 ± 0.20
Mamba32.51 ± 1.5125.36 ± 1.4816.96 ± 1.5911.52 ± 1.539.31 ± 2.545.24 ± 2.18
SSD-LTC41.23 ± 0.4138.11 ± 0.3934.38 ± 0.4230.93 ± 0.3820.71 ± 0.3915.36 ± 0.53

SSD-LTC achieves 30.93% exact recall at 1,024 steps and 15.36% at 4,096 steps, compared with 11.52% and 5.24% for Mamba. SSD-LTC outperforms the evaluated baselines at every reported delay, including 64 times the maximum training delay. Together with the component-exchange experiment, these results support the role of the persistent component as memory on this task.

Response to Q4: Scope of the LTC trade-off.

The trade-off analyzed in Section 2 follows from the standard LTC parameterization in Eq. 1, where the conductance controls both the input-driven contribution and attenuation of the previous state. Our analysis characterizes the explicit state-carry path of individual LTC coordinates.

We agree that interactions between recurrent units can generate slow collective dynamics or attractor-like behavior. The derived bound does not exclude such mechanisms at the network level. SSD-LTC addresses the identified coupling by replacing the LTC dynamics in a persistent subspace with a controlled update whose carry coefficient can approach one independently of the intrinsic LTC decay. We clarify the scope of the analysis in Section 2.

Presentation of Tables 1 and 2.

We thank the reviewer for the suggestion. We made the rankings and relative improvements over LTC easier to identify in Tables 1 and 2 in the revised manuscript.

References

[1] Neural Circuit Policies (NCPS): PyTorch and TensorFlow implementations of NCP, LTC, and CfC neural models. https://github.com/mlech26l/ncps

[2] Swain, B. R., Cheoi, K. J., & Ko, J. (2026). Coupling Liquid Time-Constant Encoders with Modern Hopfield Memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.

I thank the authors for their response and for the additional experiments.

Q1. The comparison with ODE-LSTM answers my question, provided that the baselines are correctly trained. However, in the new delayed-recall experiment, ODE-LSTM remains close to the chance level (0.39%) for all delays. This is surprising, since ODE-LSTM was designed for long-term dependencies and solves such tasks in Lechner and Hasani (2020). This raises the same concern as point 2.

Q2. I thank the authors for the description of the training procedure. As the data pipeline is the same as in Hasani et al. (2021), the difference of 1 to 3 points with the reported values probably comes from the training of the baselines. With the published values, SSD-LTC would not be the best model on sequential MNIST and activity recognition. Due to the double-blind process, it is difficult to assess whether [2] independently confirms the lower values. The paper should at least mention the difference between its values and the published ones.

Q3. The new results answer my question.

Q4. The clarification answers my question.

For the moment, I keep my rating, pending point 2.


[2] Swain, B. R., Cheoi, K. J., & Ko, J. (2026). Coupling Liquid Time-Constant Encoders with Modern Hopfield Memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.

We thank the reviewer for raising this concern. Below we address the remaining concerns.

Response to Q1

  • We used the unmodified official ODE-LSTM cell (mlech26l/ode-lstms) [1], retaining its LSTM gates, Linear–Tanh–Linear ODE field, and fixed RK4 solver with three unfolds. Only a task-shared input embedding and output head were added.
  • To verify the implementation, we reproduced the original bit-stream classification experiment of Lechner and Hasani (2020) [2], obtaining 100.00% accuracy with dense encoding and 98.84% with event-based encoding. This is consistent with the published results.
  • One possible explanation for the difference in delayed-recall performance lies in the different memory requirements of the two tasks. The original benchmark [2] requires computing the parity of a 32-bit sequence, which can be represented by a single binary state. Our delayed-recall task requires identifying four marked symbols, preserving their values and order, and reproducing them after a sequence of blank observations.

Although ODE-LSTM maintains its cell state between observations, each new observation triggers another gated memory update. Therefore, preserving the selected symbols requires the forget gates to maintain the stored information and the input gates to suppress irrelevant updates throughout the delay. Even small changes to the memory state can accumulate over repeated updates. In contrast, SSD-LTC uses a separate persistent update whose write rate can approach zero, allowing stored information to remain nearly unchanged while subsequent observations are processed.

We also examined ODE-LSTM at the delays used during training and observed that it learns the selective-copying task, but its recall accuracy decreases as the delay increases. This suggests that the near-chance performance at longer unseen delays reflects a difficulty in maintaining the selected information over repeated observations. We do not claim that ODE-LSTM cannot solve selective copying, but that its performance under our evaluated configuration deteriorates with increasing delay.

Response to Q2

  • We recognize that reproducibility requires more than an internal verification. We therefore clarified the implementation details, optimization settings, hyperparameter-selection procedures, and evaluation protocol in the revised manuscript, so that the comparison can be independently assessed. We will also be sharing the the source code to reproduce the results under evaluated settings.

  • We agree with the reviewer that the differences from the published baseline results should be explicitly acknowledged. We have revised Section 4.1 to report these differences, including the LSTM and CT-GRU results on Sequential MNIST and the Neural ODE result on Activity Recognition that exceed SSD-LTC's performance. We have also clarified the experimental settings in Appendix E.1 to specify that SSD-LTC achieves the best performance among models evaluated under our experimental settings.

We appreciate the reviewer's constructive feedback and the opportunity to clarify and strengthen our work. Beyond the regular benchmarks, SSD-LTC shows consistent improvements across eight irregularly sampled datasets. The delayed-recall and component-exchange experiments further demonstrate that the persistent subspace acts as memory. We believe these experimental findings, together with the theoretical analysis and additional evaluations (including SSD-CfC), provide a meaningful contribution to continuous-time sequence modeling.

References

[1] mlech26l/ode-lstms: Code repository of the paper Learning Long-Term Dependencies in Irregularly-Sampled Time Series. https://github.com/mlech26l/ode-lstms

[2] Lechner, M., & Hasani, R. (2020). Learning long-term dependencies in irregularly-sampled time series. arXiv preprint arXiv:2006.04418.

Reviewer_sGuCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the long-term retention problem in Liquid Time-Constant networks. The proposed SSD-LTC splits the hidden state into an adaptive liquid subspace and a persistent subspace with input-dependent retention, aiming to separate fast adaptation from memory preservation. The method is supported by theoretical analysis and evaluated on regular and irregular time-series benchmarks, together with controlled recall experiments showing that the persistent component can act as memory.

Critical strengths
  1. The motivation is clear and the proposed decomposition directly addresses the writing–retention trade-off in LTCs. The method is simple, technically sound, and the theoretical analysis is well aligned with the architecture.

  2. The empirical evaluation is fairly thorough, covering both regular and irregular time-series benchmarks. I also found the delayed-recall and component-exchange experiments particularly useful, as they provide direct evidence that the persistent subspace is actually carrying memory.

Critical weaknesses
  1. The main idea feels somewhat incremental relative to existing approaches that separate fast-changing dynamics from persistent memory. In particular, ODE-LSTM is conceptually quite close, but it is only discussed in related work and is not included in the main comparison.

  2. The dimension of the persistent subspace rr must be specified manually and may be task-dependent. The theory also establishes that the persistent branch can support long-term memory, but does not guarantee that training will learn task-relevant information in that subspace.

Questions for authors
  1. Since ODE-LSTM also separates continuous dynamics from a persistent memory path, could the authors provide a direct comparison or clarify more concretely when SSD-LTC should be expected to outperform it?

  2. How sensitive is performance to the choice of the persistent dimension rr, and is there a principled way to choose or adapt rr for a new task?

3 comments

We thank the reviewer for providing a positive assessment of our work, especially regarding the delayed-recall and component-exchange experiments. Below, we clarify the contribution and address the questions about ODE-LSTM and the persistent subspace dimension.

Core contribution: Selective writing and retention within the existing LTC hidden state.

We agree that separating evolving dynamics from memory is an established idea, and we do not claim that gating, orthogonal decomposition, or learning a subspace basis is novel in isolation. Our contribution is a structured update mechanism that assigns distinct dynamics to two components of the existing LTC hidden state, without increasing its total dimensionality. Specifically, the adaptive component follows the projected LTC dynamics, while the persistent component uses an elapsed-time-dependent gated update to control writing and retention. The write-rate controller depends on the previous liquid component and the current input, rather than directly on the persistent coordinates. The components are then recombined to form the state used by the subsequent LTC update.

Our proposed method addresses the coupling between state attenuation and input-driven updates in the standard LTC parameterization. While LTC networks can learn slow dynamics, SSD-LTC explicitly separates the update mechanisms for the two state components. Our fixed-basis ablation shows that this structured update remains effective even without learning the subspace orientation. The new SSD-CfC results below further examine whether this design principle extends beyond LTC.

Response to W1, Q1: Comparison with ODE-LSTM.

ODE-LSTM combines continuous-time hidden-state dynamics with a separate LSTM cell state that is updated when observations arrive and remains unchanged between observations. In SSD-LTC, the persistent component occupies a learned subspace of the recurrent hidden state and follows an elapsed-time-dependent update. The persistent and adaptive components are recombined to form the state used by the subsequent LTC update.

To directly address the reviewer's concern, we conduct additional experiments comparing SSD-LTC with ODE-LSTM and CfC on two classification and two irregularly sampled forecasting tasks. ODE-LSTM and SSD-LTC are parameter-matched, with less than 1% difference in their parameter counts. Results are reported as mean ± standard deviation over five seeds.

DatasetMetricCfCODE-LSTMSSD-LTC
Activity RecognitionAccuracy ↑94.48 ±\pm 0.5494.63 ±\pm 0.4196.69 ±\pm 0.35
Sequential MNISTAccuracy ↑95.20 ±\pm 0.2495.84 ±\pm 0.2898.19 ±\pm 0.19
CESNETMSE ↓0.8468$\pm$0.03410.7898 ±\pm 0.00240.6806 ±\pm 0.0022
EPA-AirMSE ↓0.4019 ±\pm 0.03170.3856 ±\pm 0.01370.3521 ±\pm 0.0112

SSD-LTC improves accuracy over ODE-LSTM by 2.06 pp on Activity Recognition and 2.35 pp on Sequential MNIST. SSD-LTC also reduces MSE by 13.8% on CESNET and 8.7% on EPA-Air compared to ODE-LSTM .

Response to W2, Q2: Selection of persistent subspace dimension.

Appendix E reports a rank sweep with hidden dimension dh=64d_h=64 and r∈{4,8,16,32}r\in\{4,8,16,32\}. Performance improves from r=4r=4 to r=16r=16 on the three reported tasks, but declines at r=32r=32. Since increasing r reduces the dimensions available to the adaptive liquid component, this experiment highlights a capacity-allocation trade-off between adaptive processing and persistent retention.

For a new task, we recommend selecting rr on a validation set while holding dhd_h fixed. Automatically adapting rr remains for future work.

Response to W2: Theoretical retention and learned memory.

We agree that the theoretical retention property establishes the capability of the persistent update but does not guarantee that training will learn to preserve task-relevant information. We investigate this empirically through the controlled delayed-recall and component-exchange experiments.

At an unseen delay of 128 steps, SSD-LTC learns a write fraction of 39.9% for cues and 0.16% for distractors, retaining 81.5% of the persistent-state carry across the delay and achieving 41.2% exact recall. The component-exchange experiment further shows that transferring the persistent component between sequences transfers the recalled content, whereas exchanging the adaptive liquid component does not. These findings provide direct evidence that the learned persistent subspace stores task-relevant information in the evaluated recall task.

We incorporate the additional ODE-LSTM comparisons and clarify the empirical scope of these findings in the revised manuscript.

Thank you for the detailed rebuttal and additional experiments. The ODE-LSTM comparisons and longer-delay recall results address several of my original concerns.

However, my main reservation remains the limited methodological novelty. The approach builds heavily on established gated-memory mechanisms, and the fixed-basis ablation suggests that much of the improvement can be achieved without learning the subspace orientation. While the empirical results are encouraging, I remain unconvinced that the proposed architecture represents a sufficiently substantial conceptual advance over existing recurrent memory designs.

I therefore maintain my rating, primarily due to concerns about novelty and scientific significance.

We thank the reviewer for clarifying their concern. We would like to organize this rebuttal reply and try to address the remaining concern. Below are the points that review agrees on the evaluation of the paper:

  1. The paper has a clear motivation. The method is technically sound, and the theoretical analysis is well aligned with the architecture.
  2. SSD-LTC directly addresses the trade-off between writing new information and retaining it in LTCs.
  3. The empirical evaluation is fairly thorough, covering both regularly and irregularly sampled time series.
  4. The delayed-recall and component-exchange experiments provide direct evidence that the persistent subspace carries recall-relevant information in the controlled task.

The main of the concern of reviewer is the methodological novelty of the paper and similarity to the ODE-LSTM. ODE-LSTM separates continuous-time dynamics from persistent memory. However, it accomplishes this using a separate LSTM cell state. In comparison, our proposed method introduces no additional hidden-state dimensions. Instead, we allocate the existing LTC hidden-state capacity between adaptive and persistent processing. These components are subsequently recombined, allowing retained information to participate directly in LTC dynamics. We further show that our method can be applied to other continuous time models like CfC, and in the rebuttal we evaluated SSD-CfC that led to an improvement of 17.1% MSE reduction on CESNET, 21.5 % MAE reduction on Electricity and + 7.24 percentage points on UCI-HAR.

We agree that gated memory is an established mechanism and also we do not claim novelty in QR decomposition or gating itself. Our contribution is to assign distinct update dynamics to complementary subspaces of the LTC hidden state. We replace the LTC dynamics in the persistent subspace with controlled retention, without introducing an additional separate memory state. The fixed-basis ablation shows that this structured update is effective even without learning the subspace orientation. Learning the basis provides a further, task-dependent benefit. These results highlight the effectiveness of assigning different dynamics to the partitioned state, with basis learning providing task-dependent improvement.

We believe our method offers novelty in assigning different update dynamics to the hidden state itself without relying on any external modules. Beyond benchmark improvements, the controlled recall experiments provide evidence of adaptive processing and memory retention of the hidden state. Our method also outperforms the evaluated baselines at every reported delay from 128 to 4096, including 64 times the maximum training delay.

We believe our work constitutes a meaningful contribution for a prestigious venue such as ICLR as our work identifies and empirically validates an alternative design for reconciling adaptive continuous-time dynamics with persistent information retention.

Comments on the paper

1 comment

Dear Reviewers,

I'm writing to gently remind you that the rebuttal discussion phase deadline is approaching. Please take a moment to read the authors' rebuttal, and feel free to respond if any concerns remain that you'd like them to address. Thank you very much for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.