acceptodds
Under review as a conference paper at ICLR 2027

State-Conditioned Spectral Propagation for Closed-Form Continuous-Time Liquid Neural Networks

Abstract

Irregular observation sequences exhibit concurrent short-term fluctuations and long-term trends. A key challenge is to equip hidden states with multiple adaptive decay timescales while preserving closed-form propagation. We propose the State-Conditioned Spectral Closed-Form Continuous-Time Network (SC-SCfC), which models hidden-state evolution toward an interval target between consecutive observations as a mixture of multiple exponential decays. While individual state components can blend fast and slow decay rates, restricting decay to fixed coordinate axes limits the model's expressiveness. To overcome this, we further propose a state-conditioned matrix allocation (SC-MA) mechanism that extends SC-SCfC. By decomposing the residual between the current state and its target into vector components in the latent space using positive semi-definite allocation matrices that sum to the identity, each component can decay at its own rate along data-dependent directions. The parameters remain fixed within each interval, yielding closed-form propagation. Experiments across four benchmark families demonstrate competitive performance in classification and remaining-useful-life regression.

Primary area:
learning on time series and dynamical systems
Keywords:
Liquid Neural Networks, Continuous-Time Models, Irregular Time Series, Multi-Timescale Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rSbiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SC-SCfC, a continuous-time network that models hidden-state evolution using multiple adaptive exponential decay timescales while preserving closed-form propagation between irregular observations. Its state-conditioned matrix allocation extension allows decay along data-dependent latent directions, with experiments across four benchmark families showing competitive performance in classification and remaining-useful-life regression.

Critical strengths
  • The paper addresses the interesting problem of modelling time series with irregularly spaced observations.
  • I believe that the proposed approach is interesting and well motivated.
Critical weaknesses
  • For me, the main issue is readability. The paper is difficult to follow in places.
  • The quantitative results do not appear particularly impressive. Statistical significance testing would help establish the reliability of the findings.
Questions for authors

In my view, the paper is borderline, and I am leaning towards rejection. However, I am open to reconsidering my score.

  1. Could you clarify Figure 1? I believe its readability could be improved.
  2. Could you comment on the identifiability of the model?
  3. Could the computations described in lines 240–250 lead to numerical instability? Please discuss this concern.
  4. Could you justify the choice of baselines? Around line 290, you state that the baselines differ across tasks. Why is this the case?
  5. Are the differences in the reported scores statistically significant?
  6. Could you report runtimes for the proposed method and the baselines?
1 comment

Thank you for the constructive review and for being open to reconsidering your score.

W1 / Q1: Readability and Figure 1. Figure 1 is a conceptual illustration, not an experimental result.

  • Axes: the horizontal axis is elapsed time τ\tau since the last observation; the vertical axis is the retained fraction of a residual component.
  • Gray bands: the desired behavior. The slow trend should stay retained across the gap, and the fast fluctuation should fade quickly.
  • Panel (a): single-rate decay gives one exponential curve, which cannot follow both bands.
  • Panel (b): a mixture of exponentials gives a curve that falls quickly and then flattens. However, it is tied to coordinate axes, so it helps only if trend and fluctuation happen to occupy different coordinates.
  • Panel (c): SC-MA lets each data-dependent direction have its own mixture.

Following your suggestion, the revised PDF contains a redrawn Figure 1:

  • Larger type: the panels are narrower and taller.
  • Less clutter: each panel now reads top to bottom (retention curves, decay matrix, inset), and each band is labeled once.
  • Labeled inset: the inset now has a title ("effective decay rate vs. τ\tau" in (a)–(b), "residual split by direction" in (c)).
  • Objectives: the bottom line of each panel now names the two design objectives from Sec. 1, "multiple timescales" and "adaptive directions".

The revised PDF also states the readout in Sec. 3.1 and rewrites the five densest sentences of Sec. 3 so that each idea comes before its notation. The prediction at event ii is yi=F([h(ti−);zi])y_i=F([h(t_i^-);z_i]), which uses the current observation.

Q2: Identifiability. Our results do not rely on parameter identifiability, and we do not claim it. Three levels should be distinguished.

  1. Retention function. Suppose a scalar retention function Γ(τ)=∑kwke−λkτ\Gamma(\tau)=\sum_kw_ke^{-\lambda_k\tau} with distinct rates and positive weights were observed exactly on an interval. Linear independence of exponentials would then identify the (rate, weight) pairs up to permutation. Equal rates merge, and zero-weight atoms are invisible. With exact observation and distinct rates, the same holds for the matrix coefficients PkP_k of G(τ)G(\tau).
  2. Generator parameters. These have the usual symmetries: atom permutation (only partly broken by the fixed anchors), shifts of the softmax logits, and changes of basis in the dictionary span. For free factors there is also Bk↦BkQB_k\mapsto B_kQ with QQ orthogonal.
  3. Task supervision. We only observe task outputs at finitely many times, not G(τ)G(\tau) itself.

Learned rates and directions should therefore be read as one functional representation that the network found, not as recovered ground-truth timescales.

Q3: Numerical stability of lines 240–250. Exact positive definiteness (S=∑kBkBk⊤+εI⪰εIS=\sum_kB_kB_k^\top+\varepsilon I\succeq\varepsilon I) guarantees that the Cholesky factor exists. In general κ2(S)≤(∑k∥Bk∥22+ε)/ε\kappa_2(S)\le(\sum_k\|B_k\|_2^2+\varepsilon)/\varepsilon, which can be large. What keeps it under control in practice:

  1. Each factor is an identity offset plus a scaled dictionary term, Bk=πkI+sBHR∑rαrDk,rB_k=\sqrt{\pi_k}I+\frac{s_B}{\sqrt{HR}}\sum_r\alpha_rD_{k,r}. When the dictionary term is zero, S=(1+ε)IS=(1+\varepsilon)I and Pk=πkIP_k=\pi_kI, so training starts perfectly conditioned. The scale 1/HR1/\sqrt{HR} keeps the offset O(1)O(1) as HH grows.
  2. We use ε=10−4\varepsilon=10^{-4}, FP32, and sB=1s_B=1 (0.30.3 on P19).
  3. The residual bound ∥e(τ)∥≤∥e(0)∥\|e(\tau)\|\le\|e(0)\| holds in exact arithmetic for any valid allocation, so propagation itself cannot amplify the state.

The revised PDF notes the identity-offset initialization in Sec. 3.3, and the final version adds the full argument.

Reviewer_z75bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SC-SCfC, a closed-form continuous-time neural network that models hidden-state evolution using mixtures of exponential decays. It further introduces SC-MA to enable state-conditioned decay directions through matrix allocation. Experiments across four benchmark families demonstrate competitive performance on classification and regression tasks.

Critical strengths
  1. Extending single-rate decay to mixtures of exponential decays provides a natural way to capture multiple timescales while preserving closed-form propagation. The state-conditioned matrix allocation further improves the flexibility of the hidden-state dynamics.

  2. The experiments cover multiple tasks and datasets, with comparisons against various continuous-time and recurrent baselines. The results demonstrate competitive performance and support the effectiveness of the proposed design.

Critical weaknesses
  1. Although the proposed exponential mixture provides a reasonable extension to single-rate decay, the paper does not clearly report the chosen number of decay components (K) or investigate its effect on prediction accuracy and computational cost. Furthermore, runtime and memory comparisons with numerical ODE-based models are missing, making the practical efficiency advantages of the proposed approach unclear.
Questions for authors

Could the authors investigate how varying the number of decay components affects prediction accuracy, runtime, and memory consumption? A direct efficiency comparison with numerical ODE-based models would also help clarify the practical benefits of the proposed approach.

1 comment

Thank you for the positive assessment of the formulation and the clarity, and for the precise questions.

Value of KK. The revised PDF now states KK in Sec. 4.1. We used K=4K=4 on QuickDraw10 and P19 and K=32K=32 on PAR and C-MAPSS. The dictionary size for SC-MA was R=2R=2 throughout.

Effect of KK on accuracy. SC-SCfC KK sweep, validation, one seed per task:

KK12481632
QuickDraw10 acc. % ↑96.2597.0896.9597.0397.1397.04
PAR acc. % ↑85.9686.0387.2387.0287.1587.42
FD002 RMSE ↓12.2910.9311.2011.1811.1211.14
FD003 RMSE ↓10.3611.379.459.208.808.84

Going from a single exponential (K=1K=1) to K≥4K\ge4 helps on every task: +0.7 pp on QuickDraw10, +1.3 pp on PAR, and −1.1 / −0.9 RMSE on FD002 / FD003. Beyond that, results are stable, so a moderate KK suffices.

For SC-MA on FD001 (one seed), K=8/32/64K=8/32/64 give test RMSE 14.27 / 15.35 / 14.49. These differ by less than one seed SD (1.10).

Effect of KK on runtime and memory. In the SC-MA event kernel (H=64H=64, R=2R=2), time per event is nearly flat in KK (0.062–0.068 ms for K=1K=1–16) because atoms are computed in parallel on the GPU. Memory grows linearly: 12, 15, 21, 33 and 58 MiB for K=1,2,4,8,16K=1,2,4,8,16. Generator parameters grow as KRH2KRH^2. The carried state stays HH-dimensional for every KK.

For SC-SCfC, an interval query costs O(HK)O(HK) once the spectrum is generated.

At full-model level, SC-MA is 1.10–1.76× slower than SC-SCfC at inference and uses 4.1–31.9× more training memory. KK therefore matters mainly for memory, through the stored H×HH\times H allocations, rather than for latency.

Full models on QuickDraw10 (batch 128 × 100 events, H=256H=256, K=4K=4, R=2R=2):

ModelParamsInference / batchTraining stepPeak training memory
SC-SCfC1.255M103 ms506 ms374 MiB
SC-MA1.256M181 ms898 ms1547 MiB

Protocol. Frozen trained checkpoints, one NVIDIA RTX PRO 6000 GPU, FP32 (TF32 off), inputs already on the GPU, medians of 20 forward passes and 5 full Adam steps after warm-up.

Comparison with numerical ODE models. Our measured comparison is at the propagation level. One SC-MA event update (K=4K=4, R=2R=2) takes 0.52×, 0.64×, 0.78× and 0.97× the time of a dense matrix-exponential propagation at H=32,64,128,256H=32, 64, 128, 256. A solver-based model's cost additionally grows with the number of solver steps per interval, i.e. with interval length and tolerance, while ours does not.

In accuracy, Neural CDE is below both of our models on QuickDraw10 (95.96 vs. 97.44 / 96.98) and in every PAR gap group.

The revised PDF states the measured cost of both of our models in Sec. 3.3. The final version adds an end-to-end runtime and memory comparison with Neural CDE under the same protocol, including its solver and tolerance settings.

We hope this answers your questions, and we would be happy to provide more detail.

Reviewer_HHfHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work studies the problem of irregularly sampled observations in a sequential modeling task (eg., clinical monitoring, event streams, streaming sensor / video representation). The goal is to keep track of hidden state that retains information across observations, even when the time between events varies irregularly. Existing sequential architectures offer single-rate decay per direction and decay directions that do not adapt sufficiently to current observations.

This work deals with the following recurrence formulation

$ h(t_i+\tau) = h_i^{\mathrm{tar}} + G_i(\tau)(h_i-h_i^{\mathrm{tar}}) $

where hih_i denotes the hidden state and hitarh_i^{\mathrm{tar}} denotes the interval target and Gi(τ)G_i(\tau) is the retention function that maps the residual hidden state to the update at interval τ\tau. Note that the boundary condition is Gi(0)=IG_i(0)=I an identity matrix.

This paper proposes two models:

  • SC-SCfc :
    • defines the retention function per coordinate Γi,j(τ)∑k=1Kwi,j,ke−λi,j,kτ,\Gamma_{i,j}(\tau)\sum_{k=1}^{K}w_{i,j,k}e^{-\lambda_{i,j,k}\tau}, where the mixture weights are nonnegative and sum to one
    • this results in decaying of faster components first and slower components dominate at longer horizon. This is a standard property of the positive exponential mixtures.
  • SC-MA:
    • defines the retention function by replacing the weights wi,j,kw_{i,j,k} with positive semi-definite matrices that sum to the identity and are generated from the event context, so that the residual decays along context-dependent directions.
    • The authors construct them through positive-definite matrix factors and a shared Cholesky-based normalization, ensuring that the allocations sum to the identity.
    • The matrices need not commute or be orthogonal projectors. Consequently, the mechanism can express time-dependent mixing between hidden coordinates rather than merely coordinate-wise forgetting.

Experiments across four benchmarks show competitive performance, including improvements on QuickDraw10 and some C-MAPSS subsets. However, improvements over the simpler SC-SCfC are generally modest and often within seed variability. SC-MA also has substantial per-event computational complexity, O(KH3)O(KH^3), with no runtime or memory measurements reported.

Critical strengths
  • SC-MA provides a clean formulation for input-conditioned state transitions, while coupling different hidden-state coordinates compared to SC-SCfc
  • This work explicitly distinguishes between the rate at which information decays and the directions carrying the information
Critical weaknesses
  • Limited novelty of multi-timescale decay. Mixtures of exponential forgetting rates have substantial precedents in CT-GRU, multi-timescale recurrent networks, and diagonal SSMs.
  • There’s not enough evidence to suggest that expensive matrix allocation is necessary in SC-MA. This work ends up increasing the complexity of the hidden state transition substantially, but the return on that complexity is not justified. SC-MA outperforms SC-SCfc in some cases, but in many other cases SC-SCfc outperforms SC-MA. One could have also opted for a diagonal + low-rank adaptation instead of the full matrix allocation.
  • There are no wall-clock, FLOPs, memory or parameter measurements in the paper, even though there are claims on efficiency of the proposed scheme.
  • At τ=0\tau=0, the new input has no effect on the state. If predictions depend on hidden state, predictions at ti+t^{+}_i cannot incorporate the new input (important for event classification and early-warning predictions).
  • Important comparisons against CRU, S5, and stronger irregular-time models are absent, despite their relevance to the paper's central claims.
  • Improvements in SC-MA are relatively small compared to other baselines
  • Missing critical ablations : no analysis of the impact of K, decay rate changes, dictionary rank R, etc.
Questions for authors
  • Can the authors report parameter counts, FLOPs, inference latency, training throughput, and peak memory against CfC, CRU, and S5 at matched model sizes? How does SC-MA scale with hidden dimension?
  • What are the effects of K=1, varying K, and changing dictionary rank R?
  • Since G(0)=I, a new observation does not immediately change the hidden state. How does this affect responsiveness under very short observation gaps, and how are event-time predictions aligned with observations?
  • How do you justify the complexity of the proposed mechanism when many baselines yield competitive performance compared to the proposed SC-MA and SC-SCfc models?
  • Are the baselines and the proposed methods (show in the main text results) using similar model footprint (compute/storage, etc.)? Did you perform a fair hyper-parameter search for different tuning parameters?
1 comment

Thank you for the detailed review and the concrete questions. We answer each below and have revised the paper accordingly.

Q1 / W3: Parameters, FLOPs, latency, throughput, memory; scaling with HH. In short:

  • SC-MA's parameter count is within 0.7% of SC-SCfC's.
  • For its event-specific directions, SC-MA uses 1.10–1.76× the inference time, 1.30–1.78× the training-step time and 4.1–31.9× the peak training memory of SC-SCfC.

Protocol. Frozen trained checkpoints, one NVIDIA RTX PRO 6000 GPU, FP32 (TF32 off), inputs already on the GPU, medians of 20 forward passes and 5 full Adam steps after warm-up.

Full models on QuickDraw10 (batch 128 × 100 events, H=256H=256, K=4K=4, R=2R=2):

ModelParamsInference / batchTraining stepPeak training memory
SC-SCfC1.255M103 ms506 ms374 MiB
SC-MA1.256M181 ms898 ms1547 MiB

Across all seven task settings (QuickDraw10, PAR, P19, FD001–FD004), SC-SCfC and SC-MA have nearly identical parameter counts (within 0.7%). SC-MA uses 1.10–1.76× the inference time and 1.30–1.78× the training-step time of SC-SCfC, and 4.1–31.9× its peak training memory, because autograd stores KK dense H×HH\times H allocation matrices per event. This is the cost of event-specific decay directions; SC-SCfC remains the lightweight option.

Cost of the matrix path. A naive construction of the allocations costs O(KH3)O(KH^3) per event. Our implementation precomputes dictionary products once per sequence and applies each PkP_k to vectors through triangular solves. This is exact: maximum output deviation is 1.6×10−41.6\times10^{-4} and relative gradient error 1.1×10−41.1\times10^{-4} in FP32. Per event, it leaves one O(H3)O(H^3) Cholesky factorization plus O(KR2H2)O(KR^2H^2) matrix–vector work, reducing 142.0 to 14.7 MFLOPs per event at H=256H=256. On the same checkpoint this gives 2.9× faster inference and 2.3× faster training steps at H=256H=256; at H=64H=64 its main benefit is 40–58% lower peak memory.

The relevant FLOP figure is the per-event operation count: 14.7 MFLOPs/event at H=256H=256 with our exact implementation, compared with 142.0 for the naive construction. The leading per-event term is one O(H3)O(H^3) Cholesky factorization. In the event kernel at K=4K=4, R=2R=2, time grows from 0.046 to 0.35 ms per event as HH goes from 32 to 256. Over that range, SC-MA stays below a dense matrix exponential (0.52–0.97× its time).

For CRU-style propagation, a fast-CRU kernel with a shared eigenbasis is 2–4.4× cheaper than SC-MA, since its basis is fixed across events. The paper's efficiency claims are structural (no solver, no matrix exponential, compact generator), and the revised PDF states the measured cost of SC-MA in Sec. 3.3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.