acceptodds
Under review as a conference paper at ICLR 2027

Decoupling Time and Space in Controlled Differential Equations for Exact Interaction-Order Decomposition

20 views

Abstract

Selective state-space models (SSMs) and linear controlled differential equations (LNCDEs) have emerged as effective approaches for long-sequence modeling. However, existing models face a trade-off: diagonal state transitions offer computational efficiency but struggle to capture cross-channel interactions, whereas dense transitions successfully capture these interactions at a computational cost that structured alternatives have since aimed to reduce. To address this, we propose the Tensor-Decoupled CDE (TD-CDE), an architecture that decouples temporal updating from spatial mixing. By restricting time evolution to diagonal, data-dependent decays, TD-CDE maintains efficient parallel scans. Cross-channel interactions are modeled along the depth dimension using Tensor Train (TT) decomposition, extracting high-order signature-type features without exponential parameter growth. Furthermore, the model employs gating via raw inputs, avoiding normalizations to preserve causality and exact, additive interpretability. Theoretically, we prove that models with commuting generators—including diagonal SSMs—realise only the shuffle subalgebra and cannot represent complex interactions like Lévy area at any width, and that TD-CDE's graded state decomposes predictions exactly by interaction order and feature words. Additionally, we formalize an order contract for selective discretisation. Empirically, TD-CDE captures cross-channel dynamics in path-dependent synthetic tasks that challenge existing models like Mamba. Evaluated on the UEA archive, TD-CDE achieves state-of-the-art accuracy with up to two orders of magnitude fewer parameters specifically on the datasets our order profiler flags as high-order, while showing no advantage on low-order ones—a separation predicted prior to training. Crucially, we demonstrate that a high-order discretisation scheme is essential for resolution invariance: TD-CDE preserves robust predictions and stable exact attributions even under severe sequence subsampling, where standard endpoint approximations fail.

Primary area:
learning on time series and dynamical systems
Keywords:
path signature, mamba

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_n29ROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TD-CDE, which decouples temporal evolution from cross-channel interaction using diagonal selective dynamics and Tensor Train decomposition. It provides theoretical analysis of high-order interactions and exact conditional attribution, and shows strong performance particularly on datasets with strong order-dependent structure.

Critical strengths
  1. Strong theoretical motivation: The paper clearly explains why diagonal or commuting dynamics can miss higher-order cross-channel interactions, and proposes a principled tensorized solution.

  2. Good alignment between theory and experiments: TD-CDE performs best on the datasets identified as having strong order-dependent structure, and the resolution-transfer experiments provide convincing evidence for the proposed high-order discretization.

Critical weaknesses
  1. The empirical gains are concentrated on a small number of datasets. TD-CDE shows large improvements mainly on SelfRegulationSCP1 and EigenWorms, while its performance is similar or worse than the strongest baseline on several other datasets.

  2. The meaning of “exact interaction-order decomposition” is somewhat conditional. The decomposition is exact for fixed gates and computational pathways, but its interpretation as interaction order in the original input space requires additional assumptions.

Questions for authors
  1. Can the authors provide additional real-world datasets with strong order-dependent structure to demonstrate that the observed gains generalize beyond the two main datasets?

  2. Can the authors clarify more explicitly when the exact decomposition corresponds to interaction order in the original input variables rather than only an exact decomposition of model computation?

  3. In camer ready, could you give us the architecture figure of your model, which can make your idea more clear?

Reviewer_CkcHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Tensor-Decoupled CDE (TD-CDE), an architecture based on state-space models (SSMs) that separates temporal state evolution from cross-channel interactions. By combining diagonal state transitions with Tensor Train decomposition, the proposed model efficiently captures the high-order temporal interactions while maintaining the computational complexity of diagonal SSMs. Theoretical analyses on the expressiveness of the method and the robustness of numerical discretization is presented, followed by empirical evaluations on multivariate time-series classification tasks and resolution robustness.

Critical strengths
  • The paper studies an interesting problem of improving the expressiveness of diagonal SSMs while maintaining computational efficiency. The proposed architecture is mathematically grounded, and the theoretical analysis provides useful insights into the representation of high-order temporal interactions.
  • The order profiler discussed in the experiment section is a good validation of the theoretical advantage of the model. Many of the prior SSM advancements I have seen are based on some new architectures with a heuristic benefit that the method should bring, but a lot of them miss clear empirical evidence to show that theoretical benefit. This paper, however, clearly connects the theoretical benefits to the empirical gains/losses by comparing the performance difference between the model and the baseline to the order-profile scores, which are a proxy for the theoretical benefit.
Critical weaknesses
  • The presentation is dense and very difficult to follow. The paper introduces a lot of notations without appropriate explanation of what they are (e.g., the notations in eq. (1)). In addition, the notation does not follow the standard notation used by everyone in the SSM community. I believe the paper could substantially improve its clarity by:
    • Presenting in equations the prior SSM designs, such as S4, S5, Mamba, before discussing the new aspect introduced in the proposed method, and more importantly, a direct comparison of the proposed method to the existing architectures.
    • Some figures that show the connections and distinctions between the architectures, which highlight the main motivation for the proposed method.
  • The theoretical analysis focuses on a simplified setting, with one-layer of SSM and no normalization or MLP post-processing. It is unclear how these results would change in multi-layer selective SSM architectures (which is what everyone uses in the real-world), where nonlinearities and interactions across layers can substantially increase expressiveness. Therefore, the theoretical limitations identified for diagonal systems do not necessarily translate into the limitations of practical deep Mamba models.
Questions for authors
  1. Could the authors provide a more intuitive explanation of the proposed architecture? It would be most efficient to start from a standard selective SSM equation (e.g., Mamba, and using the standard notations of A,B,C,\mathbf{A}, \mathbf{B}, \mathbf{C}, and D\mathbf{D}) and show how the proposed TD-CDE modifies it. A unified notation and a concrete example in a simplified setting would make the contribution much easier to understand.
  2. How do the computational costs (in terms of the wallclock measurement than an aymptotic analysis) of TD-CDE compare with conventional Mamba architectures in practice, especially when using the higher-order discretization scheme proposed in the paper?
Reviewer_7jkJOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

TD-CDE is a selective linear CDE with a graded state: each level mm evolves under a diagonal, input-dependent decay, and level mm is driven by a tensor-train core contracting level m−1m−1 with the gated path increment. Without decay and gating this computes a TT-structured linear functional of the truncated signature. The paper's second thread is how such models should be discretised: it contrasts the usual endpoint scheme with a corrected scheme that restores the within-step Chen/simplex terms, uses exponential-integrator and Gauss–Legendre weights and reconstructs the path cubically, and proposes an empirical "order contract" over five error axes. Experiments on nine UEA datasets compare accuracy against SLiCE and Mamba; a SelfRegulationSCP1 study shows A7h is nearly invariant to test-time subsampling while A0 and Mamba are not; and multilinearity in the cores gives exact additive decompositions of the logit by level, word and stratum. The theory consists of a CP-vs-TT exact-representation separation, a depth separation against depth-one diagonal models and a TT-SVD trajectory bound.

Critical strengths
  1. The discretisation of selective CDEs is taken seriously, which is rare in the SSM literature. The point that holding input-dependent gates at the step endpoint caps the global order at one regardless of how accurately the decay is integrated, and the separation of error sources into axes are useful.
  2. The interpretability claims are carefully scoped. The paper repeatedly states that word/stratum shares are conditional on fixed gates and decays, are not input-removal effects, and coincide with iterated integrals only in the ungated, zero-decay limit.
  3. The proof of Thm 4.1 is correct and clean for what it states (identifiability via the multilinear monomial, the interleaved-matricisation witness, the remark on why symmetric powers would not do), and the numerical engineering (FP64 exponent budget, chunking, scheme-agnostic scan kernel) is well documented.
Critical weaknesses
  1. The abstract's main theorem is not in the paper, and the paper's own results contradict it. The abstract claims a proof that commuting-generator models, diagonal SSMs included, "realise only the shuffle subalgebra and cannot represent … Lévy area at any width". There is no such theorem. Meanwhile Table 5(b) shows a one-level model with a live gate fitting the Lévy-area target, and the depth-one class of Definition I.1 contains ∫x1dx2−∫x2dx1\int x^1dx^2 − \int x^2dx^1 outright. The claim can only hold for CDEs with fixed commuting generators, which Mamba is not. The abstract's "synthetic tasks that challenge existing models like Mamba" has no experiment behind it either: Table 5(b) compares TD-CDE variants only.

  2. The expressivity theorems compare against the wrong classes, and the conclusion oversells them. Thm 4.1 is CP versus TT: the "diagonal" class it beats is TD-CDE at rank one, and given the linear independence of level-K signature terms the bound is Khrulkov et al.'s. The proof of Thm I.4 needs the output to be additive over concatenated path segments; that fails with input-dependent decay, and with a Mamba-style readout C(xT)C(x_T): on the paper's own witness family, $x^3_T\cdot\int x^1dx^3 reproduces the target exactly. Neither result covers a selective SSM as actually used, yet the conclusion claims TD-CDE "theoretically guarantees superior spatio-temporal expressivity beyond standard diagonal SSMs". The core idea, low-rank tensor projections of the signature built level by level, is also very close to Seq2Tens (Tóth, Bonnier & Oberhauser, ICLR 2021), which is not cited; nor are Neural CDEs (Kidger et al., 2020).

  3. The accuracy numbers are not strong enough to carry the claims. Several of the re-run baselines in Table 2 sit at chance: 48.8% and 48.3% on the two binary tasks SCP2 and MotorImagery, and 22.5% on four-class EthanolConcentration, where the published numbers are roughly 54, 54 and 34. Against those published numbers TD-CDE is clearly worse on EC, SCP2, MI and Heartbeat and about level on SCP1 and EigenWorms.

Questions for authors
  • Where is the theorem on commuting generators and the shuffle subalgebra? Which model class does it cover, and how does it reconcile with Table 5(b) and with Definition I.1 containing ∫x1dx2\int x^1dx^2?
  • Can you add a Chen-consistent, low-order, causal scheme to Table 3 to separate numerical order from the inclusion of the simplex terms (and from non-causal look-ahead)?
  • Can Table 2 be reported with seeds, standard deviations and baseline parameter counts? Why are the re-run SCP2, MI and EC baselines at chance level?
  • How do SLiCE, Log-NCDE and an S5/S4D model with Δ\Delta = physical Δ\Deltat behave under the subsampling protocol of Table 3?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.