acceptodds
Under review as a conference paper at ICLR 2027

Extending SSMs with the Exponentially Weighted Signature

49 views

Abstract

We introduce the exponentially weighted signature (EWS), a continuous-time model that computes iterated integrals of a path, where each increment is weighted by the matrix exponential of a learnable generator over elapsed clock time. We prove that it solves a linear controlled differential equation, keeps the group-like structure and the universality of the signature, and satisfies a modified Chen identity, enabling a parallel scan. At depth one the EWS is a state-space model (SSM), and we map linear time-invariant SSMs, Mamba channels and Mamba- heads to it in closed form. The EWS extends SSMs through an arbitrary matrix generator, a clock that generalises the step size to causal functionals of the input, and higher truncation depths that are non-linear in the path within a single layer. Empirically, the EWS achieves the highest average accuracy and rank on six long time-series classification datasets, where depth generally helps. Learned clocks prove necessary for state tracking on formal language tasks, and at depth one, the EWS matches or exceeds competing SSMs on regression and forecasting with far fewer parameters.

Primary area:
learning on time series and dynamical systems
Keywords:
Signatures, State-space models, neural CDEs

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_z8dyOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper defines the exponentially weighted signature; a generalization of LSSMs by increasing the depth (not layers) of an iterated integral of paths. A thorough theoretical analysis is provided in which it is demonstrated that LSSMs are equivalent to a depth-one exponentially weighted signature (EWS). The authors support their arguments experimentally via multiple results on synthetic and real-world benchmarks where EWS consistently outperformed a wide, and appropriately selected set of related models. The paper is well-written, grammatically consistent; not requiring any revision. An excellent work that, if published, provides are substantial step forward in our understanding and construction of parallelizable models for time series.

Critical strengths

Well written; mathematically consistent and more-than adequately thorough; Appendix provides additional experiments that strengthen the claims made by the authors.

Critical weaknesses

The paper would not be accessible for readers who are not familiar with controlled differential equations, even with their inclusion of a "background" section in Appendix. A lighter introduction in the main text would improve readability; an unexplained anomaly in EthanolConcentration requires an additional ablation in Appendix (after EigenWorms) as EWS dramatically outperforms the other models.

Questions for authors

A solid contribution that should be accepted, even without addressing the reviewer's concerns under critical weaknesses section.

2 comments
AuthorsOpenReview ↗

Response to reviewer z8dy

Thank you for your review and for your positive assessment of the paper. We address the accessibility of the paper in the general comment above, which lists the changes we will make, including a guide at the start of the appendix separating what a machine-learning reader needs from what is there for mathematical rigour.

EthanolConcentration. We agree that the margin on EthanolConcentration needs clearer explaining. In Appendix F.4 we retrain S5, LRU and LinOSS-IM with the same preprocessing of the input as the EWS. They reach between 26% and 32%, against 71.9% for the EWS. The EWS at depth one, where it is an LTI SSM that a single S5 layer can represent, already reaches 66.3%. The margin therefore reflects the parametrisation and training of the EWS rather than its expressivity or its preprocessing. The dataset is also ablated over depth in Tables 14 and 15, and in the revision we will refer to these results from Section 5.1. Finally, the result itself is not anomalous for this dataset. While the deep models in our comparison score much lower than the EWS, several classical, shallow classifiers, such as shapelet-based and ensemble methods, reach accuracies in a similar range to the EWS.

The reviewer would like to thanks authors for the rebuttal and for having considered the highlighted weakness about EthanolConcentration.

Reviewer_Ed1zOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The authors present a new generalized SSM architecture which contains existing architectures as special cases, but expands them to include more complex feature interactions in a single layer ("depth"), a learned clock, and more flexible memory dynamics through learned flow generators. They first evaluate their model on synthetic data to validate that their model learns (and fails to learn) sequences generated from ground truth generators in accordance with their theory. They then extend to broader public benchmarks and demonstrate that this construction enables significantly improved state tracking on formal language tasks, and sequence modeling competitive and sometimes surpassing state of the art with significantly fewer free parameters.

Critical strengths

The empirical performance is strong. Specifically the performance on state tracking tasks is notable and not something to be taken lightly. The synthetic validation and various ablations are much appreciated and further serves to support the soundness of their approach. Furthermore, their reduced parameter counts are quite significant and noticeable (e.g. Tables 5 & Table 16) compared to clearly over-parameterized models such as LinOSS. This appears to imply that the theory is functionally relevant to a variety of real world datasets, and the construction is a significant contribution worth further investigation.

The theoretical contribution is significant, timely, and relevant to the community.

Critical weaknesses

The first major limitation of the method developed in the paper is memory usage of deeper models, which is not addressed until the final sentence of the paper, and then deferred to Appendix A. I would recommend that the authors include this appendix section in the main draft if accepted (using the additional page for the camera ready) including Table 17, to make this limitation more clear.

The second major limitation is the readability of the paper. For example, Section 2 of the paper is extremely technical and analytical without a friendly introduction for a broader community. If the authors were able to somehow include portions of the current Section 4 earlier in the paper to help the SSM community familiarize themselves with the setup, that might help.

Questions for authors

None.

2 comments
AuthorsOpenReview ↗

Response to reviewer Ed1z

Thank you for your review and constructive feedback on the paper. We will adopt both of your suggestions, and address them in the general comment above. We will move the discussion of memory and a revised Table 17 into Section 5, and we will make the connection between the depth-one EWS and SSMs before developing the general theory. The general comment also reports our revised implementation, which reduces the peak memory of the EWS on all six UEA datasets.

Thanks to the authors for the updated results, and I look forward to reading the updated manuscript. The additional discussion of memory usage is welcome and appreciated.

Reviewer_4w7kOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces the concept of an "Exponentially Weighted Signature", a general mathematical object which is then interpreted as a learnable neural module, one which generalises a range of SSMs that operate on an underlying continuous-time system. The EWS is a generalisation of the notion of a path signature that allows one to more explicitly discuss systems with fading memory, and that provides parameters which are amenable to standard gradient-based optimisation. The notion of "depth" discussed in the context of the EWS is not the standard deep learning notion of a stack of layers, but is instead a "stack" of iterated integrals, which produce nonlinear effects. Standard SSMs such as e.g. Mamba and Mamba-2 map (channel-wise) to a depth-one EWS, but as is clear from the previous sentence, the EWS module supports more complex behaviour than that which can be exhibited by Mamba and Mamba-2. After a discussion on the properties of the EWS, possible parametrizations thereof, an empirical synthetic-task expressivity analysis, as well as a discussion of several mappings of common SSMs to depth-one EWS, the authors conclude with experimental results on standard time-series and formal language tasks, showing empirical benefits throughout.

Critical strengths

The exponentially weighted signature is a clean and experimentally impactful mathematical object. My lack of knowledge does not allow me to comment on how fundamentally new it is (its form certainly shares similarities with the EFM), yet there certainly seems to exist novelty in the parametrisation of the (A,B)(A,B) matrices, as well as the clock. The object generalises a range of widely-used SSMs, and analysis of SSM properties may in the future be aided by the analysis of the EWS that it induces. Again, I am not an expert in the area, but the EWS seems to have a lot of potential in future analytical work of this form.


The properties of the EWS are investigated in an amount of detail which is somewhat uncommon in current literature. Propositions 1 and 2, as well as Theorem 1, are rather significant contributions on their own. I, as a reader without significant background in the area, could for the most part appreciate their formulation and feel that I have learned something significant from reading them.


The experimental results are significant in both quantity as well as quality. The toy task constructions in section 3 are quite interesting and will certainly spur investigation in the same direction, and the remaining results in section 5 clearly demonstrate the benefits of the method, are well executed, and are reported highly informatively, transparently discussing the training time, memory, and parameter count.

Critical weaknesses

My main concern with the paper is its presentation. The text is rather difficult to follow at times, and I believe that the reason for that is suboptimal presentation rather than the difficulty of the subject. There are very clear signs of LLM generation, which, naturally, all of our work is likely to have in one way or the other, and I do not consider a weakness by itself. However, in my experience, LLMs tend do obscure the core idea by delving too deep into the technicalities.

For example, in the background section, it would have been of a lot of benefit to at least discuss why the signature is an important object, rather than immediately delving into its definition and never mentioning its significance in this section. Linear CDEs are also introduced very briefly and their significance is not discussed in much detail, and I feel the same about the section on SSMs. The excellent introductory section remedies this a bit, but I still feel like there is a good amount of room for improvement in the presentation of the background concepts.

The same general comment holds for the remainder of the text. It would be of great service to the reader to discuss the significance of the technical concepts before delving into their details. If I were to provide another example, I would refer to section 2.1, which very abruptly introduces polynomial factors before proposition 2. We never explicitly set the goal of modeling such polynomial factors, and the section does not do much to explain why this capability might be beneficial.


If I were to name one more concern, I suppose I would talk about the exponential dimensionality scaling of the state size of EWS with respect to the depth thereof. This can certainly be a concern in applications with larger models which do not allow for such a fine-grained hyper parameter tuning as can be done on the smaller tasks investigated in the paper. Because of the many possible combinations of dimension/NN depth considered in these tasks, it remains somewhat unclear to me to what degree this scaling really is an issue. The authors certainly report memory and latency numbers which mostly raise no concerns. Table 17 has a few memory numbers which may seem concerning, but I would not yet classify them as such, especially considering that they are competing with highly optimised models, and optimisation is not at the core of this paper.

Questions for authors
  1. Would you be interested in modifying the presentation of the paper to make it more accessible? If so, do you see any issues that you may rectify?
  2. How did you manage to maintain relatively low peak memory despite the exponential scaling of the effective state size? Is it due to a beneficial combination of dimensionality and NN stack depth, or is the scaling simply not an issue up to the depths that you used?
  3. Why did you not discuss the mapping of Mamba-3 to a depth-one EWS layer? Is it trivially extended from the other mappings, or is there something about it which does not permit such a mapping? I could not fully understand the portion of the paper that discusses it.
  4. You mention that "Unlike an SSM, the EWS never discretises its dynamics at depth one". However, it seems that some form of discretisation does in fact occur, as you later state that "The EWS holds xx constant between observations". Could you help me understand what you meant by these two sentences?

I hope that I have understood the paper sufficiently well and that the questions are well formed. I would appreciate any clarification the authors can provide me with, if they see that I am misunderstanding something fundamental. I am looking forward to a productive discussion period.

1 comment
AuthorsOpenReview ↗

Response to reviewer 4w7k

Thank you for your thoughtful review and encouraging assessment. Making the paper clear and accessible is important to us, and we appreciate your helpful suggestions. We address your presentation concerns (Q1) and the state size and memory (Q2) in the general comment above, which lists the changes we will make and a revised Table 17. We answer your remaining questions below.

Mamba-3. Thank you for highlighting this. We agree that our discussion of Mamba-3 could be improved, and will revise Section 4 to more clearly describe the distinction. In particular, the EWS ties rotation and decay together, whereas Mamba-3 allows them to change independently in response to the input. This is due to the EWS having a fixed generator and one scalar clock. Extending the EWS to allow independently varying rotation and decay, which would allow it to capture Mamba-3, is a natural direction for future work.

Discretisation. Thank you for pointing out this imprecise wording. We meant that, given the path built from the observations, the depth-one update is exact rather than a numerical approximation of the dynamics, as is also the case for zero-order-hold SSMs such as S4D and S5. We will revise the sentence to make this clear.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all three reviewers for their positive reviews. We are grateful that they found the theoretical contributions significant, and the empirical evaluations thorough with strong results, particularly on state tracking. Two points were raised by more than one reviewer, the presentation of the paper and its memory cost, and we address both here. Each reviewer's remaining points are answered in an individual reply.

Presentation. We agree that the technical sections need more motivation. In keeping the paper mathematically rigorous within the page limit, these sections became terse. We will make the following changes:

  • Provide additional motivation for each background concept. For example, signatures are useful because their linear functionals can approximate continuous functions of paths, while linear CDEs provide learnable dynamics that remain parallelisable.
  • Make the connection between the depth-one EWS and SSMs before developing the general theory, to give readers a familiar starting point.
  • Explain the clock as controlling how quickly the model's memory fades, and relate this to the input-dependent step sizes used in selective SSMs.
  • Explain the motivation for polynomial factors before Proposition 2. In particular, these allow memory kernels that peak at a non-zero lag, letting the model place more weight on inputs from a particular time in the past.
  • Add a short guide at the start of the appendix separating what a machine-learning reader needs from what is there for mathematical rigour.

State size and memory. We agree that the exponential growth of state size with depth is a limitation. Managing this is one of the roles of the lift map BB, which allows one to choose a width independently of the number of input channels. For example, on MotorImagery, we map 64 channels to width 16, giving 4,369 state coordinates at depth three, compared with 266,305 for the depth-three signature of the raw path. The UEA experiments also use a single EWS layer. These choices keep the state manageable at the depths considered, but do not remove the underlying scaling limitation.

We have also revised our implementation of the EWS to recompute some intermediate quantities during the backward pass rather than store them. The outputs of the model remain the same, while peak memory is reduced on all six datasets and training time is reduced on five, as shown below. Peak memory is measured at the driver level, as in Table 17.

DatasetTime (s / 1,000 steps), original → revisedTime, baselinesPeak memory (MB), original → revisedPeak memory, baselines
EigenWorms19.6 → 5.110.2–1,9554,790 → 1,6482,742–4,798
EthanolConcentration29.1 → 3.64.6–2,1166,144 → 1,134880–3,590
Heartbeat13.0 → 9.85.4–7852,748 → 1,658888–2,938
MotorImagery80.0 → 65.717.4–77934,832 → 17,0923,498–8,846
SelfRegulationSCP110.1 → 9.23.2–6162,744 → 2,704892–5,268
SelfRegulationSCP228.7 → 29.14.7–5869,532 → 2,748882–3,210

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.